imports when running CWL

How are references resolved when running CWL workflows?

I have a workflow that looks like this:

cwlVersion: v1.0
class: Workflow
inputs: []
outputs: []
steps:
  my-task:
    run: imports/myTask.cwl
    in: []
    out: []

I am submitting the workflow CWL and the imports in an "imports.zip" with structure "imports/myTask.cwl" (I have also tried with a plain structure but it did not work either).

I'm getting this error: Field run contains undefined reference to file:///tmp/imports/myTask.cwl

This happened during the "cwltool" validation step. How can references be resolved when using CWL?

More logs:

tmp/1b55881b-0741-4280-b032-36355bec37f613229185354645663150:5:1: checking field steps
tmp/1b55881b-0741-4280-b032-36355bec37f613229185354645663150:6:3: checking object tmp/1b55881b-0741-4280-b032-36355bec37f613229185354645663150#my-task
tmp/1b55881b-0741-4280-b032-36355bec37f613229185354645663150:7:5: Field run contains undefined reference to file:///tmp/imports/myTask.cwl

Thank you!

Tagged:

Best Answer

Answers

  • dyuendyuen Member

    Hi,
    I think I have a similar question for run as well. Running into the following, an issue when using imports and issue without.

    The follow-up question, I noticed the line "running cwltool on file /tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl failed with CommandResult 1"

    Is it possible I have the wrong version of cwltool? (using 1.0.20180108222053)

    :

    [email protected]:~/cromtesting$ ls
    cromwell-32.jar  cromwell-workflow-logs  dockstore-tool-md5sum.cwl  md5sum.cwl  md5sum.input  md5sum.json  womtool-32.jar
    [email protected]:~/cromtesting$ cat md5s
    md5sum.cwl    md5sum.input  md5sum.json
    [email protected]:~/cromtesting$ cat md5sum.cwl
    cwlVersion: v1.0
    class: Workflow
    
    dct:creator:
      '@id': http://orcid.org/0000-0002-7681-6415
      foaf:name: Brian O'Connor
      foaf:mbox: mailto:[email protected]
    
    dct:contributor:
      foaf:name: Denis Yuen
      foaf:mbox: mailto:[email protected]
    
    inputs:
      input_file: File
    
    outputs:
      output_file:
        type: File
        outputSource: md5sum/output_file
    
    steps:
      md5sum:
        run: dockstore-tool-md5sum.cwl
        in:
          input_file: input_file
        out: [output_file]
    [email protected]:~/cromtesting$ java -jar cromwell-32.jar run md5sum.cwl  --inputs md5sum.json
    [2018-06-01 15:55:23,84] [info] Running with database db.url = jdbc:hsqldb:mem:327f5756-a120-42e2-8b8a-be5010bdc7b4;shutdown=false;hsqldb.tx=mvcc
    [2018-06-01 15:55:28,09] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
    [2018-06-01 15:55:28,10] [info] [RenameWorkflowOptionsInMetadata] 100%
    [2018-06-01 15:55:28,17] [info] Running with database db.url = jdbc:hsqldb:mem:3b527729-8cf3-43c7-8d57-3d5a7bfb93eb;shutdown=false;hsqldb.tx=mvcc
    [2018-06-01 15:55:28,41] [info] Slf4jLogger started
    [2018-06-01 15:55:28,56] [info] Workflow heartbeat configuration:
    {
      "cromwellId" : "cromid-80b08dd",
      "heartbeatInterval" : "2 minutes",
      "ttl" : "10 minutes",
      "writeBatchSize" : 10000,
      "writeThreshold" : 10000
    }
    [2018-06-01 15:55:28,59] [info] Metadata summary refreshing every 2 seconds.
    [... snipped ...]
    [2018-06-01 15:55:29,18] [info] MaterializeWorkflowDescriptorActor [c6dc6293]: Parsing workflow as CWL v1.0
    [2018-06-01 15:55:29,24] [info] Pre-Processing /tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl
    [2018-06-01 15:55:30,18] [error] WorkflowManagerActor Workflow c6dc6293-0154-4e60-a6d0-a04242a964bd failed (during MaterializingWorkflowDescriptorState): Workflow input processing failed:
    running cwltool on file /tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl failed with CommandResult 1
    Tool definition failed validation:
    ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl:21:1: checking field steps
    ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl:22:3:   checking object ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl#md5sum
    ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl:23:5:     Field run contains undefined reference to file:///tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/dockstore-tool-md5sum.cwl
    
    cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anon$1: Workflow input processing failed:
    running cwltool on file /tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl failed with CommandResult 1
    Tool definition failed validation:
    ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl:21:1: checking field steps
    ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl:22:3:   checking object ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl#md5sum
    ../../../tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl:23:5:     Field run contains undefined reference to file:///tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/dockstore-tool-md5sum.cwl
    
            at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.cromwell$engine$workflow$lifecycle$materialization$MaterializeWorkflowDescriptorActor$$workflowInitializationFailed(MaterializeWorkflowDescriptorActor.scala:200)
            at 
    [ ... snipped ... ]
    [email protected]:~/cromtesting$ tree /tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/
    /tmp/c6dc6293-0154-4e60-a6d0-a04242a964bd.temp.8357920439348816310/
    └── c6dc6293-0154-4e60-a6d0-a04242a964bd.cwl
    
    0 directories, 1 file
    [email protected]:~/cromtesting$ java -jar cromwell-32.jar run md5sum.cwl  --imports `pwd`  --inputs md5sum.json
    [2018-06-01 15:56:05,58] [info] Running with database db.url = jdbc:hsqldb:mem:cad3d160-f0d8-45a4-a72e-5b92423168bd;shutdown=false;hsqldb.tx=mvcc
    [2018-06-01 15:56:09,90] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
    [2018-06-01 15:56:09,90] [info] [RenameWorkflowOptionsInMetadata] 100%
    [2018-06-01 15:56:09,96] [info] Running with database db.url = jdbc:hsqldb:mem:1d797a65-a4f3-43e5-a951-310e829787a8;shutdown=false;hsqldb.tx=mvcc
    [2018-06-01 15:56:10,19] [info] Slf4jLogger started
    Exception in thread "main" java.io.IOException: Is a directory
            at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
            at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
            at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
            at sun.nio.ch.IOUtil.read(IOUtil.java:197)
            at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
            at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
            at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
    

    I'm basically trying to do the Cromwell equivalent of

    [email protected]:~/cromtesting$ cwltool md5sum.cwl md5sum.json
    /usr/local/bin/cwltool 1.0.20180108222053
    Resolved 'md5sum.cwl' to 'file:///home/dyuen/cromtesting/md5sum.cwl'
    [workflow md5sum.cwl] start
    [step md5sum] start
    [job md5sum] /tmp/tmpi7fXLe$ docker \
        run \
        -i \
        --volume=/tmp/tmpi7fXLe:/var/spool/cwl:rw \
        --volume=/tmp/tmpC5yuvk:/tmp:rw \
        --volume=/home/dyuen/cromtesting/md5sum.input:/var/lib/cwl/stg68b44509-48e8-42ec-b989-372b277d02a8/md5sum.input:ro \
        --workdir=/var/spool/cwl \
        --read-only=true \
        --user=1001:1001 \
        --rm \
        --env=TMPDIR=/tmp \
        --env=HOME=/var/spool/cwl \
        quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4 \
        /bin/my_md5sum \
        /var/lib/cwl/stg68b44509-48e8-42ec-b989-372b277d02a8/md5sum.input
    [job md5sum] completed success
    [step md5sum] completed success
    [workflow md5sum.cwl] completed success
    {
        "output_file": {
            "format": "http://edamontology.org/data_3671",
            "checksum": "sha1$1250816c19c6f5524c5366b56c7a1eed6f3c3ab3",
            "basename": "md5sum.txt",
            "location": "file:///home/dyuen/cromtesting/md5sum.txt",
            "path": "/home/dyuen/cromtesting/md5sum.txt",
            "class": "File",
            "size": 33
        }
    }
    Final process status is success
    
    
  • ocampoernestoocampoernesto Member

    Hi @danb thanks for the reply. I haven't tried using run as that's not really an option in the system we are building - we aim to use Cromwell as our workflow scheduling server.

    I look forward to CWL support in server mode, thank you.

  • danbdanb Member, Broadie

    Hi @ocampoernesto , forgot to mention the submit action here, which is I think what you want.

    I.e. run cromwell w/ server command, then in another shell run submit with -t cwl and inputs and workflow, and it should figure out the relative paths for you.

  • ocampoernestoocampoernesto Member

    Hi @danb,

    The submit option worked as expected, and after some more trial and error, I managed to get the imports working via REST post as well.

    importing.cwl:

    cwlVersion: v1.0
    class: Workflow
    inputs: []
    outputs: []
    steps:
      imp:
        run: imported.cwl
        in: []
        out: []
    
    

    imported.cwl:

    cwlVersion: v1.0
    class: CommandLineTool
    baseCommand:
      - touch
      - file.txt
    inputs: []
    outputs:
      imported_out:
        type: File
        outputBinding:
          glob: file.txt
    

    submit-cwl.sh:

    rm imports-cwl.zip || true
    zip -r imports-cwl.zip wf-defs/cwl/imports/
    curl -X POST "http://my-cromwell:8000/api/workflows/v1" -H  "accept: application/json" -H  "Content-Type: multipart/form-data" -F "[email protected]${1}" -F "[email protected];type=application/zip" -F "workflowType=cwl
    

    And the run submit-cwl.sh importing.cwl.

    Interestingly, call caching is not behaving as I would expect here any change to "imported.cwl" will NOT invalidate call caching - cromwell will still use the cached results even if you submit a different definition of imported.cwl. Is there anything I'm missing here? Thank you!

Sign In or Register to comment.