Using imports and workflowDependencies with wdl_runner.sh and gcloud

Hi,
it looks like Cromwell wrapper

runners/cromwell_on_google/wdl_runner/wdl_runner.sh

as obtained from

https://github.com/broadinstitute/wdl.git

cannot handle workflowDependencies and therefore cannot be used to launch the wdl script in

https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/blob/master/germline_single_sample_workflow.wdl

Is that correct?
I could discard the imports and flatten the wdl, but it would be much better to keep the original structure. And it would be nice if I -for now- could keep on using the wrapper, without digging deeper into Cromwell :-)

My current gcloud command is based on the example below, so I guess an extra parameter would be needed to specify the five-dollar-genome-analysis-pipeline/tasks_pipelines folder with the imports?

gcloud \
alpha genomics pipelines run \
--pipeline-file wdl_pipeline.yaml \
--zones us-central1-f \
--inputs-from-file WDL=test-wdl/ga4ghMd5.wdl \
--inputs-from-file WORKFLOW_INPUTS=test-wdl/ga4ghMd5.inputs.json \
--inputs-from-file WORKFLOW_OPTIONS=test-wdl/basic.papi.us.options.json \
--inputs WORKSPACE=gs://YOUR-BUCKET/wdl_runner/work \
--inputs OUTPUTS=gs://YOUR-BUCKET/wdl_runner/output \
--logging gs://YOUR-BUCKET/wdl_runner/logging

Thanks for your help,
--luc

Answers

  • lucdhlucdh Member ✭✭
    edited March 8

    I tried to upload the tasks_pipelines files to a Google Storage bucket, share them publicly, and use

    import "https://storage.googleapis.com/..."

    statements, but that does not help apparently:

    2018-03-08 17:28:32,040 cromwell-system-akka.dispatchers.engine-dispatcher-6 ERROR - WorkflowManagerActor Workflow 9a5128f9-5939-4567-bb1e-6d97f95b4d27 failed (during MaterializingWorkflowDescriptorState): Workflow input processing failed:
    Unable to load namespace from workflow: Failed to import workflow https://storage.googleapis.com/[clipped]

  • lucdhlucdh Member ✭✭

    Just found out my question relates to the following March 2017 post:
    [WDL][Cromwell] Submitting a workflow with a subworkflow to the cloud.
    The conclusion then was that the problem was known, but ETA could not be provided.
    Has there been any progress since?

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @lucdh,

    I don't believe the wdl_runner supports workflow dependencies yet. However, If you'd like to run the $5 genome pipeline, you have a few options:
    1. Clone the pipeline repo and run the workflow using a cromwell jar directly.
    2. Switch the import statements in https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/blob/master/germline_single_sample_workflow.wdl#L29-L32 to use http imports:

    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/unmapped_bam_to_aligned_bam.wdl" as ToBam
    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/germline_variant_discovery.wdl" as Calling
    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/qc.wdl" as QC
    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/utilities.wdl" as Utils
    

    Let me know if you still experience issues.

    Thanks!

  • lucdhlucdh Member ✭✭
    edited March 16

    Hi Rui,
    thanks for your suggestions, I would like to stick to solution (2) if possible. In fact I already tried

    import "https://storage.googleapis.com/...unmapped_bam_to_aligned_bam.wdl" as toBam
    ...
    

    but that did not work. Now I copied your import lines, but still no luck:

    2018-03-16 14:34:32,786 cromwell-system-akka.dispatchers.engine-dispatcher-23 ERROR - WorkflowManagerActor Workflow 2fec3ff9-d302-4a00-ab33-0cffc6ba0982 failed (during MaterializingWorkflowDescriptorState): Workflow input processing failed:
    Unable to load namespace from workflow: Failed to import workflow https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/unmapped_bam_to_aligned_bam.wdl.:
    

    I noticed the wdl runner on
    https://github.com/broadinstitute/wdl/tree/master/runners/cromwell_on_google/wdl_runner
    uses Cromwell release 29. Could it be that the "import URI" statements require Cromwell release 30 or higher?

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @lucdh,

    Someone just pointed me to the properly published version of the $5 genome already using http imports:
    https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/blob/master/fc_germline_single_sample_workflow.wdl

Sign In or Register to comment.