Forum Login Issue:
Currently the "Log in with Google" button redirects you to a "Page not found." Our forum vendors have implemented a fix, and now we are just waiting on a patch to be released. In the meantime, while on the "Page not found" you can edit the URL to delete the second gatk, firecloud, or wdl (depending on what subforum you are acessing).
ex: https://gatkforums.broadinstitute.org/gatk/gatk/entry/...

Using imports and workflowDependencies with wdl_runner.sh and gcloud

Hi,
it looks like Cromwell wrapper

runners/cromwell_on_google/wdl_runner/wdl_runner.sh

as obtained from

https://github.com/broadinstitute/wdl.git

cannot handle workflowDependencies and therefore cannot be used to launch the wdl script in

https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/blob/master/germline_single_sample_workflow.wdl

Is that correct?
I could discard the imports and flatten the wdl, but it would be much better to keep the original structure. And it would be nice if I -for now- could keep on using the wrapper, without digging deeper into Cromwell :-)

My current gcloud command is based on the example below, so I guess an extra parameter would be needed to specify the five-dollar-genome-analysis-pipeline/tasks_pipelines folder with the imports?

gcloud \
alpha genomics pipelines run \
--pipeline-file wdl_pipeline.yaml \
--zones us-central1-f \
--inputs-from-file WDL=test-wdl/ga4ghMd5.wdl \
--inputs-from-file WORKFLOW_INPUTS=test-wdl/ga4ghMd5.inputs.json \
--inputs-from-file WORKFLOW_OPTIONS=test-wdl/basic.papi.us.options.json \
--inputs WORKSPACE=gs://YOUR-BUCKET/wdl_runner/work \
--inputs OUTPUTS=gs://YOUR-BUCKET/wdl_runner/output \
--logging gs://YOUR-BUCKET/wdl_runner/logging

Thanks for your help,
--luc

Answers

  • lucdhlucdh Member
    edited March 8

    I tried to upload the tasks_pipelines files to a Google Storage bucket, share them publicly, and use

    import "https://storage.googleapis.com/..."

    statements, but that does not help apparently:

    2018-03-08 17:28:32,040 cromwell-system-akka.dispatchers.engine-dispatcher-6 ERROR - WorkflowManagerActor Workflow 9a5128f9-5939-4567-bb1e-6d97f95b4d27 failed (during MaterializingWorkflowDescriptorState): Workflow input processing failed:
    Unable to load namespace from workflow: Failed to import workflow https://storage.googleapis.com/[clipped]

  • lucdhlucdh Member

    Just found out my question relates to the following March 2017 post:
    [WDL][Cromwell] Submitting a workflow with a subworkflow to the cloud.
    The conclusion then was that the problem was known, but ETA could not be provided.
    Has there been any progress since?

  • RuchiRuchi Member, Broadie, Moderator, Dev

    Hey @lucdh,

    I don't believe the wdl_runner supports workflow dependencies yet. However, If you'd like to run the $5 genome pipeline, you have a few options:
    1. Clone the pipeline repo and run the workflow using a cromwell jar directly.
    2. Switch the import statements in https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/blob/master/germline_single_sample_workflow.wdl#L29-L32 to use http imports:

    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/unmapped_bam_to_aligned_bam.wdl" as ToBam
    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/germline_variant_discovery.wdl" as Calling
    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/qc.wdl" as QC
    import "https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/utilities.wdl" as Utils
    

    Let me know if you still experience issues.

    Thanks!

  • lucdhlucdh Member
    edited March 16

    Hi Rui,
    thanks for your suggestions, I would like to stick to solution (2) if possible. In fact I already tried

    import "https://storage.googleapis.com/...unmapped_bam_to_aligned_bam.wdl" as toBam
    ...
    

    but that did not work. Now I copied your import lines, but still no luck:

    2018-03-16 14:34:32,786 cromwell-system-akka.dispatchers.engine-dispatcher-23 ERROR - WorkflowManagerActor Workflow 2fec3ff9-d302-4a00-ab33-0cffc6ba0982 failed (during MaterializingWorkflowDescriptorState): Workflow input processing failed:
    Unable to load namespace from workflow: Failed to import workflow https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/master/tasks_pipelines/unmapped_bam_to_aligned_bam.wdl.:
    

    I noticed the wdl runner on
    https://github.com/broadinstitute/wdl/tree/master/runners/cromwell_on_google/wdl_runner
    uses Cromwell release 29. Could it be that the "import URI" statements require Cromwell release 30 or higher?

  • RuchiRuchi Member, Broadie, Moderator, Dev

    Hey @lucdh,

    Someone just pointed me to the properly published version of the $5 genome already using http imports:
    https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/blob/master/fc_germline_single_sample_workflow.wdl

Sign In or Register to comment.