To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

How do you specify default values in a WDL runtime block?

LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev
edited August 2017 in Ask the WDL team

Please note how defaults are being specified in the runtime block below:

task PadTargets {
    File targets
    Int? padding
    File gatk_jar

    # Runtime parameters
    Int? mem
    String gatk_docker
    Int? preemptible_attempts
    Int? disk_space_gb

    # Determine output filename
    String filename = select_first([targets, ""])
    String base_filename = basename(filename, ".tsv")

    command <<<
        echo ${filename}; \
        java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \
            --targets ${targets} \
            --padding ${default=250 padding} \
            --output ${base_filename}.padded.tsv
    >>>

    runtime {
        docker: "${gatk_docker}"
        memory: "${default=2 mem+1}" + " GB"
        disks: "local-disk ${default=40 disk_space_gb} HDD"
        preemptible: "${default=2 preemptible_attempts}"
    }

    output {
        File padded_targets = "${base_filename}.padded.tsv"
    }
}

This WDL validates fine, but yields an error in cromwell at runtime:

[2017-08-09 13:58:57,55] [info] WorkflowExecutionActor-850afb1f-7ac8-4ff3-85cb-a5530b44cf0f [850afb1f]: Starting calls: CNVSomaticPanelWorkflow.PadTargets:NA:1

[2017-08-09 13:58:57,70] [error] WorkflowManagerActor Workflow 850afb1f-7ac8-4ff3-85cb-a5530b44cf0f failed (during ExecutingWorkflowState): Could not resolve variable default as a task input

wdl4s.WdlExpressionException: Could not resolve variable default as a task input
...snip...

Changing the runtime block to something more like:

    runtime {
        docker: "${gatk_docker}"
        memory: ${default=2 mem+1} + " GB"
        disks: "local-disk " + ${default=40 disk_space_gb} + " HDD"
        preemptible: ${default=2 preemptible_attempts}
    }

does not validate.

Apologies if this is a dupe.

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev

    I think this might be because the default=, as interpolated into a command line, has to be a string. Could you double check whether it works if you wrap the numbers in ""?

    Of course, the error message/error reporting could definitely be improved if that is the case.

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev
        runtime {
            docker: "${gatk_docker}"
            memory: ${default="2" mem+1} + " GB"
            disks: "local-disk " + ${default="40" disk_space_gb} + " HDD"
            preemptible: ${default="2" preemptible_attempts}
        }
    
    

    Still does not validate:

    Unrecognized token on line 31, column 17:
    
            memory: ${default="2" mem+1} + " GB"
                    ^
    
    
  • ThibThib CambridgeMember, Broadie, Dev
    edited August 2017

    You could set the default value directly in the declaration itself:

    task PadTargets {
        File targets
        Int? padding
        File gatk_jar
    
        # Runtime parameters
        Int? mem = 2
        String gatk_docker
        Int? preemptible_attempts = 2
        Int? disk_space_gb = 40
    
        # Determine output filename
        String filename = select_first([targets, ""])
        String base_filename = basename(filename, ".tsv")
    
        command <<<
            echo ${filename}; \
            java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \
                --targets ${targets} \
                --padding ${default=250 padding} \
                --output ${base_filename}.padded.tsv
        >>>
    
        runtime {
            docker: "${gatk_docker}"
            memory: "${mem+1} GB"
            disks: "local-disk ${disk_space_gb} HDD"
            preemptible: "${preemptible_attempts}"
        }
    
        output {
            File padded_targets = "${base_filename}.padded.tsv"
        }
    }
    

    The ${default=...} syntax only works in the command section.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev
    edited August 2017

    I also think the {default="" x} syntax might only work in the command block. I'm not 100% sure but I suspect it might not work at all in the runtime strings.

    In that case you can copy the select_first style you've used elsewhere, eg:

    runtime{
      memory: select_first([mem, 250]) + "GB"
    }
    
  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL FYI... The defaults in the command line should work fine. This task used to work before the runtime block was added.

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @Thib I cannot do that, because if I do, the parameters will no longer show up in wdltool inputs ....

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev

    @LeeTL1220 they used to show up with some awkward "x": "OPTIONAL" placeholder. If that's stopped then it's probably a wdltool bug.

    In the mean time, the select_first should work

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL Validates... I am running it now...

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL If I use "?" without "=x", then the placeholder shows up. As soon as I add "=x" the placeholder disappears whether the input variable is optional or not.

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL Error:

    [2017-08-09 15:07:47,52] [error] WorkflowManagerActor Workflow d08c8f7c-b4eb-4a6f-9ba5-4abb8d2d969c failed (during ExecutingWorkflowState): Sorry! Operation + is not supported on empty optional values. You might resolve this using select_first([optional, default]) to guarantee that you have a filled value.
    
    wdl4s.exception.OptionalNotSuppliedException: Sorry! Operation + is not supported on empty optional values. You might resolve this using select_first([optional, default]) to guarantee that you have a filled value.
    

    Here is my runtime block for confirmation:

        runtime {
            docker: "${gatk_docker}"
            memory: select_first([mem+1, 2]) + " GB"
            disks: "local-disk " + select_first([disk_space_gb, 40]) + " HDD"
            preemptible: select_first([preemptible_attempts, 2])
        }
    
    
  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL crud... it's the "+1"

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev
  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev

    @LeeTL1220 as I was writing this ticket I realised there's a little ambiguity about what the inputs JSON would look like. Feel free to comment https://github.com/broadinstitute/cromwell/issues/2532

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL @Thib Looks like I have made progress by using select_first in the runtime blocks. I'll keep you posted.

  • oskarvoskarv BergenMember
    edited August 2017

    Why not use the workflowoptions.json file for that instead? Create it and put this in it and it should work:

    {
      "default_runtime_attributes": {
            docker: "your/docker:image"
            memory: "8G"
            disks: "/mount/point1 HDD"
            preemptible: "2"
      }
    }
    

    Now just run:

    java -jar cromwell.jar run script.wdl inputs.json workflowoptions.json
    

    If you want to override the default you just use the runtime {} code in the wdl script like normal.

    Or define it in the inputs.json file like they do in their best practices pipeline on github, e.g

    {
    "PairedEndSingleSampleWorkflow.agg_small_disk": 200,
    }
    

    And then in the wdl script you write

    Int flowcell_small_disk
    

    where you define File, String, Array etc in the call section. Check it out:
    The .json file: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.inputs.json#L106
    The wdl script: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.wdl#L1120

    Post edited by oskarv on
  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @oskarv Unfortunately, for the usage patterns of this WDL, we cannot use a workflow options file. In fact, one of the side reasons I am working on this is to remove that as a dependency.

  • alongaloralongalor Member
    edited September 2017

    I have a different issue with the ${default="foobar" s} feature where s is declared as String? s. For some reason, when s is not input in the inputs.json file, the default function yields "None"rather than "foobar" as expected.

    I have a task that looks as follows (Note only the lines String? Ts_Filter_Level_Indels and --ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \):

    task ApplyRecalibrationIndels {
    
      File Tranches_Indels_VCF
      File Recal_Indels_VCF
      File Raw_VCF
      String? Ts_Filter_Level_Indels
      String CohortName
      String Chromosome
      File? Parallelization
    
      Map[String, String] Paths
      Array[String] RuntimeParams
    
      command {
        ${Paths["java"]} -jar ${Paths["gatk"]} \
          -T ApplyRecalibration \
          -R ${Paths["refFasta"]} \
          -input ${Raw_VCF} \
          -tranchesFile ${Tranches_Indels_VCF} \
          -recalFile ${Recal_Indels_VCF} \
          -L ${default=Chromosome Parallelization} \
          -o ${CohortName}_${Chromosome}_Indels_Filtered.vcf \
          --ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \
          -mode INDEL \
      }
    
      runtime {
        runtime_minutes: RuntimeParams[6]
        cpus: RuntimeParams[15]
        requested_memory_mb_per_core: RuntimeParams[24]
        queue: RuntimeParams[33]
      }
    
      output {
        File filtered_Indels_VCF = "${CohortName}_${Chromosome}_Indels_Filtered.vcf"
      }
    }
    

    When it is called at runtime Cromwell displays the following (Note the line --ts_filter_level \):

    [2017-09-18 15:33:05,48] [info] DispatchedConfigAsyncJobExecutionActor [40d6c2a9sub.VariantDiscoveryPerChrom.ApplyRecalibrationIndels:NA:1]: /n/data1/hms/dbmi/park/alon/0_GATK_Tools/jdk1.8.0_45/bin/java -jar /n/data1/hms/dbmi/park/alon/0_GATK_Tools/GenomeAnalysisTK.jar \
      -T ApplyRecalibration \
      -R /home/sl279/BiO/Install/GATK-bundle/2.8/b37/human_g1k_v37_decoy.fasta \
      -input /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-GenotypeGVCFs/execution/Yale_SC_All_Chroms_rawVariants.vcf \
      -tranchesFile /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-VariantRecalibratorIndels/execution/Yale_SC_All_Chroms_Indels.tranches \
      -recalFile /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-VariantRecalibratorIndels/execution/Yale_SC_All_Chroms_Indels.recal \
      -L /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/allChroms.intervals \
      -o Yale_SC_All_Chroms_Indels_Filtered.vcf \
      --ts_filter_level  \
      -mode INDEL \
    

    An error follows suit:

    ##### ERROR MESSAGE: Argument '--ts_filter_level' requires a value but none was provided
    

    My inputs file is as follows (Note "VariantDiscovery.Ts_Filter_Level_Indels": "",)

    {
      "VariantDiscovery.inputRuntimeParamsFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/runtimeParams.tsv",
      "VariantDiscovery.Ts_Filter_Level_SNPs": "",
      "VariantDiscovery.Ts_Filter_Level_Indels": "",
      "VariantDiscovery.inputSamplesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/samples.tsv",
      "VariantDiscovery.inputChromosomesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/noParalellization.tsv",
      "VariantDiscovery.inbreedingCoeff": "",
      "VariantDiscovery.callCombineVariants": "True",
      "VariantDiscovery.cohortName": "Yale_SC",
      "VariantDiscovery.Parallelization": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/allChroms.intervals",
      "VariantDiscovery.pcrFree": ""
    }
    

    Attaching my entire script for reference.

    Thanks in advance!

    Alon

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev

    @alongalor the reason this isn't working for you is that "" is considered a value, even though it's 0-length. If you want to see the behavior of the default, you should remove the line from the input JSON altogether.

    I believe from Cromwell 29 onwards you may also supply a javascript null to get the same effect.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev
    edited September 2017

    I just re-read the top half of your comment. Just to confirm, you're saying that if you had a task like this:

    task foo {
      String? bar
      command {
        echo "result: ${default="d" bar}"
      }
      output {
        String out = read_string(stdout())
      }
    }
    

    then if bar is not supplied then the output is result: None? If that's the case then it's a bug which we should fix asap!

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev

    @alongalor I just tried submitting this to Cromwell with no inputs at all, and got the expected result:

    • WDL
    task foo {
      String? bar
      command {
        echo "result: ${default="d" bar}"
      }
      output {
        String out = read_string(stdout())
      }
    }
    
    workflow foowf {
        call foo
    }
    
    • Result
    {
      "foowf.foo.out": "result: d"
    }
    

    Could you confirm what happens when you run that, and if you see result: None, what Cromwell version you're using? Thanks!

  • alongaloralongalor Member
    edited September 2017

    @ChrisL said:
    @alongalor the reason this isn't working for you is that "" is considered a value, even though it's 0-length. If you want to see the behavior of the default, you should remove the line from the input JSON altogether.

    I believe from Cromwell 29 onwards you may also supply a javascript null to get the same effect.

    Hey Chris, thanks a lot for your quick reply and your help! It seems that my issue was actually the one you address in the comment I am including above! I did not realize that! I read all the documentation I could find about the usage of ${default=....} but did not see reference of that. Is there documentation about this rule that I missed? Thanks so much!

Sign In or Register to comment.