How do you specify default values in a WDL runtime block?

LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev
edited August 2017 in Ask the WDL team

Please note how defaults are being specified in the runtime block below:

task PadTargets {
    File targets
    Int? padding
    File gatk_jar

    # Runtime parameters
    Int? mem
    String gatk_docker
    Int? preemptible_attempts
    Int? disk_space_gb

    # Determine output filename
    String filename = select_first([targets, ""])
    String base_filename = basename(filename, ".tsv")

    command <<<
        echo ${filename}; \
        java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \
            --targets ${targets} \
            --padding ${default=250 padding} \
            --output ${base_filename}.padded.tsv
    >>>

    runtime {
        docker: "${gatk_docker}"
        memory: "${default=2 mem+1}" + " GB"
        disks: "local-disk ${default=40 disk_space_gb} HDD"
        preemptible: "${default=2 preemptible_attempts}"
    }

    output {
        File padded_targets = "${base_filename}.padded.tsv"
    }
}

This WDL validates fine, but yields an error in cromwell at runtime:

[2017-08-09 13:58:57,55] [info] WorkflowExecutionActor-850afb1f-7ac8-4ff3-85cb-a5530b44cf0f [850afb1f]: Starting calls: CNVSomaticPanelWorkflow.PadTargets:NA:1

[2017-08-09 13:58:57,70] [error] WorkflowManagerActor Workflow 850afb1f-7ac8-4ff3-85cb-a5530b44cf0f failed (during ExecutingWorkflowState): Could not resolve variable default as a task input

wdl4s.WdlExpressionException: Could not resolve variable default as a task input
...snip...

Changing the runtime block to something more like:

    runtime {
        docker: "${gatk_docker}"
        memory: ${default=2 mem+1} + " GB"
        disks: "local-disk " + ${default=40 disk_space_gb} + " HDD"
        preemptible: ${default=2 preemptible_attempts}
    }

does not validate.

Apologies if this is a dupe.

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    I think this might be because the default=, as interpolated into a command line, has to be a string. Could you double check whether it works if you wrap the numbers in ""?

    Of course, the error message/error reporting could definitely be improved if that is the case.

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev
        runtime {
            docker: "${gatk_docker}"
            memory: ${default="2" mem+1} + " GB"
            disks: "local-disk " + ${default="40" disk_space_gb} + " HDD"
            preemptible: ${default="2" preemptible_attempts}
        }
    
    

    Still does not validate:

    Unrecognized token on line 31, column 17:
    
            memory: ${default="2" mem+1} + " GB"
                    ^
    
    
  • ThibThib CambridgeMember, Broadie, Dev
    edited August 2017

    You could set the default value directly in the declaration itself:

    task PadTargets {
        File targets
        Int? padding
        File gatk_jar
    
        # Runtime parameters
        Int? mem = 2
        String gatk_docker
        Int? preemptible_attempts = 2
        Int? disk_space_gb = 40
    
        # Determine output filename
        String filename = select_first([targets, ""])
        String base_filename = basename(filename, ".tsv")
    
        command <<<
            echo ${filename}; \
            java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \
                --targets ${targets} \
                --padding ${default=250 padding} \
                --output ${base_filename}.padded.tsv
        >>>
    
        runtime {
            docker: "${gatk_docker}"
            memory: "${mem+1} GB"
            disks: "local-disk ${disk_space_gb} HDD"
            preemptible: "${preemptible_attempts}"
        }
    
        output {
            File padded_targets = "${base_filename}.padded.tsv"
        }
    }
    

    The ${default=...} syntax only works in the command section.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev
    edited August 2017

    I also think the {default="" x} syntax might only work in the command block. I'm not 100% sure but I suspect it might not work at all in the runtime strings.

    In that case you can copy the select_first style you've used elsewhere, eg:

    runtime{
      memory: select_first([mem, 250]) + "GB"
    }
    
  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL FYI... The defaults in the command line should work fine. This task used to work before the runtime block was added.

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @Thib I cannot do that, because if I do, the parameters will no longer show up in wdltool inputs ....

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    @LeeTL1220 they used to show up with some awkward "x": "OPTIONAL" placeholder. If that's stopped then it's probably a wdltool bug.

    In the mean time, the select_first should work

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL Validates... I am running it now...

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL If I use "?" without "=x", then the placeholder shows up. As soon as I add "=x" the placeholder disappears whether the input variable is optional or not.

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL Error:

    [2017-08-09 15:07:47,52] [error] WorkflowManagerActor Workflow d08c8f7c-b4eb-4a6f-9ba5-4abb8d2d969c failed (during ExecutingWorkflowState): Sorry! Operation + is not supported on empty optional values. You might resolve this using select_first([optional, default]) to guarantee that you have a filled value.
    
    wdl4s.exception.OptionalNotSuppliedException: Sorry! Operation + is not supported on empty optional values. You might resolve this using select_first([optional, default]) to guarantee that you have a filled value.
    

    Here is my runtime block for confirmation:

        runtime {
            docker: "${gatk_docker}"
            memory: select_first([mem+1, 2]) + " GB"
            disks: "local-disk " + select_first([disk_space_gb, 40]) + " HDD"
            preemptible: select_first([preemptible_attempts, 2])
        }
    
    
  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL crud... it's the "+1"

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev
  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    @LeeTL1220 as I was writing this ticket I realised there's a little ambiguity about what the inputs JSON would look like. Feel free to comment https://github.com/broadinstitute/cromwell/issues/2532

  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @ChrisL @Thib Looks like I have made progress by using select_first in the runtime blocks. I'll keep you posted.

  • oskarvoskarv BergenMember
    edited August 2017

    Why not use the workflowoptions.json file for that instead? Create it and put this in it and it should work:

    {
      "default_runtime_attributes": {
            docker: "your/docker:image"
            memory: "8G"
            disks: "/mount/point1 HDD"
            preemptible: "2"
      }
    }
    

    Now just run:

    java -jar cromwell.jar run script.wdl inputs.json workflowoptions.json
    

    If you want to override the default you just use the runtime {} code in the wdl script like normal.

    Or define it in the inputs.json file like they do in their best practices pipeline on github, e.g

    {
    "PairedEndSingleSampleWorkflow.agg_small_disk": 200,
    }
    

    And then in the wdl script you write

    Int flowcell_small_disk
    

    where you define File, String, Array etc in the call section. Check it out:
    The .json file: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.inputs.json#L106
    The wdl script: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.wdl#L1120

    Post edited by oskarv on
  • LeeTL1220LeeTL1220 Arlington, MAMember, Broadie, Dev

    @oskarv Unfortunately, for the usage patterns of this WDL, we cannot use a workflow options file. In fact, one of the side reasons I am working on this is to remove that as a dependency.

  • alongaloralongalor Member
    edited September 2017

    I have a different issue with the ${default="foobar" s} feature where s is declared as String? s. For some reason, when s is not input in the inputs.json file, the default function yields "None"rather than "foobar" as expected.

    I have a task that looks as follows (Note only the lines String? Ts_Filter_Level_Indels and --ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \):

    task ApplyRecalibrationIndels {
    
      File Tranches_Indels_VCF
      File Recal_Indels_VCF
      File Raw_VCF
      String? Ts_Filter_Level_Indels
      String CohortName
      String Chromosome
      File? Parallelization
    
      Map[String, String] Paths
      Array[String] RuntimeParams
    
      command {
        ${Paths["java"]} -jar ${Paths["gatk"]} \
          -T ApplyRecalibration \
          -R ${Paths["refFasta"]} \
          -input ${Raw_VCF} \
          -tranchesFile ${Tranches_Indels_VCF} \
          -recalFile ${Recal_Indels_VCF} \
          -L ${default=Chromosome Parallelization} \
          -o ${CohortName}_${Chromosome}_Indels_Filtered.vcf \
          --ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \
          -mode INDEL \
      }
    
      runtime {
        runtime_minutes: RuntimeParams[6]
        cpus: RuntimeParams[15]
        requested_memory_mb_per_core: RuntimeParams[24]
        queue: RuntimeParams[33]
      }
    
      output {
        File filtered_Indels_VCF = "${CohortName}_${Chromosome}_Indels_Filtered.vcf"
      }
    }
    

    When it is called at runtime Cromwell displays the following (Note the line --ts_filter_level \):

    [2017-09-18 15:33:05,48] [info] DispatchedConfigAsyncJobExecutionActor [40d6c2a9sub.VariantDiscoveryPerChrom.ApplyRecalibrationIndels:NA:1]: /n/data1/hms/dbmi/park/alon/0_GATK_Tools/jdk1.8.0_45/bin/java -jar /n/data1/hms/dbmi/park/alon/0_GATK_Tools/GenomeAnalysisTK.jar \
      -T ApplyRecalibration \
      -R /home/sl279/BiO/Install/GATK-bundle/2.8/b37/human_g1k_v37_decoy.fasta \
      -input /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-GenotypeGVCFs/execution/Yale_SC_All_Chroms_rawVariants.vcf \
      -tranchesFile /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-VariantRecalibratorIndels/execution/Yale_SC_All_Chroms_Indels.tranches \
      -recalFile /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-VariantRecalibratorIndels/execution/Yale_SC_All_Chroms_Indels.recal \
      -L /n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/cromwell-executions/VariantDiscovery/7ee02f4c-746a-4c83-b27b-1ae3b12f7925/call-VariantDiscoveryPerChrom/shard-0/VariantDiscoveryPerChrom/40d6c2a9-58b0-4d6e-a300-1c0356f2d0d7/call-ApplyRecalibrationIndels/inputs/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/allChroms.intervals \
      -o Yale_SC_All_Chroms_Indels_Filtered.vcf \
      --ts_filter_level  \
      -mode INDEL \
    

    An error follows suit:

    ##### ERROR MESSAGE: Argument '--ts_filter_level' requires a value but none was provided
    

    My inputs file is as follows (Note "VariantDiscovery.Ts_Filter_Level_Indels": "",)

    {
      "VariantDiscovery.inputRuntimeParamsFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/runtimeParams.tsv",
      "VariantDiscovery.Ts_Filter_Level_SNPs": "",
      "VariantDiscovery.Ts_Filter_Level_Indels": "",
      "VariantDiscovery.inputSamplesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/samples.tsv",
      "VariantDiscovery.inputChromosomesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/noParalellization.tsv",
      "VariantDiscovery.inbreedingCoeff": "",
      "VariantDiscovery.callCombineVariants": "True",
      "VariantDiscovery.cohortName": "Yale_SC",
      "VariantDiscovery.Parallelization": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/allChroms.intervals",
      "VariantDiscovery.pcrFree": ""
    }
    

    Attaching my entire script for reference.

    Thanks in advance!

    Alon

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    @alongalor the reason this isn't working for you is that "" is considered a value, even though it's 0-length. If you want to see the behavior of the default, you should remove the line from the input JSON altogether.

    I believe from Cromwell 29 onwards you may also supply a javascript null to get the same effect.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev
    edited September 2017

    I just re-read the top half of your comment. Just to confirm, you're saying that if you had a task like this:

    task foo {
      String? bar
      command {
        echo "result: ${default="d" bar}"
      }
      output {
        String out = read_string(stdout())
      }
    }
    

    then if bar is not supplied then the output is result: None? If that's the case then it's a bug which we should fix asap!

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    @alongalor I just tried submitting this to Cromwell with no inputs at all, and got the expected result:

    • WDL
    task foo {
      String? bar
      command {
        echo "result: ${default="d" bar}"
      }
      output {
        String out = read_string(stdout())
      }
    }
    
    workflow foowf {
        call foo
    }
    
    • Result
    {
      "foowf.foo.out": "result: d"
    }
    

    Could you confirm what happens when you run that, and if you see result: None, what Cromwell version you're using? Thanks!

  • alongaloralongalor Member
    edited September 2017

    @ChrisL said:
    @alongalor the reason this isn't working for you is that "" is considered a value, even though it's 0-length. If you want to see the behavior of the default, you should remove the line from the input JSON altogether.

    I believe from Cromwell 29 onwards you may also supply a javascript null to get the same effect.

    Hey Chris, thanks a lot for your quick reply and your help! It seems that my issue was actually the one you address in the comment I am including above! I did not realize that! I read all the documentation I could find about the usage of ${default=....} but did not see reference of that. Is there documentation about this rule that I missed? Thanks so much!

Sign In or Register to comment.