Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.
How do you specify default values in a WDL runtime block?
LeeTL1220
Arlington, MAMember, Broadie, Dev ✭✭✭
Please note how defaults are being specified in the runtime block below:
task PadTargets {
File targets
Int? padding
File gatk_jar
# Runtime parameters
Int? mem
String gatk_docker
Int? preemptible_attempts
Int? disk_space_gb
# Determine output filename
String filename = select_first([targets, ""])
String base_filename = basename(filename, ".tsv")
command <<<
echo ${filename}; \
java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \
--targets ${targets} \
--padding ${default=250 padding} \
--output ${base_filename}.padded.tsv
>>>
runtime {
docker: "${gatk_docker}"
memory: "${default=2 mem+1}" + " GB"
disks: "local-disk ${default=40 disk_space_gb} HDD"
preemptible: "${default=2 preemptible_attempts}"
}
output {
File padded_targets = "${base_filename}.padded.tsv"
}
}
This WDL validates fine, but yields an error in cromwell at runtime:
[2017-08-09 13:58:57,55] [info] WorkflowExecutionActor-850afb1f-7ac8-4ff3-85cb-a5530b44cf0f [850afb1f]: Starting calls: CNVSomaticPanelWorkflow.PadTargets:NA:1 [2017-08-09 13:58:57,70] [error] WorkflowManagerActor Workflow 850afb1f-7ac8-4ff3-85cb-a5530b44cf0f failed (during ExecutingWorkflowState): Could not resolve variable default as a task input wdl4s.WdlExpressionException: Could not resolve variable default as a task input ...snip...
Changing the runtime block to something more like:
runtime {
docker: "${gatk_docker}"
memory: ${default=2 mem+1} + " GB"
disks: "local-disk " + ${default=40 disk_space_gb} + " HDD"
preemptible: ${default=2 preemptible_attempts}
}
does not validate.
Apologies if this is a dupe.
Answers
I think this might be because the
default=, as interpolated into a command line, has to be a string. Could you double check whether it works if you wrap the numbers in""?Of course, the error message/error reporting could definitely be improved if that is the case.
runtime { docker: "${gatk_docker}" memory: ${default="2" mem+1} + " GB" disks: "local-disk " + ${default="40" disk_space_gb} + " HDD" preemptible: ${default="2" preemptible_attempts} }Still does not validate:
Unrecognized token on line 31, column 17: memory: ${default="2" mem+1} + " GB" ^You could set the default value directly in the declaration itself:
task PadTargets { File targets Int? padding File gatk_jar # Runtime parameters Int? mem = 2 String gatk_docker Int? preemptible_attempts = 2 Int? disk_space_gb = 40 # Determine output filename String filename = select_first([targets, ""]) String base_filename = basename(filename, ".tsv") command <<< echo ${filename}; \ java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \ --targets ${targets} \ --padding ${default=250 padding} \ --output ${base_filename}.padded.tsv >>> runtime { docker: "${gatk_docker}" memory: "${mem+1} GB" disks: "local-disk ${disk_space_gb} HDD" preemptible: "${preemptible_attempts}" } output { File padded_targets = "${base_filename}.padded.tsv" } }The
${default=...}syntax only works in the command section.I also think the
{default="" x}syntax might only work in thecommand block. I'm not 100% sure but I suspect it might not work at all in theruntimestrings.In that case you can copy the select_first style you've used elsewhere, eg:
runtime{ memory: select_first([mem, 250]) + "GB" }@ChrisL FYI... The defaults in the command line should work fine. This task used to work before the runtime block was added.
@Thib I cannot do that, because if I do, the parameters will no longer show up in
wdltool inputs ....@LeeTL1220 they used to show up with some awkward
"x": "OPTIONAL"placeholder. If that's stopped then it's probably a wdltool bug.In the mean time, the
select_firstshould work@ChrisL Validates... I am running it now...
@ChrisL If I use "?" without "=x", then the placeholder shows up. As soon as I add "=x" the placeholder disappears whether the input variable is optional or not.
@ChrisL Error:
Here is my runtime block for confirmation:
runtime { docker: "${gatk_docker}" memory: select_first([mem+1, 2]) + " GB" disks: "local-disk " + select_first([disk_space_gb, 40]) + " HDD" preemptible: select_first([preemptible_attempts, 2]) }@ChrisL crud... it's the "+1"
@ChrisL Rerunning...
@LeeTL1220 as I was writing this ticket I realised there's a little ambiguity about what the inputs JSON would look like. Feel free to comment https://github.com/broadinstitute/cromwell/issues/2532
@ChrisL @Thib Looks like I have made progress by using select_first in the runtime blocks. I'll keep you posted.
Why not use the workflowoptions.json file for that instead? Create it and put this in it and it should work:
{ "default_runtime_attributes": { docker: "your/docker:image" memory: "8G" disks: "/mount/point1 HDD" preemptible: "2" } }Now just run:
If you want to override the default you just use the runtime {} code in the wdl script like normal.
Or define it in the inputs.json file like they do in their best practices pipeline on github, e.g
{ "PairedEndSingleSampleWorkflow.agg_small_disk": 200, }And then in the wdl script you write
where you define File, String, Array etc in the call section. Check it out:
The .json file: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.inputs.json#L106
The wdl script: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.wdl#L1120
@oskarv Unfortunately, for the usage patterns of this WDL, we cannot use a workflow options file. In fact, one of the side reasons I am working on this is to remove that as a dependency.
I have a different issue with the
${default="foobar" s}feature wheresis declared asString? s. For some reason, whensis not input in the inputs.json file, the default function yields"None"rather than"foobar"as expected.I have a task that looks as follows (Note only the lines
String? Ts_Filter_Level_Indelsand--ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \):task ApplyRecalibrationIndels { File Tranches_Indels_VCF File Recal_Indels_VCF File Raw_VCF String? Ts_Filter_Level_Indels String CohortName String Chromosome File? Parallelization Map[String, String] Paths Array[String] RuntimeParams command { ${Paths["java"]} -jar ${Paths["gatk"]} \ -T ApplyRecalibration \ -R ${Paths["refFasta"]} \ -input ${Raw_VCF} \ -tranchesFile ${Tranches_Indels_VCF} \ -recalFile ${Recal_Indels_VCF} \ -L ${default=Chromosome Parallelization} \ -o ${CohortName}_${Chromosome}_Indels_Filtered.vcf \ --ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \ -mode INDEL \ } runtime { runtime_minutes: RuntimeParams[6] cpus: RuntimeParams[15] requested_memory_mb_per_core: RuntimeParams[24] queue: RuntimeParams[33] } output { File filtered_Indels_VCF = "${CohortName}_${Chromosome}_Indels_Filtered.vcf" } }When it is called at runtime Cromwell displays the following (Note the line
--ts_filter_level \):An error follows suit:
My inputs file is as follows (Note
"VariantDiscovery.Ts_Filter_Level_Indels": "",){ "VariantDiscovery.inputRuntimeParamsFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/runtimeParams.tsv", "VariantDiscovery.Ts_Filter_Level_SNPs": "", "VariantDiscovery.Ts_Filter_Level_Indels": "", "VariantDiscovery.inputSamplesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/samples.tsv", "VariantDiscovery.inputChromosomesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/noParalellization.tsv", "VariantDiscovery.inbreedingCoeff": "", "VariantDiscovery.callCombineVariants": "True", "VariantDiscovery.cohortName": "Yale_SC", "VariantDiscovery.Parallelization": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/allChroms.intervals", "VariantDiscovery.pcrFree": "" }Attaching my entire script for reference.
Thanks in advance!
Alon
@alongalor the reason this isn't working for you is that
""is considered a value, even though it's 0-length. If you want to see the behavior of the default, you should remove the line from the input JSON altogether.I believe from Cromwell 29 onwards you may also supply a javascript
nullto get the same effect.I just re-read the top half of your comment. Just to confirm, you're saying that if you had a task like this:
task foo { String? bar command { echo "result: ${default="d" bar}" } output { String out = read_string(stdout()) } }then if
baris not supplied then the output isresult: None? If that's the case then it's a bug which we should fix asap!@alongalor I just tried submitting this to Cromwell with no inputs at all, and got the expected result:
task foo { String? bar command { echo "result: ${default="d" bar}" } output { String out = read_string(stdout()) } } workflow foowf { call foo }{ "foowf.foo.out": "result: d" }Could you confirm what happens when you run that, and if you see
result: None, what Cromwell version you're using? Thanks!Hey Chris, thanks a lot for your quick reply and your help! It seems that my issue was actually the one you address in the comment I am including above! I did not realize that! I read all the documentation I could find about the usage of ${default=....} but did not see reference of that. Is there documentation about this rule that I missed? Thanks so much!