We've moved!
For WDL questions, see the WDL specification and WDL docs.
For Cromwell questions, see the Cromwell docs and please post any issues on Github.
For WDL questions, see the WDL specification and WDL docs.
For Cromwell questions, see the Cromwell docs and please post any issues on Github.
How do you specify default values in a WDL runtime block?
LeeTL1220
Arlington, MAMember, Broadie, Dev ✭✭✭
Please note how defaults are being specified in the runtime block below:
task PadTargets {
File targets
Int? padding
File gatk_jar
# Runtime parameters
Int? mem
String gatk_docker
Int? preemptible_attempts
Int? disk_space_gb
# Determine output filename
String filename = select_first([targets, ""])
String base_filename = basename(filename, ".tsv")
command <<<
echo ${filename}; \
java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \
--targets ${targets} \
--padding ${default=250 padding} \
--output ${base_filename}.padded.tsv
>>>
runtime {
docker: "${gatk_docker}"
memory: "${default=2 mem+1}" + " GB"
disks: "local-disk ${default=40 disk_space_gb} HDD"
preemptible: "${default=2 preemptible_attempts}"
}
output {
File padded_targets = "${base_filename}.padded.tsv"
}
}
This WDL validates fine, but yields an error in cromwell at runtime:
[2017-08-09 13:58:57,55] [info] WorkflowExecutionActor-850afb1f-7ac8-4ff3-85cb-a5530b44cf0f [850afb1f]: Starting calls: CNVSomaticPanelWorkflow.PadTargets:NA:1 [2017-08-09 13:58:57,70] [error] WorkflowManagerActor Workflow 850afb1f-7ac8-4ff3-85cb-a5530b44cf0f failed (during ExecutingWorkflowState): Could not resolve variable default as a task input wdl4s.WdlExpressionException: Could not resolve variable default as a task input ...snip...
Changing the runtime block to something more like:
runtime {
docker: "${gatk_docker}"
memory: ${default=2 mem+1} + " GB"
disks: "local-disk " + ${default=40 disk_space_gb} + " HDD"
preemptible: ${default=2 preemptible_attempts}
}
does not validate.
Apologies if this is a dupe.
Answers
I think this might be because the
default=, as interpolated into a command line, has to be a string. Could you double check whether it works if you wrap the numbers in""?Of course, the error message/error reporting could definitely be improved if that is the case.
runtime { docker: "${gatk_docker}" memory: ${default="2" mem+1} + " GB" disks: "local-disk " + ${default="40" disk_space_gb} + " HDD" preemptible: ${default="2" preemptible_attempts} }Still does not validate:
Unrecognized token on line 31, column 17: memory: ${default="2" mem+1} + " GB" ^You could set the default value directly in the declaration itself:
task PadTargets { File targets Int? padding File gatk_jar # Runtime parameters Int? mem = 2 String gatk_docker Int? preemptible_attempts = 2 Int? disk_space_gb = 40 # Determine output filename String filename = select_first([targets, ""]) String base_filename = basename(filename, ".tsv") command <<< echo ${filename}; \ java -Xmx${default=1 mem}g -jar ${gatk_jar} PadTargets \ --targets ${targets} \ --padding ${default=250 padding} \ --output ${base_filename}.padded.tsv >>> runtime { docker: "${gatk_docker}" memory: "${mem+1} GB" disks: "local-disk ${disk_space_gb} HDD" preemptible: "${preemptible_attempts}" } output { File padded_targets = "${base_filename}.padded.tsv" } }The
${default=...}syntax only works in the command section.I also think the
{default="" x}syntax might only work in thecommand block. I'm not 100% sure but I suspect it might not work at all in theruntimestrings.In that case you can copy the select_first style you've used elsewhere, eg:
runtime{ memory: select_first([mem, 250]) + "GB" }@ChrisL FYI... The defaults in the command line should work fine. This task used to work before the runtime block was added.
@Thib I cannot do that, because if I do, the parameters will no longer show up in
wdltool inputs ....@LeeTL1220 they used to show up with some awkward
"x": "OPTIONAL"placeholder. If that's stopped then it's probably a wdltool bug.In the mean time, the
select_firstshould work@ChrisL Validates... I am running it now...
@ChrisL If I use "?" without "=x", then the placeholder shows up. As soon as I add "=x" the placeholder disappears whether the input variable is optional or not.
@ChrisL Error:
Here is my runtime block for confirmation:
runtime { docker: "${gatk_docker}" memory: select_first([mem+1, 2]) + " GB" disks: "local-disk " + select_first([disk_space_gb, 40]) + " HDD" preemptible: select_first([preemptible_attempts, 2]) }@ChrisL crud... it's the "+1"
@ChrisL Rerunning...
@LeeTL1220 as I was writing this ticket I realised there's a little ambiguity about what the inputs JSON would look like. Feel free to comment https://github.com/broadinstitute/cromwell/issues/2532
@ChrisL @Thib Looks like I have made progress by using select_first in the runtime blocks. I'll keep you posted.
Why not use the workflowoptions.json file for that instead? Create it and put this in it and it should work:
{ "default_runtime_attributes": { docker: "your/docker:image" memory: "8G" disks: "/mount/point1 HDD" preemptible: "2" } }Now just run:
If you want to override the default you just use the runtime {} code in the wdl script like normal.
Or define it in the inputs.json file like they do in their best practices pipeline on github, e.g
{ "PairedEndSingleSampleWorkflow.agg_small_disk": 200, }And then in the wdl script you write
where you define File, String, Array etc in the call section. Check it out:
The .json file: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.inputs.json#L106
The wdl script: https://github.com/broadinstitute/wdl/blob/develop/scripts/broad_pipelines/PublicPairedSingleSampleWf_170412.wdl#L1120
@oskarv Unfortunately, for the usage patterns of this WDL, we cannot use a workflow options file. In fact, one of the side reasons I am working on this is to remove that as a dependency.
I have a different issue with the
${default="foobar" s}feature wheresis declared asString? s. For some reason, whensis not input in the inputs.json file, the default function yields"None"rather than"foobar"as expected.I have a task that looks as follows (Note only the lines
String? Ts_Filter_Level_Indelsand--ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \):task ApplyRecalibrationIndels { File Tranches_Indels_VCF File Recal_Indels_VCF File Raw_VCF String? Ts_Filter_Level_Indels String CohortName String Chromosome File? Parallelization Map[String, String] Paths Array[String] RuntimeParams command { ${Paths["java"]} -jar ${Paths["gatk"]} \ -T ApplyRecalibration \ -R ${Paths["refFasta"]} \ -input ${Raw_VCF} \ -tranchesFile ${Tranches_Indels_VCF} \ -recalFile ${Recal_Indels_VCF} \ -L ${default=Chromosome Parallelization} \ -o ${CohortName}_${Chromosome}_Indels_Filtered.vcf \ --ts_filter_level ${default="99.0" Ts_Filter_Level_Indels} \ -mode INDEL \ } runtime { runtime_minutes: RuntimeParams[6] cpus: RuntimeParams[15] requested_memory_mb_per_core: RuntimeParams[24] queue: RuntimeParams[33] } output { File filtered_Indels_VCF = "${CohortName}_${Chromosome}_Indels_Filtered.vcf" } }When it is called at runtime Cromwell displays the following (Note the line
--ts_filter_level \):An error follows suit:
My inputs file is as follows (Note
"VariantDiscovery.Ts_Filter_Level_Indels": "",){ "VariantDiscovery.inputRuntimeParamsFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/runtimeParams.tsv", "VariantDiscovery.Ts_Filter_Level_SNPs": "", "VariantDiscovery.Ts_Filter_Level_Indels": "", "VariantDiscovery.inputSamplesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/UserInput/samples.tsv", "VariantDiscovery.inputChromosomesFile": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/noParalellization.tsv", "VariantDiscovery.inbreedingCoeff": "", "VariantDiscovery.callCombineVariants": "True", "VariantDiscovery.cohortName": "Yale_SC", "VariantDiscovery.Parallelization": "/n/data1/hms/dbmi/park/DATA/BSMCommonExperiment/Yale_SC/bam/test/bam/take6/Variant-Discovery/WDLScripts/WDLInputFiles/allChroms.intervals", "VariantDiscovery.pcrFree": "" }Attaching my entire script for reference.
Thanks in advance!
Alon
@alongalor the reason this isn't working for you is that
""is considered a value, even though it's 0-length. If you want to see the behavior of the default, you should remove the line from the input JSON altogether.I believe from Cromwell 29 onwards you may also supply a javascript
nullto get the same effect.I just re-read the top half of your comment. Just to confirm, you're saying that if you had a task like this:
task foo { String? bar command { echo "result: ${default="d" bar}" } output { String out = read_string(stdout()) } }then if
baris not supplied then the output isresult: None? If that's the case then it's a bug which we should fix asap!@alongalor I just tried submitting this to Cromwell with no inputs at all, and got the expected result:
task foo { String? bar command { echo "result: ${default="d" bar}" } output { String out = read_string(stdout()) } } workflow foowf { call foo }{ "foowf.foo.out": "result: d" }Could you confirm what happens when you run that, and if you see
result: None, what Cromwell version you're using? Thanks!Hey Chris, thanks a lot for your quick reply and your help! It seems that my issue was actually the one you address in the comment I am including above! I did not realize that! I read all the documentation I could find about the usage of ${default=....} but did not see reference of that. Is there documentation about this rule that I missed? Thanks so much!