Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Crashing wdltools

Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭
in Ask the Cromwell + WDL Team

Hi all,

Somehow I've been able to create a workflow that crashes wdltools when you try to validate it. I'm not sure I like what that says about my coding, but I have been able to boil it down to the minimal example shown below.

It crashes every action of wdltools, with the exception of parse.

The file crash_wdltool.wdl

import "report.wdl" as report

workflow wf {
    String user
    call report.makeReport as Report {
        input: user = user
    }
    output {
        File report = Report.report
    }
}

The file report.wdl

workflow makeReport {
    String user

    call Report {
        input: user = user
    }
    output {
        File report = Report.report
    }
}

When I try to validate is using wdltool version 0.10

$ wdltool validate crash_wdltool.wdl 
Exception in thread "main" java.lang.StackOverflowError
    at scala.collection.immutable.List$.newBuilder(List.scala:450)
    at scala.collection.generic.GenericTraversableTemplate.genericBuilder(GenericTraversableTemplate.scala:70)
    at scala.collection.generic.GenericTraversableTemplate.genericBuilder$(GenericTraversableTemplate.scala:70)
    at scala.collection.AbstractTraversable.genericBuilder(Traversable.scala:104)
    at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTraversableFactory.scala:57)
    at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTraversableFactory.scala:52)
    at scala.collection.TraversableLike.collect(TraversableLike.scala:270)
    at scala.collection.TraversableLike.collect$(TraversableLike.scala:269)
    at scala.collection.immutable.List.collect(List.scala:313)
    at wdl4s.Scope.resolveVariable(Scope.scala:159)
    at wdl4s.Scope.resolveVariable$(Scope.scala:153)
    at wdl4s.WorkflowOutput.resolveVariable(WorkflowOutput.scala:16)
    at wdl4s.WdlNamespace$.lookupType(WdlNamespace.scala:438)
    at wdl4s.WdlNamespace$.$anonfun$typeCheckDeclaration$2(WdlNamespace.scala:453)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$1(TypeEvaluator.scala:16)
    at scala.util.Try$.apply(Try.scala:209)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:16)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)

 <SNIPPED REST OF THE OUTPUT>
Tagged:
· Share on Twitter Share on Google+

Answers

  • Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭

    I should add, to fix it you need to change the workflow output for makeReport to any other filename, as long as it isn't report.

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    I tried validating the exact files you present here using the wdltool-0.10.jar release jar and got back an error that reads:
    ```
    ERROR: Call references a task (Report) that doesn't exist (line 4, col 10)

    call Report {
     ^

    ``
    In terms of why wdltool isn't reporting back anything, based on your stacktrace it seems like you tried running the command:
    $ wdltool validate crash_wdltool.wdl but could you please try `$ java -jar <wdltool.jar> validate crash_wdltool.wdl instead?

    · Share on Twitter Share on Google+
  • Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭

    My mistake, I did not include the full report.wdl file in my paste

    task Report {
    String user

    command {
        echo "Hello, ${user}"
    }
    output {
        File report = read_string(stdout())
    }
}

workflow makeReport {
    String user

    call Report {
        input: user = user
    }
    output {
        File report = Report.report
    }
}
    · Share on Twitter Share on Google+
  • Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭

    The wdtool command is just a wrapper that calls the wdltool.jar file.

    $ java -jar wdltool-0.10.jar validate crash_wdltool.wdl 
Exception in thread "main" java.lang.StackOverflowError
    at scala.collection.TraversableLike.collect(TraversableLike.scala:270)
    at scala.collection.TraversableLike.collect$(TraversableLike.scala:269)
    at scala.collection.immutable.List.collect(List.scala:313)
    at wdl4s.Scope.resolveVariable(Scope.scala:159)
    at wdl4s.Scope.resolveVariable$(Scope.scala:153)
    at wdl4s.WorkflowOutput.resolveVariable(WorkflowOutput.scala:16)
    at wdl4s.WdlNamespace$.lookupType(WdlNamespace.scala:438)
    at wdl4s.WdlNamespace$.$anonfun$typeCheckDeclaration$2(WdlNamespace.scala:453)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$1(TypeEvaluator.scala:16)
    at scala.util.Try$.apply(Try.scala:209)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:16)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    · Share on Twitter Share on Google+
  • Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭

    I found a second wdl that causes wdltool problems, here it just prints null when you validate it, and crashes when you call graph.

    $ cat null.wdl 
task empty{
    command {}
    output {
        File out = "${output}"
    }
}

    $ java -jar wdltool-0.10.jar validate null.wdl 
null

    $ java -jar wdltool-0.10.jar graph null.wdl 
Exception in thread "main" scala.MatchError: null
    at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:44)
    at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
    at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
    at wdl4s.expression.ValueEvaluator.replaceInterpolationTag(ValueEvaluator.scala:20)
    at wdl4s.expression.ValueEvaluator.$anonfun$interpolate$2(ValueEvaluator.scala:33)
    at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:157)
    at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:157)
    at scala.collection.Iterator.foreach(Iterator.scala:929)
    at scala.collection.Iterator.foreach$(Iterator.scala:929)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
    at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:157)
    at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:155)
    at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1406)
    at wdl4s.expression.ValueEvaluator.interpolate(ValueEvaluator.scala:33)
    at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:49)
    at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
    at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
    at wdl4s.WdlNamespace$.$anonfun$typeCheckDeclaration$1(WdlNamespace.scala:460)
    at scala.Option.flatMap(Option.scala:171)
    at wdl4s.WdlNamespace$.typeCheckDeclaration(WdlNamespace.scala:452)
    at wdl4s.WdlNamespace$.validateDeclaration(WdlNamespace.scala:432)
    at wdl4s.WdlNamespace$.$anonfun$apply$35(WdlNamespace.scala:358)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
    at scala.collection.Iterator.foreach(Iterator.scala:929)
    at scala.collection.Iterator.foreach$(Iterator.scala:929)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at wdl4s.WdlNamespace$.$anonfun$apply$34(WdlNamespace.scala:357)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
    at scala.collection.immutable.Set$Set2.foreach(Set.scala:130)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at wdl4s.WdlNamespace$.apply(WdlNamespace.scala:356)
    at wdl4s.WdlNamespace$.$anonfun$load$1(WdlNamespace.scala:178)
    at scala.util.Try$.apply(Try.scala:209)
    at wdl4s.WdlNamespace$.load(WdlNamespace.scala:178)
    at wdl4s.WdlNamespace$.loadUsingSource(WdlNamespace.scala:174)
    at wdl4s.WdlNamespaceWithWorkflow$.load(WdlNamespace.scala:555)
    at wdltool.GraphPrint$.generateWorkflowDigraph(GraphPrint.scala:14)
    at wdltool.Main$.$anonfun$graph$2(Main.scala:84)
    at wdltool.Main$.continueIf(Main.scala:94)
    at wdltool.Main$.graph(Main.scala:78)
    at wdltool.Main$.dispatchCommand(Main.scala:37)
    at wdltool.Main$.delayedEndpoint$wdltool$Main$1(Main.scala:151)
    at wdltool.Main$delayedInit$body.apply(Main.scala:12)
    at scala.Function0.apply$mcV$sp(Function0.scala:34)
    at scala.Function0.apply$mcV$sp$(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App.$anonfun$main$1$adapted(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:378)
    at scala.App.main(App.scala:76)
    at scala.App.main$(App.scala:74)
    at wdltool.Main$.main(Main.scala:12)
    at wdltool.Main.main(Main.scala)
    · Share on Twitter Share on Google+
  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev admin

    Thanks for these reports @Redmar_van_den_Berg!

    I just tried these out myself. I can reproduce what you're seeing with null.wdl and agree that the error reporting could be improved (you should be getting a "no such variable" exception). I've raised this as an issue you can track in the WDL4S repo: https://github.com/broadinstitute/wdl4s/issues/112

    When I tried your makeReport example I couldn't recreate the problem (I'm using wdltool 0.11):

    chrisl [develop] wdltool $ java -jar target/scala-2.12/wdltool-0.11.jar validate ~/myWorkflows/makeReport.wdl

chrisl [develop] wdltool $ echo $?
0
chrisl [develop] wdltool $ java -jar target/scala-2.12/wdltool-0.11.jar graph ~/myWorkflows/makeReport.wdl
digraph makeReport {
  "String makeReport.user" -> "call makeReport.Report"
  "call makeReport.Report"
}
    · Share on Twitter Share on Google+
  • Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭

    I'm still using wdltool 0.10, that is the latest release that is available on the github page https://github.com/broadinstitute/wdltool/releases. Is 0.11 still in beta or should I download wdltool from somewhere else?

    · Share on Twitter Share on Google+
  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev admin

    My mistake. 0.11 is the current development branch. Having said that, there's been no code changes to the project since 0.10 was released so it should be functionally identical. When you run your test, is it against the importing version or the all-in-one version of your workflow?

    My success was with the all-in-one workflow.

    · Share on Twitter Share on Google+
  • Redmar_van_den_BergRedmar_van_den_Berg Member ✭✭

    @ChrisL

    The all in one workflow works here as well, but the code that has import "report.wdl" as report does not validate and crashes wdltool.

    · Share on Twitter Share on Google+
Sign In or Register to comment.