Crashing wdltools

Hi all,

Somehow I've been able to create a workflow that crashes wdltools when you try to validate it. I'm not sure I like what that says about my coding, but I have been able to boil it down to the minimal example shown below.

It crashes every action of wdltools, with the exception of parse.

The file crash_wdltool.wdl

import "report.wdl" as report

workflow wf {
    String user
    call report.makeReport as Report {
        input: user = user
    }
    output {
        File report = Report.report
    }
}

The file report.wdl

workflow makeReport {
    String user

    call Report {
        input: user = user
    }
    output {
        File report = Report.report
    }
}

When I try to validate is using wdltool version 0.10

$ wdltool validate crash_wdltool.wdl 
Exception in thread "main" java.lang.StackOverflowError
    at scala.collection.immutable.List$.newBuilder(List.scala:450)
    at scala.collection.generic.GenericTraversableTemplate.genericBuilder(GenericTraversableTemplate.scala:70)
    at scala.collection.generic.GenericTraversableTemplate.genericBuilder$(GenericTraversableTemplate.scala:70)
    at scala.collection.AbstractTraversable.genericBuilder(Traversable.scala:104)
    at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTraversableFactory.scala:57)
    at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTraversableFactory.scala:52)
    at scala.collection.TraversableLike.collect(TraversableLike.scala:270)
    at scala.collection.TraversableLike.collect$(TraversableLike.scala:269)
    at scala.collection.immutable.List.collect(List.scala:313)
    at wdl4s.Scope.resolveVariable(Scope.scala:159)
    at wdl4s.Scope.resolveVariable$(Scope.scala:153)
    at wdl4s.WorkflowOutput.resolveVariable(WorkflowOutput.scala:16)
    at wdl4s.WdlNamespace$.lookupType(WdlNamespace.scala:438)
    at wdl4s.WdlNamespace$.$anonfun$typeCheckDeclaration$2(WdlNamespace.scala:453)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$1(TypeEvaluator.scala:16)
    at scala.util.Try$.apply(Try.scala:209)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:16)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
    at scala.Option.map(Option.scala:146)
    at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    at scala.util.Success.flatMap(Try.scala:247)
    at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)

 <SNIPPED REST OF THE OUTPUT>
Tagged:

Answers

  • I should add, to fix it you need to change the workflow output for makeReport to any other filename, as long as it isn't report.

  • RuchiRuchi Member, Broadie, Moderator, Dev

    I tried validating the exact files you present here using the wdltool-0.10.jar release jar and got back an error that reads:
    ```
    ERROR: Call references a task (Report) that doesn't exist (line 4, col 10)

    call Report {
         ^
    

    ``
    In terms of why wdltool isn't reporting back anything, based on your stacktrace it seems like you tried running the command:
    $ wdltool validate crash_wdltool.wdl but could you please try `$ java -jar <wdltool.jar> validate crash_wdltool.wdl instead?

  • My mistake, I did not include the full report.wdl file in my paste

    task Report {
        String user
    
        command {
            echo "Hello, ${user}"
        }
        output {
            File report = read_string(stdout())
        }
    }
    
    workflow makeReport {
        String user
    
        call Report {
            input: user = user
        }
        output {
            File report = Report.report
        }
    }
    
  • The wdtool command is just a wrapper that calls the wdltool.jar file.

    $ java -jar wdltool-0.10.jar validate crash_wdltool.wdl 
    Exception in thread "main" java.lang.StackOverflowError
        at scala.collection.TraversableLike.collect(TraversableLike.scala:270)
        at scala.collection.TraversableLike.collect$(TraversableLike.scala:269)
        at scala.collection.immutable.List.collect(List.scala:313)
        at wdl4s.Scope.resolveVariable(Scope.scala:159)
        at wdl4s.Scope.resolveVariable$(Scope.scala:153)
        at wdl4s.WorkflowOutput.resolveVariable(WorkflowOutput.scala:16)
        at wdl4s.WdlNamespace$.lookupType(WdlNamespace.scala:438)
        at wdl4s.WdlNamespace$.$anonfun$typeCheckDeclaration$2(WdlNamespace.scala:453)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$1(TypeEvaluator.scala:16)
        at scala.util.Try$.apply(Try.scala:209)
        at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:16)
        at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
        at scala.Option.map(Option.scala:146)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
        at scala.util.Success.flatMap(Try.scala:247)
        at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
        at scala.Option.map(Option.scala:146)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
        at scala.util.Success.flatMap(Try.scala:247)
        at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
        at scala.Option.map(Option.scala:146)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
        at scala.util.Success.flatMap(Try.scala:247)
        at wdl4s.expression.TypeEvaluator.evaluate(TypeEvaluator.scala:75)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$41(TypeEvaluator.scala:80)
        at scala.Option.map(Option.scala:146)
        at wdl4s.expression.TypeEvaluator.$anonfun$evaluate$39(TypeEvaluator.scala:79)
    
  • I found a second wdl that causes wdltool problems, here it just prints null when you validate it, and crashes when you call graph.

    $ cat null.wdl 
    task empty{
        command {}
        output {
            File out = "${output}"
        }
    }
    
    $ java -jar wdltool-0.10.jar validate null.wdl 
    null
    
    $ java -jar wdltool-0.10.jar graph null.wdl 
    Exception in thread "main" scala.MatchError: null
        at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:44)
        at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
        at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
        at wdl4s.expression.ValueEvaluator.replaceInterpolationTag(ValueEvaluator.scala:20)
        at wdl4s.expression.ValueEvaluator.$anonfun$interpolate$2(ValueEvaluator.scala:33)
        at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:157)
        at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:157)
        at scala.collection.Iterator.foreach(Iterator.scala:929)
        at scala.collection.Iterator.foreach$(Iterator.scala:929)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
        at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:157)
        at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:155)
        at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1406)
        at wdl4s.expression.ValueEvaluator.interpolate(ValueEvaluator.scala:33)
        at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:49)
        at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
        at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
        at wdl4s.WdlNamespace$.$anonfun$typeCheckDeclaration$1(WdlNamespace.scala:460)
        at scala.Option.flatMap(Option.scala:171)
        at wdl4s.WdlNamespace$.typeCheckDeclaration(WdlNamespace.scala:452)
        at wdl4s.WdlNamespace$.validateDeclaration(WdlNamespace.scala:432)
        at wdl4s.WdlNamespace$.$anonfun$apply$35(WdlNamespace.scala:358)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
        at scala.collection.Iterator.foreach(Iterator.scala:929)
        at scala.collection.Iterator.foreach$(Iterator.scala:929)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
        at scala.collection.IterableLike.foreach(IterableLike.scala:71)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at wdl4s.WdlNamespace$.$anonfun$apply$34(WdlNamespace.scala:357)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
        at scala.collection.immutable.Set$Set2.foreach(Set.scala:130)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at wdl4s.WdlNamespace$.apply(WdlNamespace.scala:356)
        at wdl4s.WdlNamespace$.$anonfun$load$1(WdlNamespace.scala:178)
        at scala.util.Try$.apply(Try.scala:209)
        at wdl4s.WdlNamespace$.load(WdlNamespace.scala:178)
        at wdl4s.WdlNamespace$.loadUsingSource(WdlNamespace.scala:174)
        at wdl4s.WdlNamespaceWithWorkflow$.load(WdlNamespace.scala:555)
        at wdltool.GraphPrint$.generateWorkflowDigraph(GraphPrint.scala:14)
        at wdltool.Main$.$anonfun$graph$2(Main.scala:84)
        at wdltool.Main$.continueIf(Main.scala:94)
        at wdltool.Main$.graph(Main.scala:78)
        at wdltool.Main$.dispatchCommand(Main.scala:37)
        at wdltool.Main$.delayedEndpoint$wdltool$Main$1(Main.scala:151)
        at wdltool.Main$delayedInit$body.apply(Main.scala:12)
        at scala.Function0.apply$mcV$sp(Function0.scala:34)
        at scala.Function0.apply$mcV$sp$(Function0.scala:34)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:378)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at wdltool.Main$.main(Main.scala:12)
        at wdltool.Main.main(Main.scala)
    
  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    Thanks for these reports @Redmar_van_den_Berg!

    I just tried these out myself. I can reproduce what you're seeing with null.wdl and agree that the error reporting could be improved (you should be getting a "no such variable" exception). I've raised this as an issue you can track in the WDL4S repo: https://github.com/broadinstitute/wdl4s/issues/112

    When I tried your makeReport example I couldn't recreate the problem (I'm using wdltool 0.11):

    chrisl [develop] wdltool $ java -jar target/scala-2.12/wdltool-0.11.jar validate ~/myWorkflows/makeReport.wdl
    
    chrisl [develop] wdltool $ echo $?
    0
    chrisl [develop] wdltool $ java -jar target/scala-2.12/wdltool-0.11.jar graph ~/myWorkflows/makeReport.wdl
    digraph makeReport {
      "String makeReport.user" -> "call makeReport.Report"
      "call makeReport.Report"
    }
    
    
  • I'm still using wdltool 0.10, that is the latest release that is available on the github page https://github.com/broadinstitute/wdltool/releases. Is 0.11 still in beta or should I download wdltool from somewhere else?

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    My mistake. 0.11 is the current development branch. Having said that, there's been no code changes to the project since 0.10 was released so it should be functionally identical. When you run your test, is it against the importing version or the all-in-one version of your workflow?

    My success was with the all-in-one workflow.

  • @ChrisL

    The all in one workflow works here as well, but the code that has import "report.wdl" as report does not validate and crashes wdltool.

Sign In or Register to comment.