We've moved!
For WDL questions, see the WDL specification and WDL docs.
For Cromwell questions, see the Cromwell docs and please post any issues on Github.

wdl array length not known at runtime?

esalinasesalinas BroadMember, Broadie ✭✭✭

The following WDL won't validate with wdltool 0.8 nor run with cromwell 25
(note the call inside the scatter)

task make_file_arr
    {
    File f
    command <<<
    set -x
    echo "Got localized file $f " ;
    head --verbose ${f}
    for I in `seq 0 9`; do
        echo "Hello from $I" > $I.dat ;
    done ;
    >>>

    runtime {
        docker : "ubuntu:14.04"
        disks: "local-disk 1 HDD"
        }

    output {
        Array[File] hello_arr=glob("*.dat")
        }


    }


task make_rand_arr_and_scatter_prep {

    command <<<

    #make random indices for file selection and for scattering over
    for I in `seq 0 9`; do
        echo "hi"|perl -ne "print rand()*10"|grep -Po '^\d+' >> rand_indices.dat ;
        echo "$I" >> scatter.dat ;
    done ;

    >>>

    runtime {
        docker : "ubuntu:14.04"
        disks: "local-disk 1 HDD"
        }

    output {
        Array[Int] indices_to_use=read_lines("rand_indices.dat")
        Array[Int] indices_for_scatter=read_lines("scatter.dat")
        }


    }


workflow w 
 {

    File mainInput

    call make_file_arr as first_m_f_a {
        input:
            f=mainInput
        }

    call make_rand_arr_and_scatter_prep

    scatter (i in make_rand_arr_and_scatter_prep.indices_for_scatter) {
            call make_file_arr as scatter_m_f_a {
                input:
                    f=make_file_arr.hello_arr[make_rand_arr_and_scatter_prep.indices_to_use[i]]
                }
        }
 }

Note I tried to change "make_rand_arr_and_scatter_prep.indices_to_use[i]" to a constant 0 to see if that would allow validation but it didn't.

wm8b1-75c:atanas_wdl_bug esalinas$ java -jar ~/broadinstitute/wdltool-0.8.jar validate atanas_wdl_bug.wdl 
ERROR: Unexpected symbol (line 67, col 77) when parsing 'e'.

Expected rsquare, got [.

                    f=make_file_arr.hello_arr[make_rand_arr_and_scatter_prep.indices_to_use[i]]
                                                                            ^

$e = :identifier <=> :dot :identifier -> MemberAccess( lhs=$0, rhs=$2 )

wm8b1-75c:atanas_wdl_bug esalinas$ java -jar ~/broadinstitute/wdltool-0.8.jar validate atanas_wdl_bug.wdl 
ERROR: Expression will not evaluate (line 67, col 22):

                    f=make_file_arr.hello_arr[0]
                     ^
wm8b1-75c:atanas_wdl_bug esalinas$ java -jar ~/broadinstitute/cromwell-25.jar  run atanas_wdl_bug.wdl  
[2017-04-11 10:21:35,25] [info] Slf4jLogger started
[2017-04-11 10:21:35,31] [info] RUN sub-command
[2017-04-11 10:21:35,31] [info]   WDL file: /Users/esalinas/atanas_wdl_bug/atanas_wdl_bug.wdl
[2017-04-11 10:21:35,31] [info]   Inputs: /Users/esalinas/atanas_wdl_bug/atanas_wdl_bug.inputs
[2017-04-11 10:21:35,35] [info] SingleWorkflowRunnerActor: Submitting workflow
[2017-04-11 10:21:35,39] [info] Workflow 2c59a1cd-355c-4543-8fc9-0d16de74efff submitted.
[2017-04-11 10:21:35,39] [info] SingleWorkflowRunnerActor: Workflow submitted 2c59a1cd-355c-4543-8fc9-0d16de74efff
[2017-04-11 10:21:35,85] [info] Running with database db.url = jdbc:hsqldb:mem:e202ebe5-382a-4fe0-8363-0544add52acf;shutdown=false;hsqldb.tx=mvcc
[2017-04-11 10:21:40,65] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2017-04-11 10:21:40,65] [info] [RenameWorkflowOptionsInMetadata] 100%
[2017-04-11 10:21:40,70] [info] Metadata summary refreshing every 2 seconds.
[2017-04-11 10:21:41,22] [info] 1 new workflows fetched
[2017-04-11 10:21:41,23] [info] WorkflowManagerActor Starting workflow 2c59a1cd-355c-4543-8fc9-0d16de74efff
[2017-04-11 10:21:41,23] [info] WorkflowManagerActor Successfully started WorkflowActor-2c59a1cd-355c-4543-8fc9-0d16de74efff
[2017-04-11 10:21:41,23] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2017-04-11 10:21:41,38] [error] WorkflowManagerActor Workflow 2c59a1cd-355c-4543-8fc9-0d16de74efff failed (during MaterializingWorkflowDescriptorState): Workflow input processing failed:
Unable to load namespace from workflow: ERROR: Unexpected symbol (line 67, col 77) when parsing 'e'.

Expected rsquare, got [.

                    f=make_file_arr.hello_arr[make_rand_arr_and_scatter_prep.indices_to_use[i]]
                                                                            ^

$e = :identifier <=> :dot :identifier -> MemberAccess( lhs=$0, rhs=$2 )

cromwell.engine.workflow.lifecycle.MaterializeWorkflowDescriptorActor$$anonfun$3$$anon$1: Workflow input processing failed:
Unable to load namespace from workflow: ERROR: Unexpected symbol (line 67, col 77) when parsing 'e'.

Expected rsquare, got [.

                    f=make_file_arr.hello_arr[make_rand_arr_and_scatter_prep.indices_to_use[i]]
                                                                            ^

$e = :identifier <=> :dot :identifier -> MemberAccess( lhs=$0, rhs=$2 )

    at cromwell.engine.workflow.lifecycle.MaterializeWorkflowDescriptorActor$$anonfun$3.applyOrElse(MaterializeWorkflowDescriptorActor.scala:137)
    at cromwell.engine.workflow.lifecycle.MaterializeWorkflowDescriptorActor$$anonfun$3.applyOrElse(MaterializeWorkflowDescriptorActor.scala:129)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at akka.actor.FSM$class.processEvent(FSM.scala:663)
    at cromwell.engine.workflow.lifecycle.MaterializeWorkflowDescriptorActor.akka$actor$LoggingFSM$$super$processEvent(MaterializeWorkflowDescriptorActor.scala:116)
    at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
    at cromwell.engine.workflow.lifecycle.MaterializeWorkflowDescriptorActor.processEvent(MaterializeWorkflowDescriptorActor.scala:116)
    at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
    at cromwell.engine.workflow.lifecycle.MaterializeWorkflowDescriptorActor.aroundReceive(MaterializeWorkflowDescriptorActor.scala:116)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2017-04-11 10:21:41,39] [info] WorkflowManagerActor WorkflowActor-2c59a1cd-355c-4543-8fc9-0d16de74efff is in a terminal state: WorkflowFailedState
[2017-04-11 10:21:41,40] [info] Message [cromwell.subworkflowstore.SubWorkflowStoreActor$SubWorkflowStoreCompleteSuccess] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$b#-178038196] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-2c59a1cd-355c-4543-8fc9-0d16de74efff#-1660678415] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-04-11 10:21:43,25] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
Workflow 2c59a1cd-355c-4543-8fc9-0d16de74efff transitioned to state Failed

I have a suspicion that the reason for this has something to do with the fact that cromwell does not know the size of the "make_file_arr.hello_arr" until runtime.

Atanas Kamburov was trying to write a WDL that did something like this and he got a similar error . Is the above failure to run and failure to validate a bug or issue in cromwell??? Atanas was able to get the WDL to validate by moving the "make_rand_arr_and_scatter_prep.indices_to_use[i]" to be done/evaluated by command in "make_rand_arr_and_scatter_prep"

Answers

Sign In or Register to comment.