WDL: how to input and scatter an Array[Object] file ("Can't index" error)

myourshawmyourshaw University of ColoradoMember

First, As a satisfied user of Queue, I'm very impressed with the new WDL/Cromwell stack. Congratulations and thanks!

I'm trying to write a WDL script that inputs from a file a list of file paths and related metadata. I get an error when using Array[Object] metadata = read_objects(metadata_file) and scatter (record in metadata). I'm using Cromwell 24. There was an earlier report of errors with a similar wdl, which were supposed to have been fixed in Cromwell 23/24. This error is different. Details follow.

Command:
java -jar ~/apps/cromwell.jar run test1.wdl test1_inputs.json

WDL file (same error even if call is omitted:

workflow test1 {
    File metadata_file  # list of string, integer, file
    Array[Object] metadata = read_objects(metadata_file)

    scatter (record in metadata) {
        String id = record['id']
        String type = record['type']
        File file = record['file']
        call print_string as print_id {
            input: in=id
        }
    }
}
task print_string {
    String in
    command {
        echo ${in}
    }
    output {
        File item_out = stdout()
    }
}

inputs.json:

{
  "test1.metadata_file": "./test1.tsv"
}

metadata file:

id  type    file
sampleA 1   ./test1.bam

output - error:

[2017-01-11 12:17:33,00] [info] Slf4jLogger started
[2017-01-11 12:17:33,07] [info] RUN sub-command
[2017-01-11 12:17:33,07] [info]   WDL file: /home/yoursham/hdd/dev/amc-cmoco/wdl/test1.wdl
[2017-01-11 12:17:33,07] [info]   Inputs: /home/yoursham/hdd/dev/amc-cmoco/wdl/test1_inputs.json
[2017-01-11 12:17:33,10] [info] SingleWorkflowRunnerActor: Submitting workflow
[2017-01-11 12:17:33,14] [info] Workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 submitted.
[2017-01-11 12:17:33,14] [info] SingleWorkflowRunnerActor: Workflow submitted 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3
[2017-01-11 12:17:33,59] [info] Running with database db.url = jdbc:hsqldb:mem:f08fa0c6-9176-4b6b-92fa-8634d4804320;shutdown=false;hsqldb.tx=mvcc
[2017-01-11 12:17:37,47] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2017-01-11 12:17:37,48] [info] [RenameWorkflowOptionsInMetadata] 100%
[2017-01-11 12:17:37,50] [info] Metadata summary refreshing every 2 seconds.
[2017-01-11 12:17:37,53] [info] 1 new workflows fetched
[2017-01-11 12:17:37,53] [info] WorkflowManagerActor Starting workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3
[2017-01-11 12:17:37,53] [info] WorkflowManagerActor Successfully started WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3
[2017-01-11 12:17:37,53] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2017-01-11 12:17:37,78] [info] MaterializeWorkflowDescriptorActor [838b9e0a]: Call-to-Backend assignments: test1.print_id -> Local
[2017-01-11 12:17:37,97] [info] Message [cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$DeclarationEvaluationFailedResponse] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-01-11 12:17:37,97] [info] Message [cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$DeclarationEvaluationFailedResponse] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-01-11 12:17:37,99] [info] Message [cromwell.subworkflowstore.SubWorkflowStoreActor$SubWorkflowStoreCompleteSuccess] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$b#1359240631] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#-1142450047] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-01-11 12:17:37,99] [error] WorkflowManagerActor Workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 failed (during ExecutingWorkflowState): Can't index Success(WdlObject(Map(id -> WdlString(sampleA), type -> WdlString(1), file -> WdlString(./test1.bam)))) with index Success(WdlString(file))
wdl4s.WdlExpressionException: Can't index Success(WdlObject(Map(id -> WdlString(sampleA), type -> WdlString(1), file -> WdlString(./test1.bam)))) with index Success(WdlString(file))
    at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:153)
    at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
    at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.processRunnableDynamicDeclaration(WorkflowExecutionActor.scala:433)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$18.apply(WorkflowExecutionActor.scala:362)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$18.apply(WorkflowExecutionActor.scala:353)
    at scala.collection.immutable.List.map(List.scala:273)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.cromwell$engine$workflow$lifecycle$execution$WorkflowExecutionActor$$startRunnableScopes(WorkflowExecutionActor.scala:353)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.handleExecutionSuccess(WorkflowExecutionActor.scala:326)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.cromwell$engine$workflow$lifecycle$execution$WorkflowExecutionActor$$handleDeclarationEvaluationSuccessful(WorkflowExecutionActor.scala:308)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$3.applyOrElse(WorkflowExecutionActor.scala:107)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$3.applyOrElse(WorkflowExecutionActor.scala:82)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at akka.actor.FSM$class.processEvent(FSM.scala:663)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowExecutionActor.scala:33)
    at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.processEvent(WorkflowExecutionActor.scala:33)
    at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.aroundReceive(WorkflowExecutionActor.scala:33)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2017-01-11 12:17:37,99] [info] WorkflowManagerActor WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 is in a terminal state: WorkflowFailedState
[2017-01-11 12:17:39,91] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
Workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 transitioned to state Failed
Tagged:

Best Answer

Answers

Sign In or Register to comment.