To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

WDL: how to input and scatter an Array[Object] file ("Can't index" error)

myourshawmyourshaw University of ColoradoMember

First, As a satisfied user of Queue, I'm very impressed with the new WDL/Cromwell stack. Congratulations and thanks!

I'm trying to write a WDL script that inputs from a file a list of file paths and related metadata. I get an error when using Array[Object] metadata = read_objects(metadata_file) and scatter (record in metadata). I'm using Cromwell 24. There was an earlier report of errors with a similar wdl, which were supposed to have been fixed in Cromwell 23/24. This error is different. Details follow.

Command:
java -jar ~/apps/cromwell.jar run test1.wdl test1_inputs.json

WDL file (same error even if call is omitted:

workflow test1 {
    File metadata_file  # list of string, integer, file
    Array[Object] metadata = read_objects(metadata_file)

    scatter (record in metadata) {
        String id = record['id']
        String type = record['type']
        File file = record['file']
        call print_string as print_id {
            input: in=id
        }
    }
}
task print_string {
    String in
    command {
        echo ${in}
    }
    output {
        File item_out = stdout()
    }
}

inputs.json:

{
  "test1.metadata_file": "./test1.tsv"
}

metadata file:

id  type    file
sampleA 1   ./test1.bam

output - error:

[2017-01-11 12:17:33,00] [info] Slf4jLogger started
[2017-01-11 12:17:33,07] [info] RUN sub-command
[2017-01-11 12:17:33,07] [info]   WDL file: /home/yoursham/hdd/dev/amc-cmoco/wdl/test1.wdl
[2017-01-11 12:17:33,07] [info]   Inputs: /home/yoursham/hdd/dev/amc-cmoco/wdl/test1_inputs.json
[2017-01-11 12:17:33,10] [info] SingleWorkflowRunnerActor: Submitting workflow
[2017-01-11 12:17:33,14] [info] Workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 submitted.
[2017-01-11 12:17:33,14] [info] SingleWorkflowRunnerActor: Workflow submitted 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3
[2017-01-11 12:17:33,59] [info] Running with database db.url = jdbc:hsqldb:mem:f08fa0c6-9176-4b6b-92fa-8634d4804320;shutdown=false;hsqldb.tx=mvcc
[2017-01-11 12:17:37,47] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2017-01-11 12:17:37,48] [info] [RenameWorkflowOptionsInMetadata] 100%
[2017-01-11 12:17:37,50] [info] Metadata summary refreshing every 2 seconds.
[2017-01-11 12:17:37,53] [info] 1 new workflows fetched
[2017-01-11 12:17:37,53] [info] WorkflowManagerActor Starting workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3
[2017-01-11 12:17:37,53] [info] WorkflowManagerActor Successfully started WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3
[2017-01-11 12:17:37,53] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2017-01-11 12:17:37,78] [info] MaterializeWorkflowDescriptorActor [838b9e0a]: Call-to-Backend assignments: test1.print_id -> Local
[2017-01-11 12:17:37,97] [info] Message [cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$DeclarationEvaluationFailedResponse] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-01-11 12:17:37,97] [info] Message [cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$DeclarationEvaluationFailedResponse] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3/WorkflowExecutionActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#1314870051] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-01-11 12:17:37,99] [info] Message [cromwell.subworkflowstore.SubWorkflowStoreActor$SubWorkflowStoreCompleteSuccess] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$b#1359240631] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3#-1142450047] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-01-11 12:17:37,99] [error] WorkflowManagerActor Workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 failed (during ExecutingWorkflowState): Can't index Success(WdlObject(Map(id -> WdlString(sampleA), type -> WdlString(1), file -> WdlString(./test1.bam)))) with index Success(WdlString(file))
wdl4s.WdlExpressionException: Can't index Success(WdlObject(Map(id -> WdlString(sampleA), type -> WdlString(1), file -> WdlString(./test1.bam)))) with index Success(WdlString(file))
    at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:153)
    at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
    at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.processRunnableDynamicDeclaration(WorkflowExecutionActor.scala:433)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$18.apply(WorkflowExecutionActor.scala:362)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$18.apply(WorkflowExecutionActor.scala:353)
    at scala.collection.immutable.List.map(List.scala:273)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.cromwell$engine$workflow$lifecycle$execution$WorkflowExecutionActor$$startRunnableScopes(WorkflowExecutionActor.scala:353)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.handleExecutionSuccess(WorkflowExecutionActor.scala:326)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.cromwell$engine$workflow$lifecycle$execution$WorkflowExecutionActor$$handleDeclarationEvaluationSuccessful(WorkflowExecutionActor.scala:308)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$3.applyOrElse(WorkflowExecutionActor.scala:107)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor$$anonfun$3.applyOrElse(WorkflowExecutionActor.scala:82)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at akka.actor.FSM$class.processEvent(FSM.scala:663)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowExecutionActor.scala:33)
    at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.processEvent(WorkflowExecutionActor.scala:33)
    at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at cromwell.engine.workflow.lifecycle.execution.WorkflowExecutionActor.aroundReceive(WorkflowExecutionActor.scala:33)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2017-01-11 12:17:37,99] [info] WorkflowManagerActor WorkflowActor-838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 is in a terminal state: WorkflowFailedState
[2017-01-11 12:17:39,91] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
Workflow 838b9e0a-5dc3-4fd4-8d87-ff5903a4ccf3 transitioned to state Failed
Tagged:

Best Answer

Answers

Sign In or Register to comment.