To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Cromwell hangs forever

I'm trying to run a workflow that used to work, but now just hangs forever. For some reason, it doesn't progress to the next step in the workflow, and just hangs on Status change from WaitingForReturnCodeFile to Done. How can I figure out what is causing cromwell to stall?

2018-01-31 10:23:33,701 INFO  - MaterializeWorkflowDescriptorActor [UUID(38231631)]: Call-to-Backend assignments: trim.after -> LocalExample, trim.before -> LocalExample, trim.beforeTable -> LocalExample, trim.trimmomatic -> LocalExample, trim.sampleNames -> LocalExample, trim.plotTrim -> LocalExample, trim.afterTable -> LocalExample, trim.docker -> LocalExample
2018-01-31 10:23:34,886 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.docker:NA:1
2018-01-31 10:23:35,056 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: `command1`
2018-01-31 10:23:35,072 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: executing: /bin/bash /data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-docker/execution/script
2018-01-31 10:23:35,488 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: job id: 13368
2018-01-31 10:23:35,491 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: Status change from - to Done
2018-01-31 10:23:38,822 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.sampleNames:0:1, trim.sampleNames:1:1, trim.sampleNames:2:1, trim.sampleNames:3:1, trim.sampleNames:4:1, trim.sampleNames:5:1, trim.sampleNames:6:1, trim.sampleNames:7:1, trim.sampleNames:8:1, trim.sampleNames:9:1, trim.sampleNames:10:1
2018-01-31 10:23:39,822 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.trimmomatic:0:1, trim.trimmomatic:1:1, trim.trimmomatic:2:1, trim.trimmomatic:3:1, trim.trimmomatic:4:1, trim.trimmomatic:5:1, trim.trimmomatic:6:1, trim.trimmomatic:7:1, trim.trimmomatic:8:1, trim.trimmomatic:9:1, trim.trimmomatic:10:1
2018-01-31 10:24:16,820 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.plotTrim:NA:1
2018-01-31 10:24:18,429 WARN  - BackendPreparationActor_for_38231631:trim.plotTrim:-1:1 [UUID(38231631)]: Docker lookup failed
java.lang.Exception: Unauthorized to get docker hash nvwalab/graph:0.5.1
    at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:196)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:94)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:78)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at akka.actor.FSM.processEvent(FSM.scala:665)
    at akka.actor.FSM.processEvent$(FSM.scala:662)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:39)
    at akka.actor.LoggingFSM.processEvent(FSM.scala:801)
    at akka.actor.LoggingFSM.processEvent$(FSM.scala:783)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:39)
    at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:659)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:653)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:514)
    at akka.actor.Actor.aroundReceive$(Actor.scala:512)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:39)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
    at akka.actor.ActorCell.invoke(ActorCell.scala:496)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-01-31 10:24:18,446 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: `ReadTrimmedFile.py trim --input /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-0/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-1/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-2/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-3/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-4/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-5/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-6/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-7/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-8/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-9/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-10/execution/stderr --html "Report_trim.html" --csv "Report_trim.csv" --names SRR1999981 SRR3205807 SRR3360221 SRR3360222 SRR3371812 SRR3371813 SRR3371814 SRR3951560 SRR4115648 SRR5905028 SRR5909619`
2018-01-31 10:24:18,448 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: executing: docker run \
  --rm -i \
  --user $EUID \
  --entrypoint /bin/bash \
  -v /data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim:/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim \
  nvwalab/graph:0.5.1 /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/execution/script
2018-01-31 10:24:18,564 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: job id: 13515
2018-01-31 10:24:18,565 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: Status change from - to WaitingForReturnCodeFile
2018-01-31 10:24:49,049 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: Status change from WaitingForReturnCodeFile to Done

Best Answer

Answers

Sign In or Register to comment.