Forum Login Issue:
Currently the "Log in with Google" button redirects you to a "Page not found." This is an issue that our forum vendors are working on fixing. In the meantime, while on the "Page not found" you can edit the URL to delete the second gatk, firecloud, or wdl (depending on what subforum you are acessing).
ex: https://gatkforums.broadinstitute.org/gatk/gatk/entry/...

Cromwell hangs forever

I'm trying to run a workflow that used to work, but now just hangs forever. For some reason, it doesn't progress to the next step in the workflow, and just hangs on Status change from WaitingForReturnCodeFile to Done. How can I figure out what is causing cromwell to stall?

2018-01-31 10:23:33,701 INFO  - MaterializeWorkflowDescriptorActor [UUID(38231631)]: Call-to-Backend assignments: trim.after -> LocalExample, trim.before -> LocalExample, trim.beforeTable -> LocalExample, trim.trimmomatic -> LocalExample, trim.sampleNames -> LocalExample, trim.plotTrim -> LocalExample, trim.afterTable -> LocalExample, trim.docker -> LocalExample
2018-01-31 10:23:34,886 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.docker:NA:1
2018-01-31 10:23:35,056 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: `command1`
2018-01-31 10:23:35,072 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: executing: /bin/bash /data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-docker/execution/script
2018-01-31 10:23:35,488 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: job id: 13368
2018-01-31 10:23:35,491 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.docker:NA:1]: Status change from - to Done
2018-01-31 10:23:38,822 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.sampleNames:0:1, trim.sampleNames:1:1, trim.sampleNames:2:1, trim.sampleNames:3:1, trim.sampleNames:4:1, trim.sampleNames:5:1, trim.sampleNames:6:1, trim.sampleNames:7:1, trim.sampleNames:8:1, trim.sampleNames:9:1, trim.sampleNames:10:1
2018-01-31 10:23:39,822 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.trimmomatic:0:1, trim.trimmomatic:1:1, trim.trimmomatic:2:1, trim.trimmomatic:3:1, trim.trimmomatic:4:1, trim.trimmomatic:5:1, trim.trimmomatic:6:1, trim.trimmomatic:7:1, trim.trimmomatic:8:1, trim.trimmomatic:9:1, trim.trimmomatic:10:1
2018-01-31 10:24:16,820 INFO  - WorkflowExecutionActor-38231631-7c60-4584-bf17-0b7f01443f5c [UUID(38231631)]: Starting calls: trim.plotTrim:NA:1
2018-01-31 10:24:18,429 WARN  - BackendPreparationActor_for_38231631:trim.plotTrim:-1:1 [UUID(38231631)]: Docker lookup failed
java.lang.Exception: Unauthorized to get docker hash nvwalab/graph:0.5.1
    at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:196)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:94)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:78)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at akka.actor.FSM.processEvent(FSM.scala:665)
    at akka.actor.FSM.processEvent$(FSM.scala:662)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:39)
    at akka.actor.LoggingFSM.processEvent(FSM.scala:801)
    at akka.actor.LoggingFSM.processEvent$(FSM.scala:783)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:39)
    at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:659)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:653)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:514)
    at akka.actor.Actor.aroundReceive$(Actor.scala:512)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:39)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
    at akka.actor.ActorCell.invoke(ActorCell.scala:496)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-01-31 10:24:18,446 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: `ReadTrimmedFile.py trim --input /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-0/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-1/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-2/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-3/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-4/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-5/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-6/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-7/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-8/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-9/execution/stderr /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/inputs/data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-trimmomatic/shard-10/execution/stderr --html "Report_trim.html" --csv "Report_trim.csv" --names SRR1999981 SRR3205807 SRR3360221 SRR3360222 SRR3371812 SRR3371813 SRR3371814 SRR3951560 SRR4115648 SRR5905028 SRR5909619`
2018-01-31 10:24:18,448 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: executing: docker run \
  --rm -i \
  --user $EUID \
  --entrypoint /bin/bash \
  -v /data/Experiment/redmar/Experiment_94/ecoli/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim:/cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim \
  nvwalab/graph:0.5.1 /cromwell-executions/trim/38231631-7c60-4584-bf17-0b7f01443f5c/call-plotTrim/execution/script
2018-01-31 10:24:18,564 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: job id: 13515
2018-01-31 10:24:18,565 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: Status change from - to WaitingForReturnCodeFile
2018-01-31 10:24:49,049 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(38231631)trim.plotTrim:NA:1]: Status change from WaitingForReturnCodeFile to Done

Best Answer

Answers

Sign In or Register to comment.