Outgoing request stream error when running cromwell

coyincoyin Cambridge, MAMember, Broadie
edited June 2017 in Ask the WDL team

Hello! I kept getting the following error when running cromwell (version 27) on my local machine (Mac OS) on .wdl scripts and a .json file for inputs. For example, I downloaded the workshop materials from a past FireCloud presentation (https://software.broadinstitute.org/firecloud/documentation/topic?name=firecloud-presentations: BroadE FireCloud Tool Developers Workshop: August 12th, 2016 > Workshop_Materials > task_aliasing_wdl) to test:

[email protected] ~/R/F/W/task_aliasing_wdl> cromwell run task_aliasing.wdl task_aliasing_inputs.json
[2017-06-05 16:55:53,26] [info] Slf4jLogger started
[2017-06-05 16:55:53,40] [info] RUN sub-command
[2017-06-05 16:55:53,40] [info]   WDL file: /Users/c/FireCloud/Workshop_Materials/task_aliasing_wdl/task_aliasing.wdl
[2017-06-05 16:55:53,40] [info]   Inputs: /Users/c/FireCloud/Workshop_Materials/task_aliasing_wdl/task_aliasing_inputs.json
[2017-06-05 16:55:53,52] [info] SingleWorkflowRunnerActor: Submitting workflow
[2017-06-05 16:55:53,95] [info] Running with database db.url = jdbc:hsqldb:mem:4a9c2090-80e1-43af-8439-c2ceba541ace;shutdown=false;hsqldb.tx=mvcc
[2017-06-05 16:56:02,45] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2017-06-05 16:56:02,47] [info] [RenameWorkflowOptionsInMetadata] 100%
[2017-06-05 16:56:02,64] [info] Metadata summary refreshing every 2 seconds.
[2017-06-05 16:56:02,67] [info] Workflow 3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80 submitted.
[2017-06-05 16:56:02,67] [info] SingleWorkflowRunnerActor: Workflow submitted 3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80
[2017-06-05 16:56:03,55] [info] 1 new workflows fetched
[2017-06-05 16:56:03,56] [info] WorkflowManagerActor Starting workflow 3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80
[2017-06-05 16:56:03,56] [info] WorkflowManagerActor Successfully started WorkflowActor-3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80
[2017-06-05 16:56:03,56] [info] Retrieved 1 workflows from the WorkflowStoreActor

...

[2017-06-05 16:56:10,55] [info] BackgroundConfigAsyncJobExecutionActor [3ba4074cTaskAliasingWorkflow.addNumsTask:NA:1]: job id: 32758
[2017-06-05 16:56:10,55] [info] BackgroundConfigAsyncJobExecutionActor [3ba4074cTaskAliasingWorkflow.addNumsTask:NA:1]: Status change from - to WaitingForReturnCodeFile
[2017-06-05 16:56:11,95] [info] BackgroundConfigAsyncJobExecutionActor [3ba4074cTaskAliasingWorkflow.addNumsTask:NA:1]: Status change from WaitingForReturnCodeFile to Done
[2017-06-05 16:56:12,56] [info] WorkflowExecutionActor-3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80 [3ba4074c]: Workflow TaskAliasingWorkflow complete. Final Outputs:
{
  "TaskAliasingWorkflow.SecondAdder.outNum": "7",
  "TaskAliasingWorkflow.addNumsTask.outFile": "/Users/c/FireCloud/Workshop_Materials/task_aliasing_wdl/cromwell-executions/TaskAliasingWorkflow/3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80/call-addNumsTask/execution/outFile.txt",
  "TaskAliasingWorkflow.addNumsTask.outNum": "10",
  "TaskAliasingWorkflow.addNumsTask.log": "/Users/c/FireCloud/Workshop_Materials/task_aliasing_wdl/cromwell-executions/TaskAliasingWorkflow/3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80/call-addNumsTask/execution/log.txt",
  "TaskAliasingWorkflow.FirstAdder.outNum": "3"
}
[2017-06-05 16:56:12,60] [info] WorkflowManagerActor WorkflowActor-3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80 is in a terminal state: WorkflowSucceededState
[2017-06-05 16:56:14,08] [info] SingleWorkflowRunnerActor workflow finished with status 'Succeeded'.
{
  "outputs": {
    "TaskAliasingWorkflow.SecondAdder.outNum": "7",
    "TaskAliasingWorkflow.addNumsTask.outFile": "/Users/c/FireCloud/Workshop_Materials/task_aliasing_wdl/cromwell-executions/TaskAliasingWorkflow/3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80/call-addNumsTask/execution/outFile.txt",
    "TaskAliasingWorkflow.addNumsTask.outNum": "10",
    "TaskAliasingWorkflow.addNumsTask.log": "/Users/c/FireCloud/Workshop_Materials/task_aliasing_wdl/cromwell-executions/TaskAliasingWorkflow/3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80/call-addNumsTask/execution/log.txt",
    "TaskAliasingWorkflow.FirstAdder.outNum": "3"
  },
  "id": "3ba4074c-c1e7-44b9-a64f-d2af3d4dcc80"
}
[2017-06-05 16:56:14,16] [error] Outgoing request stream error
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://cromwell-system/user/StreamSupervisor-1/flow-6-0-unknown-operation#-1334744097]] terminated abruptly
[ERROR] [06/05/2017 16:56:14.156] [cromwell-system-akka.actor.default-dispatcher-37] [akka.actor.ActorSystemImpl(cromwell-system)] Outgoing request stream error (akka.stream.AbruptTerminationException)

The workflow seems to have succeeded, but the errors at the end were not present when I did the exact same thing a month ago. I'm getting the same errors when running cromwell on other .wdl scripts. Is this something to be expected? (I'm trying to decide if this could be the cause of another of my task failing in FireCloud).

Thanks!

Best Answer

Answers

  • RuchiRuchi Member, Broadie, Moderator, Dev
    Accepted Answer

    This is a Cromwell v27 issue but that exception isn't fatal and can be ignored, it shouldn't disrupt your workflows. I've filled a bug in Cromwell: https://github.com/broadinstitute/cromwell/issues/2340 Thank you for bringing this to our attention.

  • thedamthedam BarcelonaMember

    I have the same, but im Cromwell v31
    (...)

    {
    "outputs": {
    "ngsWESdataPrepocessing_oneSample.bwaMem.outputSam": "/mnt/sas_01/pipelineTest/pipeline/cromwell-executions/ngsWESdataPrepocessing_oneSample/0a64a0cc-a9f8-4da8-ad41-e6bafea43c33/call-bwaMem/execution/49412LS.sam"
    },
    "id": "0a64a0cc-a9f8-4da8-ad41-e6bafea43c33"
    }
    [2018-04-26 13:20:09,72] [info] Message [cromwell.core.actor.StreamActorHelper$StreamFailed] without sender to Actor[akka://cromwell-system/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
    [2018-04-26 13:20:09,72] [info] Message [cromwell.core.actor.StreamActorHelper$StreamFailed] without sender to Actor[akka://cromwell-system/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
    [2018-04-26 13:20:09,73] [info] Message [cromwell.core.actor.StreamActorHelper$StreamFailed] without sender to Actor[akka://cromwell-system/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
    [2018-04-26 13:20:09,74] [error] Outgoing request stream error
    akka.stream.AbruptTerminationException: Processor actor [Actor[akka://cromwell-system/user/StreamSupervisor-1/flow-7-0-mergePreferred#-1107498721]] terminated abruptly
    [2018-04-26 13:20:09,74] [error] Outgoing request stream error
    akka.stream.AbruptTerminationException: Processor actor [Actor[akka://cromwell-system/user/StreamSupervisor-1/flow-3-0-mergePreferred#-739539807]] terminated abruptly
    [2018-04-26 13:20:09,80] [info] Automatic shutdown of the async connection
    [2018-04-26 13:20:09,80] [info] Gracefully shutdown sentry threads.
    [2018-04-26 13:20:09,80] [info] Shutdown finished.

Sign In or Register to comment.