Workflow "Running" but call stuck at "QueuedInCromwell"

Hi,
When submitting a job to the pici-firecloud/Pre-processing_b37_v3 workflow, the workflow doesn't seem to be starting even though the status is listed as running, and has been stuck at this status for over an hour.

This happened before, and was not producing any outputs, so I aborted the job so I could access the log files.
Here is a snipped of the log right before I aborted it.

2018-05-17 21:16:56,675 INFO  - WorkflowExecutionActor-ebe80853-f9d5-41be-90bb-8059e1567897 [UUID(ebe80853)]: Starting PairedEndSingleSampleWorkflow.ScatterIntervalList, PairedEndSingleSampleWorkflow.CreateSequenceGroupingTSV, PairedEndSingleSampleWorkflow.GetBwaVersion
2018-05-17 21:17:24,815 INFO  - WorkflowExecutionActor-ebe80853-f9d5-41be-90bb-8059e1567897 [UUID(ebe80853)]: Starting PairedEndSingleSampleWorkflow.CollectQualityYieldMetrics
2018-05-17 21:44:05,805 INFO  - WorkflowExecutionActor-ebe80853-f9d5-41be-90bb-8059e1567897 [UUID(ebe80853)]: Aborting workflow

SubmissionId: 9fcfc72b-3cc8-403c-b259-de505944c6d0

Any suggestions for what could be causing it to hang?
Thanks!
-Robin

Best Answer

Answers

  • RobinKRobinK Member

    Oops, I'm sorry, I just realized there was an open thread on this. I will move my question there.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Update on this: it seems that we're hitting some limits set by Google due to an individual submission that amounts to a very large number of jobs (~60k). We're working with GCP support to figure out a way forward. See the service notice I just posted on the blog and banner in the FC portal for updates.

Sign In or Register to comment.