Jobs stuck in 'starting' status
francois_a
Member, Broadie ✭✭
I've been unable to run jobs for the past ~16 hours. For successful submissions, jobs get stuck in 'starting' status. Others in the lab are seeing these problems as well, but status.firecloud.org seems to indicate that everything is fine. Is there a deadlocking issue similar to last Friday? (https://gatkforums.broadinstitute.org/firecloud/discussion/11675/error-lock-wait-timeout-exceeded-try-restarting-transaction). Thanks for looking into this.
The metadata shows the following:
{'attempt': 1, 'backend': 'JES', 'callCaching': {'allowResultReuse': True, 'effectiveCallCachingMode': 'ReadAndWriteCache'}, 'executionStatus': 'Starting', 'shardIndex': -1, 'start': '2018-03-28T11:13:33.707Z'}
Answers
Hi @francois_a can you share the submission and workflow ids?
The jobs eventually ran. Here's one of the submissions:
submission ID: 545bd87f-6fab-4f3b-8d25-97b3e3ac04fb
workflow ID: 33b5f0a8-6c1c-4c07-adae-6fa851b5b9e4
I believe this is the same issue being discussed here, which you linked above. Our developers are actively working on a fix right now, and I will pass on your submission/workflow information if they need an additional example case.