Holiday Notice:
The Frontline Support team will be offline February 18 for President's Day but will be back February 19th. Thank you for your patience as we get to all of your questions!
The Frontline Support team will be offline February 18 for President's Day but will be back February 19th. Thank you for your patience as we get to all of your questions!
Latest Release: 02/05/19
Release Notes can be found here.
Release Notes can be found here.
Jobs stuck in 'starting' status
francois_a
Member, Broadie ✭✭
I've been unable to run jobs for the past ~16 hours. For successful submissions, jobs get stuck in 'starting' status. Others in the lab are seeing these problems as well, but status.firecloud.org seems to indicate that everything is fine. Is there a deadlocking issue similar to last Friday? (https://gatkforums.broadinstitute.org/firecloud/discussion/11675/error-lock-wait-timeout-exceeded-try-restarting-transaction). Thanks for looking into this.
The metadata shows the following:
{'attempt': 1, 'backend': 'JES', 'callCaching': {'allowResultReuse': True, 'effectiveCallCachingMode': 'ReadAndWriteCache'}, 'executionStatus': 'Starting', 'shardIndex': -1, 'start': '2018-03-28T11:13:33.707Z'}
Answers
Hi @francois_a can you share the submission and workflow ids?
The jobs eventually ran. Here's one of the submissions:
submission ID: 545bd87f-6fab-4f3b-8d25-97b3e3ac04fb
workflow ID: 33b5f0a8-6c1c-4c07-adae-6fa851b5b9e4
I believe this is the same issue being discussed here, which you linked above. Our developers are actively working on a fix right now, and I will pass on your submission/workflow information if they need an additional example case.