Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.

Completed workflows marked as "Running"; outputs not written to data model

francois_afrancois_a Member, Broadie ✭✭

Hundreds of my jobs are currently in this state. Many finished running hours ago.
Under "Monitor", the status is marked a successful, and the output from the https://api.firecloud.org/#!/Submissions/workflowMetadata call confirms this.
Example workflow:
workflow ID: 46a0a335-dcc7-480e-a26e-f84f561bb225
submission ID: aedf152a-987c-405b-bf4a-bb1c4fdeab59

I there an ongoing issue responsible for this?

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I will investigate and let you know. Thank you for reporting this.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Could you share the workspace with [email protected] so we can take a look at your example?

  • francois_afrancois_a Member, Broadie ✭✭
  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    Francois, I'll help you resolve this with a workaround

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Thank you @abaumann. For general knowledge, this is a bug and we are currently working on a fix.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hey @francois_a and @abaumann, did you guys run the workaround script to attach outputs to the data model for this workspace? It looks like that stamped out the evidence we needed for troubleshooting, so this is no longer a valid bug reproduction case for debugging purposes. Unfortunately that means the dev team can't use this to work toward a solution for the underlying bug.

    That being said, we think this may have been caused by the Google bug that is causing delays, since the workflows had an incorrect "Running" status, rather than by the "outputs not written to data model" issue which previously affected "Done" workflows.

    In conclusion, for the former (Google bug) we are still waiting on a resolution from Google; while for the latter (data model not updated) we are still looking for a case that reproduces.

  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    i didn't run it, but francois may have

  • francois_afrancois_a Member, Broadie ✭✭

    Sorry for the late response to this. Yes, I had to run my script to fix the attributes. This is a production workspace and I needed to run additional jobs that required the missing attributes.

  • francois_afrancois_a Member, Broadie ✭✭

    This appears to be happening again.

    The workspace is broad-firecloud-gtex/gtex_ASE_v8_phasedVCF_1217 and I've added [email protected]; the submission ID is 4675f0b5-5281-4efd-80df-e034e564d3b3

    Note that this job was a re-run, and that the attributes have values from the previous run.

  • francois_afrancois_a Member, Broadie ✭✭

    It looks like this eventually resolved, but it took 8 hours between job completion and updating of statuses.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Thank you for reporting this; we've been looking for another reproduction of the bug in order to help diagnose and fix it.

    I'll add your information to the bug ticket we have, and you can track updates to it here.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    We have investigated and determined that the issue is due to scaling, as it did eventually resolve itself. As more and more data is run (particularly if run in large quantities in a single job), errors like this may crop up. Resolution of this issue is on our long term road map, so please do continue to report any instances in which you encounter this problem.

Sign In or Register to comment.