Latest Release: 03/12/19
Release Notes can be found here.

all workflows in a submission done, but submission not marked done and data-model not updated

esalinasesalinas BroadMember, Broadie ✭✭✭

I have a workflow c94ac5a8-f333-40dd-8f06-518a1f2a1514 that finished about 49 min ago

The workflow is the only one in the submission deed0065-ab63-47ec-9398-b2f30cd32cfa

Since the workflow is done and it's the only one in the submission, should the submission be marked done? and the "Done" status reflected in the monitor tab?

This is also preventing the data-tab from being updated with the results from the workflow

Answers

  • francois_afrancois_a Member, Broadie ✭✭

    I'm seeing the same issue with multiple submissions, including submissions with multiple workflows. For example, the 32 workflows in submission b2cb829d-1cdc-49fc-b9b9-23a11fee1cd0 all finished over an hour ago, but are still marked as Submitted and haven't written back to the data model.

  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited September 2017

    @francois_a I started observing this issue today in the afternoon. I have about 8900 submissions. Of the 8900 submissions, based on finding workflow output files in the bucket, over 8800 of the submissions are done, but in the monitor tab, over 1000 of the submissions are marked "Submitted"

  • husseinhussein DSP @ Broad InstituteMember, Broadie, Dev ✭✭

    Please share your workspaces with [email protected] I'll look into this meanwhile.

  • francois_afrancois_a Member, Broadie ✭✭

    Done. If you look at the last few submissions in the realign_0917 workspace, you'll see that submissions 8f5462df-41f2-46b8-8316-ee89abaf9bbd and 2668cedb-4190-4e26-a2c3-7663adea7778 have this issue; in the comparisons_0917 workspace, see b2cb829d-1cdc-49fc-b9b9-23a11fee1cd0. Please note that I'm about to patch the attributes in the latter through the API.

  • husseinhussein DSP @ Broad InstituteMember, Broadie, Dev ✭✭

    We bounced Rawls, so these should clear up. It looks like it was a combination of two things:

    1. Some transient database connectivity issues;
    2. Lots of single-workflow submissions. FC can get unhappy when a bunch of these complete at the same time in the same workspace. If you can stagger these it'd probably behave a little better.
  • francois_afrancois_a Member, Broadie ✭✭

    Thanks for the rapid fix!
    The large majority of my submissions contained multiple workflows, and I think @esalinas's as well.

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @francois_a did your data model get updated? Mine did not.

    For example a submission with data entity fffb519f-6aea-42bc-be7e-b314ba9d04c7

    but its row in the data model did not get updated

    I only observed a single data entity get updated:

  • husseinhussein DSP @ Broad InstituteMember, Broadie, Dev ✭✭

    @esalinas what's the submission and workflow associated with that entity that didn't write back? Can you share the workspace with [email protected] ?

  • kmegqkmegq BroadMember, Broadie

    I think I am having a similar problem - successful runs are not populating the data table in one of my workspaces:

    The outputs are listed under "Outputs" in the Monitor tab, however.

    Kate

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @hussein I gave the submission ID and workflow ID in the initial post of this thread.

    The workspace name and namespace are in the screenshots.

    I've just shared the workspace broad-firecloud-testing/hg38_PoN_Creation_copy with "[email protected]" as a READER

    Please note that the workspace is under TCGA-dbGaP-Authorized authorization domain.

    -eddie

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I wanted to update this thread to let you know our developers are still working on it. Currently we need another reproduction case, so if you have any new instances of this issue, please let us know.

    You can track the status of this fix here.

Sign In or Register to comment.