504 errors when trying to load workflow details

RLCollinsRLCollins Harvard Medical SchoolMember ✭✭

Hi FC team,

As of this morning, we are encountering 504 errors when trying to load workflow details in the FC portal.

Here is an example workflow where we encounter a 504:
https://portal.firecloud.org/#workspaces/talkowski-sv-gnomad-wgs-v2/SV_Talkowski_GNOMAD_WGS-V2/monitor/53a60272-80c0-4ccc-be49-5a9c1187fcf3/5d4230f8-1265-4cc9-a9fd-ea9a722f9d9c

The Loading workflow details... message pinwheels for about ~10 seconds before being replaced by a HTTP 2.0 / 504 message in red, after which a grey Server Unavailable banner is displayed at the top of the screen (but it only appears after the 504).

Let me know if there are any other details we can provide to help solve this issue.

Thanks,
Ryan

Tagged:

Best Answers

Answers

  • RLCollinsRLCollins Harvard Medical SchoolMember ✭✭

    Hi @dvoet,

    Thanks for the reply. That all makes sense.

    I ran the API call you linked for the following workflow:
    workspaceNamespace: talkowski-sv-gnomad-wgs-v2
    workspaceName: SV_Talkowski_GNOMAD_WGS-V2
    submissionID: 53a60272-80c0-4ccc-be49-5a9c1187fcf3
    workflowID: 5d4230f8-1265-4cc9-a9fd-ea9a722f9d9c

    I know that many of these tasks are complete, but the API call is only returning stdout/stderr/backendLogs.

    Are all outputs updated only once the entire workflow completes? So the best bet would be to just sit tight and wait for the workflow to complete?

    We expect this workflow to run for another 12-16 hours, so I'm nervous letting it run without being able to validate the intermediate steps are running & producing the expected output... but we've tested it on a smaller sample set, so it should be ok.

    Thanks,
    Ryan

  • RLCollinsRLCollins Harvard Medical SchoolMember ✭✭

    Yep, that's what I have been doing so far to monitor its progress.

    In the specific case of this workflow, there are thousands of shards, so iterating through the execution bucket directory structure to search for the presence or absence of the expected outputs would work but isn't ideal.

    I think we are content to wait this one out, especially since it sounds like future FC releases will be able to handle large workflow metadata files.

    Thanks for your help!

  • bshifawbshifaw Member, Broadie, Moderator admin

    Hi @RLCollins ,

    Looks like dvoet provided an answer and workaround until a future FC releases is available to handle viewing workspaces with large metadata. I'll mark his post as the answer.

    Thanks

Sign In or Register to comment.