Multiple issues with FC workflow execution & call caching

RLCollinsRLCollins ✭✭Harvard Medical SchoolMember ✭✭

Hi FC team,

Our team has been unable to get any workflows to successfully complete since mid-afternoon yesterday, Tues, Oct 2. We have experienced these issues across multiple workspaces, workflows, and configs.

There are no persistent errors, but here is a sampling of the issues we have encountered:

I suspect these issues could be related to the following two posts from yesterday by @jgould and @Chip:
https://gatkforums.broadinstitute.org/firecloud/discussion/13147/long-wait-times#latest
https://gatkforums.broadinstitute.org/firecloud/discussion/13148/2-hours-per-task-to-check-call-cache#latest

At this point, we are completely stalled on all workspaces, and don't want to launch any new workflows due to these unpredictable errors and long queue times.

Any idea what could be going on, or how long we should expect this behavior to persist?

Thanks a lot,
Ryan & the Talkowski lab

Best Answers

Answers

  • RLCollinsRLCollins ✭✭ Harvard Medical SchoolMember ✭✭

    Great, thank you @bshifaw. Keep us posted if there's any testing or additional information we can provide from our end.

  • RLCollinsRLCollins ✭✭ Harvard Medical SchoolMember ✭✭

    Hi @bshifaw, any updates on the prognosis for this problem?

    We have had a handful (maybe ~two dozen) tasks hit call cache over the course of the day, while dozens more are still waiting on cache or stuck in Starting mode.

    Thanks a lot for your help!

  • bshifawbshifaw admin Member, Broadie, Moderator admin

    @RLCollins , @Chip, and @jgould

    Update:
    The team is performing final validations on the hotfix, we hope to have it implemented between 6:30-7:30 today.

    Again, thank you for you patience.

    -Beri

  • RLCollinsRLCollins ✭✭ Harvard Medical SchoolMember ✭✭

    Hi @bshifaw, I’m happy to confirm that tasks appear to be launching as normal again. Looks good on our end. Thanks a lot for your efforts!

Sign In or Register to comment.