Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/17/19
Release Notes can be found here.

Multiple issues with FC workflow execution & call caching

RLCollinsRLCollins Harvard Medical SchoolMember

Hi FC team,

Our team has been unable to get any workflows to successfully complete since mid-afternoon yesterday, Tues, Oct 2. We have experienced these issues across multiple workspaces, workflows, and configs.

There are no persistent errors, but here is a sampling of the issues we have encountered:

I suspect these issues could be related to the following two posts from yesterday by @jgould and @Chip:

At this point, we are completely stalled on all workspaces, and don't want to launch any new workflows due to these unpredictable errors and long queue times.

Any idea what could be going on, or how long we should expect this behavior to persist?

Thanks a lot,
Ryan & the Talkowski lab

Best Answers


  • RLCollinsRLCollins Harvard Medical SchoolMember

    Great, thank you @bshifaw. Keep us posted if there's any testing or additional information we can provide from our end.

  • RLCollinsRLCollins Harvard Medical SchoolMember

    Hi @bshifaw, any updates on the prognosis for this problem?

    We have had a handful (maybe ~two dozen) tasks hit call cache over the course of the day, while dozens more are still waiting on cache or stuck in Starting mode.

    Thanks a lot for your help!

  • bshifawbshifaw moonMember, Broadie, Moderator admin

    @RLCollins , @Chip, and @jgould

    The team is performing final validations on the hotfix, we hope to have it implemented between 6:30-7:30 today.

    Again, thank you for you patience.


  • RLCollinsRLCollins Harvard Medical SchoolMember

    Hi @bshifaw, I’m happy to confirm that tasks appear to be launching as normal again. Looks good on our end. Thanks a lot for your efforts!

Sign In or Register to comment.