Holiday Notice:
The Frontline Support team will be slow to respond December 17-18 due to an institute-wide retreat and offline December 22- January 1, while the institute is closed. Thank you for your patience during these next few weeks. Happy Holidays!

Latest Release: 12/4/18
Release Notes can be found here.

Analysis Submissions stalled at start.

birgerbirger Member, Broadie, CGA-mod ✭✭✭

Any submission we make now make does not run. We have tried this in multiple workspaces.

THIS IS A BLOCKER FOR OUR WORK

Here is just one example

I submitted an analysis in the workspace
broad-firecloud-workshops/benchmarking-variability

The analysis runs a benchmarking workflow on a set of ten tumor/normal pairs.

The submission ID is 14784e07-0c98-4b59-9529-b2ccb9e8b3d2

FireCloud reports the submission as "Submitted". If you drill down, there are ten workflows, each in the "Submitted" state. However, for each workflow, no calls are made, and the status icon is a red triangle (See attached screen shot.)

Also, if you look at the bucket, there are not folders associated with the submission id nor the workflow id.

Tagged:

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    Ok, we'll get the team on this asap. Will update when we know more.
  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    thanks for the quick response.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @cbirger The team reset the Cromwell server; I tested a new submission and it seems to work now. Please try submitting again and let me know if you still have issues.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭
  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    OK...our workflows are running now. Any explanation of why this occurred? We had a similar incident in the past month.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    It looks like a bug in Cromwell combined with some connectivity issues to cause this -- and yes it was probably related to what happened last month. The team is close to being ready to release a new version that includes a fix for the Cromwell bug, so hopefully this shouldn't be an issue in the future.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    When will that release make it into the production version of FireCloud? Thanks.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    This is currently in the hands of the QA team; this could be a matter of a few days if the tests all come through clean. I'll let you know when we have an ETA.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    I see we are still running cromwell v 0.21. What is the status of the new release cromwell? thanks.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @birger The QA team expects to be able to greenlight the next FireCloud release tomorrow if the last round of tests that runs tonight goes through smoothly. If everything passes, then the release to the live system would happen either tomorrow or Monday (no releases can be done on Fridays). Note that this release will still include a version of Cromwell 0.21 -- not sure which hash but I will check whether it's the same as the current one or not. The devs are making a big push to integrate Cromwell 0.24, which brings some important changes; it's not clear yet whether there will be an intermediate release that will have something >21 but <24.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @birger The new release went out today. The team is now working on getting a release ready that will only update Cromwell to v24, which should prevent any recurrence of the issue reported in this thread (as well as fix the preemptible settings bug reported in your other thread). We expect that if all goes well that will go live by end of next week.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Wish the preemptible fix could get in sooner...the bug has really set us back on a tight-deadline project, but as Bill Belichick often says "it is what it is"!

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    Bill Belichick can probably afford not to use preemptibles!

    The devs are going to be loading a test environment with the new Cromwell integration today, so things are progressing. Stay tuned for updates.
Sign In or Register to comment.