FireCloud appears hung

ekofmanekofman Member, Broadie

I have several workflows in a "Running" state but that are not outputting to stderr or stdout. This is making me thing there must be an issue with FireCloud queuing, and that my jobs are actually hung and not really running. Is this currently a problem? Thanks.

Best Answer

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    stderr and stdout aren't written until the task they are for completes. Are you not seeing these files on completed tasks? As far as I know, I don't have any alerts of FireCloud being hung right now.

  • ekofmanekofman Member, Broadie

    but the JES log should be written to the whole time, right? Because ours are not updating

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Could you share your workspace in FireCloud with [email protected], and post here with the workspace name and submission ID in question. I'd like to have our developers take a look.

    We don't have any widespread alerts of queuing or hanging issues, but this could be something just affecting your workspace, so I'd like to have someone take a look to ensure your workflows are running as you expect them to.

  • jtsujijtsuji CambridgeMember, Broadie

    It appears my workflow also got stuck for hours and doesn't output anything to stderr and stdout:

    I saw this issue in other workflow I'm currently running:

    I aborted the above workflow and reran it, but no luck..

  • jtsujijtsuji CambridgeMember, Broadie

    update: it appears the jobs finished running, but the FireCloud UI didn't show the completion status and the outputs to the data model

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    We were notified late yesterday by Google that they are experiencing a higher than usual load. Unfortunately, this means that while jobs are completing correctly, it's taking some time before the status gets updated and passed back to FireCloud. The result is that jobs will be slow to start up, abort, and finish. Google is working on a fix for this, but I don't yet have a timeline for when that will be implemented.

  • ekofmanekofman Member, Broadie

    Awesome. Yes this has made debugging very difficult, hopefully it helps.

  • ekofmanekofman Member, Broadie

    I'm still not seeing JES logs, so hopefully it's just a holdover.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    The backlog is now clear. If you are still experiencing issues, let us know.

  • ekofmanekofman Member, Broadie
    edited June 2018

    @KateN I'm still not seeing live-updating JES logs in this workspace: GCT_WGS_firecloud, particularly in this submission: 0797ac03-1fdd-4622-abf1-df786b7c51a3

    Post edited by ekofman on
  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    We've been told by Google that the fix didn't work as they'd expected, and they are still experiencing issues.

    @ekofman Could you please share your workspace with [email protected] using the Share button in your FireCloud workspace? I'd like to check to be sure that your issue is related to this Google outage and that it isn't caused by something else.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    It appears to be resolved again. I'd still like to take a look at your workspace @ekofman if it still hasn't written any output logs.

  • ekofmanekofman Member, Broadie

    @KateN just shared it with you guys

  • ekofmanekofman Member, Broadie

    It's working better now

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Administrator, Broadie, Moderator admin

    @ekofman Thanks for reporting. It appears that all the logs are displaying now.

Sign In or Register to comment.