Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/10/19
Release Notes can be found here.

Is firecloud down?

bhandsakerbhandsaker Member, Broadie, Moderator admin

Or is it just me?

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    We don't yet have any reports of it. What are you seeing?

  • bhandsakerbhandsaker Member, Broadie, Moderator admin

    First, I got an error telling me my workspace bucket was inaccessible, then I couldn't see any workspaces.
    This persisted for 10 to 15 minutes or so when I tried multiple times.
    I then logged out and logged back in and the problem seemed to resolve.

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @KateN I too, in recent days (the past week or two or so?) have experienced periodic errors saying something to the effect of "the bucket for this workspace is not available" in a banner at the top.

    If I do some refreshing this seems to go away. This seems behavior similar or even the exact same as @bhandsaker described

    is it a token-expiration issue? is this a known-issue?

    -eddie

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Did you have the FireCloud page open for a long time or keep it open while your computer went to sleep? If so, it is very likely due to expired tokens, which is an issue we are aware of, and I can put a note to prioritize it higher. If not, we would like to investigate more.

  • bhandsakerbhandsaker Member, Broadie, Moderator admin

    It could be that something expired. I tend to leave windows/tabs open, not log out, etc.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    If you see the error again, please copy or screenshot the exact error messages you see. I believe this is linked to our other issue, so I will put in a note that you are encountering this difficulty.

  • francois_afrancois_a Member, Broadie

    Is FireCloud down? I'm currently getting 500s on API calls (There was an internal server error), and the submission queue has been stuck with 3435 queued for at least 1h.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Devs are investigating now.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    I've said this a number of times: we (CGA users) should not be the ones who discover FireCloud is down. Operations should be monitoring the system and identifying major outages like this ahead the users.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @birger I agree. We'll have a postmortem to find out why this was not caught earlier.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    It seems to be working now...thanks for addressing this quickly once you were informed of the problem.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    I spoke too soon....system is down again.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    We're seeing it too -- looks like it's the width of the submission that's breaking the db, so when Francois resubmitted the job, it took the system down again. The devs will address the underlying issue but in the meantime, we'll need Francois to break up the submission into subsets of ~5K.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Francois will do this.

    Apparently, the new release smoke tests include making a submission of 20K workflows. The new release obviously passed that test, but the smoke test workflow contained a small number of non-file input parameters. According to @abaumann , there is a known issue with support for large (tens of thousands workflows) submissions when the workflow's inputs include filenames. The paths of these input files on google cloud storage (especially if they are files created by workflows run earlier) can be very long (embedding the uuids of submissions and workflows). This is what, I guess, drives the width of the submission. So this problem was known to engineering.

    We need on the forum a page which describes known problems that impact how users interact with the system (e.g., size of submissions). Ideally there would be an high-visibility link on the portal's main page linking to this known problems page.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    We have some web work planned for later this week to make a known issues page. We'll see what we can do to make it sufficiently visible.
  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    That would be great! Thanks!

Sign In or Register to comment.