Or is it just me?
We don't yet have any reports of it. What are you seeing?
First, I got an error telling me my workspace bucket was inaccessible, then I couldn't see any workspaces.
This persisted for 10 to 15 minutes or so when I tried multiple times.
I then logged out and logged back in and the problem seemed to resolve.
@KateN I too, in recent days (the past week or two or so?) have experienced periodic errors saying something to the effect of "the bucket for this workspace is not available" in a banner at the top.
If I do some refreshing this seems to go away. This seems behavior similar or even the exact same as @bhandsaker described
is it a token-expiration issue? is this a known-issue?
Did you have the FireCloud page open for a long time or keep it open while your computer went to sleep? If so, it is very likely due to expired tokens, which is an issue we are aware of, and I can put a note to prioritize it higher. If not, we would like to investigate more.
It could be that something expired. I tend to leave windows/tabs open, not log out, etc.
If you see the error again, please copy or screenshot the exact error messages you see. I believe this is linked to our other issue, so I will put in a note that you are encountering this difficulty.
Is FireCloud down? I'm currently getting 500s on API calls (There was an internal server error), and the submission queue has been stuck with 3435 queued for at least 1h.
Devs are investigating now.
I've said this a number of times: we (CGA users) should not be the ones who discover FireCloud is down. Operations should be monitoring the system and identifying major outages like this ahead the users.
@birger I agree. We'll have a postmortem to find out why this was not caught earlier.
It seems to be working now...thanks for addressing this quickly once you were informed of the problem.
I spoke too soon....system is down again.
We're seeing it too -- looks like it's the width of the submission that's breaking the db, so when Francois resubmitted the job, it took the system down again. The devs will address the underlying issue but in the meantime, we'll need Francois to break up the submission into subsets of ~5K.
Francois will do this.
Apparently, the new release smoke tests include making a submission of 20K workflows. The new release obviously passed that test, but the smoke test workflow contained a small number of non-file input parameters. According to @abaumann , there is a known issue with support for large (tens of thousands workflows) submissions when the workflow's inputs include filenames. The paths of these input files on google cloud storage (especially if they are files created by workflows run earlier) can be very long (embedding the uuids of submissions and workflows). This is what, I guess, drives the width of the submission. So this problem was known to engineering.
We need on the forum a page which describes known problems that impact how users interact with the system (e.g., size of submissions). Ideally there would be an high-visibility link on the portal's main page linking to this known problems page.
That would be great! Thanks!