Getting "invalid_grant" error running method starting around November 27

pmontgompmontgom CambridgeMember, Broadie


I maintain an application that uses firecloud on the backend to run an analysis. However, I got a report recently that the analysis was failing and when I went to look at the workspace in firecloud I discovered the last successful run was at the end of Sept. The next time someone tried was nov 27th, and it has failed since.

Specifically, it is complaining of "invalid_grant" which suggests that firecloud doesn't have access to something it needs but I have no visibility into what is missing.

Here's a snippet of python which includes the namespace, workspace and the submission id as well as the error:

>>> r = firecloud.api.get_submission('broad-fc-ccl-discordance','CCL_Discordance_Portal_PROD', 'f7da407d-259a-4f77-9fcf-5c146c52f41b')
>>> [x['messages'] for x in r.json()['workflows']]
[[u'400 Bad Request\n{\n  "error" : "invalid_grant",\n  "error_description" : "Bad Request"\n}']]

I've granted read access to the workspace to [email protected] I attempted to grant WRITE access in case you'd need that but I got an error in the UI -- but I suspect that's a separate issue and a less urgent one. Just let me know if you need additional access to investigate.



  • dheimandheiman Member, Broadie ✭✭

    Hi @pmontgom, I had @birger run into this exact issue recently, it turns out that the user running the application using the firecloud API needs to have run both:
    gcloud auth login
    gcloud auth application-default login

    I've added the issue to the FISS repo, and should have a fix implemented by our next release.

  • pmontgompmontgom CambridgeMember, Broadie

    I took a look at the issue and it mentions have a google cloud account activated which I don't believe is my case.

    This is an application running in GAE, so it's using a service account. (This also means there's no host for me to run "gcloud" on.)

    Do you still think that is the issue at hand? Also, there was a long period of not getting these failures.

    If you think it's something to do with the credentials of the service account I can explore there -- I'm just puzzled as to why it is failing now.

    (And the workaround of running gcloud doesn't appear to be an option in my case.)

  • dheimandheiman Member, Broadie ✭✭
    edited December 2018

    If you're using a service account from GAE, then I'm not sure. It certainly looks like a credentialing issue. @birger, have you seen this issue crop up on any of your gcp instances that use FISS to talk to FireCloud?

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    The Cell Strainer portal app is using a service account to talk to FireCloud. Cell Strainer is successfully gaining access to FireCloud, otherwise it wouldn't even be able to attempt the launch of the workflow. I think this is something internal to FireCloud.

  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    I'm testing a few things out with Phil

  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    Ok we tested out and the permissions are all as they should be for a person to run a workflow, however Phil still gets the original error. I didn't think a svc account could run workflows due to our reliance still on refresh tokens, so I'm surprised this ever worked. If it did, then someone on the workbench team might be able to make sense of the error above and figure out what's causing that

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    Hello @pmontgom, were sorry about letting this issue get past us! I have notified team members that I believe might be able to investigate further and help resolve the error and will make sure to get back to you with any/all updates as soon as I hear back!

Sign In or Register to comment.