Workflow failure dependent on FireCloud Billing Account?

birgerbirger Member, Broadie, CGA-mod ✭✭✭

I have two identical workflows (broad-firecloud-ibmwatson/CBB_20181127_CGA_WES_Characterization_TCGAControlledAccess_test
and broad-fc-getzlab-workflows/CBB_20181127_CGA_WES_Characterization_TCGAControlledAccess_test_2), with the only difference being that one was created under the FireCloud Billing project broad-firecloud-ibmwatson and the other was created under the FC billing project broad-fc-getzlab-workflows. I am running the getz lab WES characterization workflow on a single pair in each of these workspaces. The workflow fails in the former but runs through to completion in the later. The error is reproducible. The error occurs in the the two tasks (mutect1 and mutect2) that are scattered. Each scattered task call fails. Here is one of the failure messages:

message: Call input and runtime attributes evaluation failed for Mutect1_Task
causedBy:
message: Failed to evaluate input 'diskGB' (reason 1 of 1): [Attempted 1 time(s)] - StorageException: [email protected]account.com does not have storage.objects.get access to fc-secure-b14d3c32-f817-411c-8e75-3298c52bba97/1f5413f3-b31b-4040-971c-6f7571eef537/CGA_Production_Analysis_Workflow/60e2f010-c8b3-4187-8115-9f3acd799d3a/call-CallSomaticMutations_Prepare_Task/glob-fedfc53b55ca349744292cb186d61b54/gatk-scatter.0000000010.interval_list.

In a separate workspace, however, created under the broad-firecloud-ibmwatson billing project, the same workflow on the same entity pair passed. (This separate workspace has 179 pairs, while in my two test workspaces I only placed a single pair.) If the problem is not obvious, and can't grant your support team read access to the two workspaces. Note that the data is TCGA controlled access.

Answers

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @birger We will look into the inconsistency you are reporting and get back to you!

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @birger - Sorry for the delay! Google had an issue on their end on Nov 27 that caused the exact error that you are reporting, and it only affected some but not all billing projects. We notified Google of the problem; they found the issue on their end and rolled out a fix. The rollout could have taken 24 hours. We suggest you try again, now that Google's issue is fixed and if it still is not working properly please let us know and we will look further into the issue.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    I'm rerunning the workflow and will let you know if Google's fix resolved the problem.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Still failing in same place with same error message:

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @birger
    Can you confirm if this workspace was created during the window of the Google issue, approximately November 27-28, because the bucket may be have been affected. If so, would you be able to try creating a new workspace to help us determine the error? Thank you!

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Yes, the workspace was created on 11/27 (I include creation date stamps in most of my workspaces). If I have a chance later this week, I will clone it and re-run the offending workflow. (Or I can give a member of your team read access and you can do that.)

    -Chet

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @birger If the newly cloned workspace continues to function with an error, the team would be more than happy to gain read access and help you out with the troubleshooting - it might be easier for you to test since the data is TCGA access controlled and it might get complicated to get proper permissions.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    After cloning the workspace, the workflow ran successfully in the clone.

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @birger Thanks for getting back to us with the update - glad to hear that the workflow is now running successfully!

  • emmalynchenemmalynchen Member
    edited January 15

    I am running into the same error:

    I have a workspace fccredits-radium-magenta-9081/witte-cfdna that was created on December 14, 2018 under the FireCloud Billing project fccredits-radium-magenta-9081 (free credits), and I'm trying to run my workflow with files in a google bucket. I've given my email account that I use to access Firecloud as well as my proxy group, which I found under my Firecloud profile, "storage legacy bucket owner" and "storage object creator" permissions. My fccredits billing account is linked to the GCP project where I created my bucket. I tried running the workflow in a cloned workspace, but am still getting the same error. Am I missing something obvious?

    Post edited by emmalynchen on
  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @emmalynchen I will take a closer look since you have shared all the appropriate information (thank you!) and get back to you with an update!

  • emmalynchenemmalynchen Member

    Was recommended to change the proxy group permissions to "storage object viewer", "storage legacy bucket reader", and "storage legacy bucket owner" and it looks like the workflow completed successfully!

    I don't clearly understand the roles that can be assigned, and when to assign those roles (even after looking through the roles documentation). Are there any general guidelines?

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @emmalynchen I wanted to clarify - is the google bucket that contains the data/files the one that was generated upon creation of your workspace or is it an "external" one (a bucket that is not the one that is auto created for you when you make a workspace) as you mentioned when you said "...GCP project where I created my bucket".

  • emmalynchenemmalynchen Member

    The google bucket is an external one (not the auto created one when I made the workspace).

    I initially wanted to use the auto created bucket, but I wasn't able to upload my files with gsutil cp (which was actually an issue with the specific python version that I was using on the cluster) and when I checked the permissions I get:
    I'm the owner of the workspace - is it normal to not have permission to view the bucket metadata?

    I just tried running the analysis with the files uploaded to the auto created bucket and it ran successfully.

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @emmalynchen When you create a workspace, you should be owner of the workspace and the associated bucket. I am not sure why you are seeing the error above for the workspace bucket but I am glad to hear that your workflow completed.

    For external buckets, permissions are not equivalent to project permissions. You do not need to have access to project to have access to a bucket. In these cases you will have to assign your FireCloud proxy group storage object access to your bucket at the either the bucket or project level. You probably have to get at least reader and writer privileges so that files can be added to the bucket and also can be read from the bucket.

Sign In or Register to comment.