Accessing user-pays-access bucket in FireCloud workflow?

bhaasbhaas Broad InstituteMember, Broadie

Hi - there's a bucket that I need to access in my workflow that involves user-pay access. Is there a way from within firecloud to access these data? Will FireCloud make access requests using my project ID so costs will be automatically incurred?


Best Answer


  • mnoblemnoble Broad Institute of MIT & HarvardMember, Broadie

    Does the roadmap indicate when the ability to use requestor-pays buckets is slated to appear in FireCloud?

  • birgerbirger ✭✭✭ Member, Broadie, CGA-mod ✭✭✭
  • bshifawbshifaw admin Member, Broadie, Moderator admin

    Hi @mnoble

    The plan is to have the requester-pays feature supported on FireCloud in the middle of this quarter.

    birger, I'll remind Ilyana to give a response on the thread.

  • mnoblemnoble Broad Institute of MIT & HarvardMember, Broadie
    edited October 2018

    Thanks for the info, @bshifaw!

    On a related note that would help me understand where the real constraint lies w/r/t requestor-pays: is ANY and ALL data marked as requestor-pays completely unusable (now) in FC? OR, might it be possible to use data marked as requestor-pays IF I ALSO have access to it in some other way?

    For example, suppose I own a bucket that is external to any FC workspace; I populate this bucket and update its content from time to time, and use the data therein to feed to code in FC workspaces. In other words, I have full control of the bucket, and can invite others to use it on a case-by-case basis, but it's otherwise private to me and my group.

    But now I want to open it up to the public, while avoiding egress charges. So I mark the bucket requestor-pays.

    lt seems to me that, as the owner of this bucket I should still be able to use it within my FC spaces. Is this true? In other words, why should changing the access or payment mechanism FOR OTHERS have any effect upon my ability to use my own data?

  • KateNKateN admin Cambridge, MAMember, Broadie, Moderator admin

    I'm not quite sure on that, but I will follow up with the team and get back to you soon. Thank you for your question.

  • KateNKateN admin Cambridge, MAMember, Broadie, Moderator admin

    One of our developers, @breilly, was able to find an answer to your questions:

    FireCloud performs actions using a service account for the user in the workspace's project. That service account does not have all of the user's permissions (most notably outside of the workspace's project; it's unclear what the behavior would be if the workspace and the non-FC bucket happen to be in the same Google project). Since FC does not access the bucket as the actual user, FC cannot access the requester-pays bucket without providing billing - which FC does not have support for today.

    However, the service account may be able to be given permission. FireCloud creates a group containing all FC service accounts - the so-called "proxy group" displayed on FC's user profile screen. If that group is given the resourcemanager.projects.createBillingAssignment permission (via the Project Billing Manager or similar role) then accessing the requester-pays bucket from FC might work today. We have not tested this.

    Additionally, if that works and if you'd like to extend that access to a group of FC users, you can create a managed group in FC, add all of those users to the group, and give the billing permission to that group. The permission would then trickle down to those users' FC service accounts.

    Please let us know if you have any other questions, or if that didn't fully answer everything. I'm happy to follow up.

  • mnoblemnoble Broad Institute of MIT & HarvardMember, Broadie

    Thank you @breilly and @KateN. We're about to try doing an analysis run, so I don't want to test this suggestion at the moment, but will try it early next week and get back to y'all.

Sign In or Register to comment.