Support for requestor-pays buckets for gcr.io-based docker repository

gordon123gordon123 BroadMember, Broadie
edited July 2018 in Feature Requests

I have some docker images I want to store on Google Cloud Repository (gcr.io). There is no IP precluding public access, so I don't need to jump through the hoops that would otherwise be required to make gcr.io work for Firecloud. However, I also don't want to be potentially hit with a bunch of data egress fees in case someone wants to download my images out of the Google network. I ought to be able to set the bucket to be 1) publicly readable, and 2) enable requestor-pays. Firecloud does work fine with the bucket set to publicly readable (via https://cloud.google.com/container-registry/docs/access-control), but fails when requestor-pays is also enabled:

error pulling image configuration: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<?xml version='1.0' encoding='UTF-8'?><Error><Code>UserProjectMissing</Code><Message>Bucket is a requester pays bucket but no user project provided.</Message></Error>"

Since Firecloud is running on GCP, there should be no egress charges pulling from this multi-regional bucket.

Post edited by Geraldine_VdAuwera on
Tagged:
0
0 votes

Active · Last Updated

Comments

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I will forward this feature request to our development team, thank you for the suggestion!

  • nrockweilernrockweiler Member

    Have there been any updates on this? Or are there any workarounds? I'm trying to access data in a "requester pays" bucket.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    The following situation is similar to Gordon's although it involves the storage of data files rather than docker images.

    The GTEx team stores their GTEx files on a non-workspace GCS bucket. (Their FC workspaces reference these files.) Members of the consortium have read access to the GCS bucket and thus can download these files from the bucket. To avoid having to pay data egress charges, the GTEx team recently enabled requester-pays on that bucket. Having made that change, however, they can no longer run workflows on whose inputs are files on GTEx requester-pays bucket. File delocalization fails. @francois_a has described this to me.

    We understand that currently running workflows that access files on requester pay buckets is not supported, and has been added to the backlog of feature requests. I just wanted to put my 2 cents in that this is a high-priority feature request for us. Many researchers in the GTEx community want to use FireCloud for their analysis, but are currently blocked by this issue.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Thank you all for your interest; I've passed on your comments to the teams and they will be prioritizing this work accordingly. It really helps us to know what features you are interested in, so if you see another user's request for something, and it is a feature you'd be interested in to, definitely comment to add your name in support of that feature.

    For this particular feature (enabling requester-pays functionality), our teams are currently waiting for a Cromwell functionality to be implemented. Once it is, they will begin work on this.

  • jgouldjgould GouldMember ✭✭

    Has there been any update on this? This is also an important feature for our group. Thanks.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Administrator, Broadie, Moderator admin

    Hi @jgould I am subbing in for Kate today and following up with the team now on this. I will report back as soon as I know more.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Administrator, Broadie, Moderator admin

    The engineering team is working on this feature now. They are planning to release it by the end of July. We will release documentation explaining the details of it too. @nrockweiler I believe whoever owns the bucket you want to access will need to turn off this parameter in the meantime. I know that is not a satisfying answer, but it should be available soon.

  • breardonbreardon Cambridge, MAMember, Broadie

    Has there been any update on this feature? For both images and regular files that are publicly available. I just attempted having a method localize a publicly available file with requestor pay and received the following error message

    BadRequestException: 400 Bucket is requester pays bucket but no user project provided.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Administrator, Broadie, Moderator admin

    Hi Brandon! Ilyana, the FireCloud PM, will be responding today with more detail on its status.

  • scalvoscalvo Member

    Any update on this? I'm also encountering the same problem.
    --Sarah Calvo

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Hi @scalvo,

    Unfortunately I don't have an update for you. Our project manager on this is currently out of the office, but I will be sure to follow up with her as soon as she gets back. Thank you for your patience.

  • Support for Requester Pays in FireCloud is tied with upgrading to Google Pipelines API 2.0 since PAPI V1 does not support Requester Pays. We are actively testing PAPI V2 functionality and working to upgrade to V2 this quarter. Requester Pays will be available once this upgrade is complete. Happy to connect offline to discuss more specifics or if you have further questions.

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    Hey @scalvo - as we haven't heard back from you on the forum, we will now be closing this ticket. However, if you have any new questions you can respond back and we are more than happy to take a look for you!

  • rmarencormarenco Member
    I am also very interested in this feature. We have a lot of collaborators shipping us tissue samples who want to access genomics data. It is too much of a cost for us to support free access so we are definitely in need of Requester Pay.

    I have heard this should be supported soon (last month, at the workshop, we told me something like 2-3 months).

    How can I be notified that it is out? Can we use this thread?

    Thanks,
    Remi
  • jgouldjgould GouldMember ✭✭

    FYI-I've verified that user pays works.

  • rmarencormarenco Member

    What do you mean @jgould ? Is it now implemented? I would be curious to see the documentation for setting this up

  • jgouldjgould GouldMember ✭✭

    Yes, it is implemented now. Please see https://cloud.google.com/storage/docs/access-control/making-data-public for instructions on making your data public.

Sign In or Register to comment.