Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.

Error! You do not have access to this file.

ChipChip 415M 4053Member, Broadie
edited July 2018 in Ask the FireCloud Team

Hi

Yesterday I updated 55 WGS bams in a bucket. I copied the files from /seq/picard_aggregation and verified that the files are in the bucket using gsutil from my laptop, (also from Broad servers, as well as from a google VM).

However, when running tasks in Firecloud the job response was a failure message: "Failed to localize files: failed to copy the following files: "gs://<bucket name>/<directory>/*.bam" and when I click on the bam link from the Firecloud GUI the pop up box says "Error! You do not have access to this file." The bucket is not the workspace bucket, but the bucket contains older bams that are readable from the workspace. It's just the new bams that have an access problem.

How does one tell Firecloud that he/she should have access to new files in a bucket?

Thanks,

Chip

Post edited by KateN on
Tagged:

Answers

  • mbemismbemis Cambridge, MAMember, Broadie ✭✭

    Hi Chip,

    Is the external bucket shared with your FireCloud proxy group? If so, do the newer objects have ACLs set on them that override the bucket's ACL?

  • dheimandheiman Member, Broadie ✭✭
    edited July 2018

    Hi @Chip,

    It sounds like you need to give your FireCloud proxy group storage object access to your bucket at the either the bucket or project level.

    There are two ways to do either of these.

    If there are other FireCloud users that should have access to your bucket, then you can create a group for all of them in FireCloud (if it doesn't already exist) by going here.

    If you are the only one who should have access to the bucket, then you need to find your proxy group. To do that, go here and look under Proxy Group.

    For project level access:
    Go to the console page for your storage project.
    * select IAM & admin -> IAM
    * select +ADD at the top of the screen
    * type the group/proxy group in the New members box
    * under Select a roll, choose Storage, then select the level of Storage Object access to give. Unless you are going to be giving others the capability of modifying what's in the bucket, Storage Object Viewer should suffice.

    For bucket level access:
    Go to the storage browser for your project.
    * Select Edit bucket permissions from the menu on the right side of the bucket the files are in.
    * type the group/proxy group in the Add members box
    * under Select a roll, choose Storage, then select the level of Storage Object access to give. Unless you are going to be giving others the capability of modifying what's in the bucket, Storage Object Viewer should suffice.

    Post edited by dheiman on
  • ChipChip 415M 4053Member, Broadie

    Thanks @dheiman and @mbemis

    David showed me an effective gsutil acl trick from https://cloud.google.com/storage/docs/gsutil/commands/acl

    get acls from an accessible file in the weird bucket:
    $ gsutil acl get gs://// > acl.txt

    use the acl.txt as a template for any file that can't be accessed:
    $ gsutil -m acl set acl.txt 'gs:////<file that can't be accessed>

    or any files in a path that can't be accessed:
    $ gsutil -m acl set acl.txt 'gs:///<path that can't be accessed>/**/*'

    This worked and solved my access limitations. Still not sure why the acl's were wrong to start with, but it's no longer blocking me.

  • ChipChip 415M 4053Member, Broadie

    The

    gs:////

    in the previous post should have been more like this:

    gs://<bucket>/<path>/< file>

    It seems markdown hides stuff in brackets < > including the brackets.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I've edited your original post for clarity of formatting! Markdown has its finicky bits.

Sign In or Register to comment.