We've moved!
You can find our new documentation site and support forum for posting questions here.

Permission issues transferring data from HPC or another cloud to my workspace bucket

I tried to copy data to my FC workspace bucket from a HPC cluster using gsutil and got the following error:
ResumableUploadAbortException: 401 Anonymous users does not have storage.objects.create access to bucket fc-f0bc9c42-a172-4795-........................

Also when I tried the Transfer page on the Google Cloud Platform Console it asked me to change permission on my Firecloud account for creating a transfer, didn’t give me anymore details.

Answers

  • gordon123gordon123 BroadMember, Broadie

    You need to run 'gcloud init' and authenticate with the Google account you use for Firecloud. Then 'gsutil cp' etc will let you copy files into the workspace bucket.

    Also, if you also have a non-Firecloud Google account, and it is what you used to log into Chrome initially, then you will get an authentication error when you try to open the bucket from a link within Firecloud. To fix this, click on your avatar in the upper right corner when you are trying to view the bucket, and switch accounts to the one you use for Firecloud. You should then have all the read/write privileges appropriate for that workspaces bucket.

  • durgadurga NCIMember

    Thank you @gordon123, I could transfer using gsutil from my HPC Cluster. When I tried the other method - cloud to cloud with download links, I have the following error:

    $ cat downloadLinks.txt|gsutil -m cp -I gs://my-bucket
    InvalidUrlError: Unrecognized scheme "https"
    CommandException: 1 file/object could not be transferred.

    Durga

  • gordon123gordon123 BroadMember, Broadie

    gsutil takes gs:// links, not https:// links.

  • durgadurga NCIMember

    So I am guessing my only other option is "Cloud Storage Transfer Service". I see the "edit permission required" message when I access my google cloud transfer page and "Error -Service 'storagetransfer.googleapis.com' not found or permission denied." when I tried the storage transfer service API client.
    I am following this: https://cloud.google.com/storage/transfer/

    Please let me know how to get this data transferred using the HTTPs links. I am simultaneously trying to find out if I can get the gs: links for the data origin location.

    thanks

  • gordon123gordon123 BroadMember, Broadie

    gsutil uses gs:// links, and if you configure the parallelism options right I normally see it running 20x faster than https. So, unless your data size is tiny that is the path I recommend.

    I haven't used Cloud Storage Transfer Service, but I believe it also takes gs:// paths and is typically used to transfer with another cloud like AWS.

  • durgadurga NCIMember

    Yes, I am trying to transfer data from AWS to FC-Google cloud and my data is in the 200GB range.

    The Cloud Storage Transfer Service does transfer from HTTPS links. I need to have edit permissions/authorizations on my FC project (according to the error/message). Images attached.


  • gordon123gordon123 BroadMember, Broadie
    edited September 2017
  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    Transfers do require edit permissions to the projects, which we cannot provide users in FireCloud due to our security model. The best solution here is to do the transfer to another project that you do have Edit permissions to and then either refer to the data there from FireCloud or then move it to the FireCloud Workspace's bucket

  • durgadurga NCIMember

    Thanks Alex, I will try that.

  • durgadurga NCIMember

    Hi Alex,
    I used the Cloud Storage Transfer Service (https://cloud.google.com/storage/transfer/create-manage-transfer-console) and provided a list of URLs with the destination bucket, but I'm getting an error "Object: http_list_0 Details: Unknown error."
    I tried placing my URL.tsv file in another bucket on the cloud and also on the google drive. Doesn't work, I have the same error: "Object: http_list_0 Details: Unknown error."

    tx,
    Durga

  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    Sorry I've been out sick for a bit, just getting back to this. Is this error coming about transferring to a bucket you setup outside FireCloud?

  • durgadurga NCIMember

    Yes, I created another google project under the same billing account as FC, and am trying to copy to that bucket, using a list of URLs of the files via Cloud Storage Transfer Service, such that I can then use it for running tasks in FC workspaces.

  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    I've never done this before, so I am not familiar with the errors - is it something you want to give me access to to take a look?

Sign In or Register to comment.