LATEST RELEASE: May 23, 2018
Release Notes can be found here.

[ERROR] token_pon_size_update

Hi,

I ran “token_pon_size_update” two times to build a PoN using hundreds of TCGA blood samples. But, each of them finally failed because of a file transfer issue. Should I try to run the tasks separately? Like, step 1, token_make for each bam file; step 2, aggregateTokensAndMakePon … Thanks a lot!

• message: Task token_wf.token_make:192:2 failed. JES error code 5. Message: 9: Failed to localize files: failed to copy the following files: "gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/GBM/DNA/WGS/BI/ILLUMINA/G49538.TCGA-06-0190-10B-01D-A32W-08.1.bam -> /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/GBM/DNA/WGS/BI/ILLUMINA/G49538.TCGA-06-0190-10B-01D-A32W-08.1.bam (cp failed: gsutil -q -m cp gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/GBM/DNA/WGS/BI/ILLUMINA/G49538.TCGA-06-0190-10B-01D-A32W-08.1.bam /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/GBM/DNA/WGS/BI/ILLUMINA/G49538.TCGA-06-0190-10B-01D-A32W-08.1.bam, command failed: Caught ResumableDownloadException (Transfer failed after 23 retries. Final exception: HttpError accessing <https://www.googleapis.com/download/storage/v1/b/5aa919de-0aa0-43ec-9ec3-288481102b6d/o/tcga%2FGBM%2FDNA%2FWGS%2FBI%2FILLUMINA%2FG49538.TCGA-06-0190-10B-01D-A32W-08.1.bam?generation=1459406764694000&alt=media>: response: <{'status': '503', 'x-goog-metageneration': '3', 'content-disposition': 'attachment', 'content-language': 'en', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'vary': 'Origin, X-Origin', 'content-length': '19', 'server': 'UploadServer', 'content-range': 'bytes 155369054208-181424770066/181424770067', 'etag': 'CPCzq+6p6ssCEAM=', 'x-goog-generation': '1459406764694000', 'pragma': 'no-cache', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'date': 'Fri, 22 Sep 2017 12:13:01 GMT', 'content-type': 'text/plain; charset=utf-8', 'x-guploader-uploadid': 'AEnB2UpGsqruJ_dkn5TtnT9bnKVwhWfTa89VyClt2oc3zG3FVYPEvXA-zeHS4aKhPjYG012SxcO8YryS4SXs89v2SGdMHKHueDsXFIKdD1W1e889IkW2qmo', 'x-goog-storage-class': 'DURABLE_REDUCED_AVAILABILITY'}>, content <Service Unavailable>) for download of /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/GBM/DNA/WGS/BI/ILLUMINA/G49538.TCGA-06-0190-10B-01D-A32W-08.1.bam component 3.\nResumableDownloadException: Transfer failed after 23 retries. Final exception: HttpError accessing <https://www.googleapis.com/download/storage/v1/b/5aa919de-0aa0-43ec-9ec3-288481102b6d/o/tcga%2FGBM%2FDNA%2FWGS%2FBI%2FILLUMINA%2FG49538.TCGA-06-0190-10B-01D-A32W-08.1.bam?generation=1459406764694000&alt=media>: response: <{'status': '503', 'x-goog-metageneration': '3', 'content-disposition': 'attachment', 'content-language': 'en', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'vary': 'Origin, X-Origin', 'content-length': '19', 'server': 'UploadServer', 'content-range': 'bytes 155369054208-181424770066/181424770067', 'etag': 'CPCzq+6p6ssCEAM=', 'x-goog-generation': '1459406764694000', 'pragma': 'no-cache', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'date': 'Fri, 22 Sep 2017 12:13:01 GMT', 'content-type': 'text/plain; charset=utf-8', 'x-guploader-uploadid': 'AEnB2UpGsqruJ_dkn5TtnT9bnKVwhWfTa89VyClt2oc3zG3FVYPEvXA-zeHS4aKhPjYG012SxcO8YryS4SXs89v2SGdMHKHueDsXFIKdD1W1e889IkW2qmo', 'x-goog-storage-class': 'DURABLE_REDUCED_AVAILABILITY'}>, content <Service Unavailable>\nCommandException: Some components of /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/GBM/DNA/WGS/BI/ILLUMINA/G49538.TCGA-06-0190-10B-01D-A32W-08.1.bam were not downloaded successfully. Please retry this download.\nCommandException: 1 file/object could not be transferred.\n)"

Best,
Chunyang

Best Answer

Answers

  • esalinasesalinas BroadMember, Broadie
    edited September 2017

    @cbao

    It looks that you have tried to access TCGA controlled access data. To access the data, one needs dbGAP authorization. Furthermore, to access that data via FireCloud, one must link one's dbGAP account/authorization with FireCloud. See here https://software.broadinstitute.org/firecloud/documentation/topic?name=firecloud-help-topics and click on " Controlled Access Data in FireCloud" for related documentation.

    Inspection of the error message leads to finding the string : DURABLE_REDUCED_AVAILABILITY

    Google search of the string leads to the page/link : https://cloud.google.com/storage/docs/storage-classes#dra

    It is my understanding that the dbGAP controlled access data, is, as of the current moment, hosted/stored on a "DURABLE_REDUCED_AVAILABILITY" bucket (gs://5aa919de-0aa0-43ec-9ec3-288481102b6d) . It looks to me like you tried to access the data when its availability was "reduced".

    So, unfortunately, it looks like you might have to wait and rerun when availability is better.

    @gordon123 can you comment on the status of the bucket?

    -eddie

  • esalinasesalinas BroadMember, Broadie

    @Geraldine_VdAuwera do you know of any documentation besides https://software.broadinstitute.org/firecloud/documentation/topic?name=firecloud-help-topics that should be referenced for dbGAP, controlled access data, and linking with FC?

  • cbaocbao Member, Broadie

    @esalinas said:
    @cbao

    It looks that you have tried to access TCGA controlled access data. To access the data, one needs dbGAP authorization. Furthermore, to access that data via FireCloud, one must link one's dbGAP account/authorization with FireCloud. See here https://software.broadinstitute.org/firecloud/documentation/topic?name=firecloud-help-topics and click on " Controlled Access Data in FireCloud" for related documentation.

    Inspection of the error message leads to finding the string : DURABLE_REDUCED_AVAILABILITY

    Google search of the string leads to the page/link : https://cloud.google.com/storage/docs/storage-classes#dra

    It is my understanding that the dbGAP controlled access data, is, as of the current moment, hosted/stored on a "DURABLE_REDUCED_AVAILABILITY" bucket. It looks to me like you tried to access the data when its availability was "reduced".

    So, unfortunately, it looks like you might have to wait and rerun when availability is better.

    @gordon123 can you comment on the status of the bucket?

    -eddie

    Thanks, Eddie. But, I have dbGAP authorization. And, some of the calls have been done. The workflow has been shutted down due to one failed call. So, I think it will be better if I ran them individually. Is that correct? Thank you very much!

  • cbaocbao Member, Broadie

    I see. Thanks! I will try to run them individually first by modifying the WDL script. You know, my credit will be end at 27th this month. So, I can not wait ... Thanks!

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator

    @esalinas, the only other documentation I could find was this article

  • esalinasesalinas BroadMember, Broadie

    okay @Tiffany_at_Broad . I asked because the link I gave above seems to have broken images/figures. Do you see any images here ( https://software.broadinstitute.org/firecloud/documentation/topic?name=firecloud-help-topics )

  • KateNKateN Cambridge, MAMember, Broadie, Moderator
    edited September 2017

    @esalinas
    Unfortunately the article you linked wasn't written by our team. Thus, the figures were hosted on an outside image hosting website, which has since expired. We do not have access to the original images (nor do we know what they were, since they looked like that back in April 2017 when our team began to work on FireCloud).

    We are in the process of rewriting a lot of the documentation to sync better with all the changes that have happened with FireCloud's UI, as well as to break up long articles into smaller, more easily readable ones. So that article will soon be replaced with one with non-broken links!

    @cbao
    Were you able to run your samples individually prior to the credit expiration?

  • esalinasesalinas BroadMember, Broadie

    @KateN I couldn't find any other links to offer so that's the one I gave

    -eddie

  • cbaocbao Member, Broadie

    @KateN said:
    @esalinas
    Unfortunately the article you linked wasn't written by our team. Thus, the figures were hosted on an outside image hosting website, which has since expired. We do not have access to the original images (nor do we know what they were, since they looked like that back in April 2017 when our team began to work on FireCloud).

    We are in the process of rewriting a lot of the documentation to sync better with all the changes that have happened with FireCloud's UI, as well as to break up long articles into smaller, more easily readable ones. So that article will soon be replaced with one with non-broken links!

    @cbao
    Were you able to run your samples individually prior to the credit expiration?

    @KateN said:
    @esalinas
    Unfortunately the article you linked wasn't written by our team. Thus, the figures were hosted on an outside image hosting website, which has since expired. We do not have access to the original images (nor do we know what they were, since they looked like that back in April 2017 when our team began to work on FireCloud).

    We are in the process of rewriting a lot of the documentation to sync better with all the changes that have happened with FireCloud's UI, as well as to break up long articles into smaller, more easily readable ones. So that article will soon be replaced with one with non-broken links!

    @cbao
    Were you able to run your samples individually prior to the credit expiration?

    Yes, I have. I got the PoN by running my samples individually. Thanks!

Sign In or Register to comment.