Cannot download results from FireCloud using gsutil

jemimalwhjemimalwh ChinaMember
edited November 2016 in Ask the FireCloud Team

Hi, FireCloud team. I found that I cannot download results from FireCloud using gsutil since yesterday (11/21/2016). The error message is the following:

Unable to find the server at www.googleapis.com
CommandException: 1 file/object could not be transferred.

Would you please to fix this problem? Thank you very much. Looking forward to your reply.

Best Regards,
Jemima Li.

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Were you previously able to download data from the same bucket before?

    Have you tried downloading through the GUI interface to the bucket (with the link in the summary tab for the workspace under the section titled "Google Bucket")?

    -eddie

  • jemimalwhjemimalwh ChinaMember

    @esalinas said:
    Were you previously able to download data from the same bucket before?

    Have you tried downloading through the GUI interface to the bucket (with the link in the summary tab for the workspace under the section titled "Google Bucket")?

    -eddie

    Yes, I can download previously.

    And I have tried downloading through the GUI interface to the bucket. But I have large number of result files to download, I need to use gsutil ti download them.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    You may have a networking issue. The stackoverflow posting https://stackoverflow.com/questions/40619337/gsutil-unable-to-find-the-server-at-www-googleapis-com suggesting using one google's public DNS servers: 8.8.8.8 or 8.8.4.4.

    Regards,

    Chet

  • jemimalwhjemimalwh ChinaMember

    @birger said:
    You may have a networking issue. The stackoverflow posting https://stackoverflow.com/questions/40619337/gsutil-unable-to-find-the-server-at-www-googleapis-com suggesting using one google's public DNS servers: 8.8.8.8 or 8.8.4.4.

    Regards,

    Chet

    Hi, thank you for your reply. And I have tried to add 8.8.8.8 and 8.8.4.4 to my Mac DNS setting. However, it doesn't work.

    The error message is still

    Unable to find the server at www.googleapis.com
    CommandException: 1 file/object could not be transferred.

    My situation is:

    1) I can download some of the result files through the Google Bucket GUI interface.
    2) I can ping www.googleapis.com both on my local machine and in the docker container (broadinstitute/firecloud-cli).

    [email protected]:/working# ping www.googleapis.com
    PING googleapis.l.google.com (172.217.26.106) 56(84) bytes of data.
    64 bytes from kix05s01-in-f106.1e100.net (172.217.26.106): icmp_seq=1 ttl=37 time=0.718 ms
    64 bytes from kix05s01-in-f106.1e100.net (172.217.26.106): icmp_seq=2 ttl=37 time=0.450 ms
    64 bytes from kix05s01-in-f106.1e100.net (172.217.26.106): icmp_seq=3 ttl=37 time=0.853 ms
    64 bytes from kix05s01-in-f106.1e100.net (172.217.26.106): icmp_seq=4 ttl=37 time=0.305 ms
    64 bytes from kix05s01-in-f106.1e100.net (172.217.26.106): icmp_seq=5 ttl=37 time=0.547 ms
    64 bytes from kix05s01-in-f106.1e100.net (172.217.26.106): icmp_seq=6 ttl=37 time=0.436 ms

    3) I could download results before, using the same commands (docker, gsutil). Recently I update my system to Sierra and some unexpected problems occurs...
    4) the hosts file in my machine is empty. So no redirection problem.

    Would you please to give me some guide to solve this problems. Since I'm planning to runn more than 10,000 analyses on FireCloud, and the number of the result files is more than 40,000...I have to use some tools to download the files in batch.

    Looking forward to your reply. Thank you!

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Jemima,

    Would you mind sharing your workspace with me temporarily (firecloud user account [email protected]) and provide me with the links to the results files you can and cannot download. I want to see if I can reproduce your issue here. I will probably need to ask for the assistance of our operations group in order to diagnose and fix it.

    Thanks,

    Chet

  • jemimalwhjemimalwh ChinaMember

    @birger said:
    Jemima,

    Would you mind sharing your workspace with me temporarily (firecloud user account [email protected]) and provide me with the links to the results files you can and cannot download. I want to see if I can reproduce your issue here. I will probably need to ask for the assistance of our operations group in order to diagnose and fix it.

    Thanks,

    Chet

    Hi, Chet.

    Thank you. I have shared my workspace (firecloud-seq2hlatcga/TCGA_SKCM2_Seq2HLA_index8) with you.
    The link is https://portal.firecloud.org/#workspaces/firecloud-seq2hlatcga:TCGA_SKCM2_Seq2HLA_index8

    It seems to be a network problem.

    I have also tried to reinstall google-cloud-sdk, but it does not work.

    Thank you.

Sign In or Register to comment.