Latest Release: 03/12/19
Release Notes can be found here.

error with files with // in google bucket path

danielrdanielr Broad InstituteMember, Broadie

Hi Firecloud,

I have some files in a google bucket which I am passing into a task with the following path type:

gs://[BUCKET_PATH]//[sub_directory]/mybam.bam

When running a task with these inputs, Firecloud immediately outputs an error, saying it couldn't resolve all the inputs, and outputs the problem file as:

[BUCKET_PATH]/[sub_directory]/mybam.bam

Note: The double slash (//) after [BUCKET_PATH] has been collapsed to one slash (/). It seems the full path is not the one being passed. Is this an issue with passing in files with double slashes in the middle of the full path or could it be something else?

I can provide examples if this would be helpful.

Thank you!
Daniel

Best Answer

  • danielrdanielr Broad Institute
    Accepted Answer

    Hi Tiffany,

    I received these files directly from GP and did not have write access to the bucket, so could not change the paths directly, and had to ask them to change them for me, which they were able to do. However, this is highlights an issue which I think would be good if resolved. Otherwise, it will certainly show up again at some point.

    Since the single slash / was the issue in the first place, I don't think replacing the single slash in the attribute path would have fixed the problem.

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @danielr try and see and check your paths and make sure there are no spaces? In particular make sure that the file is NOT

     gs://bucket/dir/path/to/bam.bam
    

    (see the space before "gs")

    In my experience in the past, I have seen problems when there's an extra space at the beginning and problems may result. ALSO, I have seen cases where the UI displays a string without a space BUT when you download the TSV (from either the data model OR the workspace attribute) that the space is evident; here it seems the UI removes beginning or ending whitespace and that obscures the whitespace-finding issue.

  • danielrdanielr Broad InstituteMember, Broadie

    Hi Eddie,

    The issue does not appear to be related to spaces, but rather to Firecloud editing the path, replacing a "//" with a "/".

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @danielr thanks for the clarification

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    Hi @danielr can you try to run this without the double // in the input file path (and all else the same) to see if the error still occurs? If FireCloud doesn't support the //, is this something you'd like for it to do?

  • danielrdanielr Broad InstituteMember, Broadie
    Accepted Answer

    Hi Tiffany,

    I received these files directly from GP and did not have write access to the bucket, so could not change the paths directly, and had to ask them to change them for me, which they were able to do. However, this is highlights an issue which I think would be good if resolved. Otherwise, it will certainly show up again at some point.

    Since the single slash / was the issue in the first place, I don't think replacing the single slash in the attribute path would have fixed the problem.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @danielr

    Can you confirm that the task input that has this issue is of type File and not String? The task definition should say File bam = "gs://bucket/dir/path/to/bam.bam" and not String bam = "gs://bucket/dir/path/to/bam.bam". Thanks!

  • danielrdanielr Broad InstituteMember, Broadie

    Hi Ruchi,

    The task input is of type File.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @danielr can you please share your workspace with [email protected] and I can look at it more deeply. Thanks!

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    Hi @danielr!

    Sam, who created the workspace, said he changed the paths and metadata so it no longer has two slashes. This was a mistake and shouldn't be a reoccurring problem. Are you all set?

    Thanks,
    Tiffany

  • danielrdanielr Broad InstituteMember, Broadie

    Hi @Tiffany_at_Broad ,

    Yes the problem was fixed after Sam moved the paths so there are no two slashes in the bucket path anymore. Will any file path with two slashes be unreadable by Firecloud in the future?

    Thanks,
    Daniel

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    Hi @danielr - Yes, unless the Product Team decides to prioritize this feature. Is this something that is valuable to you in FireCloud?

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    After discussing with Ruchi, we think // should be supported and that the file path was truly non-existent because it was mistyped. Could you share the workspace with [email protected] ?

Sign In or Register to comment.