Run test on sets with some lacking files

Hi all,
I am currently trying to run a method where I basically merge 300 files (.bai files for examples) from all my samples.

I get that error message:
2018/03/25 17:14:20 E: command failed: CommandException: No URLs matched: gs://XXXX
CommandException: 1 file/object could not be transferred.

My method fails because some .bai of some samples are not created yet.

Is there a way to merge the existing files anyway, without taking into account the .bai files that don't exit yet?

Thank you,
CDG

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev admin

    hi @cgdias -

    I suspect this is probably a "not yet", because WDL currently expects all File inputs to exist before it can start a task. You could make the task take a File? input, but even then if the value is specified it's expected to exist.

    What might make this easier would be some sort of File? producing optional_file(file_path_string) method that gets set if the File exists and is empty if it doesn't. Unfortunately that doesn't exist yet but you might be able to make a task or sub-workflow which does the same thing.

    Otherwise:

    • Are you trying to do this in a WDL workflow or directly in FireCloud?
    • How are you getting these paths of not-yet-created bais?
    • Is there a reason the merge step can't be scheduled into the same workflow which creates the bais, so that it's guaranteed to happen as soon as possible, but only once all of the bais are created?
  • Thanks Chris!
    I am doing this directly on Firecloud. The paths of not yet created bai are automatically retrieved when I launch an analysis with the input parameter 'this.sample.bai'.
    Otherwise, is there a way to launch an analysis on ALL samples except one (so far, I just know how to use this.samples when I launch an analysis, but maybe there is a way to specify a few samples for which I don't want the analysis to be run on)

    Thanks again Chris!

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    To launch analysis on all samples except a particular one, you will need to create a sample_set that includes all the samples in your workspace except the one you'd like to exclude. Unfortunately we don't have an easier way of doing that yet, but we hope to make it simpler in the future.

    Let me know if there are any other questions you had in this thread that Chris didn't answer for you--I've read through and it seemed he was able to answer all but your most recent question.

Sign In or Register to comment.