Can you pass an entire directory as a single input to a WDL task?

jneffjneff BostonMember, Broadie, Moderator admin

Computational biologists and software engineers have asked about this in the FireCloud forum. In Firehose, they could pass directories as string inputs. When using FireCloud, they would rather not specify each file within a Google Bucket folder as an input parameter.

The FireCloud Core Team would like to explore possible solutions/workarounds or request an enhancement.

Related questions in FireCloud Forum:

gatkforums.broadinstitute.org/firecloud/discussion/7054/google-bucket-directories-as-task-input#latest

gatkforums.broadinstitute.org/firecloud/discussion/7327/passing-a-directory-to-firecloud#latest

Tagged:

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    As WDL currently stands, there is no native way to pass entire directories in to a task. There are, however, workarounds.

    1. Specify each file individually; though tedious, this is perhaps the most straightforward method.
    2. If you are using Docker, and the directory in question is both small and static (i.e. known at the start of the script), you can bundle the directory into the Docker image.
    3. If neither of the above options are suitable, you can pass the directory in to the task in the form of a gzipped file. The task can then unzip the file, like so:
    task stepA {
        File myDir_gzipped
        command {
            tar -xzf ${myDir_gzipped}
            #insert work to be performed on directory files
        }
    }
    

    We intend to support directories natively in the future, though for now these solutions should help your users continue working.

  • jneffjneff BostonMember, Broadie, Moderator admin

    Thank you for the reply! I will relay this information.

Sign In or Register to comment.