Avoid copying input files

I noticed while testing my .wdl script that input files are copied into the respectively generated cromwell-execution subfolder. However, I expect in my actual analyses that the input files may be huge (several gigabytes per file, many files). Is there a way to avoid copying the files?

Thank you!

Comments

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    By default, Cromwell tries 3 localization strategies when gathering your input files for a run. They are applied in the following order, in succession until one works.

    • hard-link - This will create a hard link (not symbolic) link to the file
    • soft-link - Create a symbolic link to the file. This strategy is not applicable for tasks which specify a Docker image and will be ignored.
    • copy - Make a copy the file

    This default order can be overwritten by specifying a new order in the config file, or on the command line. As the config file is located within the source code of Cromwell, the simplest way to specify a new order is on the command line, e.g.

    java -Dfilesystems.local.localization="newOrder" ...
    

    I suspect a soft-link will solve your issue here. If you would like more information on modifying the config file itself, you can read this section of the Cromwell spec.

Sign In or Register to comment.