To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

call caching error with softlink output files

When call-caching is enabled for a workflow that has soft-linked files as output for some of the tasks, cromwell will not produce any output.

task softlink {
    command {
        date >> outfile.txt
        ln -s outfile.txt datum
    }
    output {
        File datum = "datum"
    }
}

workflow wf {
    call softlink
}

As far as I can tell, it attempts to link the soft-linked output file into the root directory where cromwell is run, which fails. However, it will then still report the final outputs, even though the file is missing.

redmar@hostname~/devel/wdl$ cromwell run softlink.wdl 
[2017-01-23 07:52:51,20] [warn] Localization via hard link has failed: /home/redmar/devel/wdl/cromwell-executions/wf/dbfff823-ce16-4fa8-909f-6f8ee7c763fa/call-softlink/execution/datum -> /home/redmar/devel/wdl/outfile.txt
[2017-01-23 07:52:51,21] [warn] Localization via copy has failed: /home/redmar/devel/wdl/outfile.txt
[INFO] [01/23/2017 07:52:51.289] [cromwell-system-akka.dispatchers.engine-dispatcher-33] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-dbfff823-ce16-4fa8-909f-6f8ee7c763fa/WorkflowExecutionActor-dbfff823-ce16-4fa8-909f-6f8ee7c763fa] WorkflowExecutionActor-dbfff823-ce16-4fa8-909f-6f8ee7c763fa [UUID(dbfff823)]: Workflow wf complete. Final Outputs:
{
  "wf_softlink_datum": "file:///home/redmar/devel/wdl/cromwell-executions/wf/dbfff823-ce16-4fa8-909f-6f8ee7c763fa/call-softlink/execution/datum"
}
redmar@hostname:~/devel/wdl$  cat /home/redmar/devel/wdl/cromwell-executions/wf/dbfff823-ce16-4fa8-909f-6f8ee7c763fa/call-softlink/execution/datum
cat: /home/redmar/devel/wdl/cromwell-executions/wf/dbfff823-ce16-4fa8-909f-6f8ee7c763fa/call-softlink/execution/datum: No such file or directory
Tagged:

Answers

Sign In or Register to comment.