Localization via hard link has failed
I'm using cromwell 29 with docker, and I keep getting this error:
[2017-09-15 09:08:57,11] [warn] Localization via hard link has failed: /data/test/cromwell-executions/variantDiscovery/5bb6004a-dcf9-4459-84cf-7b7ecb35d960/call-Report/variantDiscoveryReport/bc64e2c9-fd27-429c-82b2-22458b63d9eb/call-plotBam/inputs/data/test/cromwell-executions/variantDiscovery/5bb6004a-dcf9-4459-84cf-7b7ecb35d960/call-dedup/shard-1/execution/102517-23.dedup.bam -> /data/test/cromwell-executions/variantDiscovery/a4726bef-8d45-4746-ac99-9cabf9dadd36/call-dedup/shard-1/execution/102517-23.dedup.bam: Operation not permitted
Because I am using docker, soft-linking is not possible, so cromwell keeps copying over all the files it needs. I also generate a report for every step of the analysis, so cromwell makes a copy of every single output file! Because of this localization problem, this effectively doubles the disk size of every analysis, and I cannot complete my analysis because I run out of harddisk space.
I found a similar issue on the forum here https://gatkforums.broadinstitute.org/wdl/discussion/9477/localization-via-hard-link-has-failed, but the solution is not very clear. I think EADG added a user to each docker image he uses? That's not an option for me since I do not control all the docker images I use (e.g the broadinstitute images for gatk and picard).
How can I resolve this issue? Creating a copy of every bam and fastq file every time it is needed is not acceptable in my situation, and I also don't want to stop using docker, for obvious reasons.