Our staff will be observing the holiday and will be unavailable from 22nd to 25th November. This will cause a delay in reaching out to you and answering your questions immediately. Rest assured we will get back to it on Monday November 26th. We are grateful for your support and patience.
Have a great holiday everyone!!!
Localization via hard link has failed
I'm using cromwell 29 with docker, and I keep getting this error:
[2017-09-15 09:08:57,11] [warn] Localization via hard link has failed: /data/test/cromwell-executions/variantDiscovery/5bb6004a-dcf9-4459-84cf-7b7ecb35d960/call-Report/variantDiscoveryReport/bc64e2c9-fd27-429c-82b2-22458b63d9eb/call-plotBam/inputs/data/test/cromwell-executions/variantDiscovery/5bb6004a-dcf9-4459-84cf-7b7ecb35d960/call-dedup/shard-1/execution/102517-23.dedup.bam -> /data/test/cromwell-executions/variantDiscovery/a4726bef-8d45-4746-ac99-9cabf9dadd36/call-dedup/shard-1/execution/102517-23.dedup.bam: Operation not permitted
Because I am using docker, soft-linking is not possible, so cromwell keeps copying over all the files it needs. I also generate a report for every step of the analysis, so cromwell makes a copy of every single output file! Because of this localization problem, this effectively doubles the disk size of every analysis, and I cannot complete my analysis because I run out of harddisk space.
I found a similar issue on the forum here https://gatkforums.broadinstitute.org/wdl/discussion/9477/localization-via-hard-link-has-failed, but the solution is not very clear. I think EADG added a user to each docker image he uses? That's not an option for me since I do not control all the docker images I use (e.g the broadinstitute images for gatk and picard).
How can I resolve this issue? Creating a copy of every bam and fastq file every time it is needed is not acceptable in my situation, and I also don't want to stop using docker, for obvious reasons.