Error: No space left on device even block and inode are not fully used

Angry_PandaAngry_Panda Member
edited September 25 in Ask the Cromwell + WDL Team

Dear cromwell team,

I was using my cPouta Cloud VM (RAM117.2GB, 24VCPUs, disk 3T) with docker to run gatk best practice germline-snps-indels workflow with wdl and json files offered by this git link (https://github.com/gatk-workflows/gatk4-germline-snps-indels) after necessary changed all file location to my VM absolute path.

It report this error many times:
[error] BackgroundConfigAsyncJobExecutionActor [a99c2cbeHaplotypeCallerGvcf_GATK4.HaplotypeCaller:47:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:536)
at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:471)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:265)
at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:264)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:599)
at scala.util.Try$.apply(Try.scala:209)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:599)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:599)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:912)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:904)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:514)
at akka.actor.Actor.aroundReceive$(Actor.scala:512)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:193)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
Could not localize /media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-CramToBamTask/execution/NA12878_NA12878.bam -> /media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-HaplotypeCaller/shard-47/inputs/1851625158/NA12878_NA12878.bam:
/media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-CramToBamTask/execution/NA12878_NA12878.bam doesn't exist
/media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-HaplotypeCaller/shard-47/inputs/1851625158/NA12878_NA12878.bam -> /media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-CramToBamTask/execution/NA12878_NA12878.bam: Operation not permitted
/media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-CramToBamTask/execution/NA12878_NA12878.bam -> /media/volume/t2/cromwell-executions/HaplotypeCallerGvcf_GATK4/a99c2cbe-bb6b-49a7-a021-c76da3d87d68/call-HaplotypeCaller/shard-47/inputs/1851625158/NA12878_NA12878.bam.tmp: No space left on device
at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:60)
at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:56)
at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:534)

I checked my VM inode and block space both are not full:
[[email protected] ~]$ df -lih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda1 225M 1.3M 224M 1% /
devtmpfs 15M 425 15M 1% /dev
tmpfs 15M 1 15M 1% /dev/shm
tmpfs 15M 478 15M 1% /run
tmpfs 15M 16 15M 1% /sys/fs/cgroup
/dev/vdb 200M 579 200M 1% /media/volume
tmpfs 15M 1 15M 1% /run/user/1000
[[email protected] ~]$ df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 844G 592G 216G 74% /
devtmpfs 58G 0 58G 0% /dev
tmpfs 58G 0 58G 0% /dev/shm
tmpfs 58G 17M 58G 1% /run
tmpfs 58G 0 58G 0% /sys/fs/cgroup
/dev/vdb 2.0T 222G 1.8T 12% /media/volume
tmpfs 12G 0 12G 0% /run/user/1000

PS: when I only offered 3 hg38_wgs_scattered_calling_intervals (via change gatk-test-data/intervals/hg38_wgs_scattered_calling_intervals.txt), the workflow finished success.

Thanks for reading my questions and looking forward to your reply. __Angry_Panda

Post edited by Angry_Panda on

Answers

Sign In or Register to comment.