The Frontline Support team will be slow to respond December 17-18 due to an institute-wide retreat and offline December 22- January 1, while the institute is closed. Thank you for your patience during these next few weeks. Happy Holidays!
Parrallel haplotypecaller
RoosS
Nijmegen, The NetherlandsMember ✭
Hi Wdl team,
this is my first attempt to write a pipeline to use haplotypecaller on a large range of bam files. I have used your tutorials to make my wdl script.
I have tried to run it on a set of 10 bamfiles, and it worked. However when I try to upscale it to the remaining 311 bam files it crashes and tells me the job is running out of memory. I have tried to limit the cpu and amount of jobs, but I cannot get it to work. Can you help me out? I want it to run on ~10 jobs at the same time, and move to the next job when it is finished with a bam file.
Here is the input I used and the errors I got:
What am I missing here?
java -Dconfig.file=/home/data/storage/roos/GATK4_exomes/config.conf -jar /home/data/common/GATK4/cromwell-30.1.jar run /home/data/storage/roos/GATK4_exomes/haplotypecaller.wdl --inputs /home/data/storage/roos/GATK4_exomes/haplotypecaller_inputs.json --options options.json
config file:
backend {
providers {
Local {
config {
concurrent-job-limit = 10
}
}
}
}
options file:
{
"final_workflow_outputs_dir": "/home/data/storage/roos/GATK4_exomes/g.vcfs",
"final_workflow_log_dir": "/home/data/storage/roos/GATK4_exomes/g.vcfs/logs",
"final_call_logs_dir": "/home/data/storage/roos/GATK4_exomes/g.vcfs/call_logs"
}
haplotypecaller.wdl
workflow jointCallingGenotypes {
File inputSamplesFile
Array[Array[File]] inputSamples = read_tsv(inputSamplesFile)
File gatk
File refFasta
File refIndex
File refDict
File snpIndex
File refSnp
File refReg
scatter (sample in inputSamples) {
call HaplotypeCallerERC {
input: GATK=gatk,
RefFasta=refFasta,
RefIndex=refIndex,
RefDict=refDict,
RefSnp=refSnp,
SnpIndex=snpIndex,
RefReg=refReg,
sampleName=sample[0],
bamFile=sample[1],
bamIndex=sample[2]
}
}
}
task HaplotypeCallerERC {
File GATK
File RefFasta
File RefIndex
File RefDict
File RefSnp
File RefReg
String sampleName
File bamFile
File bamIndex
File SnpIndex
command {
${GATK} HaplotypeCaller \
-ERC GVCF \
-R ${RefFasta} \
-I ${bamFile} \
--dbsnp ${RefSnp} \
-L ${RefReg} \
-O ${sampleName}.raw.snps.indels.g.vcf
}
output {
File GVCF = "${sampleName}.raw.snps.indels.g.vcf"
File Index = "${sampleName}.raw.snps.indels.g.vcf.idx"
}
}
Errors
[INFO] [01/11/2018 14:36:13.217] [cromwell-system-akka.dispatchers.engine-dispatcher-27] [akka://cromwell-system/user/Si
ngleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-6e096874-1a4a-4122-a1c5-86ac3c6e4296/WorkflowExecutionActor-6
e096874-1a4a-4122-a1c5-86ac3c6e4296] WorkflowExecutionActor-6e096874-1a4a-4122-a1c5-86ac3c6e4296 [UUID(6e096874)]: Start
ing calls: jointCallingGenotypes.HaplotypeCallerERC:0:1, jointCallingGenotypes.HaplotypeCallerERC:1:1, jointCallingGenot
ypes.HaplotypeCallerERC:2:1, jointCallingGenotypes.HaplotypeCallerERC:3:1, jointCallingGenotypes.HaplotypeCallerERC:4:1,
jointCallingGenotypes.HaplotypeCallerERC:5:1, jointCallingGenotypes.HaplotypeCallerERC:6:1, jointCallingGenotypes.Haplo
typeCallerERC:7:1, jointCallingGenotypes.HaplotypeCallerERC:8:1, jointCallingGenotypes.HaplotypeCallerERC:9:1, jointCall
ingGenotypes.HaplotypeCallerERC:10:1, jointCallingGenotypes.HaplotypeCallerERC:11:1, jointCallingGenotypes.HaplotypeCall
erERC:12:1, jointCallingGenotypes.HaplotypeCallerERC:13:1, jointCallingGenotypes.HaplotypeCallerERC:14:1, jointCallingGe
notypes.HaplotypeCallerERC:15:1, jointCallingGenotypes.HaplotypeCallerERC:16:1, jointCallingGenotypes.HaplotypeCallerERC
:17:1, jointCallingGenotypes.HaplotypeCallerERC:18:1, jointCallingGenotypes.HaplotypeCallerERC:19:1, jointCallingGenotyp
es.HaplotypeCallerERC:20:1, jointCallingGenotypes.HaplotypeCallerERC:21:1, jointCallingGenotypes.HaplotypeCallerERC:22:1
, jointCallingGenotypes.HaplotypeCallerERC:23:1, jointCallingGenotypes.HaplotypeCallerERC:24:1, jointCallingGenotypes.Ha
plotypeCallerERC:25:1, jointCallingGenotypes.HaplotypeCallerERC:26:1, jointCallingGenotypes.HaplotypeCallerERC:27:1, joi
ntCallingGenotypes.HaplotypeCallerERC:28:1, jointCallingGenotypes.HaplotypeCallerERC:29:1, jointCallingGenotypes.Haploty
peCallerERC:30:1, jointCallingGenotypes.HaplotypeCallerERC:31:1, jointCallingGenotypes.HaplotypeCallerERC:32:1, jointCal
lingGenotypes.HaplotypeCallerERC:33:1, jointCallingGenotypes.HaplotypeCallerERC:34:1, jointCallingGenotypes.HaplotypeCal
lerERC:35:1, jointCallingGenotypes.HaplotypeCallerERC:36:1, jointCallingGenotypes.HaplotypeCallerERC:37:1, jointCallingG
enotypes.HaplotypeCallerERC:38:1, jointCallingGenotypes.HaplotypeCallerERC:39:1, jointCallingGenotypes.HaplotypeCallerER
C:40:1, jointCallingGenotypes.HaplotypeCallerERC:41:1, jointCallingGenotypes.HaplotypeCallerERC:42:1, jointCallingGenoty
pes.HaplotypeCallerERC:43:1, jointCallingGenotypes.HaplotypeCallerERC:44:1, jointCallingGenotypes.HaplotypeCallerERC:45:
1, jointCallingGenotypes.HaplotypeCallerERC:46:1, jointCallingGenotypes.HaplotypeCallerERC:47:1, jointCallingGenotypes.H
aplotypeCallerERC:48:1, jointCallingGenotypes.HaplotypeCallerERC:49:1
Uncaught error from thread [Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-10cromwell-s
ystem-akka.dispatchers.service-dispatcher-7Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatch
er-9]: ]: Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-11]: unable to create new nati
ve thread]: unable to create new native thread, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for, s
hutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[unable to create new native thread,
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[cromwell-systemunable to create n
ew native thread, ]
cromwell-system]
for ActorSystem[cromwell-system]
Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-88]: unable to create new native thread,
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[cromwell-system]
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at akka.dispatch.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
at akka.dispatch.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966)
at akka.dispatch.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829)
at akka.dispatch.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool.execute(ForkJoinExecutorConfigurator.scala:29)
at akka.dispatch.ExecutorServiceDelegate.execute(ThreadPoolBuilder.scala:211)
at akka.dispatch.ExecutorServiceDelegate.execute$(ThreadPoolBuilder.scala:211)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:39)
at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:115)
at akka.dispatch.Dispatcher.dispatch(Dispatcher.scala:55)
at akka.actor.dungeon.Dispatch.sendMessage(Dispatch.scala:136)
at akka.actor.dungeon.Dispatch.sendMessage$(Dispatch.scala:130)
at akka.actor.ActorCell.sendMessage(ActorCell.scala:370)
at akka.actor.Cell.sendMessage(ActorCell.scala:291)
at akka.actor.Cell.sendMessage$(ActorCell.scala:290)
at akka.actor.ActorCell.sendMessage(ActorCell.scala:370)
at akka.actor.LocalActorRef.$bang(ActorRef.scala:400)
at akka.actor.ActorRef.tell(ActorRef.scala:124)
at cromwell.services.ServiceRegistryActor$$anonfun$receive$1.applyOrElse(ServiceRegistryActor.scala:66)
at akka.actor.Actor.aroundReceive(Actor.scala:514)
at akka.actor.Actor.aroundReceive$(Actor.scala:512)
at cromwell.services.ServiceRegistryActor.aroundReceive(ServiceRegistryActor.scala:56)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Best Answer
-
kshakir
✭✭
The default
Localbackend does not support limiting resources such as memory and cpu. Once you switch to a different backend you can use runtime-attributes to communicate to a job scheduler what resources are required before a job should be executed.Job schedulers may run either on-premises (including setup on one's laptop) or in the cloud. For example, the most common cloud based scheduler with Cromwell is currently Google's Pipelines API.
For on-premises job scheduling one will need an administrator to install/configure the cluster with various job submission variables for requesting memory, cpu, etc. Once your cluster is setup and you know the variable names that Cromwell should use for submitting runtime-attributes, please follow these instructions for how to configure Cromwell as needed.
Answers
The default
Localbackend does not support limiting resources such as memory and cpu. Once you switch to a different backend you can use runtime-attributes to communicate to a job scheduler what resources are required before a job should be executed.Job schedulers may run either on-premises (including setup on one's laptop) or in the cloud. For example, the most common cloud based scheduler with Cromwell is currently Google's Pipelines API.
For on-premises job scheduling one will need an administrator to install/configure the cluster with various job submission variables for requesting memory, cpu, etc. Once your cluster is setup and you know the variable names that Cromwell should use for submitting runtime-attributes, please follow these instructions for how to configure Cromwell as needed.