Parrallel haplotypecaller

RoosSRoosS Nijmegen, The NetherlandsMember
in Ask the Cromwell + WDL Team

Hi Wdl team,

this is my first attempt to write a pipeline to use haplotypecaller on a large range of bam files. I have used your tutorials to make my wdl script.
I have tried to run it on a set of 10 bamfiles, and it worked. However when I try to upscale it to the remaining 311 bam files it crashes and tells me the job is running out of memory. I have tried to limit the cpu and amount of jobs, but I cannot get it to work. Can you help me out? I want it to run on ~10 jobs at the same time, and move to the next job when it is finished with a bam file.
Here is the input I used and the errors I got:
What am I missing here?

java -Dconfig.file=/home/data/storage/roos/GATK4_exomes/config.conf -jar /home/data/common/GATK4/cromwell-30.1.jar run /home/data/storage/roos/GATK4_exomes/haplotypecaller.wdl --inputs /home/data/storage/roos/GATK4_exomes/haplotypecaller_inputs.json --options options.json

config file:
backend {
providers {
Local {
config {
concurrent-job-limit = 10
}
}
}
}
options file:
{
"final_workflow_outputs_dir": "/home/data/storage/roos/GATK4_exomes/g.vcfs",
"final_workflow_log_dir": "/home/data/storage/roos/GATK4_exomes/g.vcfs/logs",
"final_call_logs_dir": "/home/data/storage/roos/GATK4_exomes/g.vcfs/call_logs"
}

haplotypecaller.wdl

workflow jointCallingGenotypes {

File inputSamplesFile
Array[Array[File]] inputSamples = read_tsv(inputSamplesFile)
File gatk
File refFasta
File refIndex
File refDict
File snpIndex
File refSnp
File refReg

scatter (sample in inputSamples) {
call HaplotypeCallerERC {
input: GATK=gatk,
RefFasta=refFasta,
RefIndex=refIndex,
RefDict=refDict,
RefSnp=refSnp,
SnpIndex=snpIndex,
RefReg=refReg,
sampleName=sample[0],
bamFile=sample[1],
bamIndex=sample[2]
}
}
}

task HaplotypeCallerERC {
File GATK
File RefFasta
File RefIndex
File RefDict
File RefSnp
File RefReg
String sampleName
File bamFile
File bamIndex
File SnpIndex

command {
    ${GATK} HaplotypeCaller \
    -ERC GVCF \
    -R ${RefFasta} \
    -I ${bamFile} \
    --dbsnp ${RefSnp} \
    -L ${RefReg} \
    -O ${sampleName}.raw.snps.indels.g.vcf
}
output {
File GVCF = "${sampleName}.raw.snps.indels.g.vcf"
File Index = "${sampleName}.raw.snps.indels.g.vcf.idx"
}

}

Errors
[INFO] [01/11/2018 14:36:13.217] [cromwell-system-akka.dispatchers.engine-dispatcher-27] [akka://cromwell-system/user/Si
ngleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-6e096874-1a4a-4122-a1c5-86ac3c6e4296/WorkflowExecutionActor-6
e096874-1a4a-4122-a1c5-86ac3c6e4296] WorkflowExecutionActor-6e096874-1a4a-4122-a1c5-86ac3c6e4296 [UUID(6e096874)]: Start
ing calls: jointCallingGenotypes.HaplotypeCallerERC:0:1, jointCallingGenotypes.HaplotypeCallerERC:1:1, jointCallingGenot
ypes.HaplotypeCallerERC:2:1, jointCallingGenotypes.HaplotypeCallerERC:3:1, jointCallingGenotypes.HaplotypeCallerERC:4:1,
jointCallingGenotypes.HaplotypeCallerERC:5:1, jointCallingGenotypes.HaplotypeCallerERC:6:1, jointCallingGenotypes.Haplo
typeCallerERC:7:1, jointCallingGenotypes.HaplotypeCallerERC:8:1, jointCallingGenotypes.HaplotypeCallerERC:9:1, jointCall
ingGenotypes.HaplotypeCallerERC:10:1, jointCallingGenotypes.HaplotypeCallerERC:11:1, jointCallingGenotypes.HaplotypeCall
erERC:12:1, jointCallingGenotypes.HaplotypeCallerERC:13:1, jointCallingGenotypes.HaplotypeCallerERC:14:1, jointCallingGe
notypes.HaplotypeCallerERC:15:1, jointCallingGenotypes.HaplotypeCallerERC:16:1, jointCallingGenotypes.HaplotypeCallerERC
:17:1, jointCallingGenotypes.HaplotypeCallerERC:18:1, jointCallingGenotypes.HaplotypeCallerERC:19:1, jointCallingGenotyp
es.HaplotypeCallerERC:20:1, jointCallingGenotypes.HaplotypeCallerERC:21:1, jointCallingGenotypes.HaplotypeCallerERC:22:1
, jointCallingGenotypes.HaplotypeCallerERC:23:1, jointCallingGenotypes.HaplotypeCallerERC:24:1, jointCallingGenotypes.Ha
plotypeCallerERC:25:1, jointCallingGenotypes.HaplotypeCallerERC:26:1, jointCallingGenotypes.HaplotypeCallerERC:27:1, joi
ntCallingGenotypes.HaplotypeCallerERC:28:1, jointCallingGenotypes.HaplotypeCallerERC:29:1, jointCallingGenotypes.Haploty
peCallerERC:30:1, jointCallingGenotypes.HaplotypeCallerERC:31:1, jointCallingGenotypes.HaplotypeCallerERC:32:1, jointCal
lingGenotypes.HaplotypeCallerERC:33:1, jointCallingGenotypes.HaplotypeCallerERC:34:1, jointCallingGenotypes.HaplotypeCal
lerERC:35:1, jointCallingGenotypes.HaplotypeCallerERC:36:1, jointCallingGenotypes.HaplotypeCallerERC:37:1, jointCallingG
enotypes.HaplotypeCallerERC:38:1, jointCallingGenotypes.HaplotypeCallerERC:39:1, jointCallingGenotypes.HaplotypeCallerER
C:40:1, jointCallingGenotypes.HaplotypeCallerERC:41:1, jointCallingGenotypes.HaplotypeCallerERC:42:1, jointCallingGenoty
pes.HaplotypeCallerERC:43:1, jointCallingGenotypes.HaplotypeCallerERC:44:1, jointCallingGenotypes.HaplotypeCallerERC:45:
1, jointCallingGenotypes.HaplotypeCallerERC:46:1, jointCallingGenotypes.HaplotypeCallerERC:47:1, jointCallingGenotypes.H
aplotypeCallerERC:48:1, jointCallingGenotypes.HaplotypeCallerERC:49:1
Uncaught error from thread [Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-10cromwell-s
ystem-akka.dispatchers.service-dispatcher-7Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatch
er-9]: ]: Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-11]: unable to create new nati
ve thread]: unable to create new native thread, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for, s
hutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[unable to create new native thread,
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[cromwell-systemunable to create n
ew native thread, ]
cromwell-system]
for ActorSystem[cromwell-system]
Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-88]: unable to create new native thread,
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[cromwell-system]
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at akka.dispatch.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
at akka.dispatch.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966)
at akka.dispatch.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829)
at akka.dispatch.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool.execute(ForkJoinExecutorConfigurator.scala:29)
at akka.dispatch.ExecutorServiceDelegate.execute(ThreadPoolBuilder.scala:211)
at akka.dispatch.ExecutorServiceDelegate.execute$(ThreadPoolBuilder.scala:211)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:39)
at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:115)
at akka.dispatch.Dispatcher.dispatch(Dispatcher.scala:55)
at akka.actor.dungeon.Dispatch.sendMessage(Dispatch.scala:136)
at akka.actor.dungeon.Dispatch.sendMessage$(Dispatch.scala:130)
at akka.actor.ActorCell.sendMessage(ActorCell.scala:370)
at akka.actor.Cell.sendMessage(ActorCell.scala:291)
at akka.actor.Cell.sendMessage$(ActorCell.scala:290)
at akka.actor.ActorCell.sendMessage(ActorCell.scala:370)
at akka.actor.LocalActorRef.$bang(ActorRef.scala:400)
at akka.actor.ActorRef.tell(ActorRef.scala:124)
at cromwell.services.ServiceRegistryActor$$anonfun$receive$1.applyOrElse(ServiceRegistryActor.scala:66)
at akka.actor.Actor.aroundReceive(Actor.scala:514)
at akka.actor.Actor.aroundReceive$(Actor.scala:512)
at cromwell.services.ServiceRegistryActor.aroundReceive(ServiceRegistryActor.scala:56)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

· Share on Twitter Share on Google+

Best Answer

Answers

Sign In or Register to comment.