Bug Bulletin: The recent 3.2 release fixes many issues. If you run into a problem, please try the latest version before posting a bug report, as your problem may already have been solved.

GATK 3.0 HaplotypeCaller RUNTIME ERROR

5581681555816815 NJPosts: 2Member
edited March 19 in Ask the GATK team

Hi,

I am running Haplotypecaller for ~600 bams to perform gVCF call on a cluster (SGE qsub system). Each node has 32 cores, 256 GB RAM. I am running 8 tasks pernode, so each task has 4 cores and 32GB memories.

It's been running ~30 hours now (a single BAM needs ~5 hours), out of the ~500 finished tasks there are 24 quit with error showing below. I am not sure if it is an error caused by my command or something else. Can someone give any suggestions?

The command:

java -Xmx32G GATK3.0 \
-T HaplotypeCaller \
-ERC gVCF -L EZ_Exome_v2.bed \
-variant_index_type LINEAR \
-variant_index_parameter 128000 \
-R ucsc.hg19.fasta \
-nct 4 \
--dbsnp dbsnp_138.hg19.vcf \
-I 1.recalibrated.bam \
-o 1.recalibrated.vcf
ERROR ------------------------------------------------------------------------------------------
ERROR stack trace

java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.GenotypingEngine.addMiscellaneousAllele(GenotypingEngine.java:257) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.GenotypingEngine.assignGenotypeLikelihoods(GenotypingEngine.java:227) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.HaplotypeCaller.map(HaplotypeCaller.java:872) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.HaplotypeCaller.map(HaplotypeCaller.java:141) at org.broadinstitute.sting.gatk.traversals.TraverseActiveRegions$TraverseActiveRegionMap.apply(TraverseActiveRegions.java:708) at org.broadinstitute.sting.gatk.traversals.TraverseActiveRegions$TraverseActiveRegionMap.apply(TraverseActiveRegions.java:704) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler$ReadMapReduceJob.run(NanoScheduler.java:471) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)

ERROR ------------------------------------------------------------------------------------------
ERROR A GATK RUNTIME ERROR has occurred (version 3.0-0-g6bad1c6):
ERROR
ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.
ERROR If not, please post the error message, with stack trace, to the GATK forum.
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions http://www.broadinstitute.org/gatk
ERROR
ERROR MESSAGE: Code exception (see stack trace for error itself)
ERROR ------------------------------------------------------------------------------------------

thanks a lot!

Post edited by 55816815 on
Tagged:

Best Answer

Answers

  • 5581681555816815 NJPosts: 2Member

    Hi Geraldine, Appreciated for the prompt response! I will just reload all failed jobs.

    Shuoguo

  • tbernertberner germanyPosts: 12Member

    Hi you two,

    is there anything done so far? Because we ran into the same problem using version 3.1-1-g07a4bf8. It appears on different datasets and after different progress times. Because of we have large BAM files running in single-thread mode would take weeks of runtime.

    ERROR stack trace

    java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.GenotypingEngine.addMiscellaneousAllele(GenotypingEngine.java:257) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.GenotypingEngine.assignGenotypeLikelihoods(GenotypingEngine.java:227) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.HaplotypeCaller.map(HaplotypeCaller.java:880) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.HaplotypeCaller.map(HaplotypeCaller.java:141) at org.broadinstitute.sting.gatk.traversals.TraverseActiveRegions$TraverseActiveRegionMap.apply(TraverseActiveRegions.java:708) at org.broadinstitute.sting.gatk.traversals.TraverseActiveRegions$TraverseActiveRegionMap.apply(TraverseActiveRegions.java:704) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler$ReadMapReduceJob.run(NanoScheduler.java:471) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

    ERROR ------------------------------------------------------------------------------------------
    ERROR A GATK RUNTIME ERROR has occurred (version 3.1-1-g07a4bf8):
  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 5,990Administrator, GATK Developer admin

    @tberner Not yet I'm afraid -- we haven't been able to devote any resources to this. Our current recommendation is to just drop -nct from your command if you run into this issue, or better, use a pipeline manager like Queue that parallelizes over chunks of data and restarts failed jobs automatically. Since the errors typically don't reproduce on the same chunks of data, this eventually gets everything done, and is still faster than running everything single-threaded.

    Geraldine Van der Auwera, PhD

  • MikeKMikeK Hamilton NZPosts: 3Member

    I've just found it too! ``` INFO 03:48:33,595 HelpFormatter - ------------------------------------------------------------------------------------------ INFO 03:48:33,597 HelpFormatter - The Genome Analysis Toolkit (GATK) v2014.2-3.1.7-10-g867c2fb, Compiled 2014/04/18 10:40:14 INFO 03:48:33,597 HelpFormatter - Copyright (c) 2010 The Broad Institute INFO 03:48:33,597 HelpFormatter - For support and documentation go to http://gatkdocs.appistry.com/ INFO 03:48:33,600 HelpFormatter - Program Args: -R /data/seq/indexed-genomes/bos_taurus/umd31MT/umd31MT.fa -dcov 200 -T HaplotypeCaller -BQSR /data/seq/mikee0/checkpoint/23934418.bwamem.sorted.reheadered.bam.firstpass.table -pairHMM VECTOR_LOGLESS_CACHING -nct 12 -I /data/seq/mikee0/checkpoint/23934418.bwamem.sorted.reheadered.bam.realigned.bam --dbsnp dbSNP-138-UMD3.1-no-spaces.vcf.gz --emitRefConfidence GVCF --variant_index_type LINEAR --variant_index_parameter 128000 -o 23934418.default-param.g.vcf INFO 03:48:33,605 HelpFormatter - Executing as mikee0@galvatron on Linux 3.13.0-29-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.7.0_45-b18. INFO 03:48:33,605 HelpFormatter - Date/Time: 2014/07/03 03:48:33 INFO 03:48:33,605 HelpFormatter - ------------------------------------------------------------------------------------------

    ERROR ------------------------------------------------------------------------------------------
    ERROR stack trace

    java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.GenotypingEngine.addMiscellaneousAllele(GenotypingEngine.java:257) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.GenotypingEngine.assignGenotypeLikelihoods(GenotypingEngine.java:220) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.HaplotypeCaller.map(HaplotypeCaller.java:880) at org.broadinstitute.sting.gatk.walkers.haplotypecaller.HaplotypeCaller.map(HaplotypeCaller.java:141) at org.broadinstitute.sting.gatk.traversals.TraverseActiveRegions$TraverseActiveRegionMap.apply(TraverseActiveRegions.java:708) at org.broadinstitute.sting.gatk.traversals.TraverseActiveRegions$TraverseActiveRegionMap.apply(TraverseActiveRegions.java:704) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler$ReadMapReduceJob.run(NanoScheduler.java:471) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

    ERROR ------------------------------------------------------------------------------------------
    ERROR A GATK RUNTIME ERROR has occurred (version 2014.2-3.1.7-10-g867c2fb):

    ```

    A thread safety error is a bit of a worry!

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 5,990Administrator, GATK Developer admin

    Yes it is a concern, that is why we are favoring alternative ways to speed up HC without using multithreading.

    Geraldine Van der Auwera, PhD

  • gtiaogtiao Cambridge, MAPosts: 9Member

    Just to chime in, I've run into this problem, too, on GATK 3.2-2 calling WGS samples with -nct 2. Some of my samples have run happily without any incident for nearly one day before failing.

    Geraldine, is there a current recommendation or best-practice for speeding up HC on large BAMs?

    Grace

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 5,990Administrator, GATK Developer admin

    Hi Grace,

    Right now the only recommendation we have is to use Queue to parallelize your HC jobs. And of course, make sure you're using the new workflow to run HC per sample in GVCF mode, followed by joint genotyping.

    Geraldine Van der Auwera, PhD

Sign In or Register to comment.