Java heap size error after Haplotypecaller finished traversal

prasundutta87prasundutta87 EdinburghMember

Hi,

This is strange and I am not sure if it has affected my gvcf file in any way or not, but I got the java heap size error after the traversal got over:

02:05:31.559 INFO ProgressMeter - Traversal complete. Processed 18934916 total regions in 22931.7 minutes.
02:05:49.823 INFO HaplotypeCaller - Shutting down engine
[13 March 2018 02:05:49 GMT] org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller done. Elapsed time: 22,932.27 minutes.
Runtime.totalMemory()=4284481536
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at htsjdk.tribble.index.interval.IntervalTreeIndex.setChrIndex(IntervalTreeIndex.java:101)
at htsjdk.tribble.index.interval.IntervalIndexCreator.finalizeIndex(IntervalIndexCreator.java:123)
at htsjdk.tribble.index.DynamicIndexCreator.finalizeIndex(DynamicIndexCreator.java:91)
at htsjdk.variant.variantcontext.writer.IndexingVariantContextWriter.close(IndexingVariantContextWriter.java:146)
at htsjdk.variant.variantcontext.writer.VCFWriter.close(VCFWriter.java:212)
at org.broadinstitute.hellbender.utils.variant.writers.GVCFWriter.close(GVCFWriter.java:139)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller.closeTool(HaplotypeCaller.java:224)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:897)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:136)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:153)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:195)
at org.broadinstitute.hellbender.Main.main(Main.java:277)

When I checked the gvcf file of some similar samples where Haplotypecaller finished successfully, the start and end bin of gvcf of the last call was same, and hence I came to know that the run was succesful in this case. But, is there any chance there may be some other error of some sort that I may be unaware? I gave 4 GB to my java and I am running GATK 4.0.1.2.

Best Answer

Answers

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited March 2018

    You might want to give at least 8 to 16 gbs to HaplotypeCaller even when you are running it in single threaded mode.

    BTW your GVCF is not complete. GATK stopped due to failure.

  • prasundutta87prasundutta87 EdinburghMember
    edited March 2018

    Thanks for the reply @SkyWarrior

    What about this then? (before the Java error)

    02:05:31.559 INFO ProgressMeter - Traversal complete. Processed 18934916 total regions in 22931.7 minutes.
    02:05:49.823 INFO HaplotypeCaller - Shutting down engine
    [13 March 2018 02:05:49 GMT] org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller done. Elapsed time: 22,932.27 minutes.

    What can I check in the gvcf file to know if it has formed correctly or not. The chrosmosomes (scaffolds in my case) seem to have been traversed completely.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Traversal could have been forced to complete due to error. I would not continue with any file generated after an error message. I would also suggest you to upgrade to 4.0.2.1

  • prasundutta87prasundutta87 EdinburghMember
    edited March 2018

    Ok. I will try to regenerate the file. It took 17 days to finally get an error like this. Upgrading to 4.0.2.1 is not an option for me as I have already generated 81 samples gvcfs using 4.0.1.2. Furthermore, the bug fixes in Haplotypecaller does not affect my work as I am anyway not using those options.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Do you run haplotype caller one sample at a time on your work platform. Do you have the option to run multiple samples in different instances at once to accelerate your work flow?

  • prasundutta87prasundutta87 EdinburghMember
    edited March 2018

    I use the shared oracle grid engine cluster of the University of Edinburgh. I don't run one sample at a time, but all the samples together in different nodes. The thing is that without using the spark version of the tool, which is still in beta version, I cannot use multiple threads. I am just setting pairHMM thread to 8 which again does not seem to utilize all the cores assigned to the job (8). I am having another discussion relating to that somehere else in the forum.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    Accepted Answer

    Ok it is sounds good overall. You may need to increase your -Xmx setting and you should be set for the rest of the work.

  • prasundutta87prasundutta87 EdinburghMember
    edited March 2018

    Before I start running haplotypecaller again, I validated my gvcf file using this command:

    java -Xmx4g -jar /exports/eddie3_homes_local/s0928794/tools/gatk-package-4.0.1.2-local.jar ValidateVariants -R GCF_000471725.1_UMD_CASPUR_WB_2.0_genomic.fa -V Bhadawari_male_30x_WGS_atlas.g.vcf --validate-GVCF

    I did not get any errors/warnings. I guess the error came while making .idx file.

    Is .idx file necessary downstream?

    Also, at the end, I had also got this message:

    02:05:31.551 INFO HaplotypeCaller - 285304515 read(s) filtered by: ((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter) AND GoodCigarReadFilter) AND WellformedReadFilter)
    285304515 read(s) filtered by: (((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter) AND GoodCigarReadFilter)
    285304515 read(s) filtered by: ((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter)
    285304515 read(s) filtered by: (((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter)
    285304515 read(s) filtered by: ((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter)
    115075927 read(s) filtered by: (((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter)
    115075927 read(s) filtered by: ((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter)
    115075927 read(s) filtered by: (MappingQualityReadFilter AND MappingQualityAvailableReadFilter)
    115075927 read(s) filtered by: MappingQualityReadFilter
    170228588 read(s) filtered by: NotDuplicateReadFilter

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @prasundutta87
    Hi,

    Unfortunately, @SkyWarrior is correct that you should restart your HaplotypeCaller run because you had an error message. The index file is indeed necessary for downstream analysis, but the tool should generate one for you if it runs to completion.

    -Sheila

  • prasundutta87prasundutta87 EdinburghMember

    I have already started it again after @Skywalker 's suggestion..I was just trying to understand where exactly problem arose..

  • shleeshlee CambridgeMember, Broadie, Moderator admin

    Ok @prasundutta87. Let us know if you continue to encounter this error.

  • prasundutta87prasundutta87 EdinburghMember

    Hi,

    I provided 8G as java heap size and the speed by which GVCF got made changed considerably. It got done in just 5 days (although I gave pair HMM as 16, it still used 2 cores on average). So, giving more java heap size increases the no. of sites getting processed at once increases. Although, the size of the old and the new gvcf file was 96248919K. So, I believe, duing the idx file generation, the memory issue happened.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @prasundutta87
    Hi,

    The progress meter cannot be trusted entirely. You are correct that indexing and a variety of things could be happening even after the tool completes.
    Glad to hear the tool works with more memory.

    -Sheila

Sign In or Register to comment.