Reduced Read Memory Heap Error

jfarrelljfarrell Posts: 35Member

I am running the ReduceReads (GATK version 2.3-9-ge5ebf34) on 14 bam files by chromosome. It runs fine on chr21, chr22, X, Y, and MT. Then it runs out of heap on the others(which is at 32G). Is there a memory leak somewhere? Or should I increase the amount of memory?

java -Xmx32g -Djava.io.tmpdir=/scratch -jar $GATK_PREFIX/GenomeAnalysisTK.jar \ -R $REF \ -T ReduceReads \ -I recal_bams.list \ -L $CHR \ -o reduced/saudi_arabian.$CHR.reduced.bam

INFO 20:49:41,838 ProgressMeter - chr5:44828772 2.54e+08 28.7 h 6.8 m 24.8% 4.8 d 87.0 h INFO 20:51:22,265 ProgressMeter - chr5:44851724 2.54e+08 28.7 h 6.8 m 24.8% 4.8 d 87.1 h INFO 20:53:07,958 ProgressMeter - chr5:44851724 2.54e+08 28.7 h 6.8 m 24.8% 4.8 d 87.1 h INFO 20:56:21,576 ProgressMeter - chr5:44851724 2.54e+08 28.8 h 6.8 m 24.8% 4.8 d 87.3 h INFO 20:59:56,672 ProgressMeter - chr5:44851724 2.54e+08 28.8 h 6.8 m 24.8% 4.8 d 87.5 h INFO 21:06:46,367 ProgressMeter - chr5:44851724 2.54e+08 29.0 h 6.8 m 24.8% 4.9 d 87.8 h INFO 21:20:10,834 ProgressMeter - chr5:44851724 2.54e+08 29.1 h 6.9 m 24.8% 4.9 d 88.3 h Exception in thread "ProgressMeterDaemon" ##### ERROR ------------------------------------------------------------------------------------------ java.lang.OutOfMemoryError: Java heap space at java.lang.StringBuilder.toString(StringBuilder.java:405) at org.broadinstitute.sting.utils.AutoFormattingTime.(AutoFormattingTime.java:17) at org.broadinstitute.sting.utils.AutoFormattingTime.(AutoFormattingTime.java:21) at org.broadinstitute.sting.utils.AutoFormattingTime.(AutoFormattingTime.java:25) at org.broadinstitute.sting.utils.progressmeter.ProgressMeter.printProgress(ProgressMeter.java:259) at org.broadinstitute.sting.utils.progressmeter.ProgressMeterDaemon.run(ProgressMeterDaemon.java:52)

ERROR A USER ERROR has occurred (version 2.3-9-ge5ebf34):
ERROR The invalid arguments or inputs must be corrected before the GATK can proceed
ERROR Please do not post this error to the GATK forum
ERROR
ERROR See the documentation (rerun with -h) for this tool to view allowable command-line arguments.
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions http://www.broadinstitute.org/gatk
ERROR

**##### ERROR MESSAGE: There was a failure because you did not provide enough memory to run this program. See the -Xmx JVM argument to adjust the maximum heap size provided to Java

ERROR ------------------------------------------------------------------------------------------**
Tagged:

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,423Administrator, GATK Developer admin

    Have you read this thread? http://gatkforums.broadinstitute.org/discussion/comment/3470/#Comment_3470

    Downsampling my solve your problem.

    Geraldine Van der Auwera, PhD

  • jfarrelljfarrell Posts: 35Member

    I had not seen that thread. I tried the downsampling. Since the bam files are at 40-56x, I started at -dcov 250 and then tried 40.

    When I ran it with -dcov 250, some of the jobs crashed the compute node.

    The -dcov 40 job generated this message. The heap was set at 16G for this job.

    Exception in thread "ProgressMeterDaemon" ##### ERROR ------------------------------------------------------------------------------------------
    ##### ERROR A USER ERROR has occurred (version 2.3-9-ge5ebf34)``:
    ##### ERROR The invalid arguments or inputs must be corrected before the GATK can proceed
    ##### ERROR Please do not post this error to the GATK forum
    ##### ERROR
    ##### ERROR See the documentation (rerun with -h) for this tool to view allowable command-line arguments.
    ##### ERROR Visit our website and forum for extensive documentation and answers to
    ##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
    ##### ERROR
    ##### ERROR MESSAGE: There was a failure because you did not provide enough memory to run this program.  See the -Xmx JVM argument to adjust the maximum heap size provided to Java
    ##### ERROR ------------------------------------------------------------------------------------------
    java.lang.OutOfMemoryError: GC overhead limit exceeded
            at org.broadinstitute.sting.utils.progressmeter.ProgressMeter.takeProgressSnapshot(ProgressMeter.java:311)
            at org.broadinstitute.sting.utils.progressmeter.ProgressMeter.printProgress(ProgressMeter.java:256)
            at org.broadinstitute.sting.utils.progressmeter.ProgressMeterDaemon.run(ProgressMeterDaemon.java:52)
    
  • CarneiroCarneiro Posts: 274Administrator, GATK Developer admin

    Are you sure you want to reduce them all together ? I would reduce them separately to avoid complicated memory issues, unless you really have a good reason to multi-sample reduce.

Sign In or Register to comment.