Reduced Read Memory Heap Error

jfarrell

I am running the ReduceReads (GATK version 2.3-9-ge5ebf34) on 14 bam files by chromosome. It runs fine on chr21, chr22, X, Y, and MT. Then it runs out of heap on the others(which is at 32G). Is there a memory leak somewhere? Or should I increase the amount of memory?

java -Xmx32g -Djava.io.tmpdir=/scratch -jar $GATK_PREFIX/GenomeAnalysisTK.jar \ -R $REF \ -T ReduceReads \ -I recal_bams.list \ -L $CHR \ -o reduced/saudi_arabian.$CHR.reduced.bam

INFO 20:49:41,838 ProgressMeter - chr5:44828772 2.54e+08 28.7 h 6.8 m 24.8% 4.8 d 87.0 h INFO 20:51:22,265 ProgressMeter - chr5:44851724 2.54e+08 28.7 h 6.8 m 24.8% 4.8 d 87.1 h INFO 20:53:07,958 ProgressMeter - chr5:44851724 2.54e+08 28.7 h 6.8 m 24.8% 4.8 d 87.1 h INFO 20:56:21,576 ProgressMeter - chr5:44851724 2.54e+08 28.8 h 6.8 m 24.8% 4.8 d 87.3 h INFO 20:59:56,672 ProgressMeter - chr5:44851724 2.54e+08 28.8 h 6.8 m 24.8% 4.8 d 87.5 h INFO 21:06:46,367 ProgressMeter - chr5:44851724 2.54e+08 29.0 h 6.8 m 24.8% 4.9 d 87.8 h INFO 21:20:10,834 ProgressMeter - chr5:44851724 2.54e+08 29.1 h 6.9 m 24.8% 4.9 d 88.3 h Exception in thread "ProgressMeterDaemon" ##### ERROR ------------------------------------------------------------------------------------------ java.lang.OutOfMemoryError: Java heap space at java.lang.StringBuilder.toString(StringBuilder.java:405) at org.broadinstitute.sting.utils.AutoFormattingTime.(AutoFormattingTime.java:17) at org.broadinstitute.sting.utils.AutoFormattingTime.(AutoFormattingTime.java:21) at org.broadinstitute.sting.utils.AutoFormattingTime.(AutoFormattingTime.java:25) at org.broadinstitute.sting.utils.progressmeter.ProgressMeter.printProgress(ProgressMeter.java:259) at org.broadinstitute.sting.utils.progressmeter.ProgressMeterDaemon.run(ProgressMeterDaemon.java:52)

ERROR A USER ERROR has occurred (version 2.3-9-ge5ebf34):

ERROR The invalid arguments or inputs must be corrected before the GATK can proceed

ERROR Please do not post this error to the GATK forum

ERROR

ERROR See the documentation (rerun with -h) for this tool to view allowable command-line arguments.

ERROR Visit our website and forum for extensive documentation and answers to

ERROR commonly asked questions http://www.broadinstitute.org/gatk

ERROR

**##### ERROR MESSAGE: There was a failure because you did not provide enough memory to run this program. See the -Xmx JVM argument to adjust the maximum heap size provided to Java

ERROR ------------------------------------------------------------------------------------------**

Geraldine_VdAuwera

Have you read this thread? http://gatkforums.broadinstitute.org/discussion/comment/3470/#Comment_3470

Downsampling my solve your problem.

jfarrell

I had not seen that thread. I tried the downsampling. Since the bam files are at 40-56x, I started at -dcov 250 and then tried 40.

When I ran it with -dcov 250, some of the jobs crashed the compute node.

The -dcov 40 job generated this message. The heap was set at 16G for this job.

Exception in thread "ProgressMeterDaemon" ##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 2.3-9-ge5ebf34)``:
##### ERROR The invalid arguments or inputs must be corrected before the GATK can proceed
##### ERROR Please do not post this error to the GATK forum
##### ERROR
##### ERROR See the documentation (rerun with -h) for this tool to view allowable command-line arguments.
##### ERROR Visit our website and forum for extensive documentation and answers to
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR MESSAGE: There was a failure because you did not provide enough memory to run this program.  See the -Xmx JVM argument to adjust the maximum heap size provided to Java
##### ERROR ------------------------------------------------------------------------------------------
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.broadinstitute.sting.utils.progressmeter.ProgressMeter.takeProgressSnapshot(ProgressMeter.java:311)
        at org.broadinstitute.sting.utils.progressmeter.ProgressMeter.printProgress(ProgressMeter.java:256)
        at org.broadinstitute.sting.utils.progressmeter.ProgressMeterDaemon.run(ProgressMeterDaemon.java:52)

Carneiro

Are you sure you want to reduce them all together ? I would reduce them separately to avoid complicated memory issues, unless you really have a good reason to multi-sample reduce.