Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

GATK - Exception in thread "main" java.lang.OutOfMemoryError

I am currently trying to combine 40 individuals into one big file (using 16G of memory) and this is the command line that I am using for this:

echo "Starting run at: `date`"
gatk CombineGVCFs \
-R Gac-HiC_revised_genome_assembly.fa \
--variant variant_MUI004.vcf \
--variant variant_MUI006.vcf \
--variant variant_MUI009.vcf \
--variant variant_MUI010.vcf \
--variant variant_MUI014.vcf \
--variant variant_MUI017.vcf \
--variant variant_MUI024.vcf \
--variant variant_MUI025.vcf \
--variant variant_MUI027.vcf \
--variant variant_MUI028.vcf \
--variant variant_MUI029.vcf \
--variant variant_MUI030.vcf \
--variant variant_MUI032.vcf \
--variant variant_MUI035.vcf \
--variant variant_MUI036.vcf \
--variant variant_MUI037.vcf \
--variant variant_MUI038.vcf \
--variant variant_MUI039.vcf \
--variant variant_MUI040.vcf \
--variant variant_MUI041.vcf \
--variant variant_MUI044.vcf \
--variant variant_MUI045.vcf \
--variant variant_MUI047.vcf \
--variant variant_MUI051.vcf \
--variant variant_MUI052.vcf \
--variant variant_MUI057.vcf \
--variant variant_MUI063.vcf \
--variant variant_MUI066.vcf \
--variant variant_MUI067.vcf \
--variant variant_MUI068.vcf \
--variant variant_MUI071.vcf \
--variant variant_MUI072.vcf \
--variant variant_MUI073.vcf \
--variant variant_MUI074.vcf \
--variant variant_MUI076.vcf \
--variant variant_MUI077.vcf \
--variant variant_MUI079.vcf \
--variant variant_MUI080.vcf \
--variant variant_MUI081.vcf \
--variant variant_MUI083.vcf \
-O cohort.MUIsamples_threespine_alignment.vcf \

echo "Program finished with exit code $? at: `date`"

The command runs fine for about 2 mins and then it shuts down. When looking at the output file, I get an exit code 1, and I can see the following error mentioned:

[June 30, 2019 1:10:15 AM EDT] org.broadinstitute.hellbender.tools.walkers.CombineGVCFs done. Elapsed time: 2.19 minutes.
Runtime.totalMemory()=17351741440
Exception in thread "main" java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:340)
at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:356)
at htsjdk.tribble.readers.SynchronousLineReader.readLine(SynchronousLineReader.java:51)
at htsjdk.tribble.readers.LineIteratorImpl.advance(LineIteratorImpl.java:24)
at htsjdk.tribble.readers.LineIteratorImpl.advance(LineIteratorImpl.java:11)
at htsjdk.samtools.util.AbstractIterator.next(AbstractIterator.java:57)
at htsjdk.tribble.AsciiFeatureCodec.decode(AsciiFeatureCodec.java:70)
at htsjdk.tribble.AsciiFeatureCodec.decode(AsciiFeatureCodec.java:37)
at htsjdk.tribble.TribbleIndexedFeatureReader$WFIterator.readNextRecord(TribbleIndexedFeatureReader.java:373)
at htsjdk.tribble.TribbleIndexedFeatureReader$WFIterator.next(TribbleIndexedFeatureReader.java:354)
at htsjdk.tribble.TribbleIndexedFeatureReader$WFIterator.next(TribbleIndexedFeatureReader.java:315)
at org.broadinstitute.hellbender.engine.MultiVariantDataSource$1.next(MultiVariantDataSource.java:394)
at org.broadinstitute.hellbender.engine.MultiVariantDataSource$1.next(MultiVariantDataSource.java:379)
at htsjdk.samtools.util.PeekableIterator.advance(PeekableIterator.java:71)
at htsjdk.samtools.util.PeekableIterator.next(PeekableIterator.java:57)
at htsjdk.samtools.util.MergingIterator.next(MergingIterator.java:107)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
:

....

Using GATK jar /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/gatk/4.1.0.0/gatk-package-4.1.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/gatk/4.1.0.0/gatk-package-4.1.0.0-local.jar CombineGVCFs -R Gac-HiC_revised_genome_assembly.fa --variant variant_MUI004.vcf --variant variant_MUI006.vcf --variant variant_MUI009.vcf --variant variant_MUI010.vcf --variant variant_MUI014.vcf --variant variant_MUI017.vcf --variant variant_MUI024.vcf --variant variant_MUI025.vcf --variant variant_MUI027.vcf --variant variant_MUI028.vcf --variant variant_MUI029.vcf --variant variant_MUI030.vcf --variant variant_MUI032.vcf --variant variant_MUI035.vcf --variant variant_MUI036.vcf --variant variant_MUI037.vcf --variant variant_MUI038.vcf --variant variant_MUI039.vcf --variant variant_MUI040.vcf --variant variant_MUI041.vcf --variant variant_MUI044.vcf --variant variant_MUI045.vcf --variant variant_MUI047.vcf --variant variant_MUI051.vcf --variant variant_MUI052.vcf --variant variant_MUI057.vcf --variant variant_MUI063.vcf --variant variant_MUI066.vcf --variant variant_MUI067.vcf --variant variant_MUI068.vcf --variant variant_MUI071.vcf --variant variant_MUI072.vcf --variant variant_MUI073.vcf --variant variant_MUI074.vcf --variant variant_MUI076.vcf --variant variant_MUI077.vcf --variant variant_MUI079.vcf --variant variant_MUI080.vcf --variant variant_MUI081.vcf --variant variant_MUI083.vcf -O cohort.MUIsamples_threespine_alignment.vcf
Program finished with exit code 1 at: Sun Jun 30 01:10:16 EDT 2019




I thought that by adding more memory to the specs of the job submission the problem would be solved (this is the solution I found in similar post in the GATK forum) but it does not matter how much memory I add, the result is virtually always the same: Out of Memory. I wonder if maybe there is a way to ask GATK to create a temporal folder or something that would help the program not to shut down due to lack of memory but I am clueless as to what this can be.


Any help would be greatly appreciated!

Thanks!
Tagged:

Answers

Sign In or Register to comment.