Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

SVCNVDiscovery Error "java.lang.OutOfMemoryError: Java heap space" AFTER adding "-memLimit 100" flag

Dear Genome STRiP users,

I completed the SVPreprocess to the 10686 samples successfully. When I run SVCNVDiscovery pipeline with the following script,

classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
gs_dir="/proj/yunligrp/users/minzhi/gs"
svpreprocess_dir="${gs_dir}/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success"
rundir="${gs_dir}/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000"

java -Xmx4g -cp ${classpath} \
    org.broadinstitute.gatk.queue.QCommandLine \
    -S ${SV_DIR}/qscript/discovery/cnv/CNVDiscoveryPipeline.q \
    -S ${SV_DIR}/qscript/SVQScript.q \
    -cp ${classpath} \
    -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
    -configFile ${SV_DIR}/conf/genstrip_parameters.txt \
    -R ${gs_dir}/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta \
    -I ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_sample.list \
    -genderMapFile ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map \
    -ploidyMapFile ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map \
    -md ${svpreprocess_dir}/md_tempdir \
    -tempDir ${gs_dir}/gs_tempdir/svcnv_tmp \
    -runDirectory ${rundir} \
    -jobLogDir ${rundir}/logs \
    -intervalList ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_1-500000_interval.list \
    -tilingWindowSize 1000 \
    -tilingWindowOverlap 500 \
    -maximumReferenceGapLength 1000 \
    -boundaryPrecision 100 \
    -minimumRefinedLength 500 \
    -memLimit 100 \
    -jobRunner Drmaa \
    -gatkJobRunner Drmaa \
    -jobNative "--mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8" \
    -jobQueue general \
    -run \
    || exit 1

The modification I made to the QScripts are
1. SVQScript (line 1284&1285)

        this.memoryLimit = Some(85)
        this.javaMemoryLimit = Some(85)
  1. Under "CallSampleGender()" and "ComputeDepthProfile(profilesDir: File, sequenceName: String, intervalList: List[GenomeInterval])" functions, I added the following lines to allow more memory
        this.memoryLimit = Some(10)
        this.javaMemoryLimit = Some(10)

Then I got the following errors:

  1. Non-specific error
ERROR 16:06:20,037 FunctionEdge - Error:  'java'  '-Xmx102400m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/SVQScript.q'  '-gatk' '/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs'  '-memLimit' '100.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '--mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8'  '-jobQueue' 'general'  -run  '-sequenceName' 'chr16'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-sentinelFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_sentinel_files/stage_2_seq_chr16.sent'  --disableJobReport  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-copyNumberMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gcmask.fasta'  '-readDepthMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.rdmask.bed'  '-genderMaskBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gendermask.bed'  '-vdjBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.vdjregions.bed'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  -disableGATKTraversal  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-genderGenotypeFilterFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter.txt'  '-filterDescriptionFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter_descr.txt'  '-genotypingParallelRecords' '1000'  
ERROR 16:06:20,069 FunctionEdge - Contents of /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/logs/CNVDiscoveryPipeline-5.out:
INFO  16:05:27,250 QScriptManager - Compiling 4 QScripts 
INFO  16:05:32,094 QScriptManager - Compilation complete 
INFO  16:05:32,226 HelpFormatter - ---------------------------------------------------------------------- 
INFO  16:05:32,227 HelpFormatter - Queue v3.7.GS-r1748-0-g74bfe0b, Compiled 2018/04/10 10:30:23 
INFO  16:05:32,227 HelpFormatter - Copyright (c) 2012 The Broad Institute 
INFO  16:05:32,227 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
INFO  16:05:32,227 HelpFormatter - Program Args: -cp /proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/SVQScript.q -gatk /proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar -jobLogDir /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs -memLimit 100.0 -jobRunner Drmaa -gatkJobRunner Drmaa -jobNative --mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8 -jobQueue general -run -sequenceName chr16 -runDirectory /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16 -sentinelFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_sentinel_files/stage_2_seq_chr16.sent --disableJobReport -configFile /proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt -R /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta -ploidyMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta -copyNumberMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gcmask.fasta -readDepthMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.rdmask.bed -genderMaskBedFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gendermask.bed -vdjBedFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.vdjregions.bed -genderMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map -md /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir -disableGATKTraversal -I /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam -vcf /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz -genderGenotypeFilterFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter.txt -filterDescriptionFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter_descr.txt -genotypingParallelRecords 1000 
INFO  16:05:32,228 HelpFormatter - Executing as [email protected] on Linux 3.10.0-957.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_191-b12. 
INFO  16:05:32,228 HelpFormatter - Date/Time: 2019/03/02 16:05:32 
INFO  16:05:32,228 HelpFormatter - 
  1. java.lang.OutOfMemoryError: Java heap space
ERROR 16:06:03,437 FunctionEdge - Error:  'java'  '-Xmx3072m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar' org.broadinstitute.sv.main.SVGenotyper '-T' 'SVGenotyperWalker'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-O' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/P0001.genotypes.vcf.gz'  '-disableGATKTraversal' 'true'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt' '-configFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gsparams.txt'  '-P' 'genotyping.modules:depth' '-P' 'depth.readCountCacheIgnoreGenomeMask:true'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-partitionName' 'P0001'  '-partition' 'records:1-922'  
ERROR 16:06:03,450 FunctionEdge - Contents of /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs/CNVDiscoveryStage2-1.out:
INFO  16:05:46,373 HelpFormatter - ----------------------------------------------------------------------------------------- 
INFO  16:05:46,375 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7.GS-r1748-0-g74bfe0b, Compiled 2018/04/10 10:30:23 
INFO  16:05:46,376 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute 
INFO  16:05:46,376 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk 
INFO  16:05:46,376 HelpFormatter - [Sat Mar 02 16:05:46 EST 2019] Executing on Linux 3.10.0-957.el7.x86_64 amd64 
INFO  16:05:46,376 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_191-b12 
INFO  16:05:46,380 HelpFormatter - Program Args: -T SVGenotyperWalker -R /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta -O /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/P0001.genotypes.vcf.gz -disableGATKTraversal true -md /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir -configFile /proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt -configFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gsparams.txt -P genotyping.modules:depth -P depth.readCountCacheIgnoreGenomeMask:true -runDirectory /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16 -genderMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map -ploidyMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta -vcf /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz -partitionName P0001 -partition records:1-922 -L chr1:1-1 
INFO  16:05:46,385 HelpFormatter - Executing as [email protected] on Linux 3.10.0-957.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_191-b12. 
INFO  16:05:46,385 HelpFormatter - Date/Time: 2019/03/02 16:05:46 
INFO  16:05:46,386 HelpFormatter - ----------------------------------------------------------------------------------------- 
INFO  16:05:46,386 HelpFormatter - ----------------------------------------------------------------------------------------- 
INFO  16:05:46,396 02-Mar-2019 GenomeAnalysisEngine - Strictness is SILENT
INFO  16:05:47,594 02-Mar-2019 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000
INFO  16:05:48,474 02-Mar-2019 IntervalUtils - Processing 1 bp from intervals
INFO  16:05:48,521 02-Mar-2019 GenomeAnalysisEngine - Preparing for traversal
INFO  16:05:48,522 02-Mar-2019 GenomeAnalysisEngine - Done preparing for traversal
INFO  16:05:48,522 02-Mar-2019 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO  16:05:48,522 02-Mar-2019 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining
INFO  16:05:48,522 02-Mar-2019 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime
INFO  16:05:48,528 02-Mar-2019 SVGenotyper - Opening reference sequence ...
INFO  16:05:48,528 02-Mar-2019 SVGenotyper - Opened reference sequence.
INFO  16:05:48,529 02-Mar-2019 SVGenotyper - Opening genome mask ...
INFO  16:05:48,530 02-Mar-2019 SVGenotyper - Opened genome mask.
INFO  16:05:48,532 02-Mar-2019 MetaData - Opening metadata ... 
INFO  16:05:48,532 02-Mar-2019 MetaData - Adding metadata location /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir ...
INFO  16:05:48,535 02-Mar-2019 MetaData - Opened metadata.
INFO  16:05:48,569 02-Mar-2019 SVGenotyper - Initializing input data set ...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
    at java.lang.StringBuilder.append(StringBuilder.java:136)
    at htsjdk.samtools.SAMTextHeaderCodec.advanceLine(SAMTextHeaderCodec.java:139)
    at htsjdk.samtools.SAMTextHeaderCodec.decode(SAMTextHeaderCodec.java:94)
    at htsjdk.samtools.BAMFileReader.readHeader(BAMFileReader.java:655)
    at htsjdk.samtools.BAMFileReader.<init>(BAMFileReader.java:298)
    at htsjdk.samtools.BAMFileReader.<init>(BAMFileReader.java:176)
    at htsjdk.samtools.SamReaderFactory$SamReaderFactoryImpl.open(SamReaderFactory.java:376)
    at htsjdk.samtools.SamReaderFactory$SamReaderFactoryImpl.open(SamReaderFactory.java:202)
    at org.broadinstitute.sv.dataset.SAMFileLocation.createSamFileReader(SAMFileLocation.java:97)
    at org.broadinstitute.sv.dataset.SAMLocation.createSamFileReader(SAMLocation.java:41)
    at org.broadinstitute.sv.dataset.DataSet.initInputFile(DataSet.java:138)
    at org.broadinstitute.sv.dataset.DataSet.initialize(DataSet.java:128)
    at org.broadinstitute.sv.genotyping.SVGenotyperWalker.initDataSet(SVGenotyperWalker.java:355)
    at org.broadinstitute.sv.genotyping.SVGenotyperWalker.initialize(SVGenotyperWalker.java:197)
    at org.broadinstitute.gatk.engine.executive.LinearMicroScheduler.execute(LinearMicroScheduler.java:83)
    at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:316)
    at org.broadinstitute.gatk.engine.CommandLineExecutable.execute(CommandLineExecutable.java:123)
    at org.broadinstitute.sv.main.SVCommandLine.execute(SVCommandLine.java:141)
    at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:256)
    at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:158)
    at org.broadinstitute.sv.main.SVCommandLine.main(SVCommandLine.java:91)
    at org.broadinstitute.sv.main.SVGenotyper.main(SVGenotyper.java:21) 
INFO  16:06:03,451 QGraph - Writing incremental jobs reports... 
INFO  16:06:03,452 QGraph - 8 Pend, 0 Run, 1 Fail, 0 Done 
INFO  16:06:03,453 QCommandLine - Writing final jobs report... 
INFO  16:06:03,454 QCommandLine - Done with errors 

You can find that even though I added the flag "-memLimit 100", at top of the second error, it still shows '-Xmx3072m'.

Besides, there are also two failure messages here:

The first also have shows '-Xmx3072m':

INFO  16:06:03,458 QGraph - Failed:   'java'  '-Xmx3072m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar' org.broadinstitute.sv.main.SVGenotyper '-T' 'SVGenotyperWalker'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-O' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/P0001.genotypes.vcf.gz'  '-disableGATKTraversal' 'true'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt' '-configFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gsparams.txt'  '-P' 'genotyping.modules:depth' '-P' 'depth.readCountCacheIgnoreGenomeMask:true'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-partitionName' 'P0001'  '-partition' 'records:1-922'  
INFO  16:06:03,458 QGraph - Log:     /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs/CNVDiscoveryStage2-1.out 
INFO  16:06:03,459 QCommandLine - Script failed: 8 Pend, 0 Run, 1 Fail, 0 Done 
------------------------------------------------------------------------------------------
Done. There were no warn messages.

But the second shows '-Xmx102400m':

INFO  16:06:22,771 QGraph - Failed:   'java'  '-Xmx102400m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/SVQScript.q'  '-gatk' '/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs'  '-memLimit' '100.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '--mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8'  '-jobQueue' 'general'  -run  '-sequenceName' 'chr16'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-sentinelFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_sentinel_files/stage_2_seq_chr16.sent'  --disableJobReport  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-copyNumberMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gcmask.fasta'  '-readDepthMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.rdmask.bed'  '-genderMaskBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gendermask.bed'  '-vdjBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.vdjregions.bed'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  -disableGATKTraversal  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-genderGenotypeFilterFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter.txt'  '-filterDescriptionFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter_descr.txt'  '-genotypingParallelRecords' '1000'  
INFO  16:06:22,771 QGraph - Log:     /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/logs/CNVDiscoveryPipeline-5.out 
INFO  16:06:22,772 QCommandLine - Script failed: 10 Pend, 0 Run, 1 Fail, 4 Done

Does it mean that I need more deeply edit somewhere in the Qscript to allow more memory? Because I indeed find that the first subjob (successful) trigged by the SVCNVDiscovery takes about 80 GB memory, but the other subjobs (failed) only takes about 2.9 GB memory. And especially, some errors and failure coming from QGraph steps which I cannot find in the QScript. May I have your suggestions about these errors? Thank you very much.

Best regards,
Wusheng

Sign In or Register to comment.