Attention:
The frontline support team will be unavailable to answer questions on April 15th and 17th 2019. We will be back soon after. Thank you for your patience and we apologize for any inconvenience!

Memory error - Downsampling unavailable (3.5

AlexHAlexH Toulouse, INRAMember
edited April 2016 in Ask the GATK team

Hi there,

I am working on mitochondrial genomes (250 samples) with a coverage of ~7000x.
The HC (v 3.5) is running perfectly for the firsts 248 samples but reaching the last ones, the following Java error appears :

##### ERROR MESSAGE: An error occurred because you did not provide enough memory to run this program. You can use the -Xmx argument (before the -jar argument) to adjust the maximum heap size provided to Java. Note that this is a JVM argument, not a GATK argument.

All the other samples work perfectly with ~64G of mem and v_mem so i tried 128G but it is still not sufficient for my last samples to be processed. As you probably understand, i can't extend the memory indefinitely and I would want to know if it exists a way to force the down-sampling or at least to greatly ameliorate the memory used by the HaplotypeCaller.

Thanks for your time,

Regards,

Alex H

Best Answer

Answers

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @AlexH
    Hi Alex,

    Can you please confirm that you were able to run HaplotypeCaller with no issues when you set -maxAltAlleles to a smaller number?

    Thanks,
    Sheila

  • AlexHAlexH Toulouse, INRAMember

    Hi Sheila,

    Sorry for the latency. I did not use -maxAltAlleles actually, I simply lowered the -ploidy option !

    Regards,

    Alex

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @AlexH
    Hi Alex,

    I'm happy you solved the issue :smile:

    -Sheila

  • meharmehar Member ✭✭

    Dear Shiela,

    I am having the similar error with the below command used on 1 sample:

    java -Xmx48G -jar /GATK-3.5/GenomeAnalysisTK.jar -R Fam3.fa -T HaplotypeCaller -I sorted.bam -stand_emit_conf 10 -stand_call_conf 30 -ERC GVCF --min_base_quality_score 20 --variant_index_parameter 128000 --variant_index_type LINEAR -nct 8 --genotyping_mode DISCOVERY -o GATK.g.vcf

    It gives the below error after processing for few hours:

    `##### ERROR A USER ERROR has occurred (version 3.5-0-g36282e4):

    ERROR
    ERROR This means that one or more arguments or inputs in your command are incorrect.

    INFO 05:16:38,804 ProgressMeter - chr12:26081618 9.14859792E8 9.7 h 38.0 s 40.4% 24.0 h 14.3 h

    ERROR The error message below tells you what is the problem.
    ERROR
    ERROR If the problem is an invalid argument, please check the online documentation guide

    INFO 05:19:44,926 ProgressMeter - chr12:26082605 9.14859792E8 9.7 h 38.0 s 40.4% 24.1 h 14.3 h

    ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
    ERROR

    INFO 05:21:37,126 ProgressMeter - chr12:26085496 9.14859792E8 9.8 h 38.0 s 40.4% 24.2 h 14.4 h

    ERROR Visit our website and forum for extensive documentation and answers to
    ERROR commonly asked questions http://www.broadinstitute.org/gatk
    ERROR

    INFO 05:23:12,290 ProgressMeter - chr12:26087975 9.14859792E8 9.8 h 38.0 s 40.4% 24.2 h 14.4 h

    ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
    ERROR

    INFO 05:23:41,294 HttpMethodDirector - I/O exception (javax.net.ssl.SSLException) caught when processing request: Received close_notify during handshake

    ERROR MESSAGE: An error occurred because you did not provide enough memory to run this program. You can use the -Xmx argument (before the -jar argument) to adjust the maximum heap size provided to Java. Note that this is a JVM argument, not a GATK argument.

    `

    Increasing java heap size to -Xmx96G also gives the same error. And we observed that with '-nct 1' and -Xmx16G H.C works without any error however it is very slow. So we increased nct to '-nct 8' but then this causes the above error. Could you please help to fix this issue.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @mehar
    Hi,

    Unfortunately, this is a -nct issue. The best thing to try is WDL instead. Or, run without -nct.

    -Sheila

  • meharmehar Member ✭✭

    Thanks. It works without -nct but it is very slow. However, i have the below issue while using GenotypeGVCFs with GATKv3.5 and 3.7.

    I am using GenotypeGVCFs across 38 samples and it crashes at chr12 with the following error:

    ##### ERROR MESSAGE: The provided VCF file is malformed at approximately line number 256201403: chr12 is not a valid start position in the VCF format, for input source: 426.GATK.g.vcf

    However tracing back to line number 256201403, does not have chr12 variants in the vcf file. I have excluded the header lines and it still does not have chr12 variants.

    The last variant in the combined GVCF is chr12 64582754, therefore i traced to the input vcf file in the below way:

    $ awk -F "\t" '$1=="chr12" && $2>=64582754' /proj/lohi/Canine_Projects/WGS/HL_WGS/DDT426A1/DDT426.GATK.g.vcf | head chr12 64582755 . A <NON_REF> . . END=64582757 GT:DP:GQ:MIN_DP:PL 0/0:27:72:26:0,72,1080 chr12 64582758 . G <NON_REF> . . END=64582758 GT:DP:GQ:MIN_DP:PL 0/0:26:46:26:0,46,755 chr12 64582759 . A <NON_REF> . . END=64582763 GT:DP:GQ:MIN_DP:PL 0/0:25:72:24:0,72,748 chr12 64582764 . G <NON_REF> . . END=64582764 GT:DP:GQ:MIN_DP:PL 0/0:25:69:25:0,69,1035

    and the start position look normal. Could you help how to fix this error?

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @mehar Your input file may have got corrupted. Unfortunatley that's not something we can help you with; you may need to regenerate the GVCF file for that sample.

  • meharmehar Member ✭✭

    @Geraldine_VdAuwera I did regenerated the GVCF file and now i have a new issue with GenotypeGVCFs command. Some of the GVCF files are compressed to .gz format and tabix indexed and some of them are in .vcf format. The error is shown below:

    ``##### ERROR ------------------------------------------------------------------------------------------

    ERROR A USER ERROR has occurred (version 3.7-0-gcfedb67):
    ERROR
    ERROR This means that one or more arguments or inputs in your command are incorrect.
    ERROR The error message below tells you what is the problem.
    ERROR
    ERROR If the problem is an invalid argument, please check the online documentation guide
    ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
    ERROR
    ERROR Visit our website and forum for extensive documentation and answers to
    ERROR commonly asked questions https://software.broadinstitute.org/gatk
    ERROR
    ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
    ERROR
    ERROR MESSAGE: LocationAwareSeekableRODIterator: track variant24 is out of coordinate order on contig chr13:36906251 compared to chr13:36930615

    Could you offer any clue to fix this?

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @mehar
    Hi,

    Did you output the files as gzipped files, or did you gzip them after HaplotypeCaller output them? Can you try re-indexing the GVCF files? Have a look at this thread as well.

    -Sheila

  • meharmehar Member ✭✭

    @Sheila i did not output them as gzipped files, they were gzipped from the HaplotypeCaller output. should i re-index gvcf files or gzipped vcf files?

  • meharmehar Member ✭✭

    @mehar said:
    @Sheila i did not output them as gzipped files, they were gzipped from the HaplotypeCaller output. should i re-index gvcf files or gzipped vcf files? does tabix indexed gvcf files work with GenotypeGVCFs?

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @mehar
    Hi,

    I know there were some issues with gzipping the files after HaplotypeCaller. The best thing to do is have HaplotypeCaller output them as gzipped files. You can try unzipping them, and see if the tools work.

    -Sheila

Sign In or Register to comment.