Hi GATK Users,

Happy Thanksgiving!
Our staff will be observing the holiday and will be unavailable from 22nd to 25th November. This will cause a delay in reaching out to you and answering your questions immediately. Rest assured we will get back to it on Monday November 26th. We are grateful for your support and patience.
Have a great holiday everyone!!!

Regards
GATK Staff

Variant Recalibration Timing Out and Looping?

DLD842DLD842 USAMember
edited October 2015 in Ask the GATK team

Trying to decide if this is a job submission/network problem, or a job script problem!

Running VariantRecalibrator in SNP mode on 95 joint-called genomes. The ProgressMeter runs nicely until it hits 100% and then appears to keep looping through at the same position without writing any outputs. Here are the last few of lines of the .out file ...


INFO 15:02:16,359 ProgressMeter - chrX:118789197 3.944378E7 3.3 h 5.0 m 99.8% 3.3 h 25.0 s
INFO 15:03:32,883 ProgressMeter - chrX:122953970 3.9478259E7 3.3 h 5.0 m 100.0% 3.3 h 4.0 s
INFO 15:03:33,513 VariantDataManager - DP: mean = 2891.94 standard deviation = 1541.35
INFO 15:03:34,380 VariantDataManager - QD: mean = 21.50 standard deviation = 5.52
INFO 15:03:35,231 VariantDataManager - MQRankSum: mean = -0.03 standard deviation = 0.52
INFO 15:03:36,173 VariantDataManager - ReadPosRankSum: mean = 0.30 standard deviation = 0.39
INFO 15:04:56,292 ProgressMeter - chrX:123869054 3.9488752E7 3.3 h 5.0 m 100.0% 3.3 h 0.0 s
INFO 15:06:14,605 ProgressMeter - chrX:123869054 3.9488752E7 3.3 h 5.1 m 100.0% 3.3 h 0.0 s
INFO 15:07:46,623 ProgressMeter - chrX:123869054 3.9488752E7 3.4 h 5.1 m 100.0% 3.4 h 0.0 s
INFO 15:09:08,227 ProgressMeter - chrX:123869054 3.9488752E7 3.4 h 5.2 m 100.0% 3.4 h 0.0 s
INFO 15:10:22,074 ProgressMeter - chrX:123869054 3.9488752E7 3.4 h 5.2 m 100.0% 3.4 h 0.0 s
INFO 15:11:34,295 ProgressMeter - chrX:123869054 3.9488752E7 3.4 h 5.2 m 100.0% 3.4 h 0.0 s
INFO 15:12:48,256 ProgressMeter - chrX:123869054 3.9488752E7 3.5 h 5.2 m 100.0% 3.5 h 0.0 s
INFO 15:14:01,197 ProgressMeter - chrX:123869054 3.9488752E7 3.5 h 5.3 m 100.0% 3.5 h 0.0 s
INFO 15:15:13,241 ProgressMeter - chrX:123869054 3.9488752E7 3.5 h 5.3 m 100.0% 3.5 h 0.0 s
INFO 15:15:13,611 VariantDataManager - Annotations are now ordered by their information content: [DP, QD, ReadPosRankSum, MQRankSum]
INFO 15:15:13,874 VariantDataManager - Training with 2384941 variants after standard deviation thresholding.
INFO 15:15:13,878 GaussianMixtureModel - Initializing model with 100 k-means iterations...
INFO 15:16:19,785 ProgressMeter - chrX:123869054 3.9488752E7 3.5 h 5.3 m 100.0% 3.5 h 0.0 s
INFO 15:18:46,658 ProgressMeter - chrX:123869054 3.9488752E7 3.6 h 5.4 m 100.0% 3.6 h 0.0 s
INFO 15:19:58,618 ProgressMeter - chrX:123869054 3.9488752E7 3.6 h 5.4 m 100.0% 3.6 h 0.0 s
INFO 15:21:13,231 ProgressMeter - chrX:123869054 3.9488752E7 3.6 h 5.5 m 100.0% 3.6 h 0.0 s
INFO 15:22:24,708 ProgressMeter - chrX:123869054 3.9488752E7 3.6 h 5.5 m 100.0% 3.6 h 0.0 s
INFO 15:23:40,391 ProgressMeter - chrX:123869054 3.9488752E7 3.6 h 5.5 m 100.0% 3.6 h 0.0 s
INFO 15:24:55,246 ProgressMeter - chrX:123869054 3.9488752E7 3.7 h 5.6 m 100.0% 3.7 h 0.0 s
INFO 15:26:08,867 ProgressMeter - chrX:123869054 3.9488752E7 3.7 h 5.6 m 100.0% 3.7 h 0.0 s
INFO 15:27:21,513 ProgressMeter - chrX:123869054 3.9488752E7 3.7 h 5.6 m 100.0% 3.7 h 0.0 s
INFO 15:28:42,824 ProgressMeter - chrX:123869054 3.9488752E7 3.7 h 5.6 m 100.0% 3.7 h 0.0 s
INFO 15:29:53,956 ProgressMeter - chrX:123869054 3.9488752E7 3.7 h 5.7 m 100.0% 3.7 h 0.0 s
INFO 15:32:11,877 ProgressMeter - chrX:123869054 3.9488752E7 3.8 h 5.7 m 100.0% 3.8 h 0.0 s
INFO 15:33:12,010 ProgressMeter - chrX:123869054 3.9488752E7 3.8 h 5.8 m 100.0% 3.8 h 0.0 s
INFO 15:34:19,364 ProgressMeter - chrX:123869054 3.9488752E7 3.8 h 5.8 m 100.0% 3.8 h 0.0 s
INFO 15:35:18,484 ProgressMeter - chrX:123869054 3.9488752E7 3.8 h 5.8 m 100.0% 3.8 h 0.0 s
INFO 15:36:23,992 ProgressMeter - chrX:123869054 3.9488752E7 3.8 h 5.8 m 100.0% 3.8 h 0.0 s
INFO 15:36:24,003 VariantRecalibratorEngine - Finished iteration 0.
INFO 15:37:30,932 ProgressMeter - chrX:123869054 3.9488752E7 3.9 h 5.9 m 100.0% 3.9 h 0.0 s
INFO 15:38:29,846 ProgressMeter - chrX:123869054 3.9488752E7 3.9 h 5.9 m 100.0% 3.9 h 0.0 s
INFO 15:39:30,868 ProgressMeter - chrX:123869054 3.9488752E7 3.9 h 5.9 m 100.0% 3.9 h 0.0 s
INFO 15:40:29,351 ProgressMeter - chrX:123869054 3.9488752E7 3.9 h 5.9 m 100.0% 3.9 h 0.0 s
INFO 15:41:29,067 ProgressMeter - chrX:123869054 3.9488752E7 3.9 h 6.0 m 100.0% 3.9 h 0.0 s
INFO 15:42:34,585 ProgressMeter - chrX:123869054 3.9488752E7 3.9 h 6.0 m 100.0% 3.9 h 0.0 s
INFO 15:43:33,707 ProgressMeter - chrX:123869054 3.9488752E7 4.0 h 6.0 m 100.0% 4.0 h 0.0 s
INFO 15:44:35,069 ProgressMeter - chrX:123869054 3.9488752E7 4.0 h 6.1 m 100.0% 4.0 h 0.0 s
INFO 15:45:34,865 ProgressMeter - chrX:123869054 3.9488752E7 4.0 h 6.1 m 100.0% 4.0 h 0.0 s

slurmstepd: JOB 3652350 CANCELLED AT 2015-10-07T15:45:56 DUE TO TIME LIMIT on cn0066

The job eventually gets cancelled due to wall time limits (4 hours in this case). So, the question is, is this looping at 100% completion "correct", therefore my problem is related to just giving it more computational time? Or is there some error happening and it shouldn't be cycling through like this to begin with?

Suggestions? Comments?

Thank you!!

Answers

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @DLD842
    Hi,

    Can you tell me what version of GATK you are using and the exact command you ran to get the error? Also, what kind of samples are you working with?

    -Sheila

  • DLD842DLD842 USAMember
    edited October 2015

    @Sheila

    I'm running this with a shell script of:

    --

    !/bin/sh

    module load GATK

    java -Xmx4g -jar /usr/local/apps/GATK/3.4-0/GenomeAnalysisTK.jar -T VariantRecalibrator -R /PATH.TO/cf31PMc.fa \
    -input /PATH.TO/jointcall_breeds.chrall.output.vcf \
    -resource:hdchip,known=false,training=true,truth=true,prior=15.0 /PATH.TO/CanineHD_num_order.vcf \
    -resource:dbsnp,known=true,training=true,truth=false,prior=6.0 /PATH.TO/CFA31_72.dbSNP_num_order.vcf \
    -resource:axxelsson,known=true,training=false,truth=false,prior=6.0 /PATH.TO/Axelsson.2013.SNPs.num_order.vcf \
    -an DP -an QD -an MQRankSum -an ReadPosRankSum -mode SNP -tranche 100.0 -tranche 99.9 -tranche 99.0 -tranche 90.0 \
    -recalFile /PATH.TO/allbreed_allchrom_recal_snp.output.recal \
    -tranchesFile /PATH.TO/allbreed_allchrom_recal_snp.output.tranches \
    -rscriptFile /PATH.TO/allbreed_allchrom_recal_snp.output.plots.R

    --

    Job submission command line:
    sbatch --mem=12g --time=36:00:00 /PATH.TO/VarRecalibration_snp_100715.sh

    Edit To Add:
    These are canine whole genome sequences.

  • DLD842DLD842 USAMember

    It appears as if I may have figured out my own issue. I gave it a LOT more computational power, and so far so good! Yes, it is cycling through the iterations at 100%, but doesn't appear to be at risk of timing out this time. Fingers crossed!

Sign In or Register to comment.