Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.
Attention:
We will be out of the office for a Broad Institute event from Dec 10th to Dec 11th 2019. We will be back to monitor the GATK forum on Dec 12th 2019. In the meantime we encourage you to help out other community members with their queries.
Thank you for your patience!

VariantRecalibrator error : No data found

Hi,

I did haplotypecaller -> genomicsDBimport -> genotypeGVFS ->

We only imported coding regions in genomicsDBimport.

and now I've an error at variantRecalibrator stage. Looking at the log file I've 1,266,986 variants processed. Any idea how to solve this ?

Thank you,

Nicolas


version : GATK 4.1.1.0

Here's my command :

$gatk VariantFiltration \
-V $vcf_in \
--filter-expression "ExcessHet > 54.69" --filter-name "ExcessHet" \
-O $vcf/wgs_hg19_excesshet.vcf.gz

$gatk MakeSitesOnlyVcf \
-I $vcf/wgs_hg19_excesshet.vcf.gz \
-O $vcf/wgs_hg19_excesshet.sites_only.vcf.gz 

$gatk VariantRecalibrator \
-R $ref_hg19 \
-V $vcf/wgs_hg19_excesshet.sites_only.vcf.gz \
--trust-all-polymorphic \
-tranche 100.0 -tranche 99.99 -tranche 99.95 -tranche 99.9 -tranche 99.8 -tranche 99.7 -tranche 99.6 -tranche 99.5 -tranche 99.0 -tranche 90.0 \
-an FS -an ReadPosRankSum -an MQRankSum -an QD -an SOR -an DP \
-mode SNP \
--max-gaussians 4 \
--resource:hapmap,known=false,training=true,truth=true,prior=15.0 $resources/hg19/hapmap_3.3.hg19.vcf.gz \
--resource:omni,known=false,training=true,truth=false,prior=12.0 $resources/hg19/1000G_omni2.5.hg19.vcf.gz \
--resource:1000G,known=false,training=true,truth=false,prior=10.0 $resources/hg19/1000G_phase1.snps.high_confidence.hg19.vcf.gz \
--resource:dbsnp,known=true,training=false,truth=false,prior=2.0 $resources/hg19/dbsnp_138.hg19.vcf.gz \
-O $vcf/wgs_hg19_excesshet.sites_only.snps.recal.vcf.gz \
--tranches-file $vcf/wgs_hg19_excesshet.sites_only.snps.tranches \
--rscript-file $vcf/wgs_hg19_excesshet.sites_only.snps.recalibration_plots.rscript

And the log :

Using GATK jar /gpfsuser/home/users/n/r/user/tools/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gpfsuser/home/users/n/r/user/tools/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar VariantRecalibrator -R /home/uni/gastrexp/user/genomes/hg19/gatk_resources_hg19/ucsc.hg19.fasta -V /home/uni/gastrexp/user/projects/proj/results/genotyped_vcfs/wgs_hg19_excesshet.sites_only.vcf.gz --trust-all-polymorphic -tranche 100.0 -tranche 99.99 -tranche 99.95 -tranche 99.9 -tranche 99.8 -tranche 99.7 -tranche 99.6 -tranche 99.5 -tranche 99.0 -tranche 90.0 -an FS -an ReadPosRankSum -an MQRankSum -an QD -an SOR -an DP -mode SNP --max-gaussians 4 --resource:hapmap,known=false,training=true,truth=true,prior=15.0 /home/uni/gastrexp/user/genomes/hg19/ressources/hg19/hapmap_3.3.hg19.vcf.gz --resource:omni,known=false,training=true,truth=false,prior=12.0 /home/uni/gastrexp/user/genomes/hg19/ressources/hg19/1000G_omni2.5.hg19.vcf.gz --resource:1000G,known=false,training=true,truth=false,prior=10.0 /home/uni/gastrexp/user/genomes/hg19/ressources/hg19/1000G_phase1.snps.high_confidence.hg19.vcf.gz --resource:dbsnp,known=true,training=false,truth=false,prior=2.0 /home/uni/gastrexp/user/genomes/hg19/ressources/hg19/dbsnp_138.hg19.vcf.gz -O /home/uni/gastrexp/user/projects/proj/results/genotyped_vcfs/wgs_hg19_excesshet.sites_only.snps.recal.vcf.gz --tranches-file /home/uni/gastrexp/user/projects/proj/results/genotyped_vcfs/wgs_hg19_excesshet.sites_only.snps.tranches --rscript-file /home/uni/gastrexp/user/projects/proj/results/genotyped_vcfs/wgs_hg19_excesshet.sites_only.snps.recalibration_plots.rscript
11:04:37.235 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gpfsuser/home/users/n/r/user/tools/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so 
Apr 25, 2019 11:04:39 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine INFO: Failed to detect whether we are running on Google Compute Engine. 
11:04:39.550 INFO VariantRecalibrator - ------------------------------------------------------------ 
11:04:39.551 INFO VariantRecalibrator - The Genome Analysis Toolkit (GATK) v4.1.1.0 
11:04:39.551 INFO VariantRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/ 
11:04:39.552 INFO VariantRecalibrator - Executing as [email protected] on Linux v2.6.32-696.30.1.el6.x86_64 amd64 
11:04:39.552 INFO VariantRecalibrator - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_201-b09 
11:04:39.552 INFO VariantRecalibrator - Start Date/Time: April 25, 2019 11:04:37 AM CEST 
11:04:39.553 INFO VariantRecalibrator - ------------------------------------------------------------ 
11:04:39.553 INFO VariantRecalibrator - ------------------------------------------------------------ 
11:04:39.554 INFO VariantRecalibrator - HTSJDK Version: 2.19.0 
11:04:39.554 INFO VariantRecalibrator - Picard Version: 2.19.0 
11:04:39.554 INFO VariantRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2 
11:04:39.554 INFO VariantRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 
11:04:39.554 INFO VariantRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 
11:04:39.554 INFO VariantRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 
11:04:39.554 INFO VariantRecalibrator - Deflater: IntelDeflater 
11:04:39.555 INFO VariantRecalibrator - Inflater: IntelInflater 
11:04:39.555 INFO VariantRecalibrator - GCS max retries/reopens: 20 
11:04:39.555 INFO VariantRecalibrator - Requester pays: disabled 
11:04:39.555 INFO VariantRecalibrator - Initializing engine
11:04:41.574 INFO FeatureManager - Using codec VCFCodec to read file file:///home/uni/gastrexp/user/genomes/hg19/ressources/hg19/hapmap_3.3.hg19.vcf.gz
11:04:43.238 INFO FeatureManager - Using codec VCFCodec to read file file:///home/uni/gastrexp/user/genomes/hg19/ressources/hg19/1000G_omni2.5.hg19.vcf.gz
11:04:43.470 INFO FeatureManager - Using codec VCFCodec to read file file:///home/uni/gastrexp/user/genomes/hg19/ressources/hg19/1000G_phase1.snps.high_confidence.hg19.vcf.gz
11:04:43.659 INFO FeatureManager - Using codec VCFCodec to read file file:///home/uni/gastrexp/user/genomes/hg19/ressources/hg19/dbsnp_138.hg19.vcf.gz
11:04:43.859 INFO FeatureManager - Using codec VCFCodec to read file file:///home/uni/gastrexp/user/projects/proj/results/genotyped_vcfs/wgs_hg19_excesshet.sites_only.vcf.gz
11:04:44.740 INFO VariantRecalibrator - Done initializing engine 
11:04:44.750 INFO TrainingSet - Found hapmap track: Known = false Training = true Truth = true Prior = Q15.0 
11:04:44.750 INFO TrainingSet - Found omni track: Known = false Training = true Truth = false Prior = Q12.0 
11:04:44.750 INFO TrainingSet - Found 1000G track: Known = false Training = true Truth = false Prior = Q10.0 
11:04:44.751 INFO TrainingSet - Found dbsnp track: Known = true Training = false Truth = false Prior = Q2.0 
11:04:44.797 INFO ProgressMeter - Starting traversal
11:04:44.797 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute
11:04:55.089 INFO ProgressMeter - chr1:24664239 0.2 23000 134097.8 
11:05:05.474 INFO ProgressMeter - chr1:54153724 0.3 39000 113169.2 
11:05:16.441 INFO ProgressMeter - chr1:92185522 0.5 50000 94807.7 
11:05:26.739 INFO ProgressMeter - chr1:150996502 0.7 72000 102999.4 
11:05:36.911 INFO ProgressMeter - chr1:179443937 0.9 89000 102469.6 
11:05:46.985 INFO ProgressMeter - chr1:212529959 1.0 102000 98412.9 
11:05:57.845 INFO ProgressMeter - chr1:246845971 1.2 115000 94458.4 
11:06:08.126 INFO ProgressMeter - chr2:27790203 1.4 129000 92885.9
11:06:18.688 INFO ProgressMeter - chr2:61155650 1.6 140000 89465.4 
11:06:29.599 INFO ProgressMeter - chr2:106468363 1.7 156000 89312.1 
11:06:39.774 INFO ProgressMeter - chr2:146903045 1.9 169000 88192.3
11:06:50.262 INFO ProgressMeter - chr2:185460091 2.1 180000 86079.8 
11:07:00.416 INFO ProgressMeter - chr2:224364486 2.3 192000 84943.9 
11:07:10.631 INFO ProgressMeter - chr3:10028514 2.4 205000 84342.5
11:07:20.640 INFO ProgressMeter - chr3:44771178 2.6 216000 83161.2 
11:07:32.107 INFO ProgressMeter - chr3:88515901 2.8 231000 82840.7 
11:07:42.475 INFO ProgressMeter - chr3:130722014 3.0 244000 82396.7 
11:07:52.676 INFO ProgressMeter - chr3:171179385 3.1 255000 81435.8 
11:08:03.370 INFO ProgressMeter - chr4:3635113 3.3 272000 82186.4 
11:08:13.977 INFO ProgressMeter - chr4:41633172 3.5 283000 81174.1 
11:08:24.742 INFO ProgressMeter - chr4:82524201 3.7 294000 80201.9 
11:08:35.023 INFO ProgressMeter - chr4:122068291 3.8 304000 79226.5 
11:08:45.835 INFO ProgressMeter - chr4:165674942 4.0 314000 78161.9 
11:08:56.366 INFO ProgressMeter - chr5:10482055 4.2 328000 78229.0 
11:09:06.838 INFO ProgressMeter - chr5:55224038 4.4 338000 77392.5 
11:09:17.219 INFO ProgressMeter - chr5:96364191 4.5 349000 76866.0 
11:09:27.363 INFO ProgressMeter - chr5:135388663 4.7 360000 76442.3 
11:09:37.517 INFO ProgressMeter - chr5:170209952 4.9 374000 76660.6 
11:09:47.866 INFO ProgressMeter - chr6:20212201 5.1 389000 77012.2 
11:09:57.948 INFO ProgressMeter - chr6:45544276 5.2 414000 79323.0 
11:10:08.907 INFO ProgressMeter - chr6:93952358 5.4 424000 78491.9 
11:10:19.541 INFO ProgressMeter - chr6:136664061 5.6 435000 77970.3 
11:10:29.890 INFO ProgressMeter - chr6:169613817 5.8 447000 77718.2 
11:10:40.234 INFO ProgressMeter - chr7:25397996 5.9 460000 77650.9 
11:10:50.322 INFO ProgressMeter - chr7:65112836 6.1 474000 77805.9 
11:11:00.403 INFO ProgressMeter - chr7:104758873 6.3 490000 78273.5 
11:11:10.692 INFO ProgressMeter - chr7:142032750 6.4 502000 78052.5 
11:11:20.816 INFO ProgressMeter - chr8:9945513 6.6 518000 78481.1 
11:11:31.619 INFO ProgressMeter - chr8:41566438 6.8 533000 78609.5 
11:11:42.524 INFO ProgressMeter - chr8:86572189 7.0 544000 78166.0 
11:11:52.795 INFO ProgressMeter - chr8:125823278 7.1 554000 77663.9 
11:12:03.575 INFO ProgressMeter - chr9:14587738 7.3 568000 77670.3 
11:12:14.593 INFO ProgressMeter - chr9:85670067 7.5 589000 78569.0 
11:12:25.140 INFO ProgressMeter - chr9:119050427 7.7 601000 78333.1 
11:12:35.612 INFO ProgressMeter - chr10:2114414 7.8 618000 78757.0 
11:12:46.586 INFO ProgressMeter - chr10:34600492 8.0 630000 78457.6
11:12:57.063 INFO ProgressMeter - chr10:75289842 8.2 642000 78250.4 
11:13:07.871 INFO ProgressMeter - chr10:114188086 8.4 657000 78358.3 
11:13:17.960 INFO ProgressMeter - chr11:6259803 8.6 677000 79156.1
11:13:28.290 INFO ProgressMeter - chr11:36598575 8.7 689000 78969.5 
11:13:38.290 INFO ProgressMeter - chr11:71544105 8.9 708000 79626.2
11:13:48.372 INFO ProgressMeter - chr11:107878498 9.1 720000 79474.0 
11:13:58.898 INFO ProgressMeter - chr12:3150874 9.2 736000 79696.7
11:14:09.566 INFO ProgressMeter - chr12:39688824 9.4 753000 79997.5 
11:14:20.200 INFO ProgressMeter - chr12:72094627 9.6 772000 80500.1
11:14:30.592 INFO ProgressMeter - chr12:109919480 9.8 785000 80403.6 
11:14:41.195 INFO ProgressMeter - chr13:23377686 9.9 803000 80785.1
11:14:52.705 INFO ProgressMeter - chr13:67203980 10.1 816000 80538.5 
11:15:02.914 INFO ProgressMeter - chr13:113743996 10.3 826000 80179.0
11:15:13.085 INFO ProgressMeter - chr14:54947472 10.5 845000 80695.6 
11:15:23.674 INFO ProgressMeter - chr14:89629001 10.6 859000 80672.8
11:15:33.849 INFO ProgressMeter - chr15:32879270 10.8 885000 81811.6 
11:15:44.097 INFO ProgressMeter - chr15:63119797 11.0 901000 81996.1
11:15:54.495 INFO ProgressMeter - chr15:94380279 11.2 920000 82425.3 
11:16:04.764 INFO ProgressMeter - chr16:15068837 11.3 941000 83033.4
11:16:15.568 INFO ProgressMeter - chr16:63347304 11.5 962000 83558.8 
11:16:25.821 INFO ProgressMeter - chr16:86598317 11.7 979000 83791.8
11:16:35.940 INFO ProgressMeter - chr17:20798657 11.9 1003000 84624.3 
11:16:46.525 INFO ProgressMeter - chr17:55849159 12.0 1028000 85461.7
11:16:57.319 INFO ProgressMeter - chr18:3263637 12.2 1051000 86086.2 
11:17:07.900 INFO ProgressMeter - chr18:44063598 12.4 1064000 85910.0
11:17:18.170 INFO ProgressMeter - chr19:624919 12.6 1078000 85853.9 
11:17:28.393 INFO ProgressMeter - chr19:22114812 12.7 1108000 87061.9
11:17:38.591 INFO ProgressMeter - chr19:50967686 12.9 1135000 88008.0 
11:17:48.802 INFO ProgressMeter - chr20:16651050 13.1 1160000 88774.9
11:17:58.928 INFO ProgressMeter - chr20:48739805 13.2 1176000 88852.0 
11:18:09.393 INFO ProgressMeter - chr21:31743605 13.4 1192000 88889.3
11:18:19.545 INFO ProgressMeter - chr22:23402010 13.6 1216000 89549.3 
11:18:30.239 INFO ProgressMeter - chr22:49948085 13.8 1238000 89988.2
11:18:40.559 INFO ProgressMeter - chrX:55117060 13.9 1251000 89810.4 
11:18:51.767 INFO ProgressMeter - chrX:133675593 14.1 1260000 89259.4
11:18:56.249 INFO ProgressMeter - chrY:9928999 14.2 1266986 89281.8 
11:18:56.249 INFO ProgressMeter - Traversal complete. Processed 1266986 total variants in 14.2 minutes.
11:18:56.413 INFO VariantDataManager - FS: mean = 2.48 standard deviation = 4.27
11:18:56.645 INFO VariantDataManager - ReadPosRankSum: mean = 0.17 standard deviation = 0.51
11:18:56.917 INFO VariantDataManager - MQRankSum: mean = 0.02 standard deviation = 0.44
11:18:57.187 INFO VariantDataManager - QD: mean = 16.59 standard deviation = 5.46
11:18:57.453 INFO VariantDataManager - SOR: mean = 0.75 standard deviation = 0.31
11:18:57.718 INFO VariantDataManager - DP: mean = 3178.53 standard deviation = 356.59
11:18:58.913 INFO VariantDataManager - Annotation order is: [DP, QD, SOR, MQRankSum, FS, ReadPosRankSum]
11:18:59.001 INFO VariantDataManager - Training with 655848 variants after standard deviation thresholding.
11:18:59.072 INFO GaussianMixtureModel - Initializing model with 100 k-means iterations...
11:19:48.347 INFO VariantRecalibratorEngine - Finished iteration 0. 
11:20:04.490 INFO VariantRecalibratorEngine - Finished iteration 5. Current change in mixture coefficients = 1.08336 
11:20:20.020 INFO VariantRecalibratorEngine - Finished iteration 10. Current change in mixture coefficients = 0.02013 
11:20:35.574 INFO VariantRecalibratorEngine - Finished iteration 15. Current change in mixture coefficients = 0.01176 
11:20:50.984 INFO VariantRecalibratorEngine - Finished iteration 20. Current change in mixture coefficients = 0.00188 
11:20:50.985 INFO VariantRecalibratorEngine - Convergence after 20 iterations! 
11:20:52.279 INFO VariantRecalibratorEngine - Evaluating full set of 1069287 variants... 
11:20:52.279 WARN VariantRecalibratorEngine - Evaluate datum returned a NaN. 
11:20:52.348 INFO VariantDataManager - Selected worst 0 scoring variants --> variants with LOD <= -5.0000. 
11:20:52.463 INFO VariantRecalibrator - Shutting down engine
[April 25, 2019 11:20:52 AM CEST] org.broadinstitute.hellbender.tools.walkers.vqsr.VariantRecalibrator done. Elapsed time: 16.26 minutes. 
Runtime.totalMemory()=9637462016
java.lang.IllegalArgumentException: No data found.
    at org.broadinstitute.hellbender.tools.walkers.vqsr.VariantRecalibratorEngine.generateModel(VariantRecalibratorEngine.java:34)
    at org.broadinstitute.hellbender.tools.walkers.vqsr.VariantRecalibrator.onTraversalSuccess(VariantRecalibrator.java:656)
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:986)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
    at org.broadinstitute.hellbender.Main.main(Main.java:291) 

Best Answers

  • NicoBxlNicoBxl
    Accepted Answer

    Hi @bhanuGandham

    adding -an MQ worked !

    Thank you very much

Answers

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    Hi @NicoBxl

    I will ask the dev team to look into it and get back to you shortly.

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    Hi @NicoBxl

    How many whole genome samples are you using? If you are using one sample then we recommend you use the GATK CNN tools that were specifically designed for single sample variant filtering. VQSR is meant for a bigger cohort of samples.

  • NicoBxlNicoBxl Member

    I've 80 WGS samples. I'm wondering if this step :

    $gatk MakeSitesOnlyVcf \
    -I $vcf/wgs_hg19_excesshet.vcf.gz \
    -O $vcf/wgs_hg19_excesshet.sites_only.vcf.gz
    

    is required before VariantRecalibrator (whole command in main question) ?

    Thank you

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Making sites only vcf will remove all samples and genotypes from your file thus you won't be able to filter anything. It is better to filter first then make sites only file later according to your needs.

  • NicoBxlNicoBxl Member

    Ok so if I summarize

    VariantFiltration -> VariantRecalibrator -> MakeSitesOnlyVcf

    Correct ?

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
  • NicoBxlNicoBxl Member

    @SkyWarrior said:
    Sure it is.

    But why in GATK4 best practice did they do MakeSitesOnlyVcf before VariantRecalibrator then ?

  • etrepoetrepo Member
    @SkyWarrior, I don't understand why using MakeSitesOnlyVcf before VariantRecalibrator would not be a good idea. Removing all samples and genotypes is not a problem since here the purpose is to filter variants on the whole cohort, correct?
  • NicoBxlNicoBxl Member

    @SkyWarrior said:
    Making sites only vcf will remove all samples and genotypes from your file thus you won't be able to filter anything. It is better to filter first then make sites only file later according to your needs.

    @NicoBxl said:
    Ok so if I summarize

    VariantFiltration -> VariantRecalibrator -> MakeSitesOnlyVcf

    Correct ?

    @SkyWarrior said:
    Sure it is.

    I tested by doing VariantFiltration -> VariantRecalibrator -> MakeSitesOnlyVcf and I have the same issue.

    Any other idea to solve this ?

    I will try to redo the genotypegvcfs part ; maybe something didn't go well there ...

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Sorry for my previous comment. I missed the idea totally. Is it possible that the calibration is halting due to max gaussian limitation. Can you change that parameter to 4 and try again? This kind of problems were seen before when data set contains mixed data (clean and noisy together.)

  • NicoBxlNicoBxl Member

    @SkyWarrior said:
    Sorry for my previous comment. I missed the idea totally. Is it possible that the calibration is halting due to max gaussian limitation. Can you change that parameter to 4 and try again? This kind of problems were seen before when data set contains mixed data (clean and noisy together.)

    I tried with max gaussian 4 and also with 3 : Same error :(

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    Hi @NicoBxl

    I will check with the dev team and get back to you shortly.

  • NicoBxlNicoBxl Member

    @bhanuGandham Do you have some news regarding this issue ?

    Thank you

  • NicoBxlNicoBxl Member
    Accepted Answer

    Hi @bhanuGandham

    adding -an MQ worked !

    Thank you very much

Sign In or Register to comment.