Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.
Attention:
We will be out of the office for a Broad Institute event from Dec 10th to Dec 11th 2019. We will be back to monitor the GATK forum on Dec 12th 2019. In the meantime we encourage you to help out other community members with their queries.
Thank you for your patience!

FilterMutectCalls change between 4.0.2.1 and 4.0.3.0

igorigor New YorkMember ✭✭

I tried running FilterMutectCalls on a VCF generated with Mutect2 for matched tumor-normal samples. With 4.0.2.1, I get some variants that are labeled PASS as would be expected, but with 4.0.3.0, I get 0. I could not find any documentation that explains such a big change in behavior. Is this a bug? Or did I have variants erroneously labeled as PASS before?

Best Answer

Answers

  • SheilaSheila Broad InstituteMember, Broadie admin
    edited April 2018

    @igor
    Hi,

    I know there were some changes made to the tools involved in the workflow, but I am not sure which may affect your calls. Can you post the variants that were PASS in your original callset that are now not PASS?

    Thanks,
    Sheila

  • igorigor New YorkMember ✭✭

    I checked several samples that had hundreds of variants that were passing with 4.0.2.1. None of the variants are passing with 4.0.3.0, so I don't think it's a problem with the variants themselves.

    Anyway, here are a few that were passing with 4.0.2.1:

    chr19   6544162 .   GGTGTGT G,GGTGTGTGT .   PASS    DP=358;ECNT=1;NLOD=32.18,25.97;N_ART_LOD=0.701,2.74;POP_AF=5.000e-08,5.000e-08;RPA=14,11,15;RU=GT;STR;TLOD=3.60,11.49   GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1/2:82,4,7:0.158,0.198:31,2,4:51,2,3:36,35:356,316,338:60,60:36,33:0.061,0.061,0.075:0.022,5.422e-03,0.973    0/0:116,3,3:0.129,0.154:49,0,2:67,3,1:36,36:356,378,261:60,60:38,42
    chr19   6951607 .   G   GA  .   PASS    DP=396;ECNT=1;NLOD=37.91;N_ART_LOD=0.677;POP_AF=5.000e-08;RPA=9,10;RU=A;STR;TLOD=5.87   GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:174,8:0.098:92,6:82,2:31:357,297:60:30:0.030,0.040,0.044:9.384e-03,3.729e-03,0.987  0/0:152,3:0.110:75,3:77,0:38:364,218:60:13
    chr19   9007221 .   G   T   .   PASS    DP=1675;ECNT=1;NLOD=259.93;N_ART_LOD=-2.945e+00;POP_AF=5.000e-08;TLOD=5.80  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:727,5:0.026:359,5:368,0:37:363,509:60:26:0.00,0.010,6.831e-03:3.986e-03,8.100e-04,0.995 0/0:870,2:0.026:429,0:441,2:12:348,411:60:3
    chr19   9011199 .   T   C   .   PASS    DP=1175;ECNT=2;NLOD=149.03;N_ART_LOD=2.98;POP_AF=5.000e-08;TLOD=6.52    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:537,9:0.053:274,5:263,4:36:360,307:58:60:0.020,0.010,0.016:7.795e-04,0.021,0.978    0/0:548,6:0.043:275,1:273,5:27:360,334:60:23
    chr19   10551980    .   TAA T   .   PASS    DP=369;ECNT=1;NLOD=33.95;N_ART_LOD=1.91;POP_AF=5.000e-08;RPA=15,13;RU=A;STR;TLOD=5.47   GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:100,6:0.186:47,5:53,1:27:336,185:60:27:0.051,0.040,0.057:7.077e-03,8.882e-03,0.984  0/0:146,4:0.168:59,3:87,1:31:309,287:60:27
    chr19   11478211    .   C   T   .   PASS    DP=530;ECNT=1;NLOD=79.24;N_ART_LOD=-2.427e+00;POP_AF=5.000e-08;TLOD=5.45    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:254,4:0.022:143,4:111,0:37:351,194:60:13:0.020,0.00,0.016:8.877e-04,0.083,0.916 0/0:264,0:0.013:147,0:117,0:0:324,0:0:0
    chr19   11558277    .   TG  T   .   PASS    DP=2075;ECNT=1;NLOD=295.42;N_ART_LOD=2.32;POP_AF=5.000e-08;RPA=9,8;RU=G;STR;TLOD=9.15   GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:813,20:0.062:425,11:388,9:37:350,360:60:40:0.020,0.020,0.024:0.010,8.955e-04,0.989  0/0:1075,12:0.049:526,6:549,6:38:346,333:60:42
    chr19   24381843    .   G   GA  .   PASS    DP=303;ECNT=1;NLOD=27.83;N_ART_LOD=4.99;POP_AF=5.000e-08;RPA=9,10;RU=A;STR;TLOD=5.84    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:113,6:0.105:49,2:64,4:39:343,445:60:21:0.051,0.00,0.050:4.511e-03,0.024,0.971   0/0:149,8:0.082:69,4:80,4:39:319,327:60:32
    chr19   24683016    .   A   T   .   PASS    DP=890;ECNT=1;NLOD=113.19;N_ART_LOD=2.08;POP_AF=5.000e-08;TLOD=6.25 GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:395,7:0.057:208,6:187,1:38:364,349:60:21:0.00,0.020,0.017:9.880e-03,1.696e-03,0.988 0/0:422,5:0.040:213,3:209,2:38:358,340:60:21
    

    I can also share the full VCF if that helps.

  • SheilaSheila Broad InstituteMember, Broadie admin

    @igor
    Hi,

    Can you post the exact command you ran? To confirm, the variants you posted that PASS above are present in the 4.0.3.0 VCF, but fail due to some filters?

    Thanks,
    Sheila

  • igorigor New YorkMember ✭✭

    Can you post the exact command you ran? To confirm, the variants you posted that PASS above are present in the 4.0.3.0 VCF, but fail due to some filters?

    I generated the original VCF with 4.0.3.0 in case FilterMutectCalls is looking for some flag or string that was not previously used. This is the command I used for filtering:

    /path/GenomeAnalysisTK/gatk-4.0.X.X/gatk --java-options "-Xms8G -Xms8G" FilterMutectCalls --verbosity WARNING --variant ./unfiltered.4030.vcf  --output ./filtered.40XX.vcf
    

    The following variants are PASS with 4.0.2.1 (no variants are PASS with 4.0.3.0):

    chr19   5477654 .   G   T   .   PASS    DP=207;ECNT=1;NLOD=30.93;N_ART_LOD=-2.018e+00;POP_AF=5.000e-08;TLOD=5.38    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:99,3:0.035:46,3:53,0:38:302,0:60:16:0.00,0.030,0.029:0.022,3.140e-03,0.974  0/0:103,0:6.335e-03:47,0:56,0:0:309,0:0:0
    chr19   9007221 .   G   T   .   PASS    DP=1675;ECNT=1;NLOD=259.93;N_ART_LOD=-2.945e+00;POP_AF=5.000e-08;TLOD=5.80  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:727,5:0.026:359,5:368,0:37:363,509:60:26:0.00,0.010,6.831e-03:3.986e-03,8.100e-04,0.995 0/0:870,2:0.026:429,0:441,2:12:348,411:60:3
    chr19   10942813    .   ATT A   .   PASS    DP=389;ECNT=1;NLOD=37.93;N_ART_LOD=-6.919e-01;POP_AF=5.000e-08;RPA=14,12;RU=T;STR;TLOD=5.30 GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:135,5:0.158:55,3:80,2:36:371,362:60:38:0.040,0.00,0.036:2.385e-03,0.046,0.952   0/0:148,3:0.145:75,1:73,2:35:355,326:60:34
    chr19   11478211    .   C   T   .   PASS    DP=530;ECNT=1;NLOD=79.24;N_ART_LOD=-2.427e+00;POP_AF=5.000e-08;TLOD=5.45    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:254,4:0.022:143,4:111,0:37:351,194:60:13:0.020,0.00,0.016:8.877e-04,0.083,0.916 0/0:264,0:0.013:147,0:117,0:0:324,0:0:0
    chr19   28262206    .   C   T   .   PASS    DP=1456;ECNT=2;NLOD=228.52;N_ART_LOD=-2.594e+00;POP_AF=5.000e-08;TLOD=6.48  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:596,6:0.045:286,2:310,4:36:375,229:60:32:0.010,0.010,9.967e-03:6.372e-04,0.018,0.981    0/0:762,0:0.034:388,0:374,0:0:354,0:0:0
    chr19   29387190    .   G   A   .   PASS    DP=607;ECNT=2;NLOD=80.48;N_ART_LOD=-2.440e+00;POP_AF=5.000e-08;TLOD=150.99  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:251,55:0.198:121,29:130,26:35:415,413:60:31:0.172,0.162,0.180:0.012,8.172e-03,0.979 0/0:268,0:0.030:140,0:128,0:0:406,0:0:0
    chr19   37701004    .   A   T   .   PASS    DP=1474;ECNT=1;NLOD=211.65;N_ART_LOD=-1.069e+00;POP_AF=5.000e-08;TLOD=5.66  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:667,10:0.031:311,2:356,8:36:328,264:60:21:0.010,0.010,0.015:1.219e-03,3.111e-03,0.996   0/0:729,3:0.033:356,2:373,1:36:329,525:60:21
    
  • SheilaSheila Broad InstituteMember, Broadie admin

    @igor
    Hi,

    Alright. So, you ran Mutect2 4.0.3.0 to generate a VCF. Then, you ran FilterMutectCalls from 4.0.2.1 and 4.0.3.0 on that VCF. From 4.0.2.1 filtering, you get those 7 PASS variants, but from 4.0.3.0 filtering, you get no PASS variants.

    Can you post the records that are filtered in the 4.0.3.0 filtered VCF? I am wondering why those sites were filtered, and the filter column should show which filters they failed.

    -Sheila

  • igorigor New YorkMember ✭✭
    edited April 2018

    Can you post the records that are filtered in the 4.0.3.0 filtered VCF? I am wondering why those sites were filtered, and the filter column should show which filters they failed.

    Good point. Here are those variants after 4.0.3.0 filtering:

    chr19   5477654 .   G   T   .   .   DP=207;ECNT=1;NLOD=30.93;N_ART_LOD=-2.018e+00;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-5.674e+01;TLOD=5.38    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:99,3:0.035:46,3:53,0:38:302,0:60:16:0.00,0.030,0.029:0.022,3.140e-03,0.974  0/0:103,0:6.335e-03:47,0:56,0:0:309,0:0:0
    chr19   9007221 .   G   T   .   .   DP=1675;ECNT=1;NLOD=259.93;N_ART_LOD=-2.945e+00;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-4.650e+02;TLOD=5.80  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:727,5:0.026:359,5:368,0:37:363,509:60:26:0.00,0.010,6.831e-03:3.986e-03,8.100e-04,0.995 0/0:870,2:0.026:429,0:441,2:12:348,411:60:3
    chr19   10942813    .   ATT A   .   .   DP=389;ECNT=1;NLOD=37.93;N_ART_LOD=-6.919e-01;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-6.698e+01;RPA=14,12;RU=T;STR;TLOD=5.30 GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:135,5:0.158:55,3:80,2:36:371,362:60:38:0.040,0.00,0.036:2.385e-03,0.046,0.952   0/0:148,3:0.145:75,1:73,2:35:355,326:60:34
    chr19   11478211    .   C   T   .   .   DP=530;ECNT=1;NLOD=79.24;N_ART_LOD=-2.427e+00;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-1.488e+02;TLOD=5.45    GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:254,4:0.022:143,4:111,0:37:351,194:60:13:0.020,0.00,0.016:8.877e-04,0.083,0.916 0/0:264,0:0.013:147,0:117,0:0:324,0:0:0
    chr19   28262206    .   C   T   .   .   DP=1456;ECNT=2;NLOD=228.52;N_ART_LOD=-2.594e+00;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-3.907e+02;TLOD=6.48  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:596,6:0.045:286,2:310,4:36:375,229:60:32:0.010,0.010,9.967e-03:6.372e-04,0.018,0.981    0/0:762,0:0.034:388,0:374,0:0:354,0:0:0
    chr19   29387190    .   G   A   .   .   DP=607;ECNT=2;NLOD=80.48;N_ART_LOD=-2.440e+00;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-1.109e+02;TLOD=150.99  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:251,55:0.198:121,29:130,26:35:415,413:60:31:0.172,0.162,0.180:0.012,8.172e-03,0.979 0/0:268,0:0.030:140,0:128,0:0:406,0:0:0
    chr19   37701004    .   A   T   .   .   DP=1474;ECNT=1;NLOD=211.65;N_ART_LOD=-1.069e+00;POP_AF=5.000e-08;P_CONTAM=0.00;P_GERMLINE=-3.922e+02;TLOD=5.66  GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:667,10:0.031:311,2:356,8:36:328,264:60:21:0.010,0.010,0.015:1.219e-03,3.111e-03,0.996   0/0:729,3:0.033:356,2:373,1:36:329,525:60:21
    

    They are not labeled as PASS, but also not tagged with any failed filters. If I search for "." in the FILTER field (column 7), I only get those variants. It might be that PASS was not added for some reason.

  • vsvintivsvinti Member ✭✭

    I also get

    java.lang.IllegalStateException: Key P_CONTAM found in VariantContext field INFO at 1:3000126 but this key isn't defined in the VCFHeader.  We require all VCFs to have complete VCF heade
    

    with gatk 4.0.3.0 but not with 4.0.2.1.
    The variant the error refers to don't actually have the P_CONTAM annotation in the INFO field at all, or anywhere in the vcf really. I don't know why it's looking for it,
    Using 4.0.2.1 fixed my problem.

  • SheilaSheila Broad InstituteMember, Broadie admin

    @vsvinti
    Hi,

    To confirm, when you generate a VCF using 4.0.3.0 and try to filter it with 4.0.3.0, you get that error?

    Do any of the variants have the P_CONTAM annotation?

    Thanks,
    Sheila

  • sutturkasutturka Member

    I just noticed this discussion here. I am having similar issue with P_CONTAM annotation. I see following:

    If VCF is generated with 4.0.0.0 FilterMutectCalls from 4.0.0.0 will run without errors but FilterMutectCalls in 4.0.3.0 will throw "P_CONTAM" error. Cross compatibility is lost across these two versions.

    If VCF is generated with 4.0.3.0 then FilterMutectCalls from both version runs without errors.

    VCF files with 4.0.3.0, have line ##INFO=<ID=P_CONTAM,Number=1,Type=Float,Description="Posterior probability for alt allele to be due to contamination"> but VCF files from 4.0.0.0 does not contain this line.

    In both cases, I do not see any PASS calls in tumor_only or paired_normal samples. I will post the counts of the filters applied soon.

    Thanks
    Sagar

  • vsvintivsvinti Member ✭✭

    @Sheila

    I haven't tried to go back up the pipeline to generate the vcf with 4.0.3.0. The only reason I downloaded 4.0.3.0 was because of the issue with not getting any PASS variants in the FilterMutectCalls step, I thought it's a bug and wanted to check if it was solved in 4.0.3.0 (conversation in separate thread).

  • SheilaSheila Broad InstituteMember, Broadie admin
    edited April 2018

    @sutturka
    Hi,

    The team is actively improving the tools in the workflow, and it is best to stick to one version. Have a look at this article. Unfortunately, we cannot help in this case, as the files produced by the same versions work together.

    @vsvinti
    Hi,

    I see. Is it possible to try with 4.0.3.0 again? If not, I hope the other thread has offered some useful insight.

    -Sheila

  • sutturkasutturka Member

    @Sheila
    Thank you for the comments. I do appreciate your answers which are always very useful and developers efforts for continuously improving the tools. I also installed the v4.0.3.0 to check if it could resolve the "PASS" calls issue in Mutect2. I wanted to list all the scenarios in case developers need to troubleshoot. As I mentioned above, If Mutect2 and FilterMutectCalls are run from the same version then there are no errors.

  • SheilaSheila Broad InstituteMember, Broadie admin

    @sutturka
    Hi,

    Thanks for the kind words and of course for bringing issues to our attention. The developers are indeed working hard to bring the best possible tools to you all :smile: They understand it is tough for you to keep upgrading, but they hope you all understand it is for the best.

    -Sheila

Sign In or Register to comment.