Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

Bug with version 3.6 and jexl

I recently tried to use a GaTK command I previously used with GaTK 3.5, and I'm getting a jexl error.
I don't understand why. I've updated to java 1.8.0_77
java -Xmx${MEM} -jar ${gatk_dir}/GenomeAnalysisTK.jar \
-T VariantFiltration \
-R ${genome} \
-L ${CHROM} \
-V ${data_dir}/'14a_'${abb}'HapCaller_mergedCHRs_VARIANTS'${CHROM}'.vcf' \
-G_filter "DP < 10 " \
-G_filterName "LowCov" \
-G_filter "DP > 100 " \
-G_filterName "HighCov" \
-G_filter "GQ < 20 " \
-G_filterName "LowGQ" \
--clusterWindowSize 10 --clusterSize 3 \
--filterExpression "QD < 2.0 || FS > 60.0 || MQ < 40.0 || MQRankSum < -12.5 || ReadPosRankSum < -8.0 || SOR > 4.0 " \
--filterName "gatkHardFilter" \
--setFilteredGtToNocall \
-o ${data_dir}/'14b_newgatk_'${abb}'HapCaller_mergedCHRs_VARIANTS_hard_cluster_filterflag_DPGQmissing'${CHROM}'.vcf'

The error message:

ERROR --
ERROR stack trace

java.lang.IllegalArgumentException: Invalid JEXL expression detected for HighCov with message no message
at htsjdk.variant.variantcontext.JEXLMap.evaluateExpression(JEXLMap.java:136)
at htsjdk.variant.variantcontext.JEXLMap.get(JEXLMap.java:93)
at htsjdk.variant.variantcontext.JEXLMap.get(JEXLMap.java:22)
at htsjdk.variant.variantcontext.VariantContextUtils.match(VariantContextUtils.java:323)
at org.broadinstitute.gatk.tools.walkers.filters.VariantFiltration.filter(VariantFiltration.java:433)
at org.broadinstitute.gatk.tools.walkers.filters.VariantFiltration.map(VariantFiltration.java:349)
at org.broadinstitute.gatk.tools.walkers.filters.VariantFiltration.map(VariantFiltration.java:97)
at org.broadinstitute.gatk.engine.traversals.TraverseLociNano$TraverseLociMap.apply(TraverseLociNano.java:267)
at org.broadinstitute.gatk.engine.traversals.TraverseLociNano$TraverseLociMap.apply(TraverseLociNano.java:255)
at org.broadinstitute.gatk.utils.nanoScheduler.NanoScheduler.executeSingleThreaded(NanoScheduler.java:274)
at org.broadinstitute.gatk.utils.nanoScheduler.NanoScheduler.execute(NanoScheduler.java:245)
at org.broadinstitute.gatk.engine.traversals.TraverseLociNano.traverse(TraverseLociNano.java:144)
at org.broadinstitute.gatk.engine.traversals.TraverseLociNano.traverse(TraverseLociNano.java:92)
at org.broadinstitute.gatk.engine.traversals.TraverseLociNano.traverse(TraverseLociNano.java:48)
at org.broadinstitute.gatk.engine.executive.LinearMicroScheduler.execute(LinearMicroScheduler.java:99)
at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:311)
at org.broadinstitute.gatk.engine.CommandLineExecutable.execute(CommandLineExecutable.java:113)
at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:255)
at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:157)
at org.broadinstitute.gatk.engine.CommandLineGATK.main(CommandLineGATK.java:108)

ERROR ------------------------------------------------------------------------------------------
ERROR A GATK RUNTIME ERROR has occurred (version 3.6-0-g89b7209):
ERROR
ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.
ERROR If not, please post the error message, with stack trace, to the GATK forum.
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions https://www.broadinstitute.org/gatk
ERROR
ERROR MESSAGE: Invalid JEXL expression detected for HighCov with message no message
ERROR ------------------------------------------------------------------------------------------

Answers

  • prepagamprepagam Member

    I should add that the 14a file was produced by GaTK 3.5 with HaplotypeCaller followed by selecting biallelic snps with select variaints.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Can you try running a simplified command with just that one filter?

  • prepagamprepagam Member

    Hi Geraldine,
    So it appears to be the DP filter as the GQ filter works just fine
    Any idea how to fix this?

    I can run this:
    java -Xmx${MEM} -jar ${gatk_dir}/GenomeAnalysisTK.jar \
    -T VariantFiltration \
    -R ${genome} \
    -L ${CHROM} \
    -V ${data_dir}/'14a_'${abb}'HapCaller_mergedCHRs_VARIANTS'${CHROM}'.vcf' \
    -G_filter "GQ < 20 " \
    -G_filterName "LowGQ" \
    --setFilteredGtToNocall \
    -o ${data_dir}/'newgatk_out.vcf'

    This creates an error:
    java -Xmx${MEM} -jar ${gatk_dir}/GenomeAnalysisTK.jar \
    -T VariantFiltration \
    -R ${genome} \
    -L ${CHROM} \
    -V ${data_dir}/'14a_'${abb}'HapCaller_mergedCHRs_VARIANTS'${CHROM}'.vcf' \
    -G_filter "DP > 20 " \
    -G_filterName "HighDP" \
    --setFilteredGtToNocall \
    -o ${data_dir}/'newgatk_out.vcf'

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    I just tried it myself on some local test data and I'm not getting any errors. Maybe it's choking on a particular record? If you can narrow it down to a snippet that reproduces the issue we can have a look at it.

  • prepagamprepagam Member

    Here is a snippet of my vcf - and the DP field is definitely there.
    11 33853 . C T 63.77 . AC=5;AF=0.417;AN=12;DP=189;ExcessHet=3.0103;MLEAC=1;MLEAF=
    0.500;set=variant-variant2-variant3-variant5-variant6 GT:AD:DP:GQ:PL:RGQ 0/1:26,8:34:99:115,0,573 0/
    1:19,7:26:99:125,0,443 0/0:30:30:.:.:90 0/1:29,8:37:92:92,0,604 0/1:31,11:42:99:156,0,650 0/1:13,7:2
    0:99:128,0,262
    11 33860 . A C 1271.77 . AC=12;AF=1.00;AN=12;DP=231;ExcessHet=3.0103;FS=0.000;MLEAC
    =2;MLEAF=1.00;set=Intersection GT:AD:DP:GQ:PL 1/1:0,38:38:99:1146,114,0 1/1:0,32:32:96:1058,96,0 1/
    1:0,38:38:99:940,114,0 1/1:0,46:46:99:1300,138,0 1/1:0,55:55:99:1577,164,0 1/1:0,22:22:66:686,66,0

  • SheilaSheila Broad InstituteMember, Broadie admin

    @prepagam
    Hi,

    If you can submit a bug report, the would be great so we can debug locally. Instructions are here.

    Thanks,
    Sheila

  • prepagamprepagam Member

    Will do. Thanks

  • prepagamprepagam Member

    I've now uploaded a tar.gz file
    gatk36_jexlerrorbug.tar.gz

    Issue · Github
    by Sheila

    Issue Number
    978
    State
    closed
    Last Updated
    Assignee
    Array
    Milestone
    Array
    Closed By
    chandrans
  • prepagamprepagam Member

    I'm not sure if this is too late, but I had a thought. I generate the file with Gatk35 and the for that I get calls for the hom ref like this:
    GT:AD:DP:GQ:PGT:PID:PL:RGQ 0/0:27:29:.:.:.:.:33
    You will see that the GQ field just has a .
    Perhaps GaTK36 doesn't know what to do with those.

  • sailercsailerc ETHZMember

    Hi there

    I ran into the same problem as prepagam when I switched to version 3.6.
    Are there any news about this bug that only seems to affect the format DP field and its fix?
    I attach my code and the error message.

    thanks,
    christian

  • prepagamprepagam Member

    I've not heard anything yet ...

  • SheilaSheila Broad InstituteMember, Broadie admin

    @sailerc @prepagam
    Hi,

    I am going to look into this and get back to you.

    -Sheila

  • Just to add, I run into the same issue. These are the records that trigger the bug for me

    $ grep ":.:" joint_snips.vcf
    gi|194447306|ref|NC_011083.1| 1551943 . C T 1461.03 . AC=11;AF=1.00;AN=11;DP=145;FS=0.000;MLEAC=11;MLEAF=1.00;MQ=27.25;QD=29.27;SOR=2.019 GT:AD:DP:GQ:PL .:0,0 .:0,0 .:0,0.:0,0 .:0,0 1:0,2:2:90:90,0 1:0,3:3:99:180,0 1:0,4:4:99:180,0 1:0,3:3:99:135,0 .:0,0 .:0,0 1:0,4:4:99:180,0 1:0,2:2:90:90,0 1:0,4:4:99:180,0 .:0,0 .:0,0 .:0,0.:0,0 .:0,0 .:0,0 1:0,2:2:90:90,0 .:0,0 1:0,4:4:99:180,0 .:0,0 1:0,0:.:90:90,0 .:0,0 .:0,0 .:0,0 .:0,0 1:0,2:2:90:90,0
    gi|194447306|ref|NC_011083.1| 1551946 . T C 1866.03 . AC=12;AF=1.00;AN=12;DP=135;FS=0.000;MLEAC=12;MLEAF=1.00;MQ=30.51;QD=25.02;SOR=1.675 GT:AD:DP:GQ:PL .:0,0 .:0,0 .:0,0.:0,0 .:0,0 1:0,2:2:99:135,0 1:0,4:4:99:225,0 1:0,5:5:99:225,0 1:0,3:3:99:135,0 .:0,0 .:0,0 1:0,4:4:99:180,0 1:0,2:2:90:90,0 1:0,4:4:99:180,0 .:0,0 .:0,0.:0,0 .:0,0 .:0,0 .:0,0 1:0,3:3:99:270,0 1:0,0:.:45:45,0 1:0,4:4:99:225,0 .:0,0 1:0,2:2:90:90,0 .:0,0 .:0,0 .:0,0 .:0,0 1:0,2:2:90:90,0
    gi|194447306|ref|NC_011083.1| 1551967 . A C 4701.03 . AC=14;AF=1.00;AN=14;DP=188;FS=0.000;MLEAC=14;MLEAF=1.00;MQ=39.86;QD=29.59;SOR=1.329 GT:AD:DP:GQ:PL .:0,0 .:0,0 .:0,0.:0,0 .:0,0 1:0,8:8:99:495,0 1:0,11:11:99:540,0 1:0,9:9:99:495,0 1:0,6:6:99:360,0 .:0,0 .:0,0 1:0,7:7:99:405,0 1:0,4:4:99:225,0 1:0,6:6:99:270,0 .:0,0.:0,0 .:0,0 .:0,0 .:0,0 .:0,0 1:0,8:8:99:450,0 1:0,1:1:99:135,0 1:0,15:15:99:720,0 1:0,0:.:45:45,0 1:0,3:3:99:180,0 1:0,2:2:90:90,0 .:0,0 .:0,0 .:0,0 1:0,6:6:99:315,0
    gi|194447306|ref|NC_011083.1| 4229249 . A C 84.95 . AC=2;AF=0.069;AN=29;DP=126;FS=0.000;MLEAC=2;MLEAF=0.069;MQ=10.32;QD=32.59;SOR=2.303 GT:AD:DP:GQ:PL 1:0,0:.:45:45,0 1:0,2:2:90:90,0 .:0,0 0:5,0:5:99:0,199 0:3,0:3:99:0,101 0:3,0:3:99:0,114 0:5,0:5:99:0,135 0:3,0:3:99:0,99 0:3,0:3:99:0,118 0:3,0:3:99:0,119 0:3,0:3:99:0,119 0:3,0:3:99:0,121 0:3,0:3:99:0,119 0:3,0:3:99:0,118 0:3,0:3:99:0,112 0:3,0:3:99:0,117 0:4,0:4:99:0,135 0:4,0:4:99:0,135 0:3,0:3:99:0,113 0:3,0:3:99:0,123 0:3,0:3:99:0,119 0:3,0:3:99:0,106 0:4,0:4:99:0,135 0:3,0:3:99:0,103 0:3,0:3:99:0,119 0:3,0:3:99:0,103 0:3,0:3:99:0,121 0:3,0:3:99:0,119 0:3,0:3:99:0,103 0:3,0:3:99:0,116
    gi|194447306|ref|NC_011083.1| 4229254 . C T 119.95 . AC=3;AF=0.100;AN=30;DP=91;FS=0.000;MLEAC=3;MLEAF=0.100;MQ=45.00;QD=23.76;SOR=2.833 GT:AD:DP:GQ:PL 1:0,1:1:45:45,0 1:0,2:2:90:90,0 1:0,0:.:45:45,0 0:5,0:5:99:0,199 0:3,0:3:99:0,101 0:3,0:3:99:0,114 0:5,0:5:99:0,135 0:3,0:3:99:0,99 0:3,0:3:99:0,118 0:3,0:3:99:0,119 0:3,0:3:99:0,119 0:3,0:3:99:0,121 0:3,0:3:99:0,119 0:3,0:3:99:0,118 0:3,0:3:99:0,112 0:3,0:3:99:0,117 0:4,0:4:99:0,135 0:4,0:4:99:0,135 0:3,0:3:99:0,113 0:3,0:3:99:0,123 0:3,0:3:99:0,119 0:3,0:3:99:0,106 0:4,0:4:99:0,135 0:3,0:3:99:0,103 0:3,0:3:99:0,119 0:3,0:3:99:0,103 0:3,0:3:99:0,121 0:3,0:3:99:0,119 0:3,0:3:99:0,103 0:3,0:3:99:0,116

    I've checked a vcf file generated with 3.5, it also has these records, but I don't recall running into the DP filtering issue then. I can't verify that easily though, since I've switched to java 8 so I can't juse gatk3.5

  • HeidiLingHeidiLing Member ✭✭

    Hope this helps for those before the bug gets fixed.

    I ran into the same problem when I switched to version 3.6 in order to get updated AC, AN when using setFilteredGtToNocall. The error did not show up if I use Java1.7 and GATK3.5 (but GATK 3.5 doesn't update AC, AF). In order to get around with it, my colleague and I used a nightly build version of GATK on May13th with Java1.8 and it went through.

  • SheilaSheila Broad InstituteMember, Broadie admin

    @HeidiLing
    Hi,

    Thanks for letting us know That will help the developers narrow down the issue.

    -Sheila

  • KatieKatie United StatesMember ✭✭

    Hi, I am also having issues with VariantFiltration in GATK 3.6 with a genotypeFilterExpression "DP <5".

    My call: java -jar "/home/ksw9/bin/GenomeAnalysisTK.jar" -T VariantFiltration -R $REF -V "/ycga-ba/home/ksw9/mapping/Bb/vars/"${NAME}.raw.${PLOIDY}.vcf --genotypeFilterExpression "DP < 5" --genotypeFilterName "lowCov"

    I have a strange problem where the VariantFiltration appears to be working on part of my VCF file and then truncates with the error message:

    ERROR stack trace

    java.lang.IllegalArgumentException: Invalid JEXL expression detected for lowCov with message no message
    at htsjdk.variant.variantcontext.JEXLMap.evaluateExpression(JEXLMap.java:136)
    at htsjdk.variant.variantcontext.JEXLMap.get(JEXLMap.java:93)
    at htsjdk.variant.variantcontext.JEXLMap.get(JEXLMap.java:22)
    at htsjdk.variant.variantcontext.VariantContextUtils.match(VariantContextUtils.java:323)
    at org.broadinstitute.gatk.tools.walkers.filters.VariantFiltration.filter(VariantFiltration.java:433)
    at org.broadinstitute.gatk.tools.walkers.filters.VariantFiltration.map(VariantFiltration.java:349)
    at org.broadinstitute.gatk.tools.walkers.filters.VariantFiltration.map(VariantFiltration.java:97)
    at org.broadinstitute.gatk.engine.traversals.TraverseLociNano$TraverseLociMap.apply(TraverseLociNano.java:267)
    at org.broadinstitute.gatk.engine.traversals.TraverseLociNano$TraverseLociMap.apply(TraverseLociNano.java:255)
    at org.broadinstitute.gatk.utils.nanoScheduler.NanoScheduler.executeSingleThreaded(NanoScheduler.java:274)
    at org.broadinstitute.gatk.utils.nanoScheduler.NanoScheduler.execute(NanoScheduler.java:245)
    at org.broadinstitute.gatk.engine.traversals.TraverseLociNano.traverse(TraverseLociNano.java:144)
    at org.broadinstitute.gatk.engine.traversals.TraverseLociNano.traverse(TraverseLociNano.java:92)
    at org.broadinstitute.gatk.engine.traversals.TraverseLociNano.traverse(TraverseLociNano.java:48)
    at org.broadinstitute.gatk.engine.executive.LinearMicroScheduler.execute(LinearMicroScheduler.java:99)
    at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:311)
    at org.broadinstitute.gatk.engine.CommandLineExecutable.execute(CommandLineExecutable.java:113)
    at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:255)
    at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:157)
    at org.broadinstitute.gatk.engine.CommandLineGATK.main(CommandLineGATK.java:108)

    ERROR ------------------------------------------------------------------------------------------
    ERROR A GATK RUNTIME ERROR has occurred (version 3.6-0-g89b7209):
    ERROR
    ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.
    ERROR If not, please post the error message, with stack trace, to the GATK forum.
    ERROR Visit our website and forum for extensive documentation and answers to
    ERROR commonly asked questions https://www.broadinstitute.org/gatk
    ERROR
    ERROR MESSAGE: Invalid JEXL expression detected for lowCov with message no message

    Thanks in advance for your help!

  • SheilaSheila Broad InstituteMember, Broadie admin

    @Katie
    Hi,

    Yes, this is a known issue. You can keep track of it here.

    -Sheila

  • SheilaSheila Broad InstituteMember, Broadie admin

    @prepagam
    Hi,

    We have a fix for this that is pending review. Can you confirm the data can be used in public tests? We will strip the names of the samples and can blur out the site.

    Thanks,
    Sheila

  • prepagamprepagam Member

    Sure as long as there is no identifyable features (samples names, paths) ...

  • SheilaSheila Broad InstituteMember, Broadie admin

    @prepagam
    Hi,

    Actually, it seems they may not even have to use your test case. Thanks for the confirmation :smile:

    -Sheila

  • SheilaSheila Broad InstituteMember, Broadie admin

    @prepagam @Katie @HeidiLing @Redmar_van_den_Berg @sailerc
    Hi everyone,

    You can try the latest nightly build with a fix for this issue! :smile:

    -Sheila

Sign In or Register to comment.