Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

VariantRecalibrator: java.nio.file.NoSuchFileException

manolismanolis Member ✭✭

GATK 4.1.1.0, bash, linux server

Hi,

I have the following error/ exception... maybe is always related to this [thread]!?(https://gatkforums.broadinstitute.org/gatk/discussion/10797/picard-revertsam-java-nio-file-nosuchfileexception#latest "thread")!?

Best

Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx20g -XX:GCTimeLimit=50 -XX:GCHeapFreeLimit=10 -XX:ConcGCThreads=1 -XX:ParallelGCThreads=2 -jar /share/apps/bio/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar VariantRecalibrator -V 3.rawHFSO.vcf.gz -O 4i.rawHFSO-iVR.vcf.gz --tranches-file 4i.indel.tranches --trust-all-polymorphic -tranche 100.0 -tranche 99.95 -tranche 99.9 -tranche 99.5 -tranche 99.0 -tranche 97.0 -tranche 96.0 -tranche 95.0 -tranche 94.0 -tranche 93.5 -tranche 93.0 -tranche 92.0 -tranche 91.0 -tranche 90.0 -an FS -an ReadPosRankSum -an MQRankSum -an QD -an SOR -mode INDEL --max-gaussians 4 -resource:mills,known=false,training=true,truth=true,prior=12 /shared/resources/gatk4hg38db/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz -resource:axiomPoly,known=false,training=true,truth=false,prior=10 /shared/resources/gatk4hg38db/Axiom_Exome_Plus.genotypes.all_populations.poly.hg38.vcf.gz -resource:dbsnp,known=true,training=false,truth=false,prior=2 /shared/resources/gatk4hg38db/Homo_sapiens_assembly38.dbsnp138.vcf --rscript-file indel.plots.R --tmp-dir /home/manolis/GATK4/tmp/
06:29:53.083 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/share/apps/bio/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so

Exception in thread "Thread-1" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /home/manolis/GATK4/tmp/Rlib.667992084951395367
    at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1346)
    at org.broadinstitute.hellbender.utils.io.IOUtils.deleteRecursively(IOUtils.java:1061)
    at org.broadinstitute.hellbender.utils.io.DeleteRecursivelyOnExitPathHook.runHooks(DeleteRecursivelyOnExitPathHook.java:56)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: /home/manolis/GATK4/tmp/Rlib.667992084951395367
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1344)
    ... 3 more

Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx100g -XX:GCTimeLimit=50 -XX:GCHeapFreeLimit=10 -XX:ConcGCThreads=1 -XX:ParallelGCThreads=2 -jar /share/apps/bio/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar VariantRecalibrator -V 3.rawHFSO.vcf.gz -O 4s.rawHFSO-sVR.vcf.gz --tranches-file 4s.snp.tranches -trust-all-polymorphic -tranche 100.0 -tranche 99.95 -tranche 99.9 -tranche 99.8 -tranche 99.6 -tranche 99.5 -tranche 99.4 -tranche 99.3 -tranche 99.0 -tranche 98.0 -tranche 97.0 -tranche 90.0 -an QD -an MQRankSum -an ReadPosRankSum -an FS -an MQ -an SOR -mode SNP --max-gaussians 6 -resource:hapmap,known=false,training=true,truth=true,prior=15 /shared/resources/gatk4hg38db/hapmap_3.3.hg38.vcf.gz -resource:omni,known=false,training=true,truth=true,prior=12 /shared/resources/gatk4hg38db/1000G_omni2.5.hg38.vcf.gz -resource:1000G,known=false,training=true,truth=false,prior=10 /shared/resources/gatk4hg38db/1000G_phase1.snps.high_confidence.hg38.vcf.gz -resource:dbsnp,known=true,training=false,truth=false,prior=7 /shared/resources/gatk4hg38db/Homo_sapiens_assembly38.dbsnp138.vcf --rscript-file snp.plots.R --tmp-dir /home/manolis/GATK4/tmp/

Exception in thread "Thread-1" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /home/manolis/GATK4/tmp/Rlib.9072785099028549732
    at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1346)
    at org.broadinstitute.hellbender.utils.io.IOUtils.deleteRecursively(IOUtils.java:1061)
    at org.broadinstitute.hellbender.utils.io.DeleteRecursivelyOnExitPathHook.runHooks(DeleteRecursivelyOnExitPathHook.java:56)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: /home/manolis/GATK4/tmp/Rlib.9072785099028549732
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1344)
    ... 3 more

Best Answer

Answers

  • init_jsinit_js Member
    edited May 15

    I get the same issue. I've also tried running without the --tmp-dir option, and the same stack trace is produced. I'm wondering if in this case it would be fine to keep going, or if there's another problem lurking before this exception.

    In my case it runs through, and seems to run Rscript for generating the plots, and then crashes.

    Update: Found this comment here (in the link provided in the OP question), which points to a likely cause. https://gatkforums.broadinstitute.org/gatk/discussion/comment/58104/#Comment_58104 .
    The issue being tracked is https://github.com/broadinstitute/gatk/issues/5893 and seems to match the symptoms. Got this error on 4.1.1.0 as well. Will try again on a more recent build and report.

    ...
    SNP_11:38:59.767 INFO  VariantRecalibrator - Building AS_SOR x AS_ReadPosRankSum plot...
    SNP_11:38:59.768 INFO  VariantRecalibratorEngine - Evaluating full set of 3660 variants...
    SNP_11:39:01.171 INFO  VariantRecalibratorEngine - Evaluating full set of 3660 variants...
    SNP_11:39:01.649 INFO  VariantRecalibrator - Building AS_MQRankSum x AS_ReadPosRankSum plot...
    SNP_11:39:01.650 INFO  VariantRecalibratorEngine - Evaluating full set of 3600 variants...
    SNP_11:39:02.974 INFO  VariantRecalibratorEngine - Evaluating full set of 3600 variants...
    SNP_11:39:03.444 INFO  VariantRecalibrator - Executing: Rscript /localscratch/jslegare.21052947.0/tmp.recalibrate.PXTBRx/snp.recal.Rscript
    SNP_11:39:52.997 INFO  VariantRecalibrator - Executing: Rscript (resource)org/broadinstitute/hellbender/tools/walkers/vqsr/plot_Tranches.R /localscratch/jslegare.21052947.0
    /tmp.recalibrate.PXTBRx/snp.tranches 2.15
    SNP_11:39:54.422 INFO  VariantRecalibrator - Shutting down engine
    SNP_[May 15, 2019 11:39:54 AM GMT] org.broadinstitute.hellbender.tools.walkers.vqsr.VariantRecalibrator done. Elapsed time: 137.93 minutes.
    SNP_Runtime.totalMemory()=64259358720
    SNP_Tool returned:
    SNP_true
    SNP_Exception in thread "Thread-1" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /tmp/Rlib.3127958760928906000
    SNP_at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1346)
    SNP_at org.broadinstitute.hellbender.utils.io.IOUtils.deleteRecursively(IOUtils.java:1061)
    SNP_at org.broadinstitute.hellbender.utils.io.DeleteRecursivelyOnExitPathHook.runHooks(DeleteRecursivelyOnExitPathHook.java:56)
    SNP_at java.lang.Thread.run(Thread.java:748)
    SNP_Caused by: java.nio.file.NoSuchFileException: /tmp/Rlib.3127958760928906000
    SNP_at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    SNP_at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    SNP_at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    SNP_at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    SNP_at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    SNP_at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    SNP_at java.nio.file.Files.readAttributes(Files.java:1737)
    SNP_at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
    SNP_at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    SNP_at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    SNP_at java.nio.file.Files.walkFileTree(Files.java:2662)
    SNP_at java.nio.file.Files.walkFileTree(Files.java:2742)
    SNP_at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1344)
    SNP_... 3 more
    SNP_Using GATK jar /gatk/gatk-package-4.1.1.0-local.jar
    SNP_Running:
    SNP_java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -Xmx121856m -jar /gatk/gatk-package-4.1.1.0-local.jar VariantRecalibrator --arguments_file /localscratch/jslegare.21052947.0/tmp.recalibrate.PXTBRx/arguments.txt --resource:GOLD,known=false,training=true,truth=true,prior=10.0 /localscratch/jslegare.21052947.0/tmp.recalibrate.PXTBRx/gold.snps.vcf.gz --mode SNP -an AS_QD -an AS_MQRankSum -an AS_ReadPosRankSum -an AS_FS -an AS_MQ -an AS_SOR -an DP --trust-all-polymorphic --truth-sensitivity-tranche 100.0 --truth-sensitivity-tranche 99.0 --truth-sensitivity-tranche 90.0 --truth-sensitivity-tranche 70.0 --truth-sensitivity-tranche 50.0 --max-gaussians 6 --rscript-file /localscratch/jslegare.21052947.0/tmp.recalibrate.PXTBRx/snp.recal.Rscript --tranches-file /localscratch/jslegare.21052947.0/tmp.recalibrate.PXTBRx/snp.tranches -AS --output /localscratch/jslegare.21052947.0/tmp.recalibrate.PXTBRx/snp.recal.vcf.gz
    
    Post edited by init_js on
  • init_jsinit_js Member
    edited May 16

    Retrying with broadinstitute/gatk:4.1.2.0 produced the same error outcome. (The patch for the bug was applied 16 days ago, and 4.1.2.0 was released 22 days ago, so not a huge surprise). Will try the gatk-nightly build. (tag 2019-05-15-4.1.2.0-7-ga229b5646-NIGHTLY-SNAPSHOT)

    If the said patch is truly a fix for this, the error would be with RScriptExecutor. As a workaround, it might be possible to simply remove Rscript from $PATH just before calling VariantRecalibrator, and then following that with the equivalent Rscript commands:

    Rscript <recal.Rscript>
    #and
    Rscript (resource)org/broadinstitute/hellbender/tools/walkers/vqsr/plot_Tranches.R <tranches> 2.15
    

    where <recal.Rscript> is the argument to --rscript-file, and <tranches> is the argument to --tranches-file.

    the plot_Tranches.R is deep in the jar file, but it can be extracted with:

    unzip -p /gatk/gatk.jar org/broadinstitute/hellbender/tools/walkers/vqsr/plot_Tranches.R > plot_Tranches.R
    
    Post edited by init_js on
  • AdelaideRAdelaideR Unconfirmed, Member, Broadie, Moderator admin

    This may be the same issue. I opened a new issue in case it is not:

    You can track the progress on this issue here

  • manolismanolis Member ✭✭
Sign In or Register to comment.