Our documentation websites are currently offline due to a data center fire. We do not yet have an ETA for restoring service; we’ll update this message when we know more.

Picard RevertSam java.nio.file.NoSuchFileException

Hi,

I'm starting to process a set of bams following the best practices and beginning from bams that were processed by someone else. Thus, I'm attempting to generate unmapped BAMs following this post, and using the latest version of Picard (2.15.0). Unfortunately, Picard gives an exception that shows it is unable to find temporary files it is writing. I know there's space for these files and in fact, I now have version 1.141 of Picard running without issue. The output from version 2.15.0 is below.

15:34:14.012 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/REDACTED/bin/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Sat Nov 18 15:34:14 CST 2017] RevertSam INPUT=/REDACTED.bam OUTPUT=/REDACTED/808302_LP6008048-DNA_B02.bam SORT_ORDER=queryname RESTORE_ORIGINAL_QUALITIES=true REMOVE_DUPLICATE_INFORMATION=true REMOVE_ALIGNMENT_INFORMATION=true ATTRIBUTE_TO_CLEAR=[NM, UQ, PG, MD, MQ, SA, MC, AS, XT, XN, AS, OC, OP] SANITIZE=true MAX_DISCARD_FRACTION=0.005 TMP_DIR=[/REDACTED/tmp] VALIDATION_STRINGENCY=LENIENT OUTPUT_BY_READGROUP=false OUTPUT_BY_READGROUP_FILE_FORMAT=dynamic VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Sat Nov 18 15:34:14 CST 2017] Executing as awilliams@REDACTED on Linux 3.10.0-229.7.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12; Deflater: Intel; Inflater: Intel; Picard version: 2.15.0-SNAPSHOT
[Sat Nov 18 15:34:30 CST 2017] picard.sam.RevertSam done. Elapsed time: 0.27 minutes.
Runtime.totalMemory()=1272971264
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /REDACTED/tmp/awilliams/sortingcollection.728972638772980431.tmp
at htsjdk.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:246)
at htsjdk.samtools.util.SortingCollection.add(SortingCollection.java:166)
at picard.sam.RevertSam$RevertSamSorter.add(RevertSam.java:637)
at picard.sam.RevertSam.doWork(RevertSam.java:260)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:268)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)
Caused by: java.nio.file.NoSuchFileException: /REDACTED/tmp/awilliams/sortingcollection.728972638772980431.tmp
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
at java.nio.file.Files.createTempFile(Files.java:852)
at htsjdk.samtools.util.IOUtil.newTempPath(IOUtil.java:316)
at htsjdk.samtools.util.SortingCollection.newTempFile(SortingCollection.java:255)
at htsjdk.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:220)
... 6 more

Issue · Github
by Sheila

Issue Number
2692
State
closed
Last Updated
Assignee
Array
Milestone
Array
Closed By
sooheelee

Answers

  • I have the same problem with MarkDuplicates. It seems that the output bam file may be okay -- samtools quickcheck did not complain and picard ValidateSamFile looked normal to me. But it still makes me uncomfortable. I can see the tmp files during the run and they are gone after the run, but the error complains that they cannot be found.

    java version 1.8, picard version 2.15.0
    REMOVE_DUPLICATES=true
    ASSUME_SORTED=true
    Other parameters not set (default)

    ...
    INFO 2017-11-20 17:44:00 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:03:06s. Time for last 10,000,000: 186s. Last read position: 12:2,865,454
    INFO 2017-11-20 17:45:39 MarkDuplicates Before output close freeMemory: 2126725344; totalMemory: 2144862208; maxMemory: 2144862208
    INFO 2017-11-20 17:45:39 MarkDuplicates After output close freeMemory: 2126725344; totalMemory: 2144862208; maxMemory: 2144862208
    [Mon Nov 20 17:45:39 EST 2017] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 8.52 minutes.
    Runtime.totalMemory()=2144862208
    Exception in thread "Thread-1" Exception in thread "Thread-3" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /.../sortingcollection.4908715021524747428.tmp
    at htsjdk.samtools.util.IOUtil$DeletePathThread.run(IOUtil.java:374)
    Caused by: java.nio.file.NoSuchFileException: /.../sortingcollection.4908715021524747428.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at htsjdk.samtools.util.IOUtil$DeletePathThread.run(IOUtil.java:372)
    ...
    Same error for 2 more tmp files.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @amywilliams @vwboston
    Hi,

    I think your issue may be related to this thread. It looks like one of the user's in that thread reported using USE_JDK_DEFLATER=true USE_JDK_INFLATER=true to solve the issue. This thread may also help.

    -Sheila

  • amywilliamsamywilliams Ithaca, NYMember

    I just tried this and got the same error. It dies after only a few seconds. I would think that this is a separate issue than inflation/deflation as it seems to be unable to locate a file written to temporary space.

  • My feeling is that it's trying to remove tmp files that have already been removed.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @amywilliams @vwboston
    Hi,

    Hmm. Okay, let me check with the team and get back to you. Indeed if this works with a previous version but not the latest, it may be a bug so I may need some test files from you.

    -Sheila

  • amywilliamsamywilliams Ithaca, NYMember

    I just checked and version 2.14.1 seems to be working fine. At least, it's been running now for several minutes and hasn't crashed.

  • shleeshlee CambridgeMember, Broadie, Moderator

    Hi @amywilliams, Can you see if you get the same error when you do not setTMP_DIR=[/REDACTED/tmp] ?

    @vwboston, are you also setting a different location for the TMP_DIR? If so, can you also see if not setting this results in the same error? Thanks.

  • amywilliamsamywilliams Ithaca, NYMember
    edited November 2017

    @shlee, I just tried and got the same error (different file location of course).

    FYI, I'm using Picard 2.14.1 to do this. I assume nothing important changed in RevertSam between these versions, but please let me know if I should hold off.

  • I get the same error if I set TMP_DIR or not. I also tried setting java tmp dir by java -Djava.io.tmpdir, with or without setting TMP_DIR to the same location, all gave the same errors.

  • The error seems to only occur when I submit the job to the cluster. If I run it on a node directly via command line then everything is fine.

  • Tried again on the cluster, magically problem disappeared. Will report back if happens again.

  • Hi, I have the same problem when I use the FastqToSam to convert my fastqfiles to sam files, I use the latest picard version.

    I use the command line below:
    java -jar picard.jar FastqToSam F1=94210_CGTACTAG_S2_L001_R1_001.fastq F2=94210_CGTACTAG_S2_L001_R2_001.fastq O=fastq_to_bam.bam USE_SEQUENTIAL_FASTQS=true QUALITY_FORMAT=Illumina ALLOW_AND_IGNORE_EMPTY_LINES=true USE_JDK_DEFLATER=true USE_JDK_INFLATER=true SM=for_tool_testing

    And it has the bug like these
    at htsjdk.samtools.util.IOUtil$DeletePathThread.run(IOUtil.java:374)
    Caused by: java.nio.file.NoSuchFileException: /tmp/yingma/sortingcollection.2893939607853080593.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at htsjdk.samtools.util.IOUtil$DeletePathThread.run(IOUtil.java:372)
    Exception in thread "Thread-128" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /tmp/yingma/sortingcollection.2692803932741015812.tmp

    I don't know what are the problems. Could you please help me?

    Issue · Github
    by shlee

    Issue Number
    1000
    State
    closed
    Last Updated
    Assignee
    Array
    Closed By
    sooheelee
  • shleeshlee CambridgeMember, Broadie, Moderator
    edited November 2017

    @vwboston, glad to hear the issue magically solved itself for runs on the cluster.

    Since I assume @amywilliams still has the bug and now @yingma1993 is reporting this, I think it best to consult with developers what may be going on.

    Can you tell me if you have tried to run the error-producing command multiple times, and if so whether the NoSuchFileException occurs each time or occasionally? Since @vwboston says their commands succeed directly on a node and not cluster, I think a bug report will not enable us to recapitulate the exception. Rather, can you tell us the details of your system setup, e.g. clusters, servers, cloud, etc?

    Also, @yingma1993 can you tell us which version of Picard you are using?

    Thanks.

  • amywilliamsamywilliams Ithaca, NYMember

    @shlee, it happens every time I use Picard 2.15.0 and absent when I use version 2.14.1 (so I've switched to using that version)

  • shleeshlee CambridgeMember, Broadie, Moderator

    Hi @amywilliams and @yingma1993. Our developer confirms this is a bug that relates to code changes, specifically a version of htsjdk that Picard v2.15.0 is using. Please continue to use Picard v2.14.1 until a newer version of Picard is released. The newer version should use an updated version of htsjdk that fixes the bug.

  • insectopaloinsectopalo Bergen, NorwayMember

    Hi,

    The issue on GitHub was closed, but I'm still getting the same error with Picard 2.16. Not consistently, though, only for some samples.

    Thank you

  • shleeshlee CambridgeMember, Broadie, Moderator

    Hi @insectopalo,

    I'll check with the developer what may be going on with Picard v2.16. In the meanwhile, can you see if v2.14.1 gives you the bug with the samples that are erring?

  • shleeshlee CambridgeMember, Broadie, Moderator

    @insectopalo, the fixes have yet to appear in Picard, meaning the next Picard release is what will fix it, not the most current.

  • AmandezAmandez CanadaMember

    I have the same error with picard v2.17.3
    Thank you

    Issue · Github
    by shlee

    Issue Number
    2978
    State
    closed
    Last Updated
    Assignee
    Array
    Closed By
    chandrans
  • shleeshlee CambridgeMember, Broadie, Moderator

    @Amandez,

    Can you post your exact command that causes the error and the error stack trace? Thanks.

  • AmandezAmandez CanadaMember

    @shlee
    Thank you for your answer. The command is MergeSamFiles

    [Wed Feb 28 12:35:51 PST 2018] MergeSamFiles INPUT=[/home/bemmo/projects/rrg-jdrouin/bemmo/IAD/results/bwa/005.SS6.AB-Hu-Ex-A107/005.SS6.AB-Hu-Ex-A107.sam, /home/bemmo/projects/rrg-jdrouin/bemmo/IAD/results/bwa/006.SS6.AB-Hu-Ex-A107/006.SS6.AB-Hu-Ex-A107.sam, /home/bemmo/projects/rrg-jdrouin/bemmo/IAD/results/bwa/007.SS6.AB-Hu-Ex-A107/007.SS6.AB-Hu-Ex-A107.sam] OUTPUT=/home/bemmo/projects/rrg-jdrouin/bemmo/IAD/results/variants/AB-Hu-Ex-A107/AB-Hu-Ex-A107.bam USE_JDK_DEFLATER=true USE_JDK_INFLATER=true SORT_ORDER=coordinate ASSUME_SORTED=false MERGE_SEQUENCE_DICTIONARIES=false USE_THREADING=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
    [Wed Feb 28 12:35:51 PST 2018] Executing as bemmo@cdr920.int.cedar.computecanada.ca on Linux 3.10.0-693.11.6.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Deflater: Jdk; Inflater: Jdk; Picard version: 2.17.3-SNAPSHOT
    INFO 2018-02-28 12:35:51 MergeSamFiles Sorting input files using temp directory [/tmp/bemmo]
    INFO 2018-02-28 12:35:58 MergeSamFiles Processed 1,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 6s. Last read position: chr15:26,889,500
    INFO 2018-02-28 12:36:06 MergeSamFiles Processed 2,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 7s. Last read position: chr6:35,231,712
    INFO 2018-02-28 12:36:13 MergeSamFiles Processed 3,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 6s. Last read position: chr3:47,622,264
    INFO 2018-02-28 12:36:19 MergeSamFiles Processed 4,000,000 records. Elapsed time: 00:00:27s. Time for last 1,000,000: 6s. Last read position: chr7:15,613,034
    INFO 2018-02-28 12:36:26 MergeSamFiles Processed 5,000,000 records. Elapsed time: 00:00:34s. Time for last 1,000,000: 6s. Last read position: chr17:38,490,029
    INFO 2018-02-28 12:36:33 MergeSamFiles Processed 6,000,000 records. Elapsed time: 00:00:41s. Time for last 1,000,000: 7s. Last read position: chr15:81,358,297
    INFO 2018-02-28 12:36:40 MergeSamFiles Processed 7,000,000 records. Elapsed time: 00:00:48s. Time for last 1,000,000: 7s. Last read position: chr1:1,228,522
    INFO 2018-02-28 12:36:47 MergeSamFiles Processed 8,000,000 records. Elapsed time: 00:00:55s. Time for last 1,000,000: 7s. Last read position: chr6:87,589,803
    INFO 2018-02-28 12:36:55 MergeSamFiles Processed 9,000,000 records. Elapsed time: 00:01:03s. Time for last 1,000,000: 7s. Last read position: chr1:42,610,992
    INFO 2018-02-28 12:37:02 MergeSamFiles Processed 10,000,000 records. Elapsed time: 00:01:10s. Time for last 1,000,000: 6s. Last read position: chr1:235,728,122
    /var/spool/slurmd/job5614122/slurm_script: line 27: 42247 Killed java -jar /cvmfs/soft.mugqic/CentOS6/software/picard/picard-2.17.3/picard.jar MergeSamFiles $file O=/home/bemmo/projects/rrg-jdrouin/bemmo/IAD/results/variants/AB-Hu-Ex-A107/AB-Hu-Ex-A107.bam USE_JDK_DEFLATER=true USE_JDK_INFLATER=true
    12:37:10.830 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/cvmfs/soft.mugqic/CentOS6/software/picard/picard-2.17.3/picard.jar!/com/intel/gkl/native/libgkl_compression.so

    Issue · Github
    by shlee

    Issue Number
    1129
    State
    closed
    Last Updated
    Assignee
    Array
    Closed By
    sooheelee
  • LouisBLouisB Broad InstituteMember, Broadie, Dev

    What makes you think this is the same error? The only error condition I see in the output you posted is:

    /var/spool/slurmd/job5614122/slurm_script: line 27: 42247 Killed java -jar 
    

    That makes me think that your cluster killed the job for some reason. Maybe it went over a memory limit?

    If the job is producing a stacktrace that would be helpful to debug it.

  • shleeshlee CambridgeMember, Broadie, Moderator

    Thanks @Amandez. I'm consulting our developers on your line 27: 42247 Killed java -jar exception. In the meanwhile, it looks like you are running this tool on a SLURM cluster. Is that so? What kind of memory are you providing Java to run this tool?

  • mmterpstrammterpstra NetherlandsMember
    edited March 7

    This is a slurm error. Here is the documentation: https://www.slurm.schedmd.com/documentation.html

    This is a memory issue of slurm your job exeeds the default settings, i think the default java max vmsize can be really large thats why i advise you to change the settings to match my example (and maybe further reduce the amount of mem though trail and error).

    This needs to be set for java with -Xmx8g in your java command like java -Xmx8g -jar picard.jar .

    For slurm it is the --mem 10gb with plus 1 or 2 because overhead this can be set in your script with #SBATCH --mem 10gb or as option to srun.

    For the admin plz add the the starting post or the first reply to github as an issue. This one is just an slurm error.

  • shleeshlee CambridgeMember, Broadie, Moderator

    Thanks @mmterpstra for clarifying @Amandez's issue. @Amandez, please let us know if you run into an actual Picard tool error.

  • AmandezAmandez CanadaMember

    @shlee yes I run my command it is on a slurm cluster.
    @mmterpstra @LouisB thank you for your input. For slurm, I set --mem 12gb, but I didn't set the -Xmx parameter for java. I will increase this. Thank you

Sign In or Register to comment.