We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

IlluminaBasecallsToSam crashing

I have successfully run ExtractIlluminaBarcodes but when I run IlluminaBasecallsToSam on the same data it crashes. I have tried upping both MAX_READS_IN_RAM_PER_TILE and MAX_RECORDS_IN_RAM to no avail.

Thanks,

Adam

Here is the call:

java -jar /mnt/disk4/labs/salipante/programs/picard-tools-2.17.2/picard-2.17.2.jar IlluminaBasecallsToSam BASECALLS_DIR=/mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls LANE=001 READ_STRUCTURE=4M145T10B10B4M145T RUN_BARCODE=runxx IGNORE_UNEXPECTED_BARCODES=true LIBRARY_PARAMS=library.params

Here is library.params:
BARCODE_1 BARCODE_2 OUTPUT SAMPLE_ALIAS LIBRARY_NAME
AAGGTTAAGT GAGCGCTCGC Colon-POS-19.bam Colon-POS-19 lib
AATACCTGCC ACGATCCGTA Colon-NEG-22.bam Colon-NEG-22 lib
ACGGACCTGC TCCGTCCAAG Colon-POS-14.bam Colon-POS-14 lib
AGGCAGTACC AGGTCGAACC EndoPos22.bam EndoPos22 lib2
AGTCGCGTCG AACTAGAGAC LS174t_01_c.bam LS174t_01_c lib3
ATCCGGATAA TGACCTACGA LS174t_001_b.bam LS174t_001_b lib3
ATTGCATAGT GTTGCCTCGG LS174t_01_b.bam LS174t_01_b lib3
CCTATTGAGT ATGGATCATC EndoNeg08.bam EndoNeg08 lib2
CGAGGCCGGA TGCTGACTTG OTHER-POS-01.bam OTHER-POS-01 lib4
CGCTACGATT TGGTTCTGGT EndoPos17.bam EndoPos17 lib2
CTAATATGAT TTGGAATTCC NA12878.bam NA12878 lib3
CTAGCTTCTT AGTAACGAGT Endopos10.bam Endopos10 lib2
CTTCTCTTAG TGCTATGATG EndoNeg07.bam EndoNeg07 lib2
GAATGAATAA CCAGCAATGG Colon-NEG-19.bam Colon-NEG-19 lib
GAGCTCAGCC TCTTCGGAGT Colon-POS-31.bam Colon-POS-31 lib
GCGGCGCAAG GTATTGCCAT EndoPos11.bam EndoPos11 lib2
GCGTAAGATC ACTGCGAATA OTHER-POS-02.bam OTHER-POS-02 lib4
GGATTCTCGG TCCGGCAGCA Colon-NEG-16.bam Colon-NEG-16 lib
GTTAATTCTA TTCGCTGCCA Colon-NEG-29.bam Colon-NEG-29 lib
GTTCCGTATC TATAACCAGC LS174t_001_a.bam LS174t_001_a lib3
TATTACTCTA CGTCTGAGTT LS174t_001_c.bam LS174t_001_c lib3
TGGCTGACCA TGAAGAGCTC LS174t_01_a.bam LS174t_01_a lib3
TTATAGCTCT GAGCCTTATT EndoNeg02.bam EndoNeg02 lib2
N N non_indexed.bam non_indexed lib6

Here are the first page of errors:
14:43:14.522 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/disk4/labs/salipante/programs/picard-tools-2.17.2/picard-2.17.2.jar!/com/intel/gkl/native/libgkl_compression.so
[Mon Jan 08 14:43:14 PST 2018] IlluminaBasecallsToSam BASECALLS_DIR=/mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls LANE=1 RUN_BARCODE=runxx READ_STRUCTURE=4M145T10B10B4M145T LIBRARY_PARAMS=library.params IGNORE_UNEXPECTED_BARCODES=true SEQUENCING_CENTER=BI PLATFORM=illumina ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2, FLUIDIGM] NUM_PROCESSORS=0 FORCE_GC=true APPLY_EAMSS_FILTER=true MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 INCLUDE_NON_PF_READS=true MOLECULAR_INDEX_TAG=RX MOLECULAR_INDEX_BASE_QUALITY_TAG=QX VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Mon Jan 08 14:43:14 PST 2018] Executing as [email protected] on Linux 4.4.0-45-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12; Deflater: Intel; Inflater: Intel; Picard version: 2.17.2-SNAPSHOT
INFO 2018-01-08 14:43:14 IlluminaBasecallsToSam DONE_READING STRUCTURE IS 4M145T10B10B4M145T
ERROR 2018-01-08 14:43:17 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:612)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:237)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:267)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:269)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)
Exception in thread "pool-2-thread-69" Exception in thread "pool-2-thread-5" htsjdk.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls/L001/0045.bcl.bgzf (Too many open files)
at htsjdk.samtools.util.IOUtil.maybeBufferedSeekableStream(IOUtil.java:177)
at picard.illumina.parser.readers.BaseBclReader.open(BaseBclReader.java:66)
at picard.illumina.parser.readers.BclReader.(BclReader.java:94)
at picard.illumina.parser.readers.BclReader.makeSeekable(BclReader.java:259)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:59)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:134)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:77)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:133)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:433)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:289)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:466)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$1.run(IlluminaBasecallsConverter.java:563)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls/L001/0045.bcl.bgzf (Too many open files)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at htsjdk.samtools.seekablestream.SeekableFileStream.(SeekableFileStream.java:47)
at htsjdk.samtools.util.IOUtil.maybeBufferedSeekableStream(IOUtil.java:175)
... 19 more
Exception in thread "pool-2-thread-46" Exception in thread "pool-2-thread-63" Exception in thread "pool-2-thread-44" Exception in thread "pool-2-thread-22" Exception in thread "pool-2-thread-11" htsjdk.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls/L001/0069.bcl.bgzf (Too many open files)
at htsjdk.samtools.util.IOUtil.maybeBufferedSeekableStream(IOUtil.java:177)
at picard.illumina.parser.readers.BaseBclReader.open(BaseBclReader.java:66)
at picard.illumina.parser.readers.BclReader.(BclReader.java:94)
at picard.illumina.parser.readers.BclReader.makeSeekable(BclReader.java:259)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:59)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:134)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:77)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:133)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)

Best Answer

Answers

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    ulimit is usually problematic unless you are the root user. You may 'su' yourself and become a temporary root for file limit therefore ulimit becomes unlimited. I solve my too many files open problems with su. Beware greater privileges come with greater responsibility!

Sign In or Register to comment.