Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

IlluminaBasecallsToSam crashing

I have successfully run ExtractIlluminaBarcodes but when I run IlluminaBasecallsToSam on the same data it crashes. I have tried upping both MAX_READS_IN_RAM_PER_TILE and MAX_RECORDS_IN_RAM to no avail.

Thanks,

Adam

Here is the call:

java -jar /mnt/disk4/labs/salipante/programs/picard-tools-2.17.2/picard-2.17.2.jar IlluminaBasecallsToSam BASECALLS_DIR=/mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls LANE=001 READ_STRUCTURE=4M145T10B10B4M145T RUN_BARCODE=runxx IGNORE_UNEXPECTED_BARCODES=true LIBRARY_PARAMS=library.params

Here is library.params:
BARCODE_1 BARCODE_2 OUTPUT SAMPLE_ALIAS LIBRARY_NAME
AAGGTTAAGT GAGCGCTCGC Colon-POS-19.bam Colon-POS-19 lib
AATACCTGCC ACGATCCGTA Colon-NEG-22.bam Colon-NEG-22 lib
ACGGACCTGC TCCGTCCAAG Colon-POS-14.bam Colon-POS-14 lib
AGGCAGTACC AGGTCGAACC EndoPos22.bam EndoPos22 lib2
AGTCGCGTCG AACTAGAGAC LS174t_01_c.bam LS174t_01_c lib3
ATCCGGATAA TGACCTACGA LS174t_001_b.bam LS174t_001_b lib3
ATTGCATAGT GTTGCCTCGG LS174t_01_b.bam LS174t_01_b lib3
CCTATTGAGT ATGGATCATC EndoNeg08.bam EndoNeg08 lib2
CGAGGCCGGA TGCTGACTTG OTHER-POS-01.bam OTHER-POS-01 lib4
CGCTACGATT TGGTTCTGGT EndoPos17.bam EndoPos17 lib2
CTAATATGAT TTGGAATTCC NA12878.bam NA12878 lib3
CTAGCTTCTT AGTAACGAGT Endopos10.bam Endopos10 lib2
CTTCTCTTAG TGCTATGATG EndoNeg07.bam EndoNeg07 lib2
GAATGAATAA CCAGCAATGG Colon-NEG-19.bam Colon-NEG-19 lib
GAGCTCAGCC TCTTCGGAGT Colon-POS-31.bam Colon-POS-31 lib
GCGGCGCAAG GTATTGCCAT EndoPos11.bam EndoPos11 lib2
GCGTAAGATC ACTGCGAATA OTHER-POS-02.bam OTHER-POS-02 lib4
GGATTCTCGG TCCGGCAGCA Colon-NEG-16.bam Colon-NEG-16 lib
GTTAATTCTA TTCGCTGCCA Colon-NEG-29.bam Colon-NEG-29 lib
GTTCCGTATC TATAACCAGC LS174t_001_a.bam LS174t_001_a lib3
TATTACTCTA CGTCTGAGTT LS174t_001_c.bam LS174t_001_c lib3
TGGCTGACCA TGAAGAGCTC LS174t_01_a.bam LS174t_01_a lib3
TTATAGCTCT GAGCCTTATT EndoNeg02.bam EndoNeg02 lib2
N N non_indexed.bam non_indexed lib6

Here are the first page of errors:
14:43:14.522 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/disk4/labs/salipante/programs/picard-tools-2.17.2/picard-2.17.2.jar!/com/intel/gkl/native/libgkl_compression.so
[Mon Jan 08 14:43:14 PST 2018] IlluminaBasecallsToSam BASECALLS_DIR=/mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls LANE=1 RUN_BARCODE=runxx READ_STRUCTURE=4M145T10B10B4M145T LIBRARY_PARAMS=library.params IGNORE_UNEXPECTED_BARCODES=true SEQUENCING_CENTER=BI PLATFORM=illumina ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2, FLUIDIGM] NUM_PROCESSORS=0 FORCE_GC=true APPLY_EAMSS_FILTER=true MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 INCLUDE_NON_PF_READS=true MOLECULAR_INDEX_TAG=RX MOLECULAR_INDEX_BASE_QUALITY_TAG=QX VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Mon Jan 08 14:43:14 PST 2018] Executing as [email protected] on Linux 4.4.0-45-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12; Deflater: Intel; Inflater: Intel; Picard version: 2.17.2-SNAPSHOT
INFO 2018-01-08 14:43:14 IlluminaBasecallsToSam DONE_READING STRUCTURE IS 4M145T10B10B4M145T
ERROR 2018-01-08 14:43:17 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:612)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:237)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:267)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:269)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)
Exception in thread "pool-2-thread-69" Exception in thread "pool-2-thread-5" htsjdk.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls/L001/0045.bcl.bgzf (Too many open files)
at htsjdk.samtools.util.IOUtil.maybeBufferedSeekableStream(IOUtil.java:177)
at picard.illumina.parser.readers.BaseBclReader.open(BaseBclReader.java:66)
at picard.illumina.parser.readers.BclReader.(BclReader.java:94)
at picard.illumina.parser.readers.BclReader.makeSeekable(BclReader.java:259)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:59)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:134)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:77)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:133)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:433)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:289)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:466)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$1.run(IlluminaBasecallsConverter.java:563)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls/L001/0045.bcl.bgzf (Too many open files)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at htsjdk.samtools.seekablestream.SeekableFileStream.(SeekableFileStream.java:47)
at htsjdk.samtools.util.IOUtil.maybeBufferedSeekableStream(IOUtil.java:175)
... 19 more
Exception in thread "pool-2-thread-46" Exception in thread "pool-2-thread-63" Exception in thread "pool-2-thread-44" Exception in thread "pool-2-thread-22" Exception in thread "pool-2-thread-11" htsjdk.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /mnt/disk4/labs/salipante/data/sequencing_runs/171120_NS500359_0227_AHVHYKAFXX/Data/Intensities/BaseCalls/L001/0069.bcl.bgzf (Too many open files)
at htsjdk.samtools.util.IOUtil.maybeBufferedSeekableStream(IOUtil.java:177)
at picard.illumina.parser.readers.BaseBclReader.open(BaseBclReader.java:66)
at picard.illumina.parser.readers.BclReader.(BclReader.java:94)
at picard.illumina.parser.readers.BclReader.makeSeekable(BclReader.java:259)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:59)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:134)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:77)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:133)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)

Best Answer

Answers

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    ulimit is usually problematic unless you are the root user. You may 'su' yourself and become a temporary root for file limit therefore ulimit becomes unlimited. I solve my too many files open problems with su. Beware greater privileges come with greater responsibility!

Sign In or Register to comment.