IlluminaData is missing error in Picard IlluminaBasecallsToFastq

dkotliardkotliar Cambridge, MAMember

I am using Picard 2.7.1 with Java 1.8 to demultiplex Illumina sequencing data from BCL to FASTQ. I am using the commands:

java -Xmx2000m -jar ExtractIlluminaBarcodes LANE=1 OUTPUT_DIR=path_to_output NUM_PROCESSORS=2 BASECALLS_DIR=path_to_basecalls read_structure=126T6B126T BARCODE_FILE=path_to_barcodes METRICS_FILE=path_to_metrics MAX_MISMATCHES=0

java -Xmx2000m -jar IlluminaBasecallsToFastq BARCODES_DIR=path_to_barcode_dir BASECALLS_DIR=path_to_basecalls LANE=1 read_structure=126T6B126T READ_NAME_FORMAT=ILLUMINA GZIP=true MAX_READS_IN_RAM_PER_TILE=1000000 MAX_RECORDS_IN_RAM=1000000 NUM_PROCESSORS=2 MULTIPLEX_PARAMS=path_to_multiplex_params

This works great for the vast majority of lanes. However for one lane (lane 3), I get the error message copied below:

lluminaData is missing in lane 3 at directory location path_to_basecalls/L003 (full trace at the end). If I run CheckIlluminaDirectory:

java -jar ~/Desktop/picard.jar CheckIlluminaDirectory BASECALLS_DIR=path_to_basecalls LANES=3 READ_STRUCTURE=126T6B126T

I get the output that everything is fine:

INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected cycles: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Checking lane 3
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected tiles: 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 2310, 2311, 2312, 2313, 2314, 2315, 2316
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory Lane 3 SUCCEEDED
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory SUCCEEDED! All required files are present and non-empty.
[Mon Nov 28 14:20:12 EST 2016] picard.illumina.CheckIlluminaDirectory done. Elapsed time: 0.06 minutes.
Runtime.totalMemory()=128974848

Any idea what is going on or what I can do to fix it?

Thanks,
Dylan


ERROR 2016-11-28 14:16:23 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:709)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:318)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-1" java.util.NoSuchElementException: IlluminaData is missing in lane 3 at directory location path_to_basecalls/L003
at picard.illumina.parser.PerTileCycleParser.next(PerTileCycleParser.java:150)
at picard.illumina.parser.BclParser.next(BclParser.java:90)
at picard.illumina.parser.BclParser.next(BclParser.java:44)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[Mon Nov 28 14:16:23 UTC 2016] picard.illumina.IlluminaBasecallsToFastq done. Elapsed time: 0.78 minutes.
Runtime.totalMemory()=1385693184
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:321)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-2" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1102_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.next(AbstractIterator.java:57)
at picard.illumina.parser.readers.BarcodeFileReader.next(BarcodeFileReader.java:29)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator$1.getBarcode(BarcodeParser.java:75)
at picard.illumina.parser.IlluminaDataProvider.addData(IlluminaDataProvider.java:178)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:145)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 12 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 13 more
Exception in thread "pool-1-thread-3" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1103_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.hasNext(AbstractIterator.java:44)
at picard.illumina.parser.readers.BarcodeFileReader.hasNext(BarcodeFileReader.java:42)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator.hasNext(BarcodeParser.java:69)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:120)
at picard.illumina.parser.PerTileParser.maybeAdvance(PerTileParser.java:99)
at picard.illumina.parser.PerTileParser.next(PerTileParser.java:109)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 15 more

Issue · Github
by Sheila

Issue Number
1488
State
closed
Last Updated
Assignee
Array
Milestone
Array
Closed By
vdauwera

Answers

  • dkotliardkotliar Cambridge, MAMember

    Update that I get the same error with IlluminaBasecallsToSam:

    java -jar -Xmx4000m ~/Desktop/picard.jar IlluminaBasecallsToSam BASECALLS_DIR=path_to_basecalls LANE=003 READ_STRUCTURE=126T6B126T RUN_BARCODE=XXX BARCODES_DIR=path_to_barcodes LIBRARY_PARAMS=path_to_library_params_L003.txt

    it happens around reading 102,000,000 records

  • EADGEADG KielMember ✭✭✭

    HI dkotliar,

    did you try demultiplexing it with bcl2fastq ? Maybe you get a clue from the illumnia demultiplexer what is wrong (ot not) with your file.

    Greetings EADG

  • dkotliardkotliar Cambridge, MAMember

    Hi EADG and thanks for the response. I run bcl2fastq v2.18 with the following command:
    ./bcl2fastq --runfolder-dir path_to_runfolder --output-dir path_to_outdir --tiles s_3

    and get the error copied at the bottom. Does this mean that one of the files in Intensities/Basecalls/L003/C78.1 is corrupted? Is there anyway to continue nevertheless?

    Thanks,
    Dylan

    2016-11-30 01:08:32 [7f489c0b3700] ERROR: Thread: 0 caught an exception first: /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

    2016-11-30 01:08:32 [12e9880] WARNING: Rethrowing a thread exception
    2016-11-30 01:08:32 [12e9880] ERROR: bcl2fastq::common::Exception: 2016-Nov-30 01:08:32: Success (0): /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

  • EADGEADG KielMember ✭✭✭

    Hi Dylan,

    can you ensure that your data is not corrupted ? Can you compare the hash-value for the files on the sequencer and your "local" (copied) files ?

    Greetings

    EADG

Sign In or Register to comment.