Service notice: Several of our team members are on vacation so service will be slow through at least July 13th, possibly longer depending on how much backlog accumulates during that time. This means that for a while it may take us more time than usual to answer your questions. Thank you for your patience.

IlluminaData is missing error in Picard IlluminaBasecallsToFastq

dkotliardkotliar Cambridge, MAMember

I am using Picard 2.7.1 with Java 1.8 to demultiplex Illumina sequencing data from BCL to FASTQ. I am using the commands:

java -Xmx2000m -jar ExtractIlluminaBarcodes LANE=1 OUTPUT_DIR=path_to_output NUM_PROCESSORS=2 BASECALLS_DIR=path_to_basecalls read_structure=126T6B126T BARCODE_FILE=path_to_barcodes METRICS_FILE=path_to_metrics MAX_MISMATCHES=0

java -Xmx2000m -jar IlluminaBasecallsToFastq BARCODES_DIR=path_to_barcode_dir BASECALLS_DIR=path_to_basecalls LANE=1 read_structure=126T6B126T READ_NAME_FORMAT=ILLUMINA GZIP=true MAX_READS_IN_RAM_PER_TILE=1000000 MAX_RECORDS_IN_RAM=1000000 NUM_PROCESSORS=2 MULTIPLEX_PARAMS=path_to_multiplex_params

This works great for the vast majority of lanes. However for one lane (lane 3), I get the error message copied below:

lluminaData is missing in lane 3 at directory location path_to_basecalls/L003 (full trace at the end). If I run CheckIlluminaDirectory:

java -jar ~/Desktop/picard.jar CheckIlluminaDirectory BASECALLS_DIR=path_to_basecalls LANES=3 READ_STRUCTURE=126T6B126T

I get the output that everything is fine:

INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected cycles: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Checking lane 3
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected tiles: 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 2310, 2311, 2312, 2313, 2314, 2315, 2316
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory Lane 3 SUCCEEDED
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory SUCCEEDED! All required files are present and non-empty.
[Mon Nov 28 14:20:12 EST 2016] picard.illumina.CheckIlluminaDirectory done. Elapsed time: 0.06 minutes.
Runtime.totalMemory()=128974848

Any idea what is going on or what I can do to fix it?

Thanks,
Dylan


ERROR 2016-11-28 14:16:23 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:709)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:318)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-1" java.util.NoSuchElementException: IlluminaData is missing in lane 3 at directory location path_to_basecalls/L003
at picard.illumina.parser.PerTileCycleParser.next(PerTileCycleParser.java:150)
at picard.illumina.parser.BclParser.next(BclParser.java:90)
at picard.illumina.parser.BclParser.next(BclParser.java:44)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[Mon Nov 28 14:16:23 UTC 2016] picard.illumina.IlluminaBasecallsToFastq done. Elapsed time: 0.78 minutes.
Runtime.totalMemory()=1385693184
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:321)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-2" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1102_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.next(AbstractIterator.java:57)
at picard.illumina.parser.readers.BarcodeFileReader.next(BarcodeFileReader.java:29)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator$1.getBarcode(BarcodeParser.java:75)
at picard.illumina.parser.IlluminaDataProvider.addData(IlluminaDataProvider.java:178)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:145)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 12 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 13 more
Exception in thread "pool-1-thread-3" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1103_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.hasNext(AbstractIterator.java:44)
at picard.illumina.parser.readers.BarcodeFileReader.hasNext(BarcodeFileReader.java:42)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator.hasNext(BarcodeParser.java:69)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:120)
at picard.illumina.parser.PerTileParser.maybeAdvance(PerTileParser.java:99)
at picard.illumina.parser.PerTileParser.next(PerTileParser.java:109)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 15 more

Issue · Github
by Sheila

Issue Number
1488
State
closed
Last Updated
Assignee
Array
Milestone
Array
Closed By
vdauwera

Answers

  • dkotliardkotliar Cambridge, MAMember

    Update that I get the same error with IlluminaBasecallsToSam:

    java -jar -Xmx4000m ~/Desktop/picard.jar IlluminaBasecallsToSam BASECALLS_DIR=path_to_basecalls LANE=003 READ_STRUCTURE=126T6B126T RUN_BARCODE=XXX BARCODES_DIR=path_to_barcodes LIBRARY_PARAMS=path_to_library_params_L003.txt

    it happens around reading 102,000,000 records

  • EADGEADG KielMember

    HI dkotliar,

    did you try demultiplexing it with bcl2fastq ? Maybe you get a clue from the illumnia demultiplexer what is wrong (ot not) with your file.

    Greetings EADG

  • dkotliardkotliar Cambridge, MAMember

    Hi EADG and thanks for the response. I run bcl2fastq v2.18 with the following command:
    ./bcl2fastq --runfolder-dir path_to_runfolder --output-dir path_to_outdir --tiles s_3

    and get the error copied at the bottom. Does this mean that one of the files in Intensities/Basecalls/L003/C78.1 is corrupted? Is there anyway to continue nevertheless?

    Thanks,
    Dylan

    2016-11-30 01:08:32 [7f489c0b3700] ERROR: Thread: 0 caught an exception first: /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

    2016-11-30 01:08:32 [12e9880] WARNING: Rethrowing a thread exception
    2016-11-30 01:08:32 [12e9880] ERROR: bcl2fastq::common::Exception: 2016-Nov-30 01:08:32: Success (0): /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

  • EADGEADG KielMember

    Hi Dylan,

    can you ensure that your data is not corrupted ? Can you compare the hash-value for the files on the sequencer and your "local" (copied) files ?

    Greetings

    EADG

Sign In or Register to comment.