The current GATK version is 3.7-0
Examples: Monday, today, last week, Mar 26, 3/26/04

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Powered by Vanilla. Made with Bootstrap.
GATK 3.7 is here! Be sure to read the Version Highlights and optionally the full Release Notes.
Register now for the upcoming GATK Best Practices workshop, Feb 20-22 in Leuven, Belgium. Open to all comers! More info and signup at http://bit.ly/2i4mGxz

IlluminaData is missing error in Picard IlluminaBasecallsToFastq

dkotliardkotliar Cambridge, MAMember Posts: 3

I am using Picard 2.7.1 with Java 1.8 to demultiplex Illumina sequencing data from BCL to FASTQ. I am using the commands:

java -Xmx2000m -jar ExtractIlluminaBarcodes LANE=1 OUTPUT_DIR=path_to_output NUM_PROCESSORS=2 BASECALLS_DIR=path_to_basecalls read_structure=126T6B126T BARCODE_FILE=path_to_barcodes METRICS_FILE=path_to_metrics MAX_MISMATCHES=0

java -Xmx2000m -jar IlluminaBasecallsToFastq BARCODES_DIR=path_to_barcode_dir BASECALLS_DIR=path_to_basecalls LANE=1 read_structure=126T6B126T READ_NAME_FORMAT=ILLUMINA GZIP=true MAX_READS_IN_RAM_PER_TILE=1000000 MAX_RECORDS_IN_RAM=1000000 NUM_PROCESSORS=2 MULTIPLEX_PARAMS=path_to_multiplex_params

This works great for the vast majority of lanes. However for one lane (lane 3), I get the error message copied below:

lluminaData is missing in lane 3 at directory location path_to_basecalls/L003 (full trace at the end). If I run CheckIlluminaDirectory:

java -jar ~/Desktop/picard.jar CheckIlluminaDirectory BASECALLS_DIR=path_to_basecalls LANES=3 READ_STRUCTURE=126T6B126T

I get the output that everything is fine:

INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected cycles: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Checking lane 3
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected tiles: 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 2310, 2311, 2312, 2313, 2314, 2315, 2316
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory Lane 3 SUCCEEDED
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory SUCCEEDED! All required files are present and non-empty.
[Mon Nov 28 14:20:12 EST 2016] picard.illumina.CheckIlluminaDirectory done. Elapsed time: 0.06 minutes.
Runtime.totalMemory()=128974848

Any idea what is going on or what I can do to fix it?

Thanks,
Dylan


ERROR 2016-11-28 14:16:23 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:709)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:318)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-1" java.util.NoSuchElementException: IlluminaData is missing in lane 3 at directory location path_to_basecalls/L003
at picard.illumina.parser.PerTileCycleParser.next(PerTileCycleParser.java:150)
at picard.illumina.parser.BclParser.next(BclParser.java:90)
at picard.illumina.parser.BclParser.next(BclParser.java:44)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[Mon Nov 28 14:16:23 UTC 2016] picard.illumina.IlluminaBasecallsToFastq done. Elapsed time: 0.78 minutes.
Runtime.totalMemory()=1385693184
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:321)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-2" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1102_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.next(AbstractIterator.java:57)
at picard.illumina.parser.readers.BarcodeFileReader.next(BarcodeFileReader.java:29)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator$1.getBarcode(BarcodeParser.java:75)
at picard.illumina.parser.IlluminaDataProvider.addData(IlluminaDataProvider.java:178)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:145)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 12 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 13 more
Exception in thread "pool-1-thread-3" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1103_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.hasNext(AbstractIterator.java:44)
at picard.illumina.parser.readers.BarcodeFileReader.hasNext(BarcodeFileReader.java:42)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator.hasNext(BarcodeParser.java:69)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:120)
at picard.illumina.parser.PerTileParser.maybeAdvance(PerTileParser.java:99)
at picard.illumina.parser.PerTileParser.next(PerTileParser.java:109)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 15 more

Issue · Github
by Sheila

Issue Number
1488
State
open
Last Updated
Assignee
Array
Milestone
Array

Answers

  • dkotliardkotliar Cambridge, MAMember Posts: 3

    Update that I get the same error with IlluminaBasecallsToSam:

    java -jar -Xmx4000m ~/Desktop/picard.jar IlluminaBasecallsToSam BASECALLS_DIR=path_to_basecalls LANE=003 READ_STRUCTURE=126T6B126T RUN_BARCODE=XXX BARCODES_DIR=path_to_barcodes LIBRARY_PARAMS=path_to_library_params_L003.txt

    it happens around reading 102,000,000 records

  • EADGEADG KielMember Posts: 75

    HI dkotliar,

    did you try demultiplexing it with bcl2fastq ? Maybe you get a clue from the illumnia demultiplexer what is wrong (ot not) with your file.

    Greetings EADG

  • dkotliardkotliar Cambridge, MAMember Posts: 3

    Hi EADG and thanks for the response. I run bcl2fastq v2.18 with the following command:
    ./bcl2fastq --runfolder-dir path_to_runfolder --output-dir path_to_outdir --tiles s_3

    and get the error copied at the bottom. Does this mean that one of the files in Intensities/Basecalls/L003/C78.1 is corrupted? Is there anyway to continue nevertheless?

    Thanks,
    Dylan

    2016-11-30 01:08:32 [7f489c0b3700] ERROR: Thread: 0 caught an exception first: /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

    2016-11-30 01:08:32 [12e9880] WARNING: Rethrowing a thread exception
    2016-11-30 01:08:32 [12e9880] ERROR: bcl2fastq::common::Exception: 2016-Nov-30 01:08:32: Success (0): /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

  • EADGEADG KielMember Posts: 75

    Hi Dylan,

    can you ensure that your data is not corrupted ? Can you compare the hash-value for the files on the sequencer and your "local" (copied) files ?

    Greetings

    EADG

Sign In or Register to comment.