The current GATK version is 3.7-0
Examples: Monday, today, last week, Mar 26, 3/26/04

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Get notifications!


You can opt in to receive email notifications, for example when your questions get answered or when there are new announcements, by following the instructions given here.

Did you remember to?


1. Search using the upper-right search box, e.g. using the error message.
2. Try the latest version of tools.
3. Include tool and Java versions.
4. Tell us whether you are following GATK Best Practices.
5. Include relevant details, e.g. platform, DNA- or RNA-Seq, WES (+capture kit) or WGS (PCR-free or PCR+), paired- or single-end, read length, expected average coverage, somatic data, etc.
6. For tool errors, include the error stacktrace as well as the exact command.
7. For format issues, include the result of running ValidateSamFile for BAMs or ValidateVariants for VCFs.
8. For weird results, include an illustrative example, e.g. attach IGV screenshots according to Article#5484.
9. For a seeming variant that is uncalled, include results of following Article#1235.

Did we ask for a bug report?


Then follow instructions in Article#1894.

Formatting tip!


Wrap blocks of code, error messages and BAM/VCF snippets--especially content with hashes (#)--with lines with three backticks ( ``` ) each to make a code block as demonstrated here.

Jump to another community
Picard 2.9.0 is now available. Download and read release notes here.
GATK 3.7 is here! Be sure to read the Version Highlights and optionally the full Release Notes.

IlluminaData is missing error in Picard IlluminaBasecallsToFastq

dkotliardkotliar Cambridge, MAPosts: 3

I am using Picard 2.7.1 with Java 1.8 to demultiplex Illumina sequencing data from BCL to FASTQ. I am using the commands:

java -Xmx2000m -jar ExtractIlluminaBarcodes LANE=1 OUTPUT_DIR=path_to_output NUM_PROCESSORS=2 BASECALLS_DIR=path_to_basecalls read_structure=126T6B126T BARCODE_FILE=path_to_barcodes METRICS_FILE=path_to_metrics MAX_MISMATCHES=0

java -Xmx2000m -jar IlluminaBasecallsToFastq BARCODES_DIR=path_to_barcode_dir BASECALLS_DIR=path_to_basecalls LANE=1 read_structure=126T6B126T READ_NAME_FORMAT=ILLUMINA GZIP=true MAX_READS_IN_RAM_PER_TILE=1000000 MAX_RECORDS_IN_RAM=1000000 NUM_PROCESSORS=2 MULTIPLEX_PARAMS=path_to_multiplex_params

This works great for the vast majority of lanes. However for one lane (lane 3), I get the error message copied below:

lluminaData is missing in lane 3 at directory location path_to_basecalls/L003 (full trace at the end). If I run CheckIlluminaDirectory:

java -jar ~/Desktop/picard.jar CheckIlluminaDirectory BASECALLS_DIR=path_to_basecalls LANES=3 READ_STRUCTURE=126T6B126T

I get the output that everything is fine:

INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected cycles: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Checking lane 3
INFO 2016-11-28 14:20:09 CheckIlluminaDirectory Expected tiles: 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 2310, 2311, 2312, 2313, 2314, 2315, 2316
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory Lane 3 SUCCEEDED
INFO 2016-11-28 14:20:12 CheckIlluminaDirectory SUCCEEDED! All required files are present and non-empty.
[Mon Nov 28 14:20:12 EST 2016] picard.illumina.CheckIlluminaDirectory done. Elapsed time: 0.06 minutes.
Runtime.totalMemory()=128974848

Any idea what is going on or what I can do to fix it?

Thanks,
Dylan


ERROR 2016-11-28 14:16:23 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:709)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:318)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-1" java.util.NoSuchElementException: IlluminaData is missing in lane 3 at directory location path_to_basecalls/L003
at picard.illumina.parser.PerTileCycleParser.next(PerTileCycleParser.java:150)
at picard.illumina.parser.BclParser.next(BclParser.java:90)
at picard.illumina.parser.BclParser.next(BclParser.java:44)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[Mon Nov 28 14:16:23 UTC 2016] picard.illumina.IlluminaBasecallsToFastq done. Elapsed time: 0.78 minutes.
Runtime.totalMemory()=1385693184
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:321)
at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:221)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-2" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1102_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.next(AbstractIterator.java:57)
at picard.illumina.parser.readers.BarcodeFileReader.next(BarcodeFileReader.java:29)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator$1.getBarcode(BarcodeParser.java:75)
at picard.illumina.parser.IlluminaDataProvider.addData(IlluminaDataProvider.java:178)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:145)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 12 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 13 more
Exception in thread "pool-1-thread-3" picard.PicardException: Error reading from file path_to_barcodes/L003/s_3_1103_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.hasNext(AbstractIterator.java:44)
at picard.illumina.parser.readers.BarcodeFileReader.hasNext(BarcodeFileReader.java:42)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator.hasNext(BarcodeParser.java:69)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:120)
at picard.illumina.parser.PerTileParser.maybeAdvance(PerTileParser.java:99)
at picard.illumina.parser.PerTileParser.next(PerTileParser.java:109)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 15 more

Issue · Github
by Sheila

Issue Number
1488
State
open
Last Updated
Assignee
Array
Milestone
Array

Answers

  • dkotliardkotliar Cambridge, MAPosts: 3

    Update that I get the same error with IlluminaBasecallsToSam:

    java -jar -Xmx4000m ~/Desktop/picard.jar IlluminaBasecallsToSam BASECALLS_DIR=path_to_basecalls LANE=003 READ_STRUCTURE=126T6B126T RUN_BARCODE=XXX BARCODES_DIR=path_to_barcodes LIBRARY_PARAMS=path_to_library_params_L003.txt

    it happens around reading 102,000,000 records

  • EADGEADG KielPosts: 109 ✭✭

    HI dkotliar,

    did you try demultiplexing it with bcl2fastq ? Maybe you get a clue from the illumnia demultiplexer what is wrong (ot not) with your file.

    Greetings EADG

  • dkotliardkotliar Cambridge, MAPosts: 3

    Hi EADG and thanks for the response. I run bcl2fastq v2.18 with the following command:
    ./bcl2fastq --runfolder-dir path_to_runfolder --output-dir path_to_outdir --tiles s_3

    and get the error copied at the bottom. Does this mean that one of the files in Intensities/Basecalls/L003/C78.1 is corrupted? Is there anyway to continue nevertheless?

    Thanks,
    Dylan

    2016-11-30 01:08:32 [7f489c0b3700] ERROR: Thread: 0 caught an exception first: /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

    2016-11-30 01:08:32 [12e9880] WARNING: Rethrowing a thread exception
    2016-11-30 01:08:32 [12e9880] ERROR: bcl2fastq::common::Exception: 2016-Nov-30 01:08:32: Success (0): /TeamCityBuildAgent/work/ca410b6b30c4ffb6/src/cxx/lib/conversion/BclLoader.cpp(613): Throw in function void bcl2fastq::conversion::BclLoader::bclMismatchCount(std::string, bcl2fastq::common::CycleNumber, std::vector::size_type, std::vector::size_type)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: Mismatching cluster count in BCL file: Cycle #78: bytes_real=894226 bytes_expected=1223444

  • EADGEADG KielPosts: 109 ✭✭

    Hi Dylan,

    can you ensure that your data is not corrupted ? Can you compare the hash-value for the files on the sequencer and your "local" (copied) files ?

    Greetings

    EADG

Sign In or Register to comment.