IlluminaBasecallsToSam Fails, trying to figure out why

concon PennsylvaniaMember

Hello,

I ran ExtractIlluminaBarcodes and came out with the following metrics file:

METRICS CLASS picard.illumina.ExtractIlluminaBarcodes$BarcodeMetric

BARCODE BARCODE_NAME LIBRARY_NAME READS PF_READS PERFECT_MATCHES PF_PERFECT_MATCHES ONE_MISMATCH_MATCHES PF_ONE_MISMATCH_MATCHES PCT_MATCHES RATIO_THIS_BARCODE_TO_BEST_BARCODE_PCT PF_PCT_MATCHES PF_RATIO_THIS_BARCODE_TO_BEST_BARCODE_PCT PF_NORMALIZED_MATCHES
TAAGGC 3787945 3787945 3283555 3283555 504390 504390 0.0245 1 0.024741 1 1.283624
AGGCAG 2902034 2902034 2473189 2473189 428845 428845 0.01877 0.766124 0.018955 0.766124 0.983415
CGTACT 2162953 2162953 1869231 1869231 293722 293722 0.01399 0.57101 0.014127 0.57101 0.732962
NNNNNN 145758436 144249478 0 0 0 0 0.942741 38.479554 0.942176 38.081197 0

now I need to run IlluminaBasecallsToSam, with

java -jar 3rdParty/picard/Old_picard.jar IlluminaBasecallsToSam BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/Basecalls LANE=1 READ_STRUCTURE=20T6B55T RUN_BARCODE=run15 LIBRARY_PARAMS=library_params.tsv

where library_params.tsv looks like

BARCODE OUTPUT SAMPLE_ALIAS LIBRARY_NAME
TAAGGC TAAGGC.bam TAAGGC TAAGGC
AGGCAG AGGCAG.bam AGGCAG AGGCAG
CGTACT CGTACT.bam CGTACT CGTACT
NNNNNN NNNNNN.bam NNNNNN NNNNNN

but this generates a huge list of errors, how can I get IlluminaBasecallsToSam to run correctly?

[email protected]:~/Alex$ java -jar 3rdParty/picard/Old_picard.jar IlluminaBasecallsToSam BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/Basecalls LANE=1 READ_STRUCTURE=20T6B55T RUN_BARCODE=TAAGGC LIBRARY_PARAMS=library_params.tsv
[Thu Dec 22 08:32:30 EST 2016] picard.illumina.IlluminaBasecallsToSam BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/Basecalls LANE=1 RUN_BARCODE=TAAGGC READ_STRUCTURE=20T6B55T LIBRARY_PARAMS=library_params.tsv SEQUENCING_CENTER=BI PLATFORM=illumina ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2, FLUIDIGM] NUM_PROCESSORS=0 FORCE_GC=true APPLY_EAMSS_FILTER=true MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 INCLUDE_NON_PF_READS=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
[Thu Dec 22 08:32:30 EST 2016] Executing as [email protected] on Linux 4.4.0-57-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14; Picard version: 1.138(89618e408692ff6288c7c880658f32f16fcbec53_1441135673) IntelDeflater
INFO 2016-12-22 08:32:30 IlluminaBasecallsToSam DONE_READING STRUCTURE IS 20T6B55T
ERROR 2016-12-22 08:32:31 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:695)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:311)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:213)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:206)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-1" picard.PicardException: Read records with barcode null, but this barcode was not expected. (Is it referenced in the parameters file?)
at picard.illumina.IlluminaBasecallsConverter$TileProcessingRecord.addRecord(IlluminaBasecallsConverter.java:430)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:546)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-1-thread-7" Exception in thread "pool-1-thread-2" picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0055.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
Exception in thread "pool-1-thread-6" Exception in thread "pool-1-thread-5" Exception in thread "pool-1-thread-8" Exception in thread "pool-1-thread-3" picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0060.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:971)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:114)
... 18 more
Exception in thread "pool-1-thread-4" picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0063.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:971)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:114)
... 18 more
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0046.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0065.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0049.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
[Thu Dec 22 08:32:31 EST 2016] picard.illumina.IlluminaBasecallsToSam done. Elapsed time: 0.01 minutes.
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0051.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
Runtime.totalMemory()=438304768
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:314)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:213)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:206)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    Oy that's a long stack trace. Well, buried in there are several errors that say
    "IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0060.bcl.bgzf.bci" which to me suggest that some files are missing or corrupted. Did you run CheckIlluminaDirectory to verify that everything is accounted for?
  • concon PennsylvaniaMember

    @Geraldine_VdAuwera said:
    Oy that's a long stack trace. Well, buried in there are several errors that say
    "IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0060.bcl.bgzf.bci" which to me suggest that some files are missing or corrupted. Did you run CheckIlluminaDirectory to verify that everything is accounted for?

    Hi Geraldine,

    thanks for looking into my problem.

    Unfortunately (fortunately?) everything looks ok (picard's output at the end)

    I'm curious about the "READ_STRUCTURE" argument, from RunInfo.xml I get

      <Read Number="1" NumCycles="20" IsIndexedRead="N" />
      <Read Number="2" NumCycles="6" IsIndexedRead="Y" />
      <Read Number="3" NumCycles="55" IsIndexedRead="N" />
    

    I got READ_STRUCTURE=20T6B55T perhaps this is a problem?

    the output from CheckIlluminaDirectory:

    [email protected]:~/Alex$ java -jar 3rdParty/picard/Old_picard.jar CheckIlluminaDirectory BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/BaseCalls LANES=1 DATA_TYPES=BaseCalls READ_STRUCTURE=20T6B55T
    [Thu Dec 22 10:42:13 EST 2016] picard.illumina.CheckIlluminaDirectory BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/BaseCalls DATA_TYPES=[BaseCalls] READ_STRUCTURE=20T6B55T LANES=[1] FAKE_FILES=false LINK_LOCS=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
    [Thu Dec 22 10:42:13 EST 2016] Executing as [email protected] on Linux 4.4.0-57-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14; Picard version: 1.138(89618e408692ff6288c7c880658f32f16fcbec53_1441135673) IntelDeflater
    INFO 2016-12-22 10:42:13 CheckIlluminaDirectory Checking lanes(1 in basecalls directory (/home/con/Alex/Files/Data/Intensities/BaseCalls)

    INFO 2016-12-22 10:42:13 CheckIlluminaDirectory Expected cycles: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81
    INFO 2016-12-22 10:42:13 CheckIlluminaDirectory Checking lane 1
    INFO 2016-12-22 10:42:13 CheckIlluminaDirectory Expected tiles: 11101, 11102, 11103, 11104, 11105, 11106, 11107, 11108, 11109, 11110, 11111, 11112, 11201, 11202, 11203, 11204, 11205, 11206, 11207, 11208, 11209, 11210, 11211, 11212, 11301, 11302, 11303, 11304, 11305, 11306, 11307, 11308, 11309, 11310, 11311, 11312, 12101, 12102, 12103, 12104, 12105, 12106, 12107, 12108, 12109, 12110, 12111, 12112, 12201, 12202, 12203, 12204, 12205, 12206, 12207, 12208, 12209, 12210, 12211, 12212, 12301, 12302, 12303, 12304, 12305, 12306, 12307, 12308, 12309, 12310, 12311, 12312, 13101, 13102, 13103, 13104, 13105, 13106, 13107, 13108, 13109, 13110, 13111, 13112, 13201, 13202, 13203, 13204, 13205, 13206, 13207, 13208, 13209, 13210, 13211, 13212, 13301, 13302, 13303, 13304, 13305, 13306, 13307, 13308, 13309, 13310, 13311, 13312, 21101, 21102, 21103, 21104, 21105, 21106, 21107, 21108, 21109, 21110, 21111, 21112, 21201, 21202, 21203, 21204, 21205, 21206, 21207, 21208, 21209, 21210, 21211, 21212, 21301, 21302, 21303, 21304, 21305, 21306, 21307, 21308, 21309, 21310, 21311, 21312, 22101, 22102, 22103, 22104, 22105, 22106, 22107, 22108, 22109, 22110, 22111, 22112, 22201, 22202, 22203, 22204, 22205, 22206, 22207, 22208, 22209, 22210, 22211, 22212, 22301, 22302, 22303, 22304, 22305, 22306, 22307, 22308, 22309, 22310, 22311, 22312, 23101, 23102, 23103, 23104, 23105, 23106, 23107, 23108, 23109, 23110, 23111, 23112, 23201, 23202, 23203, 23204, 23205, 23206, 23207, 23208, 23209, 23210, 23211, 23212, 23301, 23302, 23303, 23304, 23305, 23306, 23307, 23308, 23309, 23310, 23311, 23312
    INFO 2016-12-22 10:42:13 CheckIlluminaDirectory Lane 1 SUCCEEDED
    INFO 2016-12-22 10:42:13 CheckIlluminaDirectory SUCCEEDED! All required files are present and non-empty.
    [Thu Dec 22 10:42:13 EST 2016] picard.illumina.CheckIlluminaDirectory done. Elapsed time: 0.00 minutes.
    Runtime.totalMemory()=1247805

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hmm. I see you're using something called Old_picard.jar. Have you tried with the latest release?

  • concon PennsylvaniaMember

    @Geraldine_VdAuwera said:
    Hmm. I see you're using something called Old_picard.jar. Have you tried with the latest release?

    Hi Geraldine,

    the errors are still the same, I've attached the file (probably less ugly than posting here)

    thanks for all your help, I really appreciate it

    -Dave

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hmm actually this error message may be the one causing all the interrupts:

    Exception in thread "pool-1-thread-1" picard.PicardException: Read records with barcode null, but this barcode was not expected. (Is it referenced in the parameters file?)
    

    This suggests some of your read records are missing barcodes -- but to be frank I'm not very familiar with this tool myself and everyone's pretty much gone for the holidays at this point, so I don't think we can get you any expert help until January...

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @Con
    Hi Dave,
    Were you able to solve the issue? Did you fix the missing barcode issue?
    Thanks,
    Sheila

  • karthigayinikarthigayini TempeMember

    Hi

    I have same issue when I use illuminaBaseCalls toFastq.
    the command is:
    java -Xmx10g -jar ~/tools/picard-tools-2.1.0/picard.jar IlluminaBasecallsToFastq BASECALLS_DIR=$run_folder/Datantensities/BaseCalls/ LANE=1 RUN_BARCODE=$run_barcode FLOWCELL_BARCODE=$flowcell MACHINE_NAME=$machine READ_STRUCTURE=300T8B8B300T INCLUDE_NON_PF_READS=false COMPRESS_OUTPUTS=true ADAPTERS_TO_CHECK=null NUM_PROCESSORS=4 TMP_DIR=../tmp MULTIPLEX_PARAMS=./nextera_barcodes_demultiplex.txt

    And error:
    INFO 2017-07-24 11:57:11 IlluminaBasecallsToFastq READ STRUCTURE IS 300T8B8B300T
    Exception in thread "pool-1-thread-4" ERROR 2017-07-24 11:57:27 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
    java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:711)
    at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:320)
    at picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:185)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:209)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
    picard.PicardException: Read records with barcode null, but this barcode was not expected. (Is it referenced in the parameters file?)
    at picard.illumina.IlluminaBasecallsConverter$TileProcessingRecord.addRecord(IlluminaBasecallsConverter.java:445)
    at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:562)
    at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:659)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    Any thoughts on this?

    Issue · Github
    by shlee

    Issue Number
    889
    State
    open
    Last Updated
    Assignee
    Array
  • shleeshlee CambridgeMember, Broadie, Moderator admin

    Hi @karthigayini,

    I will consult with a developer on your question. Thanks for your patience.

  • shleeshlee CambridgeMember, Broadie, Moderator admin

    Hi @karthigayini,

    Our developer states:

    Perhaps the run did not have an index on the sequencer but one was specified in nextera_barcodes_demultiplex.txt. If it was an unindexed run, the headers BARCODE_1 or BARCODE_2 should not appear in the parameters file. If that's not the case, then the next step is to run CheckIlluminaDirectory on the run folder.

Sign In or Register to comment.