The current GATK version is 3.7-0
Examples: Monday, today, last week, Mar 26, 3/26/04

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Get notifications!


You can opt in to receive email notifications, for example when your questions get answered or when there are new announcements, by following the instructions given here.

Did you remember to?


1. Search using the upper-right search box, e.g. using the error message.
2. Try the latest version of tools.
3. Include tool and Java versions.
4. Tell us whether you are following GATK Best Practices.
5. Include relevant details, e.g. platform, DNA- or RNA-Seq, WES (+capture kit) or WGS (PCR-free or PCR+), paired- or single-end, read length, expected average coverage, somatic data, etc.
6. For tool errors, include the error stacktrace as well as the exact command.
7. For format issues, include the result of running ValidateSamFile for BAMs or ValidateVariants for VCFs.
8. For weird results, include an illustrative example, e.g. attach IGV screenshots according to Article#5484.
9. For a seeming variant that is uncalled, include results of following Article#1235.

Did we ask for a bug report?


Then follow instructions in Article#1894.

Formatting tip!


Wrap blocks of code, error messages and BAM/VCF snippets--especially content with hashes (#)--with lines with three backticks ( ``` ) each to make a code block as demonstrated here.

Jump to another community
Picard 2.9.0 is now available. Download and read release notes here.
GATK 3.7 is here! Be sure to read the Version Highlights and optionally the full Release Notes.

Insert size out of range

liuhuiliuhui ChinaMember Posts: 14

Hi,
I am running picard to sort sam file generated from bwa mem. But I got errors as the following lines show. Could you please help me in solving this problem?
Thank you very much.
Hui Liu

java -jar /home/liuhui/bin/picard-tools-2.4.1/picard.jar SortSam \
INPUT=lib_1.sam \
OUTPUT=lib_1_corSort.bam \
SORT_ORDER=coordinate

[Fri Jun 17 21:12:47 CST 2016] picard.sam.SortSam INPUT=lib_1.sam OUTPUT=lib_1_corSort.bam SORT_ORDER=coordinate VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
[Fri Jun 17 21:12:47 CST 2016] Executing as liuhui@ginkgo on Linux 3.8.0-44-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14; Picard version: 2.4.1(7c4d36e011df1aec4689b51efcada44e92d1817f_1464389670) JdkDeflater
[Fri Jun 17 21:12:47 CST 2016] picard.sam.SortSam done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=2058354688
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.SAMFormatException: Error parsing text SAM file. Insert size out of range; File lib_1.sam; Line 10072
Line: HWI-ST1283:245:C4K2TACXX:6:1101:8926:2934 65 Scaffold_1000 77285907 0 100M = 657067308 579781402 CTCATACCTTGAAACATGGGAGGGCTAGCACCATTTTCAGTGCCCTCAATGTTGGAGCAGAATTAGACATTCACACCACCAGGACACCCATGAGCCCCAT CCCFFFFFHHHHHJJIJJJJJJIIIJJJJIJJIJJJJJIIGJJJHEHEHGGGEHICGGIIDGIGGGEFFHGHF<@CB???AA???=?58??A@C3:2529 NM:i:11 MD:Z:12G14A0A5C4A1T7A0A12C20G9A5 AS:i:45 XS:i:41 RG:Z:lib1 XA:Z:Scaffold_1000,+129324137,61M39S,4;Scaffold_920,+15157804,6S51M43S,2;Scaffold_1000,-653898065,37S50M13S,2;
at htsjdk.samtools.SAMLineParser.reportErrorParsingLine(SAMLineParser.java:439)
at htsjdk.samtools.SAMLineParser.parseLine(SAMLineParser.java:341)
at htsjdk.samtools.SAMTextReader$RecordIterator.parseLine(SAMTextReader.java:248)
at htsjdk.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:236)
at htsjdk.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:212)
at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:545)
at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:519)
at picard.sam.SortSam.doWork(SortSam.java:91)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:209)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

Answers

  • liuhuiliuhui ChinaMember Posts: 14

    Hi,
    I have tried to fix the problem but it seems that there are something error.
    Could you please give me advice?

    java -jar picard.jar FixMateInformation INPUT=lib_1.bam >OUTPUT=lib_1.fixmate.bam SO=coordinate VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true

    INFO 2016-06-18 00:59:58 FixMateInformation Processed 100,000,000 records. Elapsed time: 00:28:01s. Time for last 1,000,000: 17s. Last read position: Scaffold_802:43,574,088
    INFO 2016-06-18 01:00:23 FixMateInformation Processed 101,000,000 records. Elapsed time: 00:28:25s. Time for last 1,000,000: 24s. Last read position: Scaffold_1000:263,056,652
    INFO 2016-06-18 01:00:43 FixMateInformation Finished processing reads; re-sorting output file.
    [Sat Jun 18 01:26:13 CST 2016] picard.sam.FixMateInformation done. Elapsed time: 78.74 minutes.
    Runtime.totalMemory()=9843507200
    To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
    Exception in thread "main" htsjdk.samtools.SAMException: Exception when processing alignment for BAM index HWI-ST1283:245:C4K2TACXX:6:1115:4128:65038 1/2 100b aligned read.
    at htsjdk.samtools.BAMFileWriter.writeAlignment(BAMFileWriter.java:124)
    at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:214)
    at picard.sam.FixMateInformation.closeWriter(FixMateInformation.java:277)
    at picard.sam.FixMateInformation.doWork(FixMateInformation.java:217)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:209)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
    Caused by: htsjdk.samtools.SAMException: Exception creating BAM index for record HWI-ST1283:245:C4K2TACXX:6:1115:4128:65038 1/2 100b aligned read.
    at htsjdk.samtools.BAMIndexer.processAlignment(BAMIndexer.java:110)
    at htsjdk.samtools.BAMFileWriter.writeAlignment(BAMFileWriter.java:121)
    ... 6 more
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 32770
    at htsjdk.samtools.BinningIndexBuilder.processFeature(BinningIndexBuilder.java:136)
    at htsjdk.samtools.BAMIndexer$BAMIndexBuilder.processAlignment(BAMIndexer.java:213)
    at htsjdk.samtools.BAMIndexer.processAlignment(BAMIndexer.java:108)
    ... 7 more

  • SheilaSheila Broad InstituteMember, Broadie, Moderator Posts: 4,723 admin
    edited June 2016

    @liuhui
    Hi Hui,

    Can you tell us exactly how you generated your BAM file? What aligner did you use and did you do any processing after the reads were aligned to your reference?

    Thanks,
    Sheila

  • liuhuiliuhui ChinaMember Posts: 14

    HI, @Sheila,
    I generated the BAM file following this article "Calling variants in RNAseq" (https://www.broadinstitute.org/gatk/guide/article?id=3891).

    Maybe the errors arose from long intron size of my reference genome, with 2,741 bp and 318,524 bp of average and maximum intron size.
    So, I used picard AddOrReplaceReadGroups to add read groups with SILENT mode to avoid this problem. But I can not fix the problem when using picard MarkDuplicates to remove duplicates with BAM file, I have to use fastuniq (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0052249) to remove duplicates with FASTQ file.

    But another error occurs to me after I used samtools to build index for BAM file (follow the article I mention above), which have added read groups using picard with SILENT mode.

    Please see the details of the error in the following lines.

    Thank you very much!
    Hui Liu

    /media/10TB/call_snp/2pass$ java -Xmx64g -Djava.io.tmpdir=/media/10TB/tmp/ \
    -jar /home/liuhui/bin/GATK/GenomeAnalysisTK.jar \
    -T SplitNCigarReads \
    -R /media/10TB/conifer_genome/Ptaeda/STAR/Ptaeda.v1.01_1kScaffolds.fa \
    -I NCPT_rg_added_sorted.bam \
    -o NCPT_split.bam \
    -rf ReassignOneMappingQuality \
    -RMQF 255 \
    -RMQT 60 \
    -U ALLOW_N_CIGAR_READS

    INFO 01:41:17,465 HelpFormatter - --------------------------------------------------------------------------------
    INFO 01:41:17,471 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.6-0-g89b7209, Compiled 2016/06/01 22:27:29
    INFO 01:41:17,471 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute
    INFO 01:41:17,472 HelpFormatter - For support and documentation go to https://www.broadinstitute.org/gatk
    INFO 01:41:17,472 HelpFormatter - [Tue Jun 21 01:41:17 CST 2016] Executing on Linux 3.8.0-44-generic amd64
    INFO 01:41:17,473 HelpFormatter - Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14 JdkDeflater
    INFO 01:41:17,481 HelpFormatter - Program Args: -T SplitNCigarReads -R /media/10TB/conifer_genome/Ptaeda/STAR/Ptaeda.v1.01_1kScaffolds.fa -I NCPT_rg_added_sorted.bam -o NCPT_split.bam -rf ReassignOneMappingQuality -RMQF 255 -RMQT 60 -U ALLOW_N_CIGAR_READS
    INFO 01:41:17,488 HelpFormatter - Executing as liuhui@ginkgo on Linux 3.8.0-44-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14.
    INFO 01:41:17,489 HelpFormatter - Date/Time: 2016/06/21 01:41:17
    INFO 01:41:17,496 HelpFormatter - --------------------------------------------------------------------------------
    INFO 01:41:17,497 HelpFormatter - --------------------------------------------------------------------------------
    INFO 01:41:17,641 GenomeAnalysisEngine - Strictness is SILENT
    INFO 01:41:18,382 GenomeAnalysisEngine - Downsampling Settings: No downsampling
    INFO 01:41:18,394 SAMDataSource$SAMReaders - Initializing SAMRecords in serial
    INFO 01:41:18,544 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.15
    INFO 01:41:19,166 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files
    INFO 01:41:19,173 GenomeAnalysisEngine - Done preparing for traversal
    INFO 01:41:19,174 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
    INFO 01:41:19,174 ProgressMeter - | processed | time | per 1M | | total | remaining
    INFO 01:41:19,174 ProgressMeter - Location | reads | elapsed | reads | completed | runtime | runtime
    INFO 01:41:19,269 ReadShardBalancer$1 - Loading BAM index data
    INFO 01:41:19,272 ReadShardBalancer$1 - Done loading BAM index data
    DEBUG 2016-06-21 01:41:19 BlockCompressedOutputStream Using deflater: Deflater
    INFO 01:41:49,181 ProgressMeter - Scaffold_678:168913 846274.0 30.0 s 35.0 s 9.7% 5.2 m 4.7 m
    INFO 01:42:21,949 ProgressMeter - Scaffold_688:122535532 1215790.0 62.0 s 51.0 s 14.2% 7.3 m 6.3 m
    INFO 01:42:51,953 ProgressMeter - Scaffold_690:234587947 1504466.0 92.0 s 61.0 s 16.3% 9.4 m 7.9 m
    ...
    INFO 02:25:09,445 ProgressMeter - Scaffold_999:335768183 1.8039611E7 43.8 m 2.4 m 95.0% 46.1 m 2.3 m
    INFO 02:25:39,447 ProgressMeter - Scaffold_999:371353609 1.8139615E7 44.3 m 2.4 m 95.2% 46.6 m 2.2 m
    INFO 02:26:09,448 ProgressMeter - Scaffold_999:407077720 1.8339697E7 44.8 m 2.4 m 95.4% 47.0 m 2.2 m
    INFO 02:26:39,450 ProgressMeter - Scaffold_999:443018290 1.8439698E7 45.3 m 2.5 m 95.6% 47.4 m 2.1 m
    INFO 02:27:09,451 ProgressMeter - Scaffold_999:483310400 1.8539699E7 45.8 m 2.5 m 95.8% 47.9 m 2.0 m
    INFO 02:27:39,453 ProgressMeter - Scaffold_999:516929057 1.8639701E7 46.3 m 2.5 m 96.0% 48.3 m 116.0 s
    INFO 02:28:09,454 ProgressMeter - Scaffold_1000:12823346 1.8892732E7 46.8 m 2.5 m 96.3% 48.6 m 108.0 s
    INFO 02:28:39,457 ProgressMeter - Scaffold_1000:61098466 1.9093928E7 47.3 m 2.5 m 96.6% 49.0 m 101.0 s
    INFO 02:29:09,459 ProgressMeter - Scaffold_1000:89146671 1.9193929E7 47.8 m 2.5 m 96.7% 49.5 m 97.0 s
    INFO 02:29:39,461 ProgressMeter - Scaffold_1000:130553246 1.9293932E7 48.3 m 2.5 m 96.9% 49.9 m 91.0 s
    INFO 02:30:09,466 ProgressMeter - Scaffold_1000:160725207 1.9393936E7 48.8 m 2.5 m 97.1% 50.3 m 87.0 s
    INFO 02:30:39,467 ProgressMeter - Scaffold_1000:196903763 1.9593938E7 49.3 m 2.5 m 97.3% 50.7 m 82.0 s
    INFO 02:31:09,469 ProgressMeter - Scaffold_1000:240750793 1.9693942E7 49.8 m 2.5 m 97.5% 51.1 m 76.0 s
    ...
    INFO 02:39:09,515 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 57.8 m 2.7 m 100.0% 57.8 m 0.0 s
    INFO 02:39:39,517 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 58.3 m 2.7 m 100.0% 58.3 m 0.0 s
    INFO 02:40:09,519 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 58.8 m 2.7 m 100.0% 58.8 m 0.0 s
    INFO 02:40:39,520 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 59.3 m 2.7 m 100.0% 59.3 m 0.0 s
    INFO 02:41:09,536 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 59.8 m 2.8 m 100.0% 59.8 m 0.0 s
    INFO 02:41:39,537 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 60.3 m 2.8 m 100.0% 60.3 m 0.0 s
    INFO 02:42:09,540 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 60.8 m 2.8 m 100.0% 60.8 m 0.0 s
    INFO 02:42:39,542 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 61.3 m 2.8 m 100.0% 61.3 m 0.0 s
    INFO 02:43:09,543 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 61.8 m 2.9 m 100.0% 61.8 m 0.0 s
    INFO 02:43:39,545 ProgressMeter - Scaffold_1000:700088491 2.1695664E7 62.3 m 2.9 m 100.0% 62.3 m 0.0 s

    ERROR ------------------------------------------------------------------------------------------
    ERROR A BAM/CRAM ERROR has occurred (version 3.6-0-g89b7209):
    ERROR
    ERROR This means that there is something wrong with the BAM/CRAM file(s) you provided.
    ERROR The error message below tells you what is the problem.
    ERROR
    ERROR Visit our website and forum for extensive documentation and answers to
    ERROR commonly asked questions https://www.broadinstitute.org/gatk
    ERROR
    ERROR Please do NOT post this error to the GATK forum until you have followed these instructions:
    ERROR - Make sure that your BAM file is well-formed by running Picard's validator on it
    ERROR (see http://picard.sourceforge.net/command-line-overview.shtml#ValidateSamFile for details)
    ERROR - Ensure that your BAM index is not corrupted: delete the current one and regenerate it with 'samtools index'
    ERROR - Ensure that your CRAM index is not corrupted: delete the current one and regenerate it with
    ERROR 'java -jar cramtools-3.0.jar index --bam-style-index --input-file --reference-fasta-file '
    ERROR (see https://github.com/enasequence/cramtools/tree/v3.0 for details)
    ERROR
    ERROR MESSAGE: Exception when processing alignment for BAM index K00159:42:H5K2GBBXX:2:1227:13311:5042 1/2 14b aligned read.
    ERROR ------------------------------------------------------------------------------------------
  • SheilaSheila Broad InstituteMember, Broadie, Moderator Posts: 4,723 admin

    @liuhui
    Hi Hui,

    I'm not sure I understand why you used Picard's AddOrReplaceReadGroups to add read groups with SILENT mode to avoid long intron issues? Can you tell us the exact command you ran?

    Thanks,
    Sheila

  • liuhuiliuhui ChinaMember Posts: 14

    @Sheila

    Hi Sheila,
    Thank you for you reply to me.

    Here is the command I used to add read groups with SILENT mode.

    java -Xmx64g -Djava.io.tmpdir=/media/10TB/tmp/ \
    -jar /home/liuhui/bin/picard-tools-2.4.1/picard.jar AddOrReplaceReadGroups \
    I=NCPT_2pass_Aligned.out.sam \
    O=NCPT_rg_added_sorted.bam \
    SO=coordinate \
    VALIDATION_STRINGENCY=SILENT \
    RGID=NCPT RGLB=NCPT RGPL=illumina RGPU=machine RGSM=NCPT

    Thank you very much,
    Hui Liu

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie Posts: 11,669 admin
    Suppressing validation errors solves nothing. You need to deal with whatever is wrong before you can proceed with your analysis. Run ValidateSamFile in summary mode first to assess problems.

    Geraldine Van der Auwera, PhD

  • 570932004570932004 ChinaMember Posts: 10

    I see you have a Chromosome/Scaffold larger than 512MB .
    Do not create bam index with java-based tools(Picard and certain GATK tools), use samtools instead.

Sign In or Register to comment.