Holiday Notice:
The Frontline Support team will be slow to respond December 17-18 due to an institute-wide retreat and offline December 22- January 1, while the institute is closed. Thank you for your patience during these next few weeks. Happy Holidays!

Picard MergeBamAlignment issue

Hi,

I am attempting to merge the output of tophat in order to run some RNASeq
QC metrics. This is a single read 50 bp on a Hiseq. In order to get past
the fact that tophat gives a MAPQ of 255 to unmapped reads (not 0 as
expected by Picard) I used the following tool (
https://github.com/cbrueffer/tophat-recondition) to change it.

Once completed, I added read groups using picard and then sorted the
accepted_hits.bam by coordinate and sorted the unmapped reads by queryname.

tophat-recondition.py /home/ryan/NGS_Data/No_Dox

java -Xmx2g -jar /home/ryan/jar/picard-tools-1.129/picard.jar \
AddOrReplaceReadGroups \
I=/home/ryan/NGS_Data/unmapped_fixup.bam \
O=/home/ryan/NGS_Data/unmapped_fixup-RG.bam \
RGID=No_Dox RGLB=No_Dox RGPL=illumina RGPU=GCCAAT RGSM=No_Dox

java -Xmx2g -jar /home/ryan/jar/picard-tools-1.129/picard.jar \
AddOrReplaceReadGroups \
I=/home/ryan/NGS_Data/accepted_hits.bam \
O=/home/ryan/NGS_Data/accepted_hits-RG.bam \
SORT_ORDER=coordinate \
RGID=No_Dox RGLB=No_Dox RGPL=illumina RGPU=GCCAAT RGSM=No_Dox \
CREATE_INDEX=true

java -Xmx2g -jar /home/ryan/jar/picard-tools-1.129/picard.jar \
SortSam \
I=/home/ryan/NGS_Data/unmapped_fixup-RG.bam \
O=/home/ryan/NGS_Data/unmapped_fixup-RG-sorted.bam \
SORT_ORDER=queryname

java -Xmx2g -jar /home/ryan/jar/picard-tools-1.129/picard.jar \
SortSam \
I=/home/ryan/NGS_Data/accepted_hits-RG.bam \
O=/home/ryan/NGS_Data/accepted_hits-RG-sorted.bam \
SORT_ORDER=coordinate \
CREATE_INDEX=true

java -Xmx2g -jar /home/ryan/jar/picard-tools-1.129/picard.jar \
MergeBamAlignment \
UNMAPPED_BAM=/home/ryan/NGS_Data/unmapped_fixup-RG-sorted.bam \
ALIGNED_BAM=/home/ryan/NGS_Data/accepted_hits-RG-sorted.bam \
O=/home/ryan/NGS_Data/merge_unmapped_accepted_hits_No_Dox.bam \
SORT_ORDER=coordinate \
REFERENCE_SEQUENCE=/home/ryan/Reference/human_spikein/hg19_spikein.fa \
PROGRAM_RECORD_ID=Tophat \
PROGRAM_GROUP_VERSION=0.1 \
PROGRAM_GROUP_COMMAND_LINE=tophat/dummy \
PAIRED_RUN=false \
VALIDATION_STRINGENCY=LENIENT \
CREATE_INDEX=true

I then get the following warning followed by the error:

WARNING 2015-03-17 09:44:22 SamAlignmentMerger Exception merging
bam alignment - attempting to sort aligned reads and try again: Aligned
record iterator (HS3:608:C6LNLACXX:7:2101:9946:63417) is behind the
unmapped reads (HS3:608:C6LNLACXX:7:2101:9947:11009)
INFO 2015-03-17 09:44:33 SamAlignmentMerger Read 1000000 records
from alignment SAM/BAM.
INFO 2015-03-17 09:44:43 SamAlignmentMerger Read 2000000 records
from alignment SAM/BAM.
....
INFO 2015-03-17 09:58:01 SamAlignmentMerger Read 96000000 records
from alignment SAM/BAM.
INFO 2015-03-17 09:58:09 SamAlignmentMerger Read 97000000 records
from alignment SAM/BAM.
INFO 2015-03-17 09:58:15 SamAlignmentMerger Finished reading
97571897 total records from alignment SAM/BAM.
[Tue Mar 17 09:58:16 PDT 2015] picard.sam.MergeBamAlignment done. Elapsed
time: 14.32 minutes.
Runtime.totalMemory()=1908932608
To get help, see
http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.IllegalStateException: Aligned record
iterator (HS3:608:C6LNLACXX:7:1101:10000:11036) is behind the unmapped
reads (HS3:608:C6LNLACXX:7:1101:10000:48402)
at
picard.sam.AbstractAlignmentMerger.mergeAlignment(AbstractAlignmentMerger.java:383)
at
picard.sam.SamAlignmentMerger.mergeAlignment(SamAlignmentMerger.java:153)
at picard.sam.MergeBamAlignment.doWork(MergeBamAlignment.java:248)
at
picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:187)
at
picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

I have searched and have been unsuccessful at resolving this problem. Any
ideas?

I am running picard 1.129 using java version "1.7.0_60"

ValidateSamFile on both inputs into MergeBamAlignment returns no errors in
those files. I am at a lost and seeking for help!

Thanks,

Ryan

Tagged:

Issue · Github
by Geraldine_VdAuwera

Issue Number
186
State
closed
Last Updated
Assignee
Array
Closed By
chandrans

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hi @Ryan ,

    I think the problem is that of the two sams you're merging, one is sorted by queryname and the other by coordinate. I expect if you change that to have both sorted the same way the error will go away.

  • RyanRyan Member

    Hi Geraldine,

    Thanks for thehelp. Unfortunately sorting both inputs by queryname does not resolve the issue and the same error pops up:

    WARNING 2015-03-18 09:51:14 SamAlignmentMerger Exception merging bam alignment - attempting to sort aligned reads and try again: Aligned record iterator (HS3:608:C6LNLACXX:7:1101:10000:11036) is behind the unmapped reads (HS3:608:C6LNLACXX:7:1101:10000:48402)
    INFO 2015-03-18 09:51:20 SamAlignmentMerger Read 1000000 records from alignment SAM/BAM.
    INFO 2015-03-18 09:51:26 SamAlignmentMerger Read 2000000 records from alignment SAM/BAM.
    ....
    INFO 2015-03-18 10:02:52 SamAlignmentMerger Read 97000000 records from alignment SAM/BAM.
    INFO 2015-03-18 10:02:59 SamAlignmentMerger Finished reading 97571897 total records from alignment SAM/BAM.
    [Wed Mar 18 10:02:59 PDT 2015] picard.sam.MergeBamAlignment done. Elapsed time: 11.75 minutes.
    Runtime.totalMemory()=1908932608
    To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
    Exception in thread "main" java.lang.IllegalStateException: Aligned record iterator (HS3:608:C6LNLACXX:7:1101:10000:11036) is behind the unmapped reads (HS3:608:C6LNLACXX:7:1101:10000:48402)
    at picard.sam.AbstractAlignmentMerger.mergeAlignment(AbstractAlignmentMerger.java:383)
    at picard.sam.SamAlignmentMerger.mergeAlignment(SamAlignmentMerger.java:153)
    at picard.sam.MergeBamAlignment.doWork(MergeBamAlignment.java:248)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:187)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

    I am not sure what is causing this.

    Ryan

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hi @Ryan,

    Would you be able to share some snippets of data that reproduce the problem so we can debug this locally? Instructions are here: https://www.broadinstitute.org/gatk/guide/article?id=1894

  • RyanRyan Member

    Hi Geraldine,

    Thanks for the help.
    I have uploaded snippets of data to your FTP server. It is named PICARD_MergeBam_error.tar.gz
    I really appreciate you looking into this. Let me know if you need additional information.

    Ryan

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Thanks @Ryan, we'll try to get to these asap.

  • 5581681555816815 TNMember
    edited May 2016

    @Ryan @Geraldine_VdAuwera
    Is this resolved? I got the same error for qn sorted bams.
    Thanks,

  • shleeshlee CambridgeMember, Broadie, Moderator admin

    Can you check to see if the read records that give the exception are present in both your uBAM and your aligned BAM? All reads in the aligned BAM must be in the uBAM.

  • 5581681555816815 TNMember
    edited May 2016

    Thanks @shlee
    Actually I resolved the issue by sorting my ubam file using picard...previously i used samtools.

    The queryname sorted bam are quite different detween picard and samtools.
    samtools seems to be numeric sorting on tile number
    picard seems to be text based like E00438:46:HMTV7CCXX:8:1101:10632:1661 is before E00438:46:HMTV7CCXX:8:1101:1073:10802

  • shleeshlee CambridgeMember, Broadie, Moderator admin

    Ah yes. There are minor inconsistencies between the tools so to avoid problems with GATK we recommend folks use Picard for data transformations. Good to hear you've sorted it out.

Sign In or Register to comment.