MarkDuplicates on reads with different lengths

SvyatoslavSidorov

Dear GATK team,

Am I right that since MarkDuplicates considers only 5' coordinates of reads, it should work properly on reads (both paired-end and single-end) that have different lengths (due to quality trimming from 3')?

Geraldine_VdAuwera

No, both ends are considered so quality trimming by hard-clipping will negate the tool's ability to identify duplicates. Soft-clipping is okay because the tool is able to use the soft-clipped sequence.<\del>

edit: my original response was incorrect, see discussion below

Quality trimming is a legacy practice dating back to the time when analysis tools did not take individual base qualities into account. Now, most established tools have the ability to weigh the evidence appropriately and quality trimming is no longer useful. At this point it has more downsides than advantages, so we recommend just not doing it.

jalves

@Geraldine_VdAuwera said:

No, both ends are considered so quality trimming by hard-clipping will negate the tool's ability to identify duplicates. Soft-clipping is okay because the tool is able to use the soft-clipped sequence.

Hi Geraldine. Are you sure that Markduplicates consideres both ends for the detection of duplicates? According to the Picard Wiki page it only appears to take into account the 5' coordinate of the reads:

"Essentially what it does (for pairs; single-end data is also handled) is to find the 5' coordinates and mapping orientations of each read pair. When doing this it takes into account all clipping that has taking place as well as any gaps or jumps in the alignment. You can thus think of it as determining "if all the bases from the read were aligned, where would the 5' most base have been aligned".
Source: https://sourceforge.net/p/picard/wiki/Main_Page/#q-how-does-markduplicates-work"

Geraldine_VdAuwera

You're absolutely right, @jalves -- I'm not sure what I was thinking. Sorry about that.

@SvyatoslavSidorov my answer to you was wrong, please see this correction.

I still don't recommend hard clipping for quality though.

Geraldine_VdAuwera

No, both ends are considered so quality trimming by hard-clipping will negate the tool's ability to identify duplicates. Soft-clipping is okay because the tool is able to use the soft-clipped sequence.<\del>

edit: my original response was incorrect, see discussion below

Quality trimming is a legacy practice dating back to the time when analysis tools did not take individual base qualities into account. Now, most established tools have the ability to weigh the evidence appropriately and quality trimming is no longer useful. At this point it has more downsides than advantages, so we recommend just not doing it.

SvyatoslavSidorov

Geraldine, thank you very much!

jalves

@Geraldine_VdAuwera said:

No, both ends are considered so quality trimming by hard-clipping will negate the tool's ability to identify duplicates. Soft-clipping is okay because the tool is able to use the soft-clipped sequence.

Hi Geraldine. Are you sure that Markduplicates consideres both ends for the detection of duplicates? According to the Picard Wiki page it only appears to take into account the 5' coordinate of the reads:

"Essentially what it does (for pairs; single-end data is also handled) is to find the 5' coordinates and mapping orientations of each read pair. When doing this it takes into account all clipping that has taking place as well as any gaps or jumps in the alignment. You can thus think of it as determining "if all the bases from the read were aligned, where would the 5' most base have been aligned".
Source: https://sourceforge.net/p/picard/wiki/Main_Page/#q-how-does-markduplicates-work"

Geraldine_VdAuwera

You're absolutely right, @jalves -- I'm not sure what I was thinking. Sorry about that.

@SvyatoslavSidorov my answer to you was wrong, please see this correction.

I still don't recommend hard clipping for quality though.

SvyatoslavSidorov

@jalves and @Geraldine_VdAuwera, thanks for your corrections!