The current GATK version is 3.3-0

#### Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

# BaseRecalibrator - Trade-Off between runtime and accuracy

Posts: 3Member
edited August 2012

Hi,

We are working with Illumina HiSeq 2000 paired-end data and as time goes by, lanes yield more and more sequences.

We are processing data at the lane BAM level (only one read group). The procedure, among others, does BWA mapping, Indel realignment, duplicates flagging and base quality recalibration. This is, as expected, a long process to complete but clearly the base recalibration stage is the longest by far, especially when lanes contain many sequences. We are using QualityScoreCovariate, ReadGroupCovariate, ContextCovariate and CycleCovariate covariates.

For instance, we have quite big lanes :

1 lane of 140,000,000 pairs (280,000,000 reads) : ~36 hours for recalibration

1 lane of 185,000,000 pairs (370,000,000 reads) : ~48 hours for recalibration

We obviously wish to reduce this run time and I found in the following link a small chapter on the topic (at the very end of the page) :
http://gatk.vanillaforums.com/discussion/44/base-quality-score-recalibrator#latest

So, we are really keen on downsampling our BAM files to reduce run time but at the same time we want our data as accurate as possible to help us for instance in the task of diminishing false positive substitutions rate. So if it is worth to wait, we wait.

Nevertheless, in the plot shown in the previous link, the x axis stops at 5,000,000 reads, where the RMSE value seems to have reached a "plateau".

1) We were thus wondering if there is a read count threshold (empirical value) above which the accuracy of the recalibration is no more improved ?

2) If such a threshold exists, I can not find the '--process_nth_locus' switch described in the link above, should I use '-dt', '-dfrac', '-dcov' options instead to downsample ?

3 ) Is the '--num_threads' working with BaseRecalibrator Walker ? Up to how many threads ?

Thanks a lot,

Best Regards,

Anthony

PS : GATK version used is v2.0-23-ge9a19be

Post edited by tony on
Tagged:

Good questions.

1) Theoretically, the more data you have, the more accurate your recalibration will be. But the marginal difference in accuracy is very small (as shown by the plots in the thread you referenced) once you have enough data. By default the GATK engine will downsample a position to 1000x coverage, but you could certainly knock that number down without harm. You can use the plot at the bottom of the other thread to determine how much you'll want to downsample.

2) -dcov. --process_nth_locus is no longer a recommended/valid option and I've updated the other post. Thanks for the heads up.

3) Yes. As many as your machine can handle well (every system is different) - but you'll need to bump up the memory accordingly.

Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

• Posts: 1Member

I have the same question.
Avinash

Good questions.

1) Theoretically, the more data you have, the more accurate your recalibration will be. But the marginal difference in accuracy is very small (as shown by the plots in the thread you referenced) once you have enough data. By default the GATK engine will downsample a position to 1000x coverage, but you could certainly knock that number down without harm. You can use the plot at the bottom of the other thread to determine how much you'll want to downsample.

2) -dcov. --process_nth_locus is no longer a recommended/valid option and I've updated the other post. Thanks for the heads up.

3) Yes. As many as your machine can handle well (every system is different) - but you'll need to bump up the memory accordingly.

Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

• Posts: 3Member

Just a quick point :
From the same link again, it is mention that one can downsample by selecting a genome interval with -L option. Would you rather recommend to downsample with -dcov but using the whole genome or it is just fine to use only chromosome 1 for instance if enough reads were mapped back to this genomic region ?

Ideally you'd use the whole genome with downsampling (because then you see more instances for the context covariate).

Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

• Posts: 9Member

Hi,

I was wondering which version of GATK you were referring to when you pointed out that one could use -nt to parallelize this walker, I get an error message, see below:

INFO 08:42:39,778 HelpFormatter - ---------------------------------------------------------------------------------
INFO 08:42:39,782 HelpFormatter - The Genome Analysis Toolkit (GATK) v2.2-10-gbbafb72, Compiled 2012/11/26 03:37:03
[...]

##### ERROR ------------------------------------------------------------------------------------------

Cheers,
Paul

• Posts: 19Member

GATK 2.2+ use the -nct option for parallelization, instead of -nt. See the release notes on GATK 2.2, specifically the 'Multi-threading power returns to BaseRecalibrator' section for more details:

Brad Chapman, Bioinformatics Core at Harvard School of Public Health

• Posts: 43GATK Developer mod

Just a quick note: you can use both -nt and -nct in GATK 2.2+ (each one by itself or even together). They provide different types of parallelization.
BQSR on the other hand can work only with -nct, and this is the reason for the error you got.