US Holiday notice: this Thursday and Friday (Nov 25-26) the forum will be unattended. Normal service will resume Monday Nov 29. Happy Thanksgiving!

BaseRecalibrator - Trade-Off between runtime and accuracy

tonytony Posts: 3Member
edited August 2012 in Ask the GATK team

Hi,

We are working with Illumina HiSeq 2000 paired-end data and as time goes by, lanes yield more and more sequences.

We are processing data at the lane BAM level (only one read group). The procedure, among others, does BWA mapping, Indel realignment, duplicates flagging and base quality recalibration. This is, as expected, a long process to complete but clearly the base recalibration stage is the longest by far, especially when lanes contain many sequences. We are using QualityScoreCovariate, ReadGroupCovariate, ContextCovariate and CycleCovariate covariates.

For instance, we have quite big lanes :

1 lane of 140,000,000 pairs (280,000,000 reads) : ~36 hours for recalibration

1 lane of 185,000,000 pairs (370,000,000 reads) : ~48 hours for recalibration

We obviously wish to reduce this run time and I found in the following link a small chapter on the topic (at the very end of the page) : http://gatk.vanillaforums.com/discussion/44/base-quality-score-recalibrator#latest

So, we are really keen on downsampling our BAM files to reduce run time but at the same time we want our data as accurate as possible to help us for instance in the task of diminishing false positive substitutions rate. So if it is worth to wait, we wait.

Nevertheless, in the plot shown in the previous link, the x axis stops at 5,000,000 reads, where the RMSE value seems to have reached a "plateau".

1) We were thus wondering if there is a read count threshold (empirical value) above which the accuracy of the recalibration is no more improved ?

2) If such a threshold exists, I can not find the '--process_nth_locus' switch described in the link above, should I use '-dt', '-dfrac', '-dcov' options instead to downsample ?

3 ) Is the '--num_threads' working with BaseRecalibrator Walker ? Up to how many threads ?

Thanks a lot,

Best Regards,

Anthony

PS : GATK version used is v2.0-23-ge9a19be

Post edited by tony on

Best Answer

  • ebanksebanks Posts: 684GATK Developer mod
    Answer ✓

    Good questions.

    1) Theoretically, the more data you have, the more accurate your recalibration will be. But the marginal difference in accuracy is very small (as shown by the plots in the thread you referenced) once you have enough data. By default the GATK engine will downsample a position to 1000x coverage, but you could certainly knock that number down without harm. You can use the plot at the bottom of the other thread to determine how much you'll want to downsample.

    2) -dcov. --process_nth_locus is no longer a recommended/valid option and I've updated the other post. Thanks for the heads up.

    3) Yes. As many as your machine can handle well (every system is different) - but you'll need to bump up the memory accordingly.

    Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

Answers

Sign In or Register to comment.