BaseRecalibrator - Trade-Off between runtime and accuracy

tonytony Posts: 3Member
edited August 2012 in Ask the GATK team

Hi,

We are working with Illumina HiSeq 2000 paired-end data and as time goes by, lanes yield more and more sequences.

We are processing data at the lane BAM level (only one read group). The procedure, among others, does BWA mapping, Indel realignment, duplicates flagging and base quality recalibration. This is, as expected, a long process to complete but clearly the base recalibration stage is the longest by far, especially when lanes contain many sequences. We are using QualityScoreCovariate, ReadGroupCovariate, ContextCovariate and CycleCovariate covariates.

For instance, we have quite big lanes :

1 lane of 140,000,000 pairs (280,000,000 reads) : ~36 hours for recalibration

1 lane of 185,000,000 pairs (370,000,000 reads) : ~48 hours for recalibration

We obviously wish to reduce this run time and I found in the following link a small chapter on the topic (at the very end of the page) :
http://gatk.vanillaforums.com/discussion/44/base-quality-score-recalibrator#latest

So, we are really keen on downsampling our BAM files to reduce run time but at the same time we want our data as accurate as possible to help us for instance in the task of diminishing false positive substitutions rate. So if it is worth to wait, we wait.

Nevertheless, in the plot shown in the previous link, the x axis stops at 5,000,000 reads, where the RMSE value seems to have reached a "plateau".

1) We were thus wondering if there is a read count threshold (empirical value) above which the accuracy of the recalibration is no more improved ?

2) If such a threshold exists, I can not find the '--process_nth_locus' switch described in the link above, should I use '-dt', '-dfrac', '-dcov' options instead to downsample ?

3 ) Is the '--num_threads' working with BaseRecalibrator Walker ? Up to how many threads ?

Thanks a lot,

Best Regards,

Anthony

PS : GATK version used is v2.0-23-ge9a19be

Post edited by tony on

Best Answer

  • ebanksebanks Broad InstitutePosts: 684Member, Administrator, GATK Developer, Broadie, Moderator, DSDE Member, GP Member admin
    Answer ✓

    Good questions.

    1) Theoretically, the more data you have, the more accurate your recalibration will be. But the marginal difference in accuracy is very small (as shown by the plots in the thread you referenced) once you have enough data. By default the GATK engine will downsample a position to 1000x coverage, but you could certainly knock that number down without harm. You can use the plot at the bottom of the other thread to determine how much you'll want to downsample.

    2) -dcov. --process_nth_locus is no longer a recommended/valid option and I've updated the other post. Thanks for the heads up.

    3) Yes. As many as your machine can handle well (every system is different) - but you'll need to bump up the memory accordingly.

    Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

Answers

  • avinashavinash Posts: 1Member

    I have the same question.
    Avinash

  • ebanksebanks Broad InstitutePosts: 684Member, Administrator, GATK Developer, Broadie, Moderator, DSDE Member, GP Member admin
    Answer ✓

    Good questions.

    1) Theoretically, the more data you have, the more accurate your recalibration will be. But the marginal difference in accuracy is very small (as shown by the plots in the thread you referenced) once you have enough data. By default the GATK engine will downsample a position to 1000x coverage, but you could certainly knock that number down without harm. You can use the plot at the bottom of the other thread to determine how much you'll want to downsample.

    2) -dcov. --process_nth_locus is no longer a recommended/valid option and I've updated the other post. Thanks for the heads up.

    3) Yes. As many as your machine can handle well (every system is different) - but you'll need to bump up the memory accordingly.

    Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

  • tonytony Posts: 3Member

    Just a quick point :
    From the same link again, it is mention that one can downsample by selecting a genome interval with -L option. Would you rather recommend to downsample with -dcov but using the whole genome or it is just fine to use only chromosome 1 for instance if enough reads were mapped back to this genomic region ?

  • ebanksebanks Broad InstitutePosts: 684Member, Administrator, GATK Developer, Broadie, Moderator, DSDE Member, GP Member admin

    Ideally you'd use the whole genome with downsampling (because then you see more instances for the context covariate).

    Eric Banks, PhD -- Senior Group Leader, MPG Analysis, Broad Institute of Harvard and MIT

  • pylpyl Posts: 9Member

    Hi,

    I was wondering which version of GATK you were referring to when you pointed out that one could use -nt to parallelize this walker, I get an error message, see below:

    INFO 08:42:39,778 HelpFormatter - ---------------------------------------------------------------------------------
    INFO 08:42:39,782 HelpFormatter - The Genome Analysis Toolkit (GATK) v2.2-10-gbbafb72, Compiled 2012/11/26 03:37:03
    [...]

    ERROR
    ERROR MESSAGE: Invalid command line: Argument nt has a bad value: The analysis BaseRecalibrator currently does not support parallel execution with nt. Please run your analysis without the nt option.
    ERROR ------------------------------------------------------------------------------------------

    Cheers,
    Paul

  • chapmanbchapmanb Posts: 19Member

    GATK 2.2+ use the -nct option for parallelization, instead of -nt. See the release notes on GATK 2.2, specifically the 'Multi-threading power returns to BaseRecalibrator' section for more details:

    http://www.broadinstitute.org/gatk/guide/version-history

    Brad Chapman, Bioinformatics Core at Harvard School of Public Health

  • amiami Posts: 43GATK Developer mod

    Just a quick note: you can use both -nt and -nct in GATK 2.2+ (each one by itself or even together). They provide different types of parallelization.
    BQSR on the other hand can work only with -nct, and this is the reason for the error you got.

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,427Administrator, GATK Developer admin

    We'll have a document outlining which tools can use which modes up soon, possibly later today.

    Geraldine Van der Auwera, PhD

Sign In or Register to comment.