We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

How long does it take, Part 2

Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
edited December 2017 in Announcements


A while back, I posted this article about work done by the Intel Bio Team to benchmark the speed and resource utilization of each step in the per-sample segment of the germline variation pipeline (from BWA to HaplotypeCaller; FASTQ to GVCF). They published their results as a white paper on the Intel Life Sciences website, which has a section dedicated to GATK (which makes us feel all warm and tingly).

Now the Intel team has published an updated version of the white paper here that extends the work, originally done on a WGS trio, to a cohort of 50 exomes and adds the joint analysis segment of the pipeline (GenotypeGVCFs to VQSR; GVCFs to filtered multisample VCF) for both datasets.

As previously, the paper does a great job of showing where are the performance bottlenecks and where you can get the biggest speed increases by parallelizing execution.

My commentary from the previous post still applies pretty much equally to this updated version, except now we have performance profiles for GenotypeGVCFs and the VQSR tools as well, which I'll comment on briefly (using WGS but the profiles are similar for exomes).


The biggest takeaway here is that the runtime of GenotypeGVCFs scales down almost linearly with how widely you parallelize it, which is obviously great news if you're in a rush and you have access to lots of machines. But pay attention here to the meaning of "thread count" in the context of the paper! As a reminder, most of the parallelization it presents is achieved through scatter-gather (parallelizing over predetermined genomic intervals), not by multithreading using -nt and/or nct. In our own production pipelines we don't use -nt/-nct multithreading at all, and in GATK4 we're abandoning them and replacing the functionality with Spark support wherever it makes sense. Why am I pointing this out here? Because we're finding that GenotypeGVCFs is especially difficult to parallelize through multithreading, due to the complexity of dealing with overlapping events across multiple samples (the occurrence of which increases with cohort size). In the recent GATK 3.7 release, we added some functionality to deal better with overlapping deletions -- and now we're getting reports that this breaks when multithreading is turned on (cue the poop emoji). The safest way to deal with this? Don't use multithreading with GenotypeGVCFs; use scatter-gather instead (ask me how in the comments).

Also, don't parallelize VQSR. Look at the graph; it's not worth it. VQSR needs to see all of the things most of the time.

Finally, I should add that having the exome numbers to compare to the WGS numbers is a big upgrade -- it really gives you sense of scale of the practical implications of choosing to work with one datatype versus the other. All other sciencey considerations being equal (which they're not, but let's pretend) the computational resource commitment is massively different. Which is hardly news to our Ops team that processed Daniel MacArthur's ludicrously large gnomAD dataset, let me tell you -- for reference, the final joint VCF on that was ~22TB for 20K genomes. That's a big part of why we run our whole genomes on the cloud. It's real Big Data, no hype needed.

Post edited by Geraldine_VdAuwera on


  • ying_sheng_1ying_sheng_1 Member ✭✭
    edited January 2017

    Nice presentation. Is it possible to have these data for GATK4?

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    We haven't done this for GATK4 yet but we plan to do it once we have the full pipeline validated in GATK4. Stay tuned for an upcoming announcement of our calendar for GATK4 release.
  • Thanks for your presentation.
    I have two questions:
    1. How to parallelize GenotypeGVCFs by scatter-gather? (I guess you want us to ask ;) )
    2. In White paper , it used the following arguments:

    GenotypeGVCFs: Merges gVCF's to create a genotyped VCF:
    -nt "NUMTHREADS"
    -D "DBSN VCF"
    -V "input"

    It used -nt which is different with your presentation, so I am confused.

    Looking forward to your reply,
    Thank you!

    Issue · Github
    by Sheila

    Issue Number
    Last Updated
    Closed By
  • SheilaSheila Broad InstituteMember, Broadie ✭✭✭✭✭


    I will ask Geraldine to get back to you soon.


  • SheilaSheila Broad InstituteMember, Broadie ✭✭✭✭✭

    Hi again,

    Another teammate jumped in. It seems "For the Joint Analysis pipeline, VariantRecalibrator is currently unable to conduct process level parallelism and a comparison between both thread and process level parallelism techniques for the rest of the tools showed no significant improvement in time. Thus, all the tools in the Joint Analysis portion uses GATK's integrated -nt argument to apply thread level parallelism." This is from the white paper.

    I hope that helps.


  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    To clarify @Sheila's comment, we realized that there was indeed a contradiction between my blog post and the white paper. It seems I misunderstood a communication I had at the time with the authors of the white paper. We had discussed the steps earlier in the pipeline that are parallelized using scatter-gather, and I thought that applied to all steps, but the joint calling part (including GenotypeGVCFs) is in fact parallelized using -nt. I will amend my blog post accordingly. My apologies for the confusion, and thank you for pointing out this error!

Sign In or Register to comment.