We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

What would cause Mutect2 to run super slow?

alanhoylealanhoyle UNC Lineberger Member

I'm running Mutect2 from GATK 4.1.3.0 on a cohort of TCGA T/N pairs. About half of them finished in 8-16 hours, but the other half have been running extremely slowly. They are verging on about 4 solid days of compute, and seem to be only on chromosome 3.

I'm running with the --germline-resource af-only-gnomad.hg38.vcf.gz downloaded from the Broad, and a set of merged intervals of hg38 exons (306k regions total). and using the GDC GRCh38 reference fasta.

As far as I can tell, the compute nodes are running find and not overloaded. Mutect has plenty of RAM

Here's a section of the stdout/stderr, which seems to not list any errors, just some warning about lack of AVX instructions preciptating a "MUCH slower LOGLESS_CACHING implementation!"

18:26:42.675 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
18:27:04.057 INFO ProgressMeter - chr1:184914 0.4 120 336.8
18:27:16.799 INFO ProgressMeter - chr1:841791 0.6 410 720.9
18:27:28.881 INFO ProgressMeter - chr1:941067 0.8 560 727.2
18:27:48.734 INFO ProgressMeter - chr1:944364 1.1 580 526.8
18:28:00.608 INFO ProgressMeter - chr1:957920 1.3 620 477.3

[...]

15:36:21.119 INFO ProgressMeter - chr3:138324466 5589.6 224460 40.2
15:36:35.178 INFO ProgressMeter - chr3:138398460 5589.9 224490 40.2
15:36:47.696 INFO ProgressMeter - chr3:138467552 5590.1 224540 40.2

After the first 5 minutes of processing, the "Regions/Minute" has dropped below 200, and has mostly hovered in the ~40 range, with a minimum of 39.3.

Any ideas? How much -Xmx should I be giving Mutect2 with a standard human genome?

Best Answer

Answers

  • alanhoylealanhoyle UNC Lineberger Member

    The half of the cohort that has finished appears to have run on nodes that have the AVX instructions. does that completely explain the relative slowness of the ones that have it?

  • alanhoylealanhoyle UNC Lineberger Member

    @davidben Am I correct in thinking that "Terra" is something that's local to the Broad? Since I'm at a remote university I should check with my admins creating a partition on our cluster that only includes machines that include AVX support and forcing the Mutect jobs to run on that partition?

  • davidbendavidben BostonMember, Broadie, Dev ✭✭✭

    @alanhoyle Terra (https://app.terra.bio/) is the new Firecloud -- if that means anything; otherwise it's a site for running pipelines on the cloud -- and is very much a public resource, although you do have to pay for your own compute and storage. That is, Terra is a very convenient wrapper for Google Cloud services so that you don't have to know anything about running on VMs etc, but you still have to pay Google.

    If you could get a partition with only AVX nodes that should work. BTW the same goes for HaplotypeCaller.

  • alanhoylealanhoyle UNC Lineberger Member

    Thanks @davidben. I did a side by side test with the same settings, and I had more than 10X speedup with a processor that had AVX vs those that lacked it, so I've been talking to our cluster admins.

Sign In or Register to comment.