interval lists and -nct threading

Regarding the observation elsewhere: "HaplotypeCaller is slower when restricted to intervals": My test-bed is a 48 core, 100GB RAM CentOS 7 system with GenomeAnalysisTK-3.5 on it. If i submit the command:

java -Xmx16g -jar GenomeAnalysisTK.jar -T HaplotypeCaller -I chr12.bam -R hg19.fasta -o out.vcf -L chr12:40600000-40800000 -nct 36

I can see that the system is indeed using all 36 cores through completion.

But if I choose a small subset of ranges such as

chr12:40634200-40634514
chr12:40637254-40637647
chr12:40643603-40643770
chr12:40644926-40645491
chr12:40646596-40646822
chr12:40650957-40651309
chr12:40653220-40653421
chr12:40657469-40657806
chr12:40668336-40668853
chr12:40671529-40672115

, put them in the file test.list, then the command

java -Xmx16g -jar GenomeAnalysisTK.jar -T HaplotypeCaller -I chr12.bam -R hg19.fasta -o out.vcf -L test.list -nct 36

executes, but quickly drops to one core usage for almost the entire run, taking an unexpectedly long time to complete. This does not seem like the efficiency gain anticipated. I have tested this with other interval file formats, but the problem persists and is easily reproducible. This is indeed unfortunate behavior. Examining just the exons of a region should take less time than examining the whole region, not more.

Is there any hope that this will ever operate as expected, or is the problem too deeply embedded in the threading algorithm? Some explanation would be useful. I am doing everything I can to speed up the GATK best practices methods to use on whole exome datasets, but things like this are frustrating.

-- Fred P.

Best Answer

Answers

  • pishottapishotta UBCMember

    Believe it or not, I think I figured out why this is happening. Something like: When you supply a list of regions, the caller steps through that list in sequence, using a number of threads it thinks appropriate for that region. If the regions are small, like the above, it will use just a small number of computational threads, not necessarily the number specified in the -nct option. So in the above example, it used just 1 or 2 threads consecutively for each of the 10 regions, which is why the command with "-L test.list" takes so much longer that "-L chr12:40634200-40672115" which is bigger in total, but therefore able to elicit more simultaneous threads.

    Misleading behavior, but I see no way to force the number of threads used. -nct only seems to set an upper bound. Bottom line: Looks like -nct can hurt you if you are checking many small regions like in an exome analysis.

    Have to rethink this. Maybe it I targeted whole genes and filtered out introns afterwards... Hardly ideal, but it's a thought.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    The implementation of multithreading in HC is not very efficient, as you observe. We've seen some benchmarking results that suggest there is no point to using more than 4 to 8 threads with HC.

    In our own work we use scatter-gather parallelism to speed up execution of HC, and scatter by exome target.

Sign In or Register to comment.