Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.
Attention:
We will be out of the office on November 11th and 13th 2019, due to the U.S. holiday(Veteran's day) and due to a team event(Nov 13th). We will return to monitoring the GATK forum on November 12th and 14th respectively. Thank you for your patience.

Limiting GATK's number of threads

Hi,

first of all, yes, I am aware of this post ( http://gatkforums.broadinstitute.org/discussion/comment/20469/#Comment_20469 ) and its recommendation of using -XX:ParallelGCThreads.

I have been testing my pipeline to run HaplotypeCaller and, through testing, I have found java/GATK opening too many ¿unnecessary? threads. This leads to increased CPU metrics on our cluster and a not-so-happy sysadmin.

An example of my command is as follows:

java -Xmx20G -XX:ParallelGCThreads=${gct} -Djava.io.tmpdir=${TMPDIR} -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar \
-T HaplotypeCaller \
-R ${ref} \
-I ${input} \
-L ${reg} \
-ERC GVCF \
-nct ${nct} \
--genotyping_mode DISCOVERY \
-stand_emit_conf 10 \
-stand_call_conf 30 \
-o ${name}.raw_variants.annotated.g.vcf \
-A QualByDepth -A RMSMappingQuality -A MappingQualityRankSumTest -A ReadPosRankSumTest -A FisherStrand -A StrandOddsRatio -A Coverage -A InbreedingCoeff

For example, when running this in the cluster with nct=1 and ParallelGCThreads=4, top shows the following:

$ top -c -u theredia -H -b -n1
top - 20:25:40 up 15 days, 7:00, 1 user, load average: 5.95, 5.41, 5.15
Tasks: 520 total, 7 running, 513 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.3%us, 0.1%sy, 0.1%ni, 88.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 99192688k total, 10184220k used, 89008468k free, 886776k buffers
Swap: 10239992k total, 0k used, 10239992k free, 800664k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29376 theredia 20 0 21.9g 657m 12m R 97.0 0.7 2:13.99 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29377 theredia 20 0 21.9g 657m 12m S 1.9 0.7 0:00.79 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29378 theredia 20 0 21.9g 657m 12m S 1.9 0.7 0:00.79 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29379 theredia 20 0 21.9g 657m 12m S 1.9 0.7 0:00.79 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
30210 theredia 20 0 17424 1504 868 R 1.9 0.0 0:00.01 top -c -u theredia -H -b -n1
29375 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.00 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29380 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.77 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29381 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.35 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29382 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.03 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29383 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.05 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29384 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.00 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29385 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:12.40 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29386 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:10.34 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29387 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.00 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29388 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.06 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29429 theredia 20 0 21.9g 657m 12m S 0.0 0.7 0:00.00 java -Xmx20G -XX:ParallelGCThreads=4 -Djava.io.tmpdir=/tmp/16535.1.lab_dcexs.q -jar /aplic/noarch/GATK/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller
29747 theredia 20 0 112m 1968 888 S 0.0 0.0 0:00.02 sshd: [email protected]/0
29748 theredia 20 0 114m 6060 1460 S 0.0 0.0 0:00.05 -bash

All in all, a count of the number of threads tested is:

12-core node:
nct=1 GC=NO : 21
nct=1 GC=4 : 15
nct=1 GC=8 : 19
nct=1 GC=12 : 23
nct=4 GC=NO : 26
nct=4 GC=4 : 20

20-core node:
nct=1 GC=NO : 26
nct=1 GC=4 : 15
nct=1 GC=8 : 19
nct=1 GC=12 : 23
nct=1 GC=15 : 26
nct=1 GC=20 : 31
nct=4 GC=NO : 31
nct=4 GC=4 : 20
nct=4 GC=8 : 24
nct=8 GC=4 : 24
nct=8 GC=8 : 28

In order to run the calling pipeline for the chromosome 1 of a single 40x sample, I have to request -Xmx20G and ask SunGridEngine to reserve 4 threads for me, even when requesing -nct=1 and parallelGCThreads=4. Any request lower than this ends up with the job stalled at some point with progress reporter stuck at the same point for 3 days before I have to kill the job.

Our cluster has core-binding enabled. Thus, if I request only 1 thread/core, but GATK opens 15/20 threads, all these 19 extra threads have to fight the main thread for cpu time and the main process slows down a lot.

Does GATK really need this amount of resources? I there a way I can further reduce the amount of threads java/GATK is spawning? Our sysadmins are not happy because this rises the cpu load levels of the involved nodes and rises several alarms with no need to.

Thanks in advance,

Txema

Answers

  • tommycarstensentommycarstensen United KingdomMember ✭✭✭

    @txemaheredia It's not necessary for me to use java -XX:ParallelGCThreads, when I run HC with -nct greater than 1. Have you tried without it?

  • txemaherediatxemaheredia Member

    -nct 4 and no -XX:ParallelGCThreads tag leads to 26 threads in a 12-core node, and 31 threads in a 20-core node.

    The point is: I can run GATK in whatever fashion, but, in order for it to finish, I have to reserve 4 execution cores for all these "extra" threads open, even if I use -nct 1.

    In fact, a pipeline execution granting 4 cores and using -nct 1, took ~33 hours to finish, but it used 62 cpu hours ( <33h main thread + >29h "other" threads). Another execution lasting 36 "wallclock" hours, consumed 72 cpu hours.

    So there is a lot going on inside the guts of java/GATK, and that "a lot" does not always respect the number of threads requested. This is a real problem when working in a cluster.

    Issue · Github
    by Sheila

    Issue Number
    74
    State
    closed
    Last Updated
    Closed By
    vdauwera
  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @txemaheredia The behavior you're observing is related to multithreading in the JVM itself, which is outside of GATK control and may require tuning the JVM parameters for java to behave as desired on your particular system. I can only recommend asking your sysadmin or IT staff for help as this is outside the scope of support we can provide.

Sign In or Register to comment.