Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

Exception in PostprocessGermlineCNVCalls by CNV calling on gatk 4.1.1 and gatk 4.1.2

Hi team,

I tried to run GATK4 pipeline on HPC with PBSPro and everything's good except to the quite last step (I can't make a successful run so far that I don't know the actual final step yet) that got the exception in the PostprocessGermlineCNVCalls step. I tried with 4 BAM files of exome data.

I tried with both gatk-package-4.1.1.0-local.jar and gatk-package-4.1.2.0-local.jar and got different exceptions. In both cases, both genotyped-segments and genotyped-intervals vcf.gz could be generated successfully.

4.1.2

If using gatk-package-4.1.2.0-local.jar, I got the exception:
23:32:28.928 INFO PostprocessGermlineCNVCalls - Generating intervals VCF file...
23:32:28.928 WARN PostprocessGermlineCNVCalls - A variant index will not be created - a sequence dictionary is required to create an output index
23:32:28.950 INFO PostprocessGermlineCNVCalls - Writing intervals VCF file to /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/genotyped-intervals-RAP083-PRO.chr9.sorted.markdup.realign.recal.vcf.gz...
23:32:28.950 INFO PostprocessGermlineCNVCalls - Analyzing shard 0 / 21...
23:32:29.935 INFO PostprocessGermlineCNVCalls - Analyzing shard 1 / 21...
23:32:30.451 INFO PostprocessGermlineCNVCalls - Analyzing shard 2 / 21...
23:32:30.877 INFO PostprocessGermlineCNVCalls - Analyzing shard 3 / 21...
23:32:31.116 INFO PostprocessGermlineCNVCalls - Analyzing shard 4 / 21...
23:32:31.349 INFO PostprocessGermlineCNVCalls - Analyzing shard 5 / 21...
23:32:31.561 INFO PostprocessGermlineCNVCalls - Analyzing shard 6 / 21...
23:32:31.834 INFO PostprocessGermlineCNVCalls - Analyzing shard 7 / 21...
23:32:32.067 INFO PostprocessGermlineCNVCalls - Analyzing shard 8 / 21...
23:32:32.263 INFO PostprocessGermlineCNVCalls - Analyzing shard 9 / 21...
23:32:32.500 INFO PostprocessGermlineCNVCalls - Analyzing shard 10 / 21...
23:32:32.821 INFO PostprocessGermlineCNVCalls - Analyzing shard 11 / 21...
23:32:33.219 INFO PostprocessGermlineCNVCalls - Analyzing shard 12 / 21...
23:32:33.492 INFO PostprocessGermlineCNVCalls - Analyzing shard 13 / 21...
23:32:33.694 INFO PostprocessGermlineCNVCalls - Analyzing shard 14 / 21...
23:32:33.880 INFO PostprocessGermlineCNVCalls - Analyzing shard 15 / 21...
23:32:34.068 INFO PostprocessGermlineCNVCalls - Analyzing shard 16 / 21...
23:32:34.255 INFO PostprocessGermlineCNVCalls - Analyzing shard 17 / 21...
23:32:34.439 INFO PostprocessGermlineCNVCalls - Analyzing shard 18 / 21...
23:32:34.780 INFO PostprocessGermlineCNVCalls - Analyzing shard 19 / 21...
23:32:35.194 INFO PostprocessGermlineCNVCalls - Analyzing shard 20 / 21...
23:32:35.552 INFO PostprocessGermlineCNVCalls - Generating segments VCF file...
23:32:38.485 INFO PostprocessGermlineCNVCalls - Shutting down engine
[May 14, 2019 11:32:38 PM HKT] org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls done. Elapsed time: 0.89 minutes.
Runtime.totalMemory()=2228224000
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
python exited with 1
Command Line: python /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/tmp.02bb7386/segment_gcnv_calls.1253628288024294994.py --ploidy_calls_path /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/extracted-contig-ploidy-calls --model_shards /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_0 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_1 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_2 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_3 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_4 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_5 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_6 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_7 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_8 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_9 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_10 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_11 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_12 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_13 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_14 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_15 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_16 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_17 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_18 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_19 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/MODEL_20 --calls_shards /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_0 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_1 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_2 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_3 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_4 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_5 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_6 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_7 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_8 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_9 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_10 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_11 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_12 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_13 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_14 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_15 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_16 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_17 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_18 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_19 /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/execution/CALLS_20 --output_path /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/tmp.02bb7386/gcnv-segmented-calls7338880877476252409 --sample_index 1
Stdout:
Stderr: Traceback (most recent call last):
File "/home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/e9cb1678-4365-45a5-a92f-8a0da98ad6b7/call-PostprocessGermlineCNVCalls/shard-1/tmp.02bb7386/segment_gcnv_calls.1253628288024294994.py", line 9, in
import gcnvkernel
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/init.py", line 1, in
from pymc3 import version as pymc3_version
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/pymc3/init.py", line 5, in
from .distributions import *
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/init.py", line 1, in
from . import timeseries
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/timeseries.py", line 5, in
from .continuous import get_tau_sd, Normal, Flat
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/continuous.py", line 16, in
from pymc3.theanof import floatX
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/pymc3/theanof.py", line 89, in
empty_gradient = tt.zeros(0, dtype='float32')
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/tensor/basic.py", line 2558, in zeros
return alloc(np.array(0, dtype=dtype), *shape)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/tensor/basic.py", line 3091, in call
ret = super(Alloc, self).call(val, *shapes, **kwargs)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 670, in __call__
no_recycling=[])
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 955, in make_thunk
no_recycling)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 858, in make_c_thunk
output_storage=node_output_storage)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk
keep_lock=keep_lock)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in __compile__
keep_lock=keep_lock)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1623, in cthunk_factory
module = get_module_cache().module_from_key(
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 48, in get_module_cache
return cmodule.get_module_cache(config.compiledir, init_args=init_args)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1587, in get_module_cache
_module_cache = ModuleCache(dirname, **init_args)
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 703, in __init__
self.refresh()
File "/home/mullinyu/anaconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 794, in refresh
files = os.listdir(root)
FileNotFoundError: [Errno 2] No such file or directory: '/home/mullinyu/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.2-64/tmp9k7blvt4'

at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:151)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:121)
at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.executeSegmentGermlineCNVCallsPythonScript(PostprocessGermlineCNVCalls.java:509)
at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.generateSegmentsVCFFileFromAllShards(PostprocessGermlineCNVCalls.java:447)
at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.onTraversalSuccess(PostprocessGermlineCNVCalls.java:304)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1041)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
at org.broadinstitute.hellbender.Main.main(Main.java:291)

Using GATK jar /home/mullinyu/software/gatk4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx7000m -jar /home/mullinyu/software/gatk4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar PostprocessGermlineCNVCalls --calls-shard-path CALLS_0 --calls-shard-path CALLS_1 --calls-shard-path CALLS_2 --calls-shard-path CALLS_3 --calls-shard-path CALLS_4 --calls-shard-path CALLS_5 --calls-shard-path CALLS_6 --calls-shard-path CALLS_7 --calls-shard-path CALLS_8 --calls-shard-path CALLS_9 --calls-shard-path CALLS_10 --calls-shard-path CALLS_11 --calls-shard-path CALLS_12 --calls-shard-path CALLS_13 --calls-shard-path CALLS_14 --calls-shard-path CALLS_15 --calls-shard-path CALLS_16 --calls-shard-path CALLS_17 --calls-shard-path CALLS_18 --calls-shard-path CALLS_19 --calls-shard-path CALLS_20 --model-shard-path MODEL_0 --model-shard-path MODEL_1 --model-shard-path MODEL_2 --model-shard-path MODEL_3 --model-shard-path MODEL_4 --model-shard-path MODEL_5 --model-shard-path MODEL_6 --model-shard-path MODEL_7 --model-shard-path MODEL_8 --model-shard-path MODEL_9 --model-shard-path MODEL_10 --model-shard-path MODEL_11 --model-shard-path MODEL_12 --model-shard-path MODEL_13 --model-shard-path MODEL_14 --model-shard-path MODEL_15 --model-shard-path MODEL_16 --model-shard-path MODEL_17 --model-shard-path MODEL_18 --model-shard-path MODEL_19 --model-shard-path MODEL_20 --allosomal-contig chrX --allosomal-contig chrY --autosomal-ref-copy-number 2 --contig-ploidy-calls extracted-contig-ploidy-calls --sample-index 1 --output-genotyped-intervals genotyped-intervals-RAP083-PRO.chr9.sorted.markdup.realign.recal.vcf.gz --output-genotyped-segments genotyped-segments-RAP083-PRO.chr9.sorted.markdup.realign.recal.vcf.gz

4.1.1

If using gatk-package-4.1.1.0-local.jar, I got the following exception like:

16:45:43.888 INFO ProgressMeter - Starting traversal
16:45:43.888 INFO ProgressMeter - Current Locus Elapsed Minutes Records Processed Records/Minute
16:45:43.889 INFO ProgressMeter - unmapped 0.0 0 NaN
16:45:43.889 INFO ProgressMeter - Traversal complete. Processed 0 total records in 0.0 minutes.
16:45:43.889 INFO PostprocessGermlineCNVCalls - Generating intervals VCF file...
16:45:43.889 WARN PostprocessGermlineCNVCalls - A variant index will not be created - a sequence dictionary is required to create an output index
16:45:43.927 INFO PostprocessGermlineCNVCalls - Writing intervals VCF file to /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/321c2174-e98e-4c8e-a9b2-2aba00cfd66a/call-PostprocessGermlineCNVCalls/shard-3/execution/genotyped-intervals-RAP083-FAT.chr9.sorted.markdup.realign.recal.vcf.gz...
16:45:43.927 INFO PostprocessGermlineCNVCalls - Analyzing shard 0 / 21...
16:45:44.782 INFO PostprocessGermlineCNVCalls - Analyzing shard 1 / 21...
16:45:45.094 INFO PostprocessGermlineCNVCalls - Analyzing shard 2 / 21...
16:45:45.362 INFO PostprocessGermlineCNVCalls - Analyzing shard 3 / 21...
16:45:45.598 INFO PostprocessGermlineCNVCalls - Analyzing shard 4 / 21...
16:45:46.000 INFO PostprocessGermlineCNVCalls - Analyzing shard 5 / 21...
16:45:46.366 INFO PostprocessGermlineCNVCalls - Analyzing shard 6 / 21...
16:45:46.556 INFO PostprocessGermlineCNVCalls - Analyzing shard 7 / 21...
16:45:46.754 INFO PostprocessGermlineCNVCalls - Analyzing shard 8 / 21...
16:45:46.946 INFO PostprocessGermlineCNVCalls - Analyzing shard 9 / 21...
16:45:47.372 INFO PostprocessGermlineCNVCalls - Analyzing shard 10 / 21...
16:45:47.808 INFO PostprocessGermlineCNVCalls - Analyzing shard 11 / 21...
16:45:48.021 INFO PostprocessGermlineCNVCalls - Analyzing shard 12 / 21...
16:45:48.243 INFO PostprocessGermlineCNVCalls - Analyzing shard 13 / 21...
16:45:48.464 INFO PostprocessGermlineCNVCalls - Analyzing shard 14 / 21...
16:45:48.648 INFO PostprocessGermlineCNVCalls - Analyzing shard 15 / 21...
16:45:48.837 INFO PostprocessGermlineCNVCalls - Analyzing shard 16 / 21...
16:45:49.022 INFO PostprocessGermlineCNVCalls - Analyzing shard 17 / 21...
16:45:49.204 INFO PostprocessGermlineCNVCalls - Analyzing shard 18 / 21...
16:45:49.387 INFO PostprocessGermlineCNVCalls - Analyzing shard 19 / 21...
16:45:49.580 INFO PostprocessGermlineCNVCalls - Analyzing shard 20 / 21...
16:45:49.786 INFO PostprocessGermlineCNVCalls - Generating segments VCF file...
16:48:49.213 INFO PostprocessGermlineCNVCalls - Writing segments VCF file to /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/321c2174-e98e-4c8e-a9b2-2aba00cfd66a/call-PostprocessGermlineCNVCalls/shard-3/execution/genotyped-segments-RAP083-FAT.chr9.sorted.markdup.realign.recal.vcf.gz...
16:48:49.214 WARN PostprocessGermlineCNVCalls - A variant index will not be created - a sequence dictionary is required to create an output index
16:48:49.236 INFO PostprocessGermlineCNVCalls - PostprocessGermlineCNVCalls complete.
16:48:49.236 INFO PostprocessGermlineCNVCalls - Shutting down engine
[May 14, 2019 4:48:49 PM HKT] org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls done. Elapsed time: 3.72 minutes.
Runtime.totalMemory()=2163212288
Exception in thread "Thread-1" htsjdk.samtools.util.RuntimeIOException: java.nio.file.DirectoryNotEmptyException: /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/321c2174-e98e-4c8e-a9b2-2aba00cfd66a/call-PostprocessGermlineCNVCalls/shard-3/tmp.080ce2f6/gcnv-segmented-calls3302613394812803063/SAMPLE_3
at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1346)
at org.broadinstitute.hellbender.utils.io.IOUtils.deleteRecursively(IOUtils.java:1061)
at org.broadinstitute.hellbender.utils.io.DeleteRecursivelyOnExitPathHook.runHooks(DeleteRecursivelyOnExitPathHook.java:56)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.DirectoryNotEmptyException: /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/321c2174-e98e-4c8e-a9b2-2aba00cfd66a/call-PostprocessGermlineCNVCalls/shard-3/tmp.080ce2f6/gcnv-segmented-calls3302613394812803063/SAMPLE_3
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1165)
at htsjdk.samtools.util.IOUtil$3.postVisitDirectory(IOUtil.java:1338)
at htsjdk.samtools.util.IOUtil$3.postVisitDirectory(IOUtil.java:1327)
at java.nio.file.Files.walkFileTree(Files.java:2688)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at htsjdk.samtools.util.IOUtil.recursiveDelete(IOUtil.java:1344)
... 3 more
Using GATK jar /home/mullinyu/software/gatk4/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx7000m -jar /home/mullinyu/software/gatk4/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar PostprocessGermlineCNVCalls --calls-shard-path CALLS_0 --calls-shard-path CALLS_1 --calls-shard-path CALLS_2 --calls-shard-path CALLS_3 --calls-shard-path CALLS_4 --calls-shard-path CALLS_5 --calls-shard-path CALLS_6 --calls-shard-path CALLS_7 --calls-shard-path CALLS_8 --calls-shard-path CALLS_9 --calls-shard-path CALLS_10 --calls-shard-path CALLS_11 --calls-shard-path CALLS_12 --calls-shard-path CALLS_13 --calls-shard-path CALLS_14 --calls-shard-path CALLS_15 --calls-shard-path CALLS_16 --calls-shard-path CALLS_17 --calls-shard-path CALLS_18 --calls-shard-path CALLS_19 --calls-shard-path CALLS_20 --model-shard-path MODEL_0 --model-shard-path MODEL_1 --model-shard-path MODEL_2 --model-shard-path MODEL_3 --model-shard-path MODEL_4 --model-shard-path MODEL_5 --model-shard-path MODEL_6 --model-shard-path MODEL_7 --model-shard-path MODEL_8 --model-shard-path MODEL_9 --model-shard-path MODEL_10 --model-shard-path MODEL_11 --model-shard-path MODEL_12 --model-shard-path MODEL_13 --model-shard-path MODEL_14 --model-shard-path MODEL_15 --model-shard-path MODEL_16 --model-shard-path MODEL_17 --model-shard-path MODEL_18 --model-shard-path MODEL_19 --model-shard-path MODEL_20 --allosomal-contig chrX --allosomal-contig chrY --autosomal-ref-copy-number 2 --contig-ploidy-calls extracted-contig-ploidy-calls --sample-index 3 --output-genotyped-intervals genotyped-intervals-RAP083-FAT.chr9.sorted.markdup.realign.recal.vcf.gz --output-genotyped-segments genotyped-segments-RAP083-FAT.chr9.sorted.markdup.realign.recal.vcf.gz

Please advise. Also, what's the expected result files if everything is running fine.

Thanks!

Tagged:

Answers

  • bshifawbshifaw Member, Broadie, Moderator admin

    Hi @mullinyu ,

    Mind attaching the wdl and the json you are executing. Also, post the command used to execute the wdl and what directory you are executing from.

    At first glance the problem might be related to the system the workflow is being run. The stderr shows that the workflow is having trouble access the systems temp folder.
    4.1.2
    FileNotFoundError: [Errno 2] No such file or directory: '/home/mullinyu/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.2-64/tmp9k7blvt4'

    4.1.1
    Exception in thread "Thread-1" htsjdk.samtools.util.RuntimeIOException: java.nio.file.DirectoryNotEmptyException: /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/321c2174-e98e-4c8e-a9b2-2aba00cfd66a/call-PostprocessGermlineCNVCalls/shard-3/tmp.080ce2f6/gcnv-segmented-calls3302613394812803063/SAMPLE_3

    What is the diskspace and memory of the HPC? How much of it is being used?

  • mullinyumullinyu HKMember
    edited May 16

    i executed gcnv.sh at the directory /home/mullinyu/project/gcnv

    I attached the files as per requested.

    You gave me a clue - Indeed, I could see that empy directory

    [[email protected] SAMPLE_3]$ls -al /home/mullinyu/project/gcnv/cromwell-executions/CNVGermlineCohortWorkflow/321c2174-e98e-4c8e-a9b2-2aba00cfd66a/call-PostprocessGermlineCNVCalls/shard-3/tmp.080ce2f6/gcnv-segmented-calls3302613394812803063/SAMPLE_3
    total 8
    drwxr-xr-x 1 mullinyu paedgrp 244 May 14 16:47 .
    drwx------ 1 mullinyu paedgrp  80 May 14 16:46 ..
    [[email protected] SAMPLE_3]$
    

    But, you can see "mullinyu" was the owner to create the directory and the account used for running this job. It should have the permissoin to remove the directory.

    Post edited by bshifaw on
  • sleeslee Member, Broadie, Dev ✭✭✭

    I suspect this may be related to https://github.com/broadinstitute/gatk/issues/5893, which was fixed after the 4.1.2.0 release (however, see https://github.com/broadinstitute/gatk/issues/5945). @mullinyu can you try running PostprocessGermlineCNVCalls with the latest master? You shouldn't have to rerun previous steps, as everything should be backwards compatible.

  • YangyxtYangyxt Member

    @bshifaw @slee ,

    Hi, I just found that in @mullinyu log file, there is a statement of :

    16:45:43.888 INFO ProgressMeter - Starting traversal 16:45:43.888 INFO ProgressMeter - Current Locus Elapsed Minutes Records Processed Records/Minute 16:45:43.889 INFO ProgressMeter - unmapped 0.0 0 NaN 16:45:43.889 INFO ProgressMeter - Traversal complete. Processed 0 total records in 0.0 minutes. 16:45:43.889 INFO PostprocessGermlineCNVCalls - Generating intervals VCF file...

    I also found this in my log file but I do not receive any errors and I got my results. Would you mind explaining what does this mean? Is this normal? FYI, I used 4.1.3.0 GATK.

    Thanks!

  • sleeslee Member, Broadie, Dev ✭✭✭

    @Yangyxt this is indeed normal. You may notice that many GATK tools perform traversals over various types of records---intervals, reads, variants, etc.---in which case the ProgressMeter provides updates on the status of the traversal. The code to perform these traversals is implemented in the GATK engine, which is also responsible for things like command-line argument parsing. Tools like PostprocessGermlineCNVCalls that do not actually traverse any records but otherwise use the engine may output trivial status updates like the one you point out---these can be safely ignored.

  • YangyxtYangyxt Member

    @slee said:
    @Yangyxt this is indeed normal. You may notice that many GATK tools perform traversals over various types of records---intervals, reads, variants, etc.---in which case the ProgressMeter provides updates on the status of the traversal. The code to perform these traversals is implemented in the GATK engine, which is also responsible for things like command-line argument parsing. Tools like PostprocessGermlineCNVCalls that do not actually traverse any records but otherwise use the engine may output trivial status updates like the one you point out---these can be safely ignored.

    Thanks @slee for the information!

Sign In or Register to comment.