Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.
Attention:
We will be out of the office on November 11th and 13th 2019, due to the U.S. holiday(Veteran's day) and due to a team event(Nov 13th). We will return to monitoring the GATK forum on November 12th and 14th respectively. Thank you for your patience.

Python error using PostprocessGermlineCNVCalls

Hello,

I set a gCNV pipeline for germline CNVs which works just fine with one of the capture kits we use, but when I started using the same pipeline on a new capture kit, the last step of the pipeline (PostprocessGermlineCNVCalls) fails with this python error:

Using GATK jar /etc/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /etc/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar PostprocessGermlineCNVCalls --calls-shard-path output_dir/normal_cohort_run-calls --model-shard-path output_dir/normal_cohort_run-model --sample-index 0 --autosomal-ref-copy-number 2 --allosomal-contig chrX --allosomal-contig chrY --contig-ploidy-calls contig_ploidy/normal_cohort-calls --output-genotyped-intervals results_vcf/OGI1058_002096.intervals_gCNV.vcf --output-genotyped-segments results_vcf/OGI1058_002096.segments_gCNV.vcf
23:39:06.718 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/etc/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
23:39:06.946 INFO PostprocessGermlineCNVCalls - ------------------------------------------------------------
23:39:06.946 INFO PostprocessGermlineCNVCalls - The Genome Analysis Toolkit (GATK) v4.1.0.0
23:39:06.947 INFO PostprocessGermlineCNVCalls - For support and documentation go to https://software.broadinstitute.org/gatk/
23:39:06.947 INFO PostprocessGermlineCNVCalls - Executing as ju[email protected] on Linux v4.9.0-8-amd64 amd64
23:39:06.947 INFO PostprocessGermlineCNVCalls - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_201-b09
23:39:06.947 INFO PostprocessGermlineCNVCalls - Start Date/Time: March 28, 2019 11:39:06 PM UTC
23:39:06.947 INFO PostprocessGermlineCNVCalls - ------------------------------------------------------------
23:39:06.947 INFO PostprocessGermlineCNVCalls - ------------------------------------------------------------
23:39:06.948 INFO PostprocessGermlineCNVCalls - HTSJDK Version: 2.18.2
23:39:06.948 INFO PostprocessGermlineCNVCalls - Picard Version: 2.18.25
23:39:06.948 INFO PostprocessGermlineCNVCalls - HTSJDK Defaults.COMPRESSION_LEVEL : 2
23:39:06.948 INFO PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
23:39:06.948 INFO PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
23:39:06.948 INFO PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
23:39:06.949 INFO PostprocessGermlineCNVCalls - Deflater: IntelDeflater
23:39:06.949 INFO PostprocessGermlineCNVCalls - Inflater: IntelInflater
23:39:06.949 INFO PostprocessGermlineCNVCalls - GCS max retries/reopens: 20
23:39:06.949 INFO PostprocessGermlineCNVCalls - Requester pays: disabled
23:39:06.949 INFO PostprocessGermlineCNVCalls - Initializing engine
23:39:11.184 INFO PostprocessGermlineCNVCalls - Done initializing engine
23:39:11.365 INFO ProgressMeter - Starting traversal
23:39:11.366 INFO ProgressMeter - Current Locus Elapsed Minutes Records Processed Records/Minute
23:39:11.366 INFO PostprocessGermlineCNVCalls - Generating intervals VCF file...
23:39:11.366 WARN PostprocessGermlineCNVCalls - An variant index will not be created - a sequence dictionary is required to create an output index
23:39:11.398 INFO PostprocessGermlineCNVCalls - Analyzing shard 0...
23:39:11.891 INFO PostprocessGermlineCNVCalls - Generating segments VCF file...
23:39:45.523 INFO PostprocessGermlineCNVCalls - Shutting down engine
[March 28, 2019 11:39:45 PM UTC] org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls done. Elapsed time: 0.65 minutes.
Runtime.totalMemory()=439353344
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
python exited with 1
Command Line: python /tmp/segment_gcnv_calls.8702070710755585046.py --ploidy_calls_path /home/jupyter-user/gCNV/contig_ploidy/normal_cohort-calls --model_shards /home/jupyter-user/gCNV/output_dir/normal_cohort_run-model --calls_shards /home/jupyter-user/gCNV/output_dir/normal_cohort_run-calls --output_path /tmp/gcnv-segmented-calls2860183369929045204 --sample_index 0
Stdout: 23:39:15.197 INFO segment_gcnv_calls - Loading ploidy calls...
23:39:15.198 INFO gcnvkernel.io.io_metadata - Loading germline contig ploidy and global read depth metadata...
23:39:15.514 INFO segment_gcnv_calls - Instantiating the Viterbi segmentation engine...
23:39:15.515 INFO gcnvkernel.postprocess.viterbi_segmentation - Assembling interval list and copy-number class posterior from model shards...
23:39:15.691 INFO gcnvkernel.structs.metadata - Generating intervals metadata...
23:39:15.723 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano forward-backward function...
23:39:21.779 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano Viterbi function...
23:39:43.142 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano variational HHMM...
23:39:43.722 INFO gcnvkernel.postprocess.viterbi_segmentation - Processing sample index: 0, sample name: OGI1058_002096...
23:39:43.739 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (1/24) (contig name: chr1)...
23:39:43.840 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (2/24) (contig name: chr2)...
23:39:43.924 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (3/24) (contig name: chr3)...
23:39:43.968 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (4/24) (contig name: chr4)...
23:39:44.022 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (5/24) (contig name: chr5)...
23:39:44.058 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (6/24) (contig name: chr6)...
23:39:44.103 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (7/24) (contig name: chr7)...
23:39:44.127 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (8/24) (contig name: chr8)...
23:39:44.159 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (9/24) (contig name: chr9)...
23:39:44.189 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (10/24) (contig name: chr10)...
23:39:44.240 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (11/24) (contig name: chr11)...
23:39:44.292 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (12/24) (contig name: chr12)...
23:39:44.337 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (13/24) (contig name: chr13)...
23:39:44.349 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (14/24) (contig name: chr14)...
23:39:44.377 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (15/24) (contig name: chr15)...
23:39:44.408 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (16/24) (contig name: chr16)...
23:39:44.455 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (17/24) (contig name: chr17)...
23:39:44.487 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (18/24) (contig name: chr18)...
23:39:44.493 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (19/24) (contig name: chr19)...
23:39:44.506 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (20/24) (contig name: chr20)...
23:39:44.526 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (21/24) (contig name: chr21)...
23:39:44.530 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (22/24) (contig name: chr22)...
23:39:44.540 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (23/24) (contig name: chrX)...
23:39:44.576 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (24/24) (contig name: chrY)...

Stderr: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/theano/compile/function_module.py", line 884, in call
self.fn() if output_subset is None else\
File "/usr/local/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 989, in rval
r = p(n, [x[0] for x in i], o)
File "/usr/local/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 978, in p
self, node)
File "theano/scan_module/scan_perform.pyx", line 215, in theano.scan_module.scan_perform.perform (/home/jupyter-user/.theano/compiledir_Linux-4.9--amd64-x86_64-with-debian-9.8--3.6.8-64/scan_perform/mod.cpp:2628)
NotImplementedError: We didn't implemented yet the case where scan do 0 iteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/segment_gcnv_calls.8702070710755585046.py", line 73, in
viterbi_engine.write_copy_number_segments()
File "/usr/local/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 234, in write_copy_number_segments
for segment in self._viterbi_segments_generator():
File "/usr/local/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 160, in _viterbi_segments_generator
log_prior_c, log_trans_contig_tcc, copy_number_log_emission_contig_tc)
File "/usr/local/lib/python3.6/site-packages/gcnvkernel/models/theano_hmm.py", line 88, in perform_forward_backward
prev_log_posterior_tc, admixing_rate, temperature)))
File "/usr/local/lib/python3.6/site-packages/theano/compile/function_module.py", line 898, in call
storage_map=getattr(self.fn, 'storage_map', None))
File "/usr/local/lib/python3.6/site-packages/theano/gof/link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python3.6/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/theano/compile/function_module.py", line 884, in call
self.fn() if output_subset is None else\
File "/usr/local/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 989, in rval
r = p(n, [x[0] for x in i], o)
File "/usr/local/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 978, in p
self, node)
File "theano/scan_module/scan_perform.pyx", line 215, in theano.scan_module.scan_perform.perform (/home/jupyter-user/.theano/compiledir_Linux-4.9--amd64-x86_64-with-debian-9.8--3.6.8-64/scan_perform/mod.cpp:2628)
NotImplementedError: We didn't implemented yet the case where scan do 0 iteration
Apply node that caused the error: forall_inplace,cpu,scan_fn}(Elemwise{minimum,no_inplace}.0, InplaceDimShuffle{0,2,1}.0, Subtensor{int64:int64:int64}.0, IncSubtensor{InplaceSet;:int64:}.0, Shape_i{0}.0)
Toposort index: 97
Inputs types: [TensorType(int64, scalar), TensorType(float64, 3D), TensorType(float64, matrix), TensorType(float64, matrix), TensorType(int64, scalar)]
Inputs shapes: [(), (0, 6, 6), (0, 6), (2, 6), ()]
Inputs strides: [(), (288, 8, 48), (48, 8), (48, 8), ()]
Inputs values: [array(0), array([], shape=(0, 6, 6), dtype=float64), array([], shape=(0, 6), dtype=float64), 'not shown', array(6)]
Outputs clients: [[Subtensor{int64:int64:int8}(forall_inplace,cpu,scan_fn}.0, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:151)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:121)
at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.executeSegmentGermlineCNVCallsPythonScript(PostprocessGermlineCNVCalls.java:499)
at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.generateSegmentsVCFFileFromAllShards(PostprocessGermlineCNVCalls.java:435)
at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.traverse(PostprocessGermlineCNVCalls.java:296)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:966)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
at org.broadinstitute.hellbender.Main.main(Main.java:291)

I tough that could be a problem related to my installation, so after the GATK workshop I used Terra to run my pipeline and I get exactly the same error.

If you guys need data to debug it, I have everything ready in Terra.

Tagged:

Best Answer

Answers

Sign In or Register to comment.