Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

CNNScoreVariants 2D Not Working

I am trying to run the pre-trained CNNScoreVariants 2D model using the GATK docker image on my local machine. I was able to successfully run the 1D model, but the 2D keeps failing. It appears the tools attempts to run, but does not produce any results. I have posted the edited log below (system does not allow me to post links). Any help would be greatly appreciated.

```
(gatk) [email protected]:/gatk# gatk CNNScoreVariants -I my_data/NA12878_IDT_CNV_GATK4.1.Recalibrated.bam -V my_data/NA12878_IDT_CNV_GATK4.1.HC.vcf.gz -R my_data/human_g1k_v37_decoy.fasta -O CNNScore2D_Variant_GATK4.1_NA12878.vcf -inference-batch-size 2 -transfer-batch-size 2 -tensor-type read_tensor

Using GATK jar /gatk/gatk-package-4.1.0.0-local.jar

Running:

java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-package-4.1.0.0-local.jar CNNScoreVariants -I my_data/NA12878_IDT_CNV_GATK4.1.Recalibrated.bam -V my_data/NA12878_IDT_CNV_GATK4.1.HC.vcf.gz -R my_data/human_g1k_v37_decoy.fasta -O CNNScore2D_Variant_GATK4.1_NA12878.vcf -inference-batch-size 2 -transfer-batch-size 2 -tensor-type read_tensor

18:33:32.413 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so

Apr 17, 2019 6:33:33 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine

WARNING: Failed to detect whether we are running on Google Compute Engine.

java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at sun.net.NetworkClient.doConnect(NetworkClient.java:175)


at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:104)

at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)

at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.runningOnComputeEngine(ComputeEngineCredentials.java:210)

at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.tryGetComputeCredentials(DefaultCredentialsProvider.java:290)

at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:207)

at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:124)

at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)

at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)

at com.google.cloud.ServiceOptions.defaultCredentials(ServiceOptions.java:304)

at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:278)

at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83)

at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31)

at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78)

at org.broadinstitute.hellbender.utils.gcs.BucketUtils.setGlobalNIODefaultOptions(BucketUtils.java:353)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:182)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)

at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)

at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)

at org.broadinstitute.hellbender.Main.main(Main.java:291)




Apr 17, 2019 6:33:33 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine

WARNING: Failed to detect whether we are running on Google Compute Engine.

java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at sun.net.NetworkClient.doConnect(NetworkClient.java:175)


at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.runningOnComputeEngine(ComputeEngineCredentials.java:210)

at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.tryGetComputeCredentials(DefaultCredentialsProvider.java:290)

at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:207)

at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:124)

at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)

at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)

at com.google.cloud.ServiceOptions.defaultCredentials(ServiceOptions.java:304)

at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:278)

at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83)

at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31)

at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78)

at org.broadinstitute.hellbender.utils.gcs.BucketUtils.setGlobalNIODefaultOptions(BucketUtils.java:353)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:182)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)

at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)

at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)

at org.broadinstitute.hellbender.Main.main(Main.java:291)




18:33:33.170 INFO CNNScoreVariants - ------------------------------------------------------------

18:33:33.171 INFO CNNScoreVariants - The Genome Analysis Toolkit (GATK) v4.1.0.0

18:33:33.171 INFO CNNScoreVariants - For support and documentation go to

18:33:33.171 INFO CNNScoreVariants - Executing as [email protected] on Linux v4.9.125-linuxkit amd64

18:33:33.171 INFO CNNScoreVariants - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12

18:33:33.172 INFO CNNScoreVariants - Start Date/Time: April 17, 2019 6:33:32 PM UTC

18:33:33.172 INFO CNNScoreVariants - ------------------------------------------------------------

18:33:33.172 INFO CNNScoreVariants - ------------------------------------------------------------

18:33:33.173 INFO CNNScoreVariants - HTSJDK Version: 2.18.2

18:33:33.173 INFO CNNScoreVariants - Picard Version: 2.18.25

18:33:33.173 INFO CNNScoreVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2

18:33:33.173 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

18:33:33.173 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

18:33:33.174 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

18:33:33.174 INFO CNNScoreVariants - Deflater: IntelDeflater

18:33:33.174 INFO CNNScoreVariants - Inflater: IntelInflater

18:33:33.174 INFO CNNScoreVariants - GCS max retries/reopens: 20

18:33:33.174 INFO CNNScoreVariants - Requester pays: disabled

18:33:33.174 INFO CNNScoreVariants - Initializing engine

WARNING: BAM index file /gatk/my_data/NA12878_IDT_CNV_GATK4.1.Recalibrated.bai is older than BAM /gatk/my_data/NA12878_IDT_CNV_GATK4.1.Recalibrated.bam

18:33:33.572 INFO FeatureManager - Using codec VCFCodec to read file file:///gatk/my_data/NA12878_IDT_CNV_GATK4.1.HC.vcf.gz

18:33:33.698 INFO CNNScoreVariants - Done initializing engine

18:33:33.699 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/gatk/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so

18:33:36.040 INFO CNNScoreVariants - Using key:CNN_2D for CNN architecture:/tmp/small_2d.1767057937227156485.json and weights:/tmp/small_2d.3995843942938171054.hd5

18:33:36.567 INFO ProgressMeter - Starting traversal

18:33:36.567 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute

18:33:36.573 INFO CNNScoreVariants - Starting first pass through the variants

18:33:43.926 INFO CNNScoreVariants - Done scoring variants with CNN.

18:33:43.945 INFO CNNScoreVariants - Shutting down engine

[April 17, 2019 6:33:44 PM UTC] org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants done. Elapsed time: 0.20 minutes.

Runtime.totalMemory()=218628096

org.broadinstitute.hellbender.exceptions.GATKException: Exception waiting for ack from Python: org.broadinstitute.hellbender.exceptions.GATKException: Expected message of length 3 but only found 0 bytes

at org.broadinstitute.hellbender.utils.runtime.StreamingProcessController.waitForAck(StreamingProcessController.java:233)

at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.waitForAck(StreamingPythonScriptExecutor.java:207)

at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.waitForPreviousBatchCompletion(StreamingPythonScriptExecutor.java:284)

at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.sendBatchIfReady(CNNScoreVariants.java:406)

at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.firstPassApply(CNNScoreVariants.java:326)

at org.broadinstitute.hellbender.engine.TwoPassVariantWalker.lambda$traverseVariants$0(TwoPassVariantWalker.java:76)

at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)

at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at java.util.Iterator.forEachRemaining(Iterator.java:116)

at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)

at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)

at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)

at org.broadinstitute.hellbender.engine.TwoPassVariantWalker.traverseVariants(TwoPassVariantWalker.java:74)

at org.broadinstitute.hellbender.engine.TwoPassVariantWalker.traverse(TwoPassVariantWalker.java:27)

at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:966)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)

at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)

at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)

at org.broadinstitute.hellbender.Main.main(Main.java:291)

Caused by: java.util.concurrent.ExecutionException: org.broadinstitute.hellbender.exceptions.GATKException: Expected message of length 3 but only found 0 bytes

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at org.broadinstitute.hellbender.utils.runtime.StreamingProcessController.waitForAck(StreamingProcessController.java:228)


Caused by: org.broadinstitute.hellbender.exceptions.GATKException: Expected message of length 3 but only found 0 bytes

at org.broadinstitute.hellbender.utils.runtime.StreamingProcessController.getBytesFromStream(StreamingProcessController.java:261)

at org.broadinstitute.hellbender.utils.runtime.StreamingProcessController.lambda$waitForAck$0(StreamingProcessController.java:208)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)
```

Answers

Sign In or Register to comment.