CNNScoreVariants Hanging in 4.1.0

I am trying to run CNNScoreVariants in GATK 4.1.0 but the tool seems to hang on the 'INFO NativeLibraryLoader - Loading libgkl_utils.so from jar' step for both the 1D and 2D models.

My issues seems similar to this post, but the hang up occurs at a different location:
(gatkforums.broadinstitute.org/gatk/discussion/12384/cnnscorevariants-hanging-in-4-0-5-2-and-4-0-6-0)

I have tried the accepted answer in the above post without success. Any help would be appreciated.

Comments

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    Hi @pbrennan

    Would you please post the exact command you are using and the entire error log.

  • RFRichholtRFRichholt Phoenix, AZMember
    edited March 1

    Came here to post the same issue, here's my logs. This was run with bash -v so that you can see the exact command that was executed. But, the project and sample name have been redacted.

    One more note, I also ran this with the --verbosity DEBUG option. With that option, it hung on this final line in the logs:

    10:53:12.091 DEBUG NativeLibraryLoader - Extracting libgkl_utils.so to /scratch/rrichholt/Phoenix-9579dca/<PROJECT>/temp/libgkl_utils8789314355369295443.so
    

    Full logs:

    module load gatk/4.1.0.0
    /cm/local/apps/environment-modules/3.2.10/Modules/$MODULE_VERSION/bin/modulecmd bash $*
    CLASSPATH=/packages/java/jdk1.8.0_91/jre/lib:/packages/java/jdk1.8.0_91/lib/tools.jar ;export CLASSPATH;JAVA_HOME=/packages/java/jre1.8.0_91:/packages/java/jdk1.8.0_91 ;export JAVA_HOME;LD_LIBRARY_PATH=/packages/R/3.3.2/lib64/R/lib:/packages/python/3.6.2/lib:/opt/slurm/current/lib64/slurm:/opt/slurm/current/lib64 ;export LD_LIBRARY_PATH;LOADEDMODULES=slurm/new:3.6.2:1.8.0_91:3.3.2:gatk/4.1.0.0 ;export LOADEDMODULES;MANPATH=/packages/R/3.3.2/share/man:/packages/python/3.6.2/share/man/:/opt/slurm/current/share/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man:/cm/local/apps/environment-modules/current/share/man ;export MANPATH;PATH=/packages/gatk/4.1.0.0:/packages/R/3.3.2/bin:/packages/java/jre1.8.0_91/bin:/packages/java/jdk1.8.0_91/bin:/packages/python/3.6.2/bin:/home/rrichholt/elasticsearch/elasticsearch-6.2.3/bin:/home/rrichholt/martian/bin:/home/rrichholt/go/go-1.10/bin:/home/rrichholt/.cargo/bin:/opt/slurm/current/sbin:/opt/slurm/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/cm/local/apps/environment-modules/3.2.10/bin:/opt/dell/srvadmin/bin:~/bin/:~/scripts/:~/.local/bin/:~/toolkit_rrichholt/ ;export PATH;PYTHONPATH=/packages/python/3.6.2 ;export PYTHONPATH;PYTHON_CONFIG=/packages/python/3.6.2/lib/python3.6/config-3.6m-x86_64-linux-gnu/python-config.py ;export PYTHON_CONFIG;_LMFILES_=/packages/modules-3.2.9/modulefiles/slurm/new:/packages/modules-3.2.9/modulefiles/python/3.6.2:/packages/modules-3.2.9/modulefiles/java/1.8.0_91:/packages/modules-3.2.9/modulefiles/R/3.3.2:/packages/modules-3.2.9/modulefiles/gatk/4.1.0.0 ;export _LMFILES_;
    
    gatk CNNScoreVariants \
      --java-options "-Xmx6G" \
      --tensor-type read_tensor \
      --reference "/home/tgenref/homo_sapiens/grch38_hg38/broad_resource_bundle/Homo_sapiens_assembly38.fasta" \
      --variant "exome/constitutional_variant_calls/haplotypecaller/<SAMPLE>/<SAMPLE>.bwa.hc.vcf.gz" \
      --input "exome/alignment/bwa/<SAMPLE>/<SAMPLE>.bwa.bam" \
      --output "exome/constitutional_variant_calls/haplotypecaller/<SAMPLE>/<SAMPLE>.bwa.scored.hc.vcf.gz"
    02:52:39.608 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/packages/gatk/4.1.0.0/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    Mar 01, 2019 2:52:39 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    WARNING: Failed to detect whether we are running on Google Compute Engine.
    java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
        at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:104)
        at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
        at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.runningOnComputeEngine(ComputeEngineCredentials.java:210)
        at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.tryGetComputeCredentials(DefaultCredentialsProvider.java:290)
        at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:207)
        at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:124)
        at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)
        at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)
        at com.google.cloud.ServiceOptions.defaultCredentials(ServiceOptions.java:304)
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:278)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31)
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78)
        at org.broadinstitute.hellbender.utils.gcs.BucketUtils.setGlobalNIODefaultOptions(BucketUtils.java:353)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:182)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
        at org.broadinstitute.hellbender.Main.main(Main.java:291)
    
    Mar 01, 2019 2:52:40 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    WARNING: Failed to detect whether we are running on Google Compute Engine.
    java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
        at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:104)
        at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
        at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.runningOnComputeEngine(ComputeEngineCredentials.java:210)
        at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.tryGetComputeCredentials(DefaultCredentialsProvider.java:290)
        at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:207)
        at shaded.cloud_nio.com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:124)
        at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)
        at shaded.cloud_nio.com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)
        at com.google.cloud.ServiceOptions.defaultCredentials(ServiceOptions.java:304)
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:278)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31)
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78)
        at org.broadinstitute.hellbender.utils.gcs.BucketUtils.setGlobalNIODefaultOptions(BucketUtils.java:353)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:182)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
        at org.broadinstitute.hellbender.Main.main(Main.java:291)
    
    02:52:40.505 INFO  CNNScoreVariants - ------------------------------------------------------------
    02:52:40.505 INFO  CNNScoreVariants - The Genome Analysis Toolkit (GATK) v4.1.0.0
    02:52:40.505 INFO  CNNScoreVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
    02:52:40.506 INFO  CNNScoreVariants - Executing as [email protected] on Linux v3.10.0-957.5.1.el7.x86_64 amd64
    02:52:40.506 INFO  CNNScoreVariants - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_91-b14
    02:52:40.506 INFO  CNNScoreVariants - Start Date/Time: March 1, 2019 2:52:39 AM MST
    02:52:40.506 INFO  CNNScoreVariants - ------------------------------------------------------------
    02:52:40.506 INFO  CNNScoreVariants - ------------------------------------------------------------
    02:52:40.507 INFO  CNNScoreVariants - HTSJDK Version: 2.18.2
    02:52:40.507 INFO  CNNScoreVariants - Picard Version: 2.18.25
    02:52:40.507 INFO  CNNScoreVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    02:52:40.507 INFO  CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    02:52:40.507 INFO  CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    02:52:40.507 INFO  CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    02:52:40.508 INFO  CNNScoreVariants - Deflater: IntelDeflater
    02:52:40.508 INFO  CNNScoreVariants - Inflater: IntelInflater
    02:52:40.508 INFO  CNNScoreVariants - GCS max retries/reopens: 20
    02:52:40.508 INFO  CNNScoreVariants - Requester pays: disabled
    02:52:40.508 INFO  CNNScoreVariants - Initializing engine
    02:52:41.437 INFO  FeatureManager - Using codec VCFCodec to read file file:///scratch/rrichholt/Phoenix-9579dca/<PROJECT>/exome/constitutional_variant_calls/haplotypecaller/<SAMPLE>/<SAMPLE>.bwa.hc.vcf.gz
    02:52:42.013 INFO  CNNScoreVariants - Done initializing engine
    02:52:42.014 INFO  NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/packages/gatk/4.1.0.0/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
    
  • pbrennanpbrennan Member
    edited March 1
    Here is the command:

    ```
    gatk CNNScoreVariants -V GIAB.vcf.gz -R human_g1k_v37_decoy.fasta -O CNNScore_Variant_GATK4.1_GIAB.vcf
    ```

    Output:
    ```
    Using GATK jar /gatk/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar CNNScoreVariants -V GIAB.vcf.gz -R human_g1k_v37_decoy.fasta -O CNNScore_Variant_GATK4.1_GIAB.vcf
    Picked up JAVA_TOOL_OPTIONS: -XX:ParallelGCThreads=1
    16:33:59.168 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    16:33:59.642 INFO CNNScoreVariants - ------------------------------------------------------------
    16:33:59.642 INFO CNNScoreVariants - The Genome Analysis Toolkit (GATK) v4.1.0.0
    16:33:59.642 INFO CNNScoreVariants - For support and documentation go to
    16:33:59.642 INFO CNNScoreVariants - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
    16:33:59.642 INFO CNNScoreVariants - Start Date/Time: March 1, 2019 4:33:59 PM EST
    16:33:59.643 INFO CNNScoreVariants - ------------------------------------------------------------
    16:33:59.643 INFO CNNScoreVariants - ------------------------------------------------------------
    16:33:59.643 INFO CNNScoreVariants - HTSJDK Version: 2.18.2
    16:33:59.643 INFO CNNScoreVariants - Picard Version: 2.18.25
    16:33:59.643 INFO CNNScoreVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    16:33:59.643 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    16:33:59.643 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    16:33:59.643 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    16:33:59.643 INFO CNNScoreVariants - Deflater: IntelDeflater
    16:33:59.643 INFO CNNScoreVariants - Inflater: IntelInflater
    16:33:59.643 INFO CNNScoreVariants - GCS max retries/reopens: 20
    16:33:59.644 INFO CNNScoreVariants - Requester pays: disabled
    16:33:59.644 INFO CNNScoreVariants - Initializing engine
    16:34:00.097 INFO FeatureManager - Using codec VCFCodec to read file GIAB.vcf.gz
    16:34:00.347 INFO CNNScoreVariants - Done initializing engine
    16:34:00.348 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file: /gatk/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
    ```
    There are no error logs. It will hang on "INFO NativeLibraryLoader - Loading libgkl_utils.so" and never complete that step.
  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    @RFRichholt and @pbrennan

    Looks like the issue here is that the gatk conda env is not active and that is causing this issue. Please follow this link for more information on how to activate the gatk conda env: https://software.broadinstitute.org/gatk/documentation/article?id=12836

    We have a github issue ticket to make this easier for the users: https://github.com/broadinstitute/gatk/issues/4127

  • RFRichholtRFRichholt Phoenix, AZMember

    Thank you @bhanuGandham, that fixed the issue.

Sign In or Register to comment.