Update: July 26, 2019
This section of the forum is no longer actively monitored. We are working on a support migration plan that we will share here shortly. Apologies for this inconvenience.

StorageException Caused by: java.net.UnknownHostException: www.googleapis.com

cwhelancwhelan Cambridge, MAMember, Broadie, Dev

Hi,

I'm trying to run a JointGenotyping workflow using my own cromwell 29 server with a JES backend. I've been seeing some GenomicsDBImport jobs that die after a while with the following exception:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell_root/tmp.2V4kyl
16:16:27.103 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/build/install/gatk/lib/gkl-0.8.2.jar!/com/intel/gkl/native/libgkl_compression.so
16:16:27.287 INFO  GenomicsDBImport - ------------------------------------------------------------
16:16:27.288 INFO  GenomicsDBImport - The Genome Analysis Toolkit (GATK) v4.0.1.0
16:16:27.288 INFO  GenomicsDBImport - For support and documentation go to https://software.broadinstitute.org/gatk/
16:16:27.290 INFO  GenomicsDBImport - Executing as [email protected] on Linux v4.9.0-0.bpo.6-amd64 amd64
16:16:27.290 INFO  GenomicsDBImport - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11
16:16:27.291 INFO  GenomicsDBImport - Start Date/Time: May 31, 2018 4:16:27 PM UTC
16:16:27.291 INFO  GenomicsDBImport - ------------------------------------------------------------
16:16:27.291 INFO  GenomicsDBImport - ------------------------------------------------------------
16:16:27.291 INFO  GenomicsDBImport - HTSJDK Version: 2.14.1
16:16:27.292 INFO  GenomicsDBImport - Picard Version: 2.17.2
16:16:27.292 INFO  GenomicsDBImport - HTSJDK Defaults.COMPRESSION_LEVEL : 1
16:16:27.292 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
16:16:27.292 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
16:16:27.292 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
16:16:27.293 INFO  GenomicsDBImport - Deflater: IntelDeflater
16:16:27.293 INFO  GenomicsDBImport - Inflater: IntelInflater
16:16:27.293 INFO  GenomicsDBImport - GCS max retries/reopens: 20
16:16:27.293 INFO  GenomicsDBImport - Using google-cloud-java patch 6d11bef1c81f885c26b2b56c8616b7a705171e4f from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
16:16:27.293 INFO  GenomicsDBImport - Initializing engine
16:16:29.295 INFO  IntervalArgumentCollection - Processing 23112114 bp from intervals
16:16:29.299 INFO  GenomicsDBImport - Done initializing engine
Created workspace /cromwell_root/genomicsdb
16:16:29.487 INFO  GenomicsDBImport - Vid Map JSON file will be written to genomicsdb/vidmap.json
16:16:29.487 INFO  GenomicsDBImport - Callset Map JSON file will be written to genomicsdb/callset.json
16:16:29.487 INFO  GenomicsDBImport - Complete VCF Header will be written to genomicsdb/vcfheader.vcf
16:16:29.487 INFO  GenomicsDBImport - Importing to array - genomicsdb/genomicsdb_array
16:16:29.505 INFO  ProgressMeter - Starting traversal
16:16:29.505 INFO  ProgressMeter -        Current Locus  Elapsed Minutes     Batches Processed   Batches/Minute
16:16:29.505 INFO  GenomicsDBImport - Starting batch input file preload
16:16:47.420 INFO  GenomicsDBImport - Finished batch preload
16:16:47.420 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
20:10:17.185 INFO  ProgressMeter -          13:63648711            233.8                     1              0.0
20:10:17.185 INFO  GenomicsDBImport - Done importing batch 1/12
20:10:17.185 INFO  GenomicsDBImport - Starting batch input file preload
20:10:37.193 INFO  GenomicsDBImport - Finished batch preload
20:10:37.193 INFO  GenomicsDBImport - Importing batch 2 with 50 samples
00:04:52.541 INFO  ProgressMeter -          13:63648711            468.4                     2              0.0
00:04:52.567 INFO  GenomicsDBImport - Done importing batch 2/12
00:04:52.567 INFO  GenomicsDBImport - Starting batch input file preload
00:05:12.681 INFO  GenomicsDBImport - Finished batch preload
00:05:12.681 INFO  GenomicsDBImport - Importing batch 3 with 50 samples
04:01:07.164 INFO  ProgressMeter -          13:63648711            704.6                     3              0.0
04:01:07.164 INFO  GenomicsDBImport - Done importing batch 3/12
04:01:07.164 INFO  GenomicsDBImport - Starting batch input file preload
04:01:26.383 INFO  GenomicsDBImport - Finished batch preload
04:01:26.383 INFO  GenomicsDBImport - Importing batch 4 with 50 samples
06:31:06.479 INFO  GenomicsDBImport - Shutting down engine
[June 1, 2018 6:31:06 AM UTC] org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport done. Elapsed time: 854.66 minutes.
Runtime.totalMemory()=4285005824
code:      0
message:   www.googleapis.com
reason:    null
location:  null
retryable: false
com.google.cloud.storage.StorageException: www.googleapis.com
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:515)
    at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
    at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
    at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
    at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124)
    at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.read(CloudStorageReadChannel.java:114)
    at htsjdk.samtools.seekablestream.SeekablePathStream.read(SeekablePathStream.java:86)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at htsjdk.samtools.seekablestream.SeekableBufferedStream.read(SeekableBufferedStream.java:108)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBytes(BlockCompressedInputStream.java:571)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBytes(BlockCompressedInputStream.java:560)
    at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:525)
    at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
    at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:241)
    at htsjdk.tribble.readers.TabixReader.readLine(TabixReader.java:215)
    at htsjdk.tribble.readers.TabixReader.access$300(TabixReader.java:48)
    at htsjdk.tribble.readers.TabixReader$IteratorImpl.next(TabixReader.java:434)
    at htsjdk.tribble.readers.TabixIteratorLineReader.readLine(TabixIteratorLineReader.java:46)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:205)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:149)
    at com.intel.genomicsdb.GenomicsDBImporterStreamWrapper.next(GenomicsDBImporterStreamWrapper.java:122)
    at com.intel.genomicsdb.GenomicsDBImporter.importBatch(GenomicsDBImporter.java:1420)
    at org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport.traverse(GenomicsDBImport.java:508)
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:893)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:136)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:152)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:195)
    at org.broadinstitute.hellbender.Main.main(Main.java:275)
Caused by: java.net.UnknownHostException: www.googleapis.com
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
    at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
    at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
    at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
    at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
    at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
    at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380)
    at shaded.cloud_nio.com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:6133)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:494)
    ... 35 more

Does anyone know what these errors are or how to avoid them?

Thanks,

Chris

Answers

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey Chris!

    Not sure how feasible this is, but have you had a chance to run this GenomicsDBImport outside of a Cromwell task? I'm trying to differentiate if the issue is with Cromwell + GCS or GenomicsDBImport + GCS.

  • cwhelancwhelan Cambridge, MAMember, Broadie, Dev

    I'll try to run one now. (Many) other shards in the workflow have completed successfully. The problem is that it takes a long time to run so I might not be able to get back to you with the results for a while. I agree that it's hard to figure out what part of the GATK/Cromwell/GCS stack is responsible for this. Seems like maybe the DNS lookup on the node is glitching and and something is not retrying against that error?

  • cwhelancwhelan Cambridge, MAMember, Broadie, Dev

    I wasn't able to reproduce the error on a GenomicsDBImport task outside of cromwell due to time constraints, but I did decrease my interval size (basically I doubled the number of interval shards, so each new shard should have been about half the size of the original ones). This got my workflow through the GenomicsDBImport stages and beyond. However, I received the same error again in the FinalGatherVcf/GatherVcfsCloud part of the workflow after running for seven hours or so:

    [June 9, 2018 6:35:25 AM UTC] org.broadinstitute.hellbender.tools.GatherVcfsCloud done. Elapsed time: 418.46 minutes.
    Runtime.totalMemory()=6433013760
    java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.storage.StorageException: www.googleapis.com
        at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher.read(SeekableByteChannelPrefetcher.java:309)
        at htsjdk.samtools.seekablestream.SeekablePathStream.read(SeekablePathStream.java:86)
        at htsjdk.samtools.util.IOUtil.transferByStream(IOUtil.java:147)
        at org.broadinstitute.hellbender.tools.GatherVcfsCloud.gatherWithBlockCopying(GatherVcfsCloud.java:434)
        at org.broadinstitute.hellbender.tools.GatherVcfsCloud.doWork(GatherVcfsCloud.java:183)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:136)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:152)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:195)
        at org.broadinstitute.hellbender.Main.main(Main.java:275)
    Caused by: java.util.concurrent.ExecutionException: com.google.cloud.storage.StorageException: www.googleapis.com
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher$WorkUnit.getBuf(SeekableByteChannelPrefetcher.java:136)
        at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher.fetch(SeekableByteChannelPrefetcher.java:255)
        at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher.read(SeekableByteChannelPrefetcher.java:300)
        ... 10 more
    Caused by: com.google.cloud.storage.StorageException: www.googleapis.com
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:515)
        at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
        at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
        at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
        at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124)
        at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.read(CloudStorageReadChannel.java:114)
        at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher$WorkUnit.call(SeekableByteChannelPrefetcher.java:131)
        at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher$WorkUnit.call(SeekableByteChannelPrefetcher.java:104)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.net.UnknownHostException: www.googleapis.com
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
        at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
        at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
        at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380)
        at shaded.cloud_nio.com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:6133)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:494)
        ... 12 more
    

    So it seems like the error might be related to the duration of the job. Any ideas on how I can figure out what's happening or work around this?

    The full log from this task is attached.

Sign In or Register to comment.