Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/10/19
Release Notes can be found here.

numerous docker pull timeouts caused numerous calls to fail

esalinasesalinas BroadMember, Broadie ✭✭✭

Yesterday I used FC, and I made two submissions. Each submission had a single workflow. EAch workflow had a scatter in it.

A number of the "scatter" calls end with something like the following in the JES log :

4ef75fcc34a4: Waiting
ba2a9f5e76ae: Waiting
89b43d834129: Waiting
fa3c6d254026: Waiting
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5b/5bb8b784523ce1586c0f923de4d1cf3013c2126cae62fbb5b61beed2afb21d36/data?Expires=1487804016&Signature=EIjH0N1cbqvz~vEOV5bXeCFttj7lxv8CPHsa1LFGDhJB0aJCbKYNrhJ28wAVXNzj5c0zK2VnTsHlI4lIJ1VVAd4KHjH52iMcy3YmJnp52FXwTDDwYqnHqovdTGVeYda2ltwnupUPIgyfGII9h5viYEm2BdSrMtwrycmGjijKt7k_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: net/http: TLS handshake timeout

2017/02/22 22:34:16 I: Copying /var/log/google-genomics/*.log to gs://fc-ef9602f6-0cbc-448c-8fd9-de9d05b66f86/62829caa-7c9d-421e-a9f1-084a8c2e7d10/CallingGroup_Workflow/aa85173c-c8d2-4f64-b3e1-a176db5b49d2/call-Mutect1_Task/shard-10/
2017/02/22 22:34:16 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-ef9602f6-0cbc-448c-8fd9-de9d05b66f86/62829caa-7c9d-421e-a9f1-084a8c2e7d10/CallingGroup_Workflow/aa85173c-c8d2-4f64-b3e1-a176db5b49d2/call-Mutect1_Task/shard-10/

Note the

error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5b/5bb8b784523ce1586c0f923de4d1cf3013c2126cae62fbb5b61beed2afb21d36/data?Expires=1487804016&Signature=EIjH0N1cbqvz~vEOV5bXeCFttj7lxv8CPHsa1LFGDhJB0aJCbKYNrhJ28wAVXNzj5c0zK2VnTsHlI4lIJ1VVAd4KHjH52iMcy3YmJnp52FXwTDDwYqnHqovdTGVeYda2ltwnupUPIgyfGII9h5viYEm2BdSrMtwrycmGjijKt7k_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: net/http: TLS handshake timeout

For such "timeout" reasons, at least one scatter in each workflow was marked as "Failed" in the UI. For that reason each of the workflows failed.

Answers

Sign In or Register to comment.