Problem with Docker-Images pulled Self-Hosted Registries

Hi,
I run into a little problem since using the latest cromewell-version 29 locally. Every time when cromewell starts a locally created docker-Container I get the following WARN-Messages:
[WARN] [09/04/2017 12:29:10.315] [cromwell-system-akka.dispatchers.engine-dispatcher-39] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-dd1c8ef5-506f-4088-89fb-fb417aa2345c/WorkflowExecutionActor-dd1c8ef5$
java.lang.Exception: Unauthorized to get docker hash transwf:170904
with a long java stack trace like:
at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:195)
at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:93)
at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:77)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
at akka.actor.FSM.processEvent(FSM.scala:663)
at akka.actor.FSM.processEvent$(FSM.scala:660)
at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:41)
at akka.actor.LoggingFSM.processEvent(FSM.scala:799)
at akka.actor.LoggingFSM.processEvent$(FSM.scala:781)
at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:41)
at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:657)
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:513)
at akka.actor.Actor.aroundReceive$(Actor.scala:511)
at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:41)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Ok that legit since local created images dont have a hash/digests. To get rid of the warn-message i created my own locally registry, as described in the docker-docs. I push and pull my image to the registry and checked with docker images --digests
if it has a hash (it has now ). I checked if I can run the container from the shell and insert the new container-name into the wdl-file and hit start. Now I get a java.lang.IllegalArgumentException from the cromewell-server:
[ERROR] [09/04/2017 12:24:27.649] [cromwell-system-akka.dispatchers.engine-dispatcher-85] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor] WorkflowManagerActor Workflow 2faa7ebb-6354-4251-baf6-9c1432f619ae failed (durin$
java.lang.IllegalArgumentException: Docker image localhost:5000/capture/transwf:170904 has an invalid syntax.
Any Idea what went wrong ?
Thank you in advance,
greetings
EADG
Best Answers
-
mcovarr Cambridge, MAMember, Broadie, Dev
Hash lookup is generally useful for provenance and reproducibility so I'm not sure we'd want to disable that just because we turn off call caching. It would certainly be reasonable to ask for an enhancement for the option to disable hash lookups with the understanding that call caching cannot be used in that mode.
Answers
Our regex does not allow the local port, but it should. I filed this bug.
Hi @danb,
thanks for that, would it also be possible to add a flag to the applicationconf.json to disable the check of the docker hash/digests ?
Greetings EADG
Turning off digest lookups would disable call caching. Would you still want to turn off lookups if call caching were disabled as a result?
Hi @mcovarr,
I currently don't use call caching, so it isn't activated. Is it possible to automatically deactivate digest lookup together with call caching? When is it only used for that purpose?
Hash lookup is generally useful for provenance and reproducibility so I'm not sure we'd want to disable that just because we turn off call caching. It would certainly be reasonable to ask for an enhancement for the option to disable hash lookups with the understanding that call caching cannot be used in that mode.
Enhancement request created:
https://github.com/broadinstitute/cromwell/issues/2600
@mcovarr
Thank you for that!
Hello, is there any solution for this problem? I should run in a local backend a pipeline using a docker from a local registry, but I reach the same problem as here. Any idea of how to go over it?
I am using cromwell-30.2...
As mentioned above the various issues in this post have been migrated over as enhancement requests. Please continue the discussion and follow the statuses on GitHub: