Problem with Docker-Images pulled Self-Hosted Registries

EADG

Hi,

I run into a little problem since using the latest cromewell-version 29 locally. Every time when cromewell starts a locally created docker-Container I get the following WARN-Messages:

[WARN] [09/04/2017 12:29:10.315] [cromwell-system-akka.dispatchers.engine-dispatcher-39] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-dd1c8ef5-506f-4088-89fb-fb417aa2345c/WorkflowExecutionActor-dd1c8ef5$
java.lang.Exception: Unauthorized to get docker hash transwf:170904

with a long java stack trace like:

        at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:195)
        at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:93)
        at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:77)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
        at akka.actor.FSM.processEvent(FSM.scala:663)
        at akka.actor.FSM.processEvent$(FSM.scala:660)
        at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:41)
        at akka.actor.LoggingFSM.processEvent(FSM.scala:799)
        at akka.actor.LoggingFSM.processEvent$(FSM.scala:781)
        at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:41)
        at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:657)
        at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
        at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
        at akka.actor.Actor.aroundReceive(Actor.scala:513)
        at akka.actor.Actor.aroundReceive$(Actor.scala:511)
        at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:41)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
        at akka.actor.ActorCell.invoke(ActorCell.scala:496)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Ok that legit since local created images dont have a hash/digests. To get rid of the warn-message i created my own locally registry, as described in the docker-docs. I push and pull my image to the registry and checked with docker images --digests if it has a hash (it has now ). I checked if I can run the container from the shell and insert the new container-name into the wdl-file and hit start. Now I get a java.lang.IllegalArgumentException from the cromewell-server:

[ERROR] [09/04/2017 12:24:27.649] [cromwell-system-akka.dispatchers.engine-dispatcher-85] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor] WorkflowManagerActor Workflow 2faa7ebb-6354-4251-baf6-9c1432f619ae failed (durin$
java.lang.IllegalArgumentException: Docker image localhost:5000/capture/transwf:170904 has an invalid syntax.

Any Idea what went wrong ?

Thank you in advance,
greetings
EADG

danb

Our regex does not allow the local port, but it should. I filed this bug.

mcovarr

Hash lookup is generally useful for provenance and reproducibility so I'm not sure we'd want to disable that just because we turn off call caching. It would certainly be reasonable to ask for an enhancement for the option to disable hash lookups with the understanding that call caching cannot be used in that mode.

danb

Our regex does not allow the local port, but it should. I filed this bug.

EADG

Hi @danb,

thanks for that, would it also be possible to add a flag to the applicationconf.json to disable the check of the docker hash/digests ?

Greetings EADG

mcovarr

Turning off digest lookups would disable call caching. Would you still want to turn off lookups if call caching were disabled as a result?

EADG

Hi @mcovarr,
I currently don't use call caching, so it isn't activated. Is it possible to automatically deactivate digest lookup together with call caching? When is it only used for that purpose?

mcovarr

Hash lookup is generally useful for provenance and reproducibility so I'm not sure we'd want to disable that just because we turn off call caching. It would certainly be reasonable to ask for an enhancement for the option to disable hash lookups with the understanding that call caching cannot be used in that mode.

mcovarr

Enhancement request created:

https://github.com/broadinstitute/cromwell/issues/2600

EADG

@mcovarr

Thank you for that!

magicDGS

Hello, is there any solution for this problem? I should run in a local backend a pipeline using a docker from a local registry, but I reach the same problem as here. Any idea of how to go over it?

I am using cromwell-30.2...

kshakir

As mentioned above the various issues in this post have been migrated over as enhancement requests. Please continue the discussion and follow the statuses on GitHub:

• https://github.com/broadinstitute/cromwell/issues/2592
• https://github.com/broadinstitute/cromwell/issues/2600