Running in server mode, jobs that have localization error become immortal

conradLconradL BrisbaneMember

Running cromwell in server mode, with default configuration in each case, I can reproduce the following behaviour in 0.18, 0.19 and 0.19_hotfix (HEAD):

Submit a workflow that has non-existent file as input to a task, e.g.:
task BillyBob { File bbInput command { echo "done" } } workflow badLocalization { call BillyBob { input: bbInput="/foo/bar/baz" } }

The server log shows "Failures during localization" error (below) - as expected initially, I guess - but then repeats the error every 30 seconds or so, forever, and hitting the API <workflowId>/status endpoint shows the job in a "Running" state, forever. I would expect this error to cause the task, and then the workflow, to die.

example of a single block of the server log error:
2016-05-27 11:08:57,269 cromwell-system-akka.actor.default-dispatcher-5 ERROR - Failures during localization java.lang.UnsupportedOperationException: Could not localize /foo/bar/baz -> /home/conradL/cromwell-executions/badLocalization/8c7774be-7917-4c6a-88c4-55e495bbb9ec/call-BillyBob/foo/bar/baz at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19] at scala.Option.getOrElse(Option.scala:121) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.localize$1(SharedFileSystem.scala:242) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustFile$1(SharedFileSystem.scala:264) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.localizeWdlValue(SharedFileSystem.scala:271) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.localizeWdlValue(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:225) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[cromwell-0.19.jar:0.19] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustSharedInputPaths(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustSharedInputPaths(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustInputPaths(LocalBackend.scala:96) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.instantiateCommand(LocalBackend.scala:246) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand$lzycompute(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:115) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:113) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[cromwell-0.19.jar:0.19] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [cromwell-0.19.jar:0.19] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [cromwell-0.19.jar:0.19] 2016-05-27 11:08:57,270 cromwell-system-akka.actor.default-dispatcher-5 ERROR - Failures during localization java.lang.UnsupportedOperationException: Could not localize /foo/bar/baz -> /home/conradL/cromwell-executions/badLocalization/8c7774be-7917-4c6a-88c4-55e495bbb9ec/call-BillyBob/foo/bar/baz at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19] at scala.Option.getOrElse(Option.scala:121) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.localize$1(SharedFileSystem.scala:242) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustFile$1(SharedFileSystem.scala:264) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.localizeWdlValue(SharedFileSystem.scala:271) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.localizeWdlValue(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:225) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[cromwell-0.19.jar:0.19] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustSharedInputPaths(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustSharedInputPaths(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustInputPaths(LocalBackend.scala:96) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.instantiateCommand(LocalBackend.scala:246) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand$lzycompute(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:115) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:113) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[cromwell-0.19.jar:0.19] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [cromwell-0.19.jar:0.19] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [cromwell-0.19.jar:0.19] 2016-05-27 11:08:57,270 cromwell-system-akka.actor.default-dispatcher-4 ERROR - Failures during localization java.lang.UnsupportedOperationException: Could not localize /foo/bar/baz -> /home/conradL/cromwell-executions/badLocalization/8c7774be-7917-4c6a-88c4-55e495bbb9ec/call-BillyBob/foo/bar/baz at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19] at scala.Option.getOrElse(Option.scala:121) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.localize$1(SharedFileSystem.scala:242) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustFile$1(SharedFileSystem.scala:264) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.localizeWdlValue(SharedFileSystem.scala:271) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.localizeWdlValue(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:225) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[cromwell-0.19.jar:0.19] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[cromwell-0.19.jar:0.19] at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19] at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustSharedInputPaths(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustSharedInputPaths(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustInputPaths(LocalBackend.scala:96) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.instantiateCommand(LocalBackend.scala:246) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand$lzycompute(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:115) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:113) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[cromwell-0.19.jar:0.19] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) ~[cromwell-0.19.jar:0.19] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [cromwell-0.19.jar:0.19] 2016-05-27 11:08:57,271 cromwell-system-akka.actor.default-dispatcher-4 ERROR - BackendCallExecutionActor [UUID(8c7774be):BillyBob]: Failures during localizationCould not localize /foo/bar/baz -> /home/conradL/cromwell-executions/badLocalization/8c7774be-7917-4c6a-88c4-55e495bbb9ec/call-BillyBob/foo/bar/baz cromwell.util.AggregatedException: Failures during localizationCould not localize /foo/bar/baz -> /home/conradL/cromwell-executions/badLocalization/8c7774be-7917-4c6a-88c4-55e495bbb9ec/call-BillyBob/foo/bar/baz at cromwell.util.TryUtil$.sequenceIterable(TryUtil.scala:118) ~[cromwell-0.19.jar:0.19] at cromwell.util.TryUtil$.sequence(TryUtil.scala:125) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.SharedFileSystem$class.adjustSharedInputPaths(SharedFileSystem.scala:228) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustSharedInputPaths(LocalBackend.scala:94) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.adjustInputPaths(LocalBackend.scala:96) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend.instantiateCommand(LocalBackend.scala:246) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand$lzycompute(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:115) ~[cromwell-0.19.jar:0.19] at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:113) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[cromwell-0.19.jar:0.19] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[cromwell-0.19.jar:0.19] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) ~[cromwell-0.19.jar:0.19] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [cromwell-0.19.jar:0.19] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [cromwell-0.19.jar:0.19]

Tagged:

Best Answer

Answers

Sign In or Register to comment.