Cromwell won't run, waiting for changelog lock

Hi all,

I stopped cromwell using ctrl+C after accidentally starting the wrong workflow. However, now it won't start again while using the mysql database. If I run the jar file directly it works, but as soon as I specify the config file that has the mysql configuration it just hangs. See the error log below. I have already rebooted the PC to no avail, so my best guess is that something in the database has gotten inconsistent.

Is there a way to get it running again? The database has a lot of cached workflow runs that I'd rather not loose (I'd estimate more than a week's worth of runtime on this PC).

I think this is the most relevant line:

Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by hostname.local.wifi (my:ipv6:adress:here%wlp5s0) since 5/26/17 9:52 AM

Starting cromwell in server mode

$ java -Dconfig.file=cromwell.conf -jar cromwell-26.jar server
[INFO] [05/26/2017 11:09:43.021] [cromwell-system-akka.actor.default-dispatcher-3] [akka://cromwell-system/user/IO-HTTP/listener-0] Bound to /0.0.0.0:8000
[INFO] [05/26/2017 11:09:43.023] [ForkJoinPool-1-worker-15] [akka.actor.ActorSystemImpl(cromwell-system)] Cromwell service started...
2017-05-26 11:09:43,131  INFO  - Running with database db.url = jdbc:mysql://localhost:3306/cromwell?useSSL=false
2017-05-26 11:09:45,264  INFO  - Waiting for changelog lock....
2017-05-26 11:09:55,265  INFO  - Waiting for changelog lock....
2017-05-26 11:10:05,266  INFO  - Waiting for changelog lock....
2017-05-26 11:10:15,268  INFO  - Waiting for changelog lock....
2017-05-26 11:10:25,269  INFO  - Waiting for changelog lock....
2017-05-26 11:10:35,270  INFO  - Waiting for changelog lock....
2017-05-26 11:10:45,271  INFO  - Waiting for changelog lock....
2017-05-26 11:10:55,272  INFO  - Waiting for changelog lock....
2017-05-26 11:11:05,273  INFO  - Waiting for changelog lock....
2017-05-26 11:11:15,274  INFO  - Waiting for changelog lock....
2017-05-26 11:11:25,275  INFO  - Waiting for changelog lock....
2017-05-26 11:11:35,277  INFO  - Waiting for changelog lock....
2017-05-26 11:11:45,278  INFO  - Waiting for changelog lock....
2017-05-26 11:11:55,279  INFO  - Waiting for changelog lock....
2017-05-26 11:12:05,280  INFO  - Waiting for changelog lock....
2017-05-26 11:12:15,282  INFO  - Waiting for changelog lock....
2017-05-26 11:12:25,283  INFO  - Waiting for changelog lock....
2017-05-26 11:12:35,284  INFO  - Waiting for changelog lock....
2017-05-26 11:12:45,285  INFO  - Waiting for changelog lock....
2017-05-26 11:12:55,286  INFO  - Waiting for changelog lock....
2017-05-26 11:13:05,288  INFO  - Waiting for changelog lock....
2017-05-26 11:13:15,289  INFO  - Waiting for changelog lock....
2017-05-26 11:13:25,290  INFO  - Waiting for changelog lock....
2017-05-26 11:13:35,291  INFO  - Waiting for changelog lock....
2017-05-26 11:13:45,292  INFO  - Waiting for changelog lock....
2017-05-26 11:13:55,294  INFO  - Waiting for changelog lock....
2017-05-26 11:14:05,295  INFO  - Waiting for changelog lock....
2017-05-26 11:14:15,296  INFO  - Waiting for changelog lock....
2017-05-26 11:14:25,297  INFO  - Waiting for changelog lock....
2017-05-26 11:14:35,298  INFO  - Waiting for changelog lock....
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.ExceptionInInitializerError
    at cromwell.server.CromwellRootActor.workflowStore$lzycompute(CromwellRootActor.scala:56)
    at cromwell.server.CromwellRootActor.workflowStore(CromwellRootActor.scala:56)
[ERROR] [SECURITY][05/26/2017 11:14:45.302] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka.actor.ActorSystemImpl(cromwell-system)] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled    at cromwell.server.CromwellRootActor.workflowStoreActor$lzycompute(CromwellRootActor.scala:57)

    at cromwell.server.CromwellRootActor.workflowStoreActor(CromwellRootActor.scala:57)
    at cromwell.server.CromwellRootActor.workflowManagerActor$lzycompute(CromwellRootActor.scala:108)
    at cromwell.server.CromwellRootActor.workflowManagerActor(CromwellRootActor.scala:106)
    at cromwell.server.CromwellServerActor.<init>(CromwellServer.scala:79)
    at cromwell.server.CromwellServerActor$$anonfun$props$1.apply(CromwellServer.scala:84)
    at cromwell.server.CromwellServerActor$$anonfun$props$1.apply(CromwellServer.scala:84)
    at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:87)
    at akka.actor.Props.newActor(Props.scala:213)
    at akka.actor.ActorCell.newActor(ActorCell.scala:562)
    at akka.actor.ActorCell.create(ActorCell.scala:588)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:461)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:483)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
    at akka.dispatch.Mailbox.run(Mailbox.scala:223)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by hostname.local.wifi (my:ipv6:adress:here%wlp5s0) since 5/26/17 9:52 AM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:190)
    at liquibase.Liquibase.update(Liquibase.java:196)
    at liquibase.Liquibase.update(Liquibase.java:192)
    at cromwell.database.migration.liquibase.LiquibaseUtils$.updateSchema(LiquibaseUtils.scala:55)
    at cromwell.database.migration.liquibase.LiquibaseUtils$.updateSchema(LiquibaseUtils.scala:28)
    at cromwell.services.ServicesStore$EnhancedSqlDatabase$$anonfun$initialized$extension$2.apply(ServicesStore.scala:19)
    at cromwell.services.ServicesStore$EnhancedSqlDatabase$$anonfun$initialized$extension$2.apply(ServicesStore.scala:19)
    at cromwell.database.slick.SlickDatabase$$anonfun$withConnection$1.apply(SlickDatabase.scala:122)
    at cromwell.database.slick.SlickDatabase$$anonfun$withConnection$1.apply(SlickDatabase.scala:122)
    at slick.jdbc.SimpleJdbcAction.run(StreamingInvokerAction.scala:70)
    at slick.jdbc.SimpleJdbcAction.run(StreamingInvokerAction.scala:69)
    at slick.backend.DatabaseComponent$DatabaseDef$$anon$2.liftedTree1$1(DatabaseComponent.scala:237)
    at slick.backend.DatabaseComponent$DatabaseDef$$anon$2.run(DatabaseComponent.scala:237)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

Running a workflow

$ java -Dconfig.file=cromwell.conf -jar cromwell-26.jar lissero.wdl lissero.json
[2017-05-26 11:24:33,96] [info] RUN sub-command
[2017-05-26 11:24:33,97] [info]   WDL file: /dev/shm/wdl/lissero.wdl
[2017-05-26 11:24:33,97] [info]   Inputs: /dev/shm/wdl/lissero.json
[INFO] [05/26/2017 11:24:34.001] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor: Submitting workflow
[INFO] [05/26/2017 11:24:34.063] [cromwell-system-akka.dispatchers.engine-dispatcher-7] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor: Workflow submitted UUID(d3c7a60c-655b-4225-9414-2fc58ea1b089)
[INFO] [05/26/2017 11:24:34.063] [cromwell-system-akka.dispatchers.api-dispatcher-9] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowStoreActor/$a] Workflow d3c7a60c-655b-4225-9414-2fc58ea1b089 submitted.
[2017-05-26 11:24:34,51] [info] Running with database db.url = jdbc:mysql://localhost:3306/cromwell?useSSL=false
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-7] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.NoClassDefFoundError: Could not initialize class cromwell.services.SingletonServicesStore$
    at cromwell.server.CromwellRootActor.callCache$lzycompute(CromwellRootActor.scala:76)
    at cromwell.server.CromwellRootActor.callCache(CromwellRootActor.scala:76)
    at cromwell.server.CromwellRootActor.callCacheReadActor$lzycompute(CromwellRootActor.scala:80)
    at cromwell.server.CromwellRootActor.callCacheReadActor(CromwellRootActor.scala:79)
    at cromwell.server.CromwellRootActor.workflowManagerActor$lzycompute(CromwellRootActor.scala:108)
    at cromwell.server.CromwellRootActor.workflowManagerActor(CromwellRootActor.scala:106)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:63)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:59)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at akka.actor.FSM$class.processEvent(FSM.scala:663)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor.akka$actor$LoggingFSM$$super$processEvent(SingleWorkflowRunnerActor.scala:37)
    at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor.processEvent(SingleWorkflowRunnerActor.scala:37)
    at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)[ERROR] [SECURITY][05/26/2017 11:29:36.759] [cromwell-system-akka.dispatchers.engine-dispatcher-7] [akka.actor.ActorSystemImpl(cromwell-system)] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-7] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)

    at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
    at cromwell.server.CromwellRootActor.aroundReceive(CromwellRootActor.scala:43)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Tagged:

Best Answer

Answers

Sign In or Register to comment.