To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Cromwell won't run, waiting for changelog lock

Hi all,

I stopped cromwell using ctrl+C after accidentally starting the wrong workflow. However, now it won't start again while using the mysql database. If I run the jar file directly it works, but as soon as I specify the config file that has the mysql configuration it just hangs. See the error log below. I have already rebooted the PC to no avail, so my best guess is that something in the database has gotten inconsistent.

Is there a way to get it running again? The database has a lot of cached workflow runs that I'd rather not loose (I'd estimate more than a week's worth of runtime on this PC).

I think this is the most relevant line:

Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by hostname.local.wifi (my:ipv6:adress:here%wlp5s0) since 5/26/17 9:52 AM

Starting cromwell in server mode

$ java -Dconfig.file=cromwell.conf -jar cromwell-26.jar server
[INFO] [05/26/2017 11:09:43.021] [cromwell-system-akka.actor.default-dispatcher-3] [akka://cromwell-system/user/IO-HTTP/listener-0] Bound to /0.0.0.0:8000
[INFO] [05/26/2017 11:09:43.023] [ForkJoinPool-1-worker-15] [akka.actor.ActorSystemImpl(cromwell-system)] Cromwell service started...
2017-05-26 11:09:43,131  INFO  - Running with database db.url = jdbc:mysql://localhost:3306/cromwell?useSSL=false
2017-05-26 11:09:45,264  INFO  - Waiting for changelog lock....
2017-05-26 11:09:55,265  INFO  - Waiting for changelog lock....
2017-05-26 11:10:05,266  INFO  - Waiting for changelog lock....
2017-05-26 11:10:15,268  INFO  - Waiting for changelog lock....
2017-05-26 11:10:25,269  INFO  - Waiting for changelog lock....
2017-05-26 11:10:35,270  INFO  - Waiting for changelog lock....
2017-05-26 11:10:45,271  INFO  - Waiting for changelog lock....
2017-05-26 11:10:55,272  INFO  - Waiting for changelog lock....
2017-05-26 11:11:05,273  INFO  - Waiting for changelog lock....
2017-05-26 11:11:15,274  INFO  - Waiting for changelog lock....
2017-05-26 11:11:25,275  INFO  - Waiting for changelog lock....
2017-05-26 11:11:35,277  INFO  - Waiting for changelog lock....
2017-05-26 11:11:45,278  INFO  - Waiting for changelog lock....
2017-05-26 11:11:55,279  INFO  - Waiting for changelog lock....
2017-05-26 11:12:05,280  INFO  - Waiting for changelog lock....
2017-05-26 11:12:15,282  INFO  - Waiting for changelog lock....
2017-05-26 11:12:25,283  INFO  - Waiting for changelog lock....
2017-05-26 11:12:35,284  INFO  - Waiting for changelog lock....
2017-05-26 11:12:45,285  INFO  - Waiting for changelog lock....
2017-05-26 11:12:55,286  INFO  - Waiting for changelog lock....
2017-05-26 11:13:05,288  INFO  - Waiting for changelog lock....
2017-05-26 11:13:15,289  INFO  - Waiting for changelog lock....
2017-05-26 11:13:25,290  INFO  - Waiting for changelog lock....
2017-05-26 11:13:35,291  INFO  - Waiting for changelog lock....
2017-05-26 11:13:45,292  INFO  - Waiting for changelog lock....
2017-05-26 11:13:55,294  INFO  - Waiting for changelog lock....
2017-05-26 11:14:05,295  INFO  - Waiting for changelog lock....
2017-05-26 11:14:15,296  INFO  - Waiting for changelog lock....
2017-05-26 11:14:25,297  INFO  - Waiting for changelog lock....
2017-05-26 11:14:35,298  INFO  - Waiting for changelog lock....
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.ExceptionInInitializerError
    at cromwell.server.CromwellRootActor.workflowStore$lzycompute(CromwellRootActor.scala:56)
    at cromwell.server.CromwellRootActor.workflowStore(CromwellRootActor.scala:56)
[ERROR] [SECURITY][05/26/2017 11:14:45.302] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka.actor.ActorSystemImpl(cromwell-system)] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled    at cromwell.server.CromwellRootActor.workflowStoreActor$lzycompute(CromwellRootActor.scala:57)

    at cromwell.server.CromwellRootActor.workflowStoreActor(CromwellRootActor.scala:57)
    at cromwell.server.CromwellRootActor.workflowManagerActor$lzycompute(CromwellRootActor.scala:108)
    at cromwell.server.CromwellRootActor.workflowManagerActor(CromwellRootActor.scala:106)
    at cromwell.server.CromwellServerActor.<init>(CromwellServer.scala:79)
    at cromwell.server.CromwellServerActor$$anonfun$props$1.apply(CromwellServer.scala:84)
    at cromwell.server.CromwellServerActor$$anonfun$props$1.apply(CromwellServer.scala:84)
    at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:87)
    at akka.actor.Props.newActor(Props.scala:213)
    at akka.actor.ActorCell.newActor(ActorCell.scala:562)
    at akka.actor.ActorCell.create(ActorCell.scala:588)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:461)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:483)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
    at akka.dispatch.Mailbox.run(Mailbox.scala:223)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by hostname.local.wifi (my:ipv6:adress:here%wlp5s0) since 5/26/17 9:52 AM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:190)
    at liquibase.Liquibase.update(Liquibase.java:196)
    at liquibase.Liquibase.update(Liquibase.java:192)
    at cromwell.database.migration.liquibase.LiquibaseUtils$.updateSchema(LiquibaseUtils.scala:55)
    at cromwell.database.migration.liquibase.LiquibaseUtils$.updateSchema(LiquibaseUtils.scala:28)
    at cromwell.services.ServicesStore$EnhancedSqlDatabase$$anonfun$initialized$extension$2.apply(ServicesStore.scala:19)
    at cromwell.services.ServicesStore$EnhancedSqlDatabase$$anonfun$initialized$extension$2.apply(ServicesStore.scala:19)
    at cromwell.database.slick.SlickDatabase$$anonfun$withConnection$1.apply(SlickDatabase.scala:122)
    at cromwell.database.slick.SlickDatabase$$anonfun$withConnection$1.apply(SlickDatabase.scala:122)
    at slick.jdbc.SimpleJdbcAction.run(StreamingInvokerAction.scala:70)
    at slick.jdbc.SimpleJdbcAction.run(StreamingInvokerAction.scala:69)
    at slick.backend.DatabaseComponent$DatabaseDef$$anon$2.liftedTree1$1(DatabaseComponent.scala:237)
    at slick.backend.DatabaseComponent$DatabaseDef$$anon$2.run(DatabaseComponent.scala:237)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

Running a workflow

$ java -Dconfig.file=cromwell.conf -jar cromwell-26.jar lissero.wdl lissero.json
[2017-05-26 11:24:33,96] [info] RUN sub-command
[2017-05-26 11:24:33,97] [info]   WDL file: /dev/shm/wdl/lissero.wdl
[2017-05-26 11:24:33,97] [info]   Inputs: /dev/shm/wdl/lissero.json
[INFO] [05/26/2017 11:24:34.001] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor: Submitting workflow
[INFO] [05/26/2017 11:24:34.063] [cromwell-system-akka.dispatchers.engine-dispatcher-7] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor: Workflow submitted UUID(d3c7a60c-655b-4225-9414-2fc58ea1b089)
[INFO] [05/26/2017 11:24:34.063] [cromwell-system-akka.dispatchers.api-dispatcher-9] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowStoreActor/$a] Workflow d3c7a60c-655b-4225-9414-2fc58ea1b089 submitted.
[2017-05-26 11:24:34,51] [info] Running with database db.url = jdbc:mysql://localhost:3306/cromwell?useSSL=false
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-7] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.NoClassDefFoundError: Could not initialize class cromwell.services.SingletonServicesStore$
    at cromwell.server.CromwellRootActor.callCache$lzycompute(CromwellRootActor.scala:76)
    at cromwell.server.CromwellRootActor.callCache(CromwellRootActor.scala:76)
    at cromwell.server.CromwellRootActor.callCacheReadActor$lzycompute(CromwellRootActor.scala:80)
    at cromwell.server.CromwellRootActor.callCacheReadActor(CromwellRootActor.scala:79)
    at cromwell.server.CromwellRootActor.workflowManagerActor$lzycompute(CromwellRootActor.scala:108)
    at cromwell.server.CromwellRootActor.workflowManagerActor(CromwellRootActor.scala:106)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:63)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:59)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at akka.actor.FSM$class.processEvent(FSM.scala:663)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor.akka$actor$LoggingFSM$$super$processEvent(SingleWorkflowRunnerActor.scala:37)
    at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
    at cromwell.engine.workflow.SingleWorkflowRunnerActor.processEvent(SingleWorkflowRunnerActor.scala:37)
    at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)[ERROR] [SECURITY][05/26/2017 11:29:36.759] [cromwell-system-akka.dispatchers.engine-dispatcher-7] [akka.actor.ActorSystemImpl(cromwell-system)] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-7] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)

    at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
    at cromwell.server.CromwellRootActor.aroundReceive(CromwellRootActor.scala:43)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Tagged:

Best Answer

Answers

Sign In or Register to comment.