OutOfMemoryError when call caching is turned on

Hi,

I'm trying to run a WDL workflow with the LSF backend and a mySQL database. When I set call-caching.enabled=false, the workflow runs fine, correctly submits jobs to LSF and writes to the mySQL database, but the pipeline state does not persist from one run to the next, as expected. When I set call-caching.enabled=true, the workflow dies right away with this error:

[2016-07-01 12:10:43,484] [info] WorkflowActor [bc069339]: Start(Some(Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor#1907937338])) message received
java.lang.OutOfMemoryError: Required array size too large
        at java.nio.file.Files.readAllBytes(Files.java:3156)
        at better.files.File$$anonfun$digest$1.apply(File.scala:168)
        at better.files.File$$anonfun$digest$1.apply(File.scala:166)
        at scala.collection.immutable.Stream.foreach(Stream.scala:594)
        at better.files.File.digest(File.scala:166)
        at better.files.File.checksum(File.scala:177)
        at better.files.File.md5(File.scala:179)
        at cromwell.engine.backend.io.package$PathEnhanced$.hash$extension(package.scala:33)
        at cromwell.engine.backend.WorkflowDescriptor$$anonfun$fileHasher$1.apply(WorkflowDescriptor.scala:65)
        at cromwell.engine.backend.WorkflowDescriptor$$anonfun$fileHasher$1.apply(WorkflowDescriptor.scala:63)
        at wdl4s.values.WdlValue$class.computeHash(WdlValue.scala:63)
        at wdl4s.values.WdlSingleFile.computeHash(WdlFile.scala:39)
        at cromwell.engine.backend.WorkflowDescriptor.hash(WorkflowDescriptor.scala:229)
        at cromwell.engine.workflow.WorkflowActor$$anonfun$57.apply(WorkflowActor.scala:1127)
        at cromwell.engine.workflow.WorkflowActor$$anonfun$57.apply(WorkflowActor.scala:1126)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at cromwell.engine.workflow.WorkflowActor.buildSymbolStoreEntries(WorkflowActor.scala:1126)
        at cromwell.engine.workflow.WorkflowActor.createWorkflow(WorkflowActor.scala:255)
        at cromwell.engine.workflow.WorkflowActor$Start$$anonfun$runInitialization$1.apply(WorkflowActor.scala:125)
        at cromwell.engine.workflow.WorkflowActor$Start$$anonfun$runInitialization$1.apply(WorkflowActor.scala:124)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Any idea what could be going wrong? I'm running Cromwell with a large JVM (15Gb).

Thanks,
Pam

Tagged:

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    At the moment, our answer is to simply throw more memory at it. Future releases will work on better optimizing the call-caching.

Sign In or Register to comment.