OutOfMemoryError when call caching is turned on

prussellprussell Denver, COMember

Hi,

I'm trying to run a WDL workflow with the LSF backend and a mySQL database. When I set call-caching.enabled=false, the workflow runs fine, correctly submits jobs to LSF and writes to the mySQL database, but the pipeline state does not persist from one run to the next, as expected. When I set call-caching.enabled=true, the workflow dies right away with this error:

[2016-07-01 12:10:43,484] [info] WorkflowActor [bc069339]: Start(Some(Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor#1907937338])) message received
java.lang.OutOfMemoryError: Required array size too large
        at java.nio.file.Files.readAllBytes(Files.java:3156)
        at better.files.File$$anonfun$digest$1.apply(File.scala:168)
        at better.files.File$$anonfun$digest$1.apply(File.scala:166)
        at scala.collection.immutable.Stream.foreach(Stream.scala:594)
        at better.files.File.digest(File.scala:166)
        at better.files.File.checksum(File.scala:177)
        at better.files.File.md5(File.scala:179)
        at cromwell.engine.backend.io.package$PathEnhanced$.hash$extension(package.scala:33)
        at cromwell.engine.backend.WorkflowDescriptor$$anonfun$fileHasher$1.apply(WorkflowDescriptor.scala:65)
        at cromwell.engine.backend.WorkflowDescriptor$$anonfun$fileHasher$1.apply(WorkflowDescriptor.scala:63)
        at wdl4s.values.WdlValue$class.computeHash(WdlValue.scala:63)
        at wdl4s.values.WdlSingleFile.computeHash(WdlFile.scala:39)
        at cromwell.engine.backend.WorkflowDescriptor.hash(WorkflowDescriptor.scala:229)
        at cromwell.engine.workflow.WorkflowActor$$anonfun$57.apply(WorkflowActor.scala:1127)
        at cromwell.engine.workflow.WorkflowActor$$anonfun$57.apply(WorkflowActor.scala:1126)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at cromwell.engine.workflow.WorkflowActor.buildSymbolStoreEntries(WorkflowActor.scala:1126)
        at cromwell.engine.workflow.WorkflowActor.createWorkflow(WorkflowActor.scala:255)
        at cromwell.engine.workflow.WorkflowActor$Start$$anonfun$runInitialization$1.apply(WorkflowActor.scala:125)
        at cromwell.engine.workflow.WorkflowActor$Start$$anonfun$runInitialization$1.apply(WorkflowActor.scala:124)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Any idea what could be going wrong? I'm running Cromwell with a large JVM (15Gb).

Thanks,
Pam

Tagged:

Answers

Sign In or Register to comment.