To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

OutOfMemoryError when call caching is turned on

prussellprussell Denver, COMember

Hi,

I'm trying to run a WDL workflow with the LSF backend and a mySQL database. When I set call-caching.enabled=false, the workflow runs fine, correctly submits jobs to LSF and writes to the mySQL database, but the pipeline state does not persist from one run to the next, as expected. When I set call-caching.enabled=true, the workflow dies right away with this error:

[2016-07-01 12:10:43,484] [info] WorkflowActor [bc069339]: Start(Some(Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor#1907937338])) message received
java.lang.OutOfMemoryError: Required array size too large
        at java.nio.file.Files.readAllBytes(Files.java:3156)
        at better.files.File$$anonfun$digest$1.apply(File.scala:168)
        at better.files.File$$anonfun$digest$1.apply(File.scala:166)
        at scala.collection.immutable.Stream.foreach(Stream.scala:594)
        at better.files.File.digest(File.scala:166)
        at better.files.File.checksum(File.scala:177)
        at better.files.File.md5(File.scala:179)
        at cromwell.engine.backend.io.package$PathEnhanced$.hash$extension(package.scala:33)
        at cromwell.engine.backend.WorkflowDescriptor$$anonfun$fileHasher$1.apply(WorkflowDescriptor.scala:65)
        at cromwell.engine.backend.WorkflowDescriptor$$anonfun$fileHasher$1.apply(WorkflowDescriptor.scala:63)
        at wdl4s.values.WdlValue$class.computeHash(WdlValue.scala:63)
        at wdl4s.values.WdlSingleFile.computeHash(WdlFile.scala:39)
        at cromwell.engine.backend.WorkflowDescriptor.hash(WorkflowDescriptor.scala:229)
        at cromwell.engine.workflow.WorkflowActor$$anonfun$57.apply(WorkflowActor.scala:1127)
        at cromwell.engine.workflow.WorkflowActor$$anonfun$57.apply(WorkflowActor.scala:1126)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at cromwell.engine.workflow.WorkflowActor.buildSymbolStoreEntries(WorkflowActor.scala:1126)
        at cromwell.engine.workflow.WorkflowActor.createWorkflow(WorkflowActor.scala:255)
        at cromwell.engine.workflow.WorkflowActor$Start$$anonfun$runInitialization$1.apply(WorkflowActor.scala:125)
        at cromwell.engine.workflow.WorkflowActor$Start$$anonfun$runInitialization$1.apply(WorkflowActor.scala:124)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Any idea what could be going wrong? I'm running Cromwell with a large JVM (15Gb).

Thanks,
Pam

Tagged:

Answers

Sign In or Register to comment.