Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Java heap space error when running a custom rna-seq WDL pipeline using Cromwell on google cloud

Hi ,
I'm getting a Java heap-space error while trying to scatter 80 samples with a concurrent job-limit of 1000 after processing 70 samples on the Google cloud using Cromwell40 on a VM instance with 240 GB ram . Reducing the concurrent job limit to 800 didn't seem to resolve the issue. What is the best way to resolve this issue. Is there a memory you would recommend to use , if I have the ability to scale up my VM. I bascially would like to process these samples in 4-6 hours , on an average it takes 3 hours to run the entire pipeline. Any fixes is greatly appreciated.

2019-05-04 23:55:11,11] [error] WriteMetadataActor Failed to properly process data
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.addBatch(Unknown Source)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.addBatch(HikariProxyPreparedStatement.java)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15(JdbcActionComponent.scala:530)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15$adapted(JdbcActionComponent.scala:528)
at scala.collection.Iterator.foreach(Iterator.scala:944)
at scala.collection.Iterator.foreach$(Iterator.scala:944)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:528)
at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
at scala.collection.Iterator.foreach(Iterator.scala:944)
at scala.collection.Iterator.foreach$(Iterator.scala:944)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:239)
at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:237)
at slick.dbio.SynchronousDatabaseAction$$anon$7.run(DBIOAction.scala:486)
at slick.basic.BasicBackend$DatabaseDef$$anon$2.liftedTree1$1(BasicBackend.scala:275)
at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:275)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.SessionData.allocateLobForResult(Unknown Source)
at org.hsqldb.Session.allocateResultLob(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.performPreExecute(Unknown Source)
... 33 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.hsqldb.persist.LobStoreMem.setBlockBytes(Unknown Source)
at org.hsqldb.persist.LobManager.setBytesISNormal(Unknown Source)
at org.hsqldb.persist.LobManager.setBytesIS(Unknown Source)
at org.hsqldb.persist.LobManager.setCharsForNewClob(Unknown Source)
at org.hsqldb.SessionData.allocateLobForResult(Unknown Source)
at org.hsqldb.Session.allocateResultLob(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.performPreExecute(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.addBatch(Unknown Source)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.addBatch(HikariProxyPreparedStatement.java)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15(JdbcActionComponent.scala:530)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15$adapted(JdbcActionComponent.scala:528)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction$$Lambda$2682/0x0000000840eca040.apply(Unknown Source)
at scala.collection.Iterator.foreach(Iterator.scala:944)
at scala.collection.Iterator.foreach$(Iterator.scala:944)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:528)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction$$Lambda$2681/0x0000000840eca840.apply(Unknown Source)
at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
at slick.dbio.DBIOAction$$anon$4$$Lambda$2677/0x0000000840ecd840.apply(Unknown Source)
at scala.collection.Iterator.foreach(Iterator.scala:944)
at scala.collection.Iterator.foreach$(Iterator.scala:944)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
[2019-05-04 23:55:11,11] [error] Metadata events permanently dropped for the following workflows: d3834604-335c-4039-afff-071416ded9fb: 335

Thanks,
Archana
Sign In or Register to comment.