This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.
how to run cromwell on large number of samples
I constructed a WES variant calling WDL pipeline to run on one sample. And I use the following command to run the pipeline in a SGE cluster
for i in ...;do nohup "java -Dconfig.file=sge.conf -jar cromwell-36.jar run fq2gvcf.wdl -i sample.$i.json" >$i.out 2>$i.err & done
But when the number of sample is large, cromwell gives the error, which is probably due to the overhead of running too many instances of cromwell.
java.lang.OutOfMemoryError: unable to create new native thread
Can someone please tell how to solve this problem? one suggests to use the
server mode instead of the
run mode, but I'm not sure how to use server mode on a cluster I can only ssh into, and how to use server mode to run large number of samples?