Cromwell performance

Hi,
I am having some issues with Cromwell performance. I was hoping for some guidance from the team.

I have cromwell running as a server, with mysql configured for caching. I have run about 4 workflows (~16 tasks) against this server/db. As I submit a job I see cpu usage spike to 100% for 2 threads of cromwell (using htop). The cpu usage eventually settles down, ~20 minutes into the job.

PID USER VIRT RES SHR S CPU% MEM% TIME+ Command
2338 evaben 11.7G 1440M 13644 S 101. 4.5 1h03:58 java -Dconfig.file=/home/evaben/cromwell/ ...
27815 evaben 11.7G 1440M 13644 R 100. 4.5 12:32.44 java -Dconfig.file=/home/evaben/cromwell/ ...

I have also attached a log, showing that 10 minutes was spent for a 'BwaMem' task to find a Cache Hit. I do not know the full mechanism of the cache. Could it be checking the md5 of my input files? Probably my two problems are related, the high cpu and the slow cache.

Please advise if these are expected times, or potentially some problem with my setup or a bug. I am using release 31.

Answers

  • On submitting a job with a scatter of 50 the usage was even higher. ~5 minutes of this:

      PID USER     CPU     
      305 evaben S 200.    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10097 evaben S 29.6    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
     9869 evaben S 22.2    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10093 evaben R 22.2    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10098 evaben R 22.2    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10091 evaben R 14.8    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10094 evaben S 14.8    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10095 evaben S 14.8    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10096 evaben R 14.8    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10099 evaben S 14.8    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10100 evaben S 14.8    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10075 evaben S  7.4    java -Dconfig.file=/home/evaben/cromwell/sge.conf -jar /home/evaben/cromwell/cromwel
    10092 evaben S  7.4    java -Dconfig.file=/home/evaben/cromwell/s
    
  • I have found and implemented the caching configuration:
    https://cromwell.readthedocs.io/en/develop/Configuring/#call-caching

    This has reduced the lag and CPU usage on the server process. I believe this option only checks the file path, not the mtime of the file. Use with caution.

Sign In or Register to comment.