How difficult would it be to get Cromwell working on Slurm?

MattBMattB NewcastleMember

Our existing cluster is SGE based, and I see in the documentation for Cromwell there is a SGE backend: https://github.com/broadinstitute/cromwell#sun-gridengine-backend which is all very nice. However looks like the current plan for our new (shiny and much bigger/faster) University wide cluster is to use the Slurm Workload Manager https://slurm.schedmd.com How hard would it be to get a Cromwell local Slurm back end up and running? Would this be something we could realistically port over locally, or is it quite involved? I'd appreciate any advice on just how much work this would be, would I simply swap out qsub and associated arguments for their Slurm equivalents within a actor factory somewhere or would it be more involved? (I've only just started looking into the nuts and bolts of Cromwell/WDL recently)

I'm asking because politically I may try argue that we need SGE on the cluster, but this will of course involve fighting it out with other factions in the university all of which have their favoured job scheduler solution. That and I need to factor that writing a Slurm back end to Cromwell/WDL is not exactly my day job!

From what I can tell Slurm is gaining in popularity, the NIH appear to use it https://slurm.schedmd.com Is anyone else in the community also in need of Slurm support / have experience implementing it for Cromwell?

Tagged:

Best Answers

Answers

Sign In or Register to comment.