Forcing cromwell to submit all possible jobs at once

When I use cromwell to submit jobs on LSF and on AWS (SGE) I often see cromwell submitting only a fraction of the available jobs to submit. So if I have a 1000 jobs often times there's only 100 jobs pending at the same time. Not only does this behavior waste time it also gives problem on AWS, where the # of queueing jobs is used to determine scaling behavior. Is there a way to force Cromwell to submit as many jobs as possible into the job scheduler?

Answers

  • kshakirkshakir Broadie, Dev ✭✭

    Hi @awacs,

    We'll need a little more info to narrow down your specific question. Specifically:

    • What version of cromwell are you running? java -jar cromwell.jar -version (--version starting in 29)
    • What does your application.conf's backend stanza look like? General note: please don't post any secrets to the forum.
    • How are your workflow calls structured? Specifically, looking for the order of the calls that aren't running but should be, and their dependencies, not necessarily the tasks.

    In the meantime, here are some general answers regarding jobs:

    • The Local/LSF/SGE/etc., i.e. any shared filesystem (SFS), backends all support a integer config value for backend.providers.<name>.config.concurrent-job-limit. If set, then only that number of jobs at a time will be dispatched for that backend. If the config value is not set, then the backend will launch an unlimited number of concurrent jobs.

    • Cromwell also doesn't submit dependent jobs until their pre-requisites are complete. Both LSF and SGE have the concept of queuing dependent jobs. Cromwell does not use these particular pre-scheduling features. Instead it waits for a job to complete successfully before adding the dependent jobs to the LSF/SGE/etc. scheduler.

    • Not for your particular situation, so mainly leaving this here for future visitors: The engine also has a system.max-concurrent-workflows that limits the number of concurrent workflows. This defaults to 5000, and you're nowhere close to the limit.

Sign In or Register to comment.