To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Something about job management

y_zhang88y_zhang88 Member
edited December 2017 in Ask the WDL team

Hi,
I am a new user and I really want to know how to manager jobs by WDL. For example, task will be run if the CPU usage is below 90% and memory is over 2G, otherwise it will be suspended until the condition is met. Does WDL contain this function?

Answers

  • kshakirkshakir Broadie, Dev

    Cromwell doesn't support this level of functionality for Local backends, but it can work with a job scheduler such as Grid Engine / SLURM / TORQUE / etc. to support resource requirements.

    WDL allows one to specify runtime attributes per task. One can configure Cromwell to pass the runtime attributes to the job scheduler during job submission. Under the documentation for setting up cromwell with an HPC custer there is an example that uses Grid Engine. Depending on who administers your cluster, most clusters usually support requesting a minimum amount of memory be available before a job will run. In the example, this is done with a flag -l mem_free= and then reading the runtime attribute memory as Float? memory_gb. NOTE: the docs under "memory" explain why there's a difference between the two names.

    The builtin cpu attribute is meant to request 1 whole CPU. Your concept of percentages is new to me, but if your job scheduler supports cpu percentages, in theory in your cromwell configuration file you could create a custom runtime-attribute, say Int? cpu_pct and then wire that in to your custom submit the same way as memory.

Sign In or Register to comment.