To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Sub-workflow/task concurrency limits?

I have a pipeline that typically splits into a couple hundred pieces when it runs, e.g. 60 tiles x 4 samples on each tile for 240 sub-workflows running at once, each composed of several tasks. Most of the individual tasks finish in relatively short time spans, from 30 seconds to 10 minutes. Those are not a problem, and need no throttling. Near the end of the pipeline, we have a stage where each task requires a GPU. These are a somewhat scarce resource for us, and each of the GPU tasks runs for about 12 hours. I'd like to put a limit on just the GPU tasks. Putting them in a sub-workflow would be fine. I know I could put a global limit on concurrency, but I don't want to do that because the GPU stage doesn't always run, and more concurrency makes a huge difference if I can take advantage of it.

Is there any way to do this in the current version of Cromwell? I'm hoping I can manage it without some sort of awkward hack in Python.

Best Answer

Answers

Sign In or Register to comment.