Sub-workflow/task concurrency limits?
I have a pipeline that typically splits into a couple hundred pieces when it runs, e.g. 60 tiles x 4 samples on each tile for 240 sub-workflows running at once, each composed of several tasks. Most of the individual tasks finish in relatively short time spans, from 30 seconds to 10 minutes. Those are not a problem, and need no throttling. Near the end of the pipeline, we have a stage where each task requires a GPU. These are a somewhat scarce resource for us, and each of the GPU tasks runs for about 12 hours. I'd like to put a limit on just the GPU tasks. Putting them in a sub-workflow would be fine. I know I could put a global limit on concurrency, but I don't want to do that because the GPU stage doesn't always run, and more concurrency makes a huge difference if I can take advantage of it.
Is there any way to do this in the current version of Cromwell? I'm hoping I can manage it without some sort of awkward hack in Python.