Forum Login Issue:
Currently the "Log in with Google" button redirects you to a "Page not found." This is an issue that our forum vendors are working on fixing. In the meantime, while on the "Page not found" you can edit the URL to delete the second gatk, firecloud, or wdl (depending on what subforum you are acessing).

Sub-workflow/task concurrency limits?

I have a pipeline that typically splits into a couple hundred pieces when it runs, e.g. 60 tiles x 4 samples on each tile for 240 sub-workflows running at once, each composed of several tasks. Most of the individual tasks finish in relatively short time spans, from 30 seconds to 10 minutes. Those are not a problem, and need no throttling. Near the end of the pipeline, we have a stage where each task requires a GPU. These are a somewhat scarce resource for us, and each of the GPU tasks runs for about 12 hours. I'd like to put a limit on just the GPU tasks. Putting them in a sub-workflow would be fine. I know I could put a global limit on concurrency, but I don't want to do that because the GPU stage doesn't always run, and more concurrency makes a huge difference if I can take advantage of it.

Is there any way to do this in the current version of Cromwell? I'm hoping I can manage it without some sort of awkward hack in Python.

Best Answer


Sign In or Register to comment.