Can (or will) Queue use PBS Professional job dependencies?

scottwoodscottwood Brisbane, AustraliaMember

Question

Hey folks,

We've run in to a situation where we have hundreds of queue managed GATK HaplotypeCaller.scala jobs which we'd like to run. This has led to hundreds of instances of Queue.jar in their post scatter watching state holding on to cores in our cluster. I'm curious about plans for Queue and whether or not a job dependency tree model has been considered for future versions, or whether or not this is already available.

By that, I mean something along the lines of the scatter kicks off the child jobs and a check/resubmit_faileds/gather job is queued on the cluster with a qsub -W depend=after:child1jobid[:child2jobid: ...:childNjobid]. The initial scatter job would end, freeing up a core, and resources would be available to the sub jobs. Then, when the child jobs finish up, the dependencies would be met and the next job would execute (when resources are available) and pick up the successful, resubmit the failed, lather, rinse , repeat.

Thanks in advance for your patience with me in the phrasing of my question, as I am approaching this as a cluster admin, not as the developer who has implemented the queue.

Cheers,
Scott

Best Answer

Answers

  • scottwoodscottwood Brisbane, AustraliaMember

    Thanks for the suggestion. I've shown your response to the developer who is working on our pipeline and they'll give it some thought. Hopefully the single watchdog, multi HaplotypeCaller solution does the job for us. Thanks again.

Sign In or Register to comment.