GATK Queue GridEngine Job submission slows dramatically over time
I'm using Queue 3.4 with a Univa GridEngine cluster. Everything runs fine, and at first the time between job submissions is only a few seconds. But the rate at which queue submits jobs to the cluster slows to a crawl - up to 40 seconds per job after a few hundred submissions, and slowly increasing. Any idea what might be happening here? With a scatterCount of 1000, it can take a full day just to submit all jobs to the cluster for a HaplotypeCaller run.