Queue job submission rate slows over time
I'm stumped by some behavior regarding job submission by Queue (although I'm not sure if the problem is related to Queue).
I wrote a QScript and tested it on a single machine running SGE; everything worked entirely as expected. I've since moved the analysis over to a cluster using UGE. Everything works mostly as expected, except for 2 issues: one I posted about on the forums here, and one where the job submissions begin taking longer over time.
When the pipeline is first started, jobs are submitted rapidly (within a few seconds of each other). Over time, jobs take increasingly longer (e.g. minutes) to submit, regardless of the job. The trend can be seen in the attached .pdf file. I can kill the pipeline, immediately restart it (both from scratch or not), and reproduce the behavior. Additionally, I can qsub the same jobs from a bash for loop without any problems.
The terminal pauses after outputting the "FunctionEdge - Output written to..." line for a minute (or more) between each submission, even with (what I think are) simple jobs:
INFO 13:07:14,093 FunctionEdge - Starting: rm 12.sam 12.sorted.bam 12.sorted.bai 12.sorted.intervals 12.sorted.realigned.bam 12.sorted.realigned.bai INFO 13:07:14,093 FunctionEdge - Output written to /data/ryanabashbash/src/Queue1104bDev/CleanupIntermediates.out INFO 13:08:08,810 DrmaaJobRunner - Submitted job id: 37461
On the single machine with SGE, these same jobs were submitted almost instantaneously, and I can qsub the same jobs from a bash for loop on the cluster just as fast. I'm guessing its some sort of interaction between Queue and the architecture that I'm overlooking? Does anyone have any suggestions on how to get some traction in tracking down the cause, or has anyone seen similar behavior before?
As an aside, my .out files all contain something similar to the following errors at the beginning of the file (I haven't been able to track down the source, and I'm not sure if it is related; it doesn't seem to affect the output of the job):
sh: module: line 1: syntax error: unexpected end of file sh: error importing function definition for `module'
Thanks for any suggestions or pointers!