ways to speed up Indel Realinger
I am trying to speed up the Indel Realigner step in my pipeline. I have tried Queue, using number of scatters (-nsc) equal to 8; I noticed the jobs that are "scattered" finish relatively quickly (within 2 hours), but the Gather step is taking the bulk of the processing time (over 6 hours). I suspect this slow-down is because the scheduler for the cluster automatically assigns the Gather job to be done on just one core.
As I don't know enough about Queue and .scala to increase the number of cores for the Gather job, I was thinking of other ways to get the same output, but save time. Reading this post by @neilw in the forum (http://gatkforums.broadinstitute.org/discussion/3051/parallelizing-printreads), I was wondering if this approach would work:
1) Use RealignmentTargetCreator to get list of target intervals for realignment
2) Run 100 parallel jobs of IndelRealigner, passing to each the same (whole) input .bam and list of target intervals from step 1, but a unique portion of the genome using the -L option (intervals)
3) Merge the 100 realigned .bams with samtools to get single re-aligned .bam
Thanks a lot for your advice,