BaseRecalibrator fails with java array index exception
Hi team members,
Like other users in past one month, I have also encountered the problem with BaseRecalibrator
ERROR A GATK RUNTIME ERROR has occurred (version 2.4-9-g532efad):
I have performed aligning of the whole genome ~ 40X coverage data with bwa 0.7.3a using bwa-mem option. Followed by Indel-realignment, fixing mate-pairs, and removing duplicates withe GATK 2.4-9-g532efad and picard tools (1.86). Outputs at all the steps look fine. However, I run into problem at the base recalibration step, which runs for about two hours on cluster with -nct 8 option before failing at the following position
INFO 16:38:42,214 ProgressMeter - 3:172666449 2.61e+08 2.0 h 27.0 s 21.4% 9.4 h 7.4 h
INFO 16:39:12,375 ProgressMeter - 3:175261930 2.63e+08 2.0 h 27.0 s 21.5% 9.4 h 7.4 h
INFO 16:39:21,706 GATKRunReport - Uploaded run statistics report to AWS S3
I did perform the validation on the bam file produced at removing duplicates step and it is fine. It has 1.25 billion reads.
There was a suggestion from Geraldine in response to earlier error report to consider using "-rf MateSameStrand" When I use the filter, base recalibrator gets done in 1.5 hours with -nct 8 option without error condition. I am printing about last dozen lines from output
INFO 18:40:49,412 ProgressMeter - GL000214.1:90167 1.11e+07 89.3 m 8.0 m 99.9% 89.4 m 7.0 s
INFO 18:41:19,416 ProgressMeter - GL000224.1:175635 1.14e+07 89.8 m 7.9 m 99.9% 89.8 m 3.0 s
INFO 18:41:49,421 ProgressMeter - GL000225.1:93582 1.14e+07 90.3 m 7.9 m 100.0% 90.3 m 1.0 s
INFO 18:42:13,521 ReadShardBalancer$1 - Loading BAM index data for next contig
INFO 18:42:14,172 BaseRecalibrator - Calculating quantized quality scores...
INFO 18:42:14,249 BaseRecalibrator - Writing recalibration report...
INFO 18:42:16,950 BaseRecalibrator - ...done!
INFO 18:42:16,951 BaseRecalibrator - Processed: 11635682 reads
INFO 18:42:16,951 ProgressMeter - done 1.16e+07 90.8 m 7.8 m 100.0% 90.8 m 0.0 s
INFO 18:42:16,952 ProgressMeter - Total runtime 5445.24 secs, 90.75 min, 1.51 hours
INFO 18:42:16,953 MicroScheduler - 34797156 reads were filtered out during traversal out of 34997160 total (99.43%)
INFO 18:42:16,953 MicroScheduler - -> 6157065 reads (17.59% of total) failing MappingQualityZeroFilter
INFO 18:42:16,953 MicroScheduler - -> 28627579 reads (81.80% of total) failing MateSameStrandFilter
INFO 18:42:16,953 MicroScheduler - -> 12512 reads (0.04% of total) failing NotPrimaryAlignmentFilter
INFO 18:42:18,549 GATKRunReport - Uploaded run statistics report to AWS S3
With "-rf MateSameStrand" filter, base recalibrator has processed only 11.6 M reads, what happened to other reads (I have 1.25 billion in my bam file). I do not understand what is going on at "MicroScheduler" it is showing " 34797156 reads were filtered out during traversal out of 34997160 total (99.43%)" . How did it get (34.99 - 11.6 M) reads? I was surprised with "28627579 reads (81.80% of total) failing MateSameStrandFilter". I thought, may be there is something wrong with my BWA alignment, but it looks fine. Here is a sample output from bwa
[M::main_mem] read 800000 sequences (80000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (58, 378407, 0, 14)
[M::mem_pestat] analyzing insert size distribution for orientation FF...
My mate pairs are aligned correctly for most part.
I also tried running the nightly GATK built and I keep getting the following message
ERROR MESSAGE: Timeout of 30000 milliseconds was reached while trying to acquire a lock on file /home/mgujral/broad_bundles/2.3/b37/dbsnp_137.b37.vcf.idx. Since the GATK uses non-blocking lock acquisition calls that are not supposed to wait, this implies a problem with the file locking support in your operating system.
I have following few questions
1) Does my base recalibrator output with "-rf MateSameStrand" look any way near acceptable?
2) Should I just wait for release 2.5 and try again?
Thank you for your patience, I apologize for this long description.