BaseRecalibrator doesn't respect -Xmx and -Xms settings

GATK version: version 2.1-12-g2d7797a

I'm using open grid engine via "starcluster", and have cluster nodes with 21G of ram. I have h_vmem as a requestable resource.

My qsub command requests "-l h_vmem=7G" and the java command is "java -Xms1G -Xmx5G -jar GenomeAnalysisTKLite.jar -T BaseRecalibrator ..."

the linux kernel/open grid engine kills the job:
main|node003|W|job 2271 exceeds job hard limit "h_vmem" of queue "[email protected]" (11716157440.00000 > limit:7516192768.00000) - sending SIGKILL

I've been reading about what I might be not specifying when trying to constrain the process, and perhaps I need to also add -XX:MaxPermSize to something specific ?

Best Answer

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MA admin
    Accepted Answer

    Hi Mike, I'm going to guess that when it invokes the Rscript, the R run is not subject to the limitations placed on the java job. And I don't think we've ever had a reason to build in an option to disable plotting. If you want to do it yourself and make a patch, we'd be happy to look at it.

Answers

  • I've been testing this a lot today -- it happens almost unpredictably, but with 1 caveat: it seems to be invoking that R script when it decides to take up 2x the amount of available heap space allocated to it. I've also added the -XX:MaxPermSize option, but the process will still take up 2x the amount of heap space allowed.

    It doesn't seem like you can disable plotting, why is that?

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    Accepted Answer

    Hi Mike, I'm going to guess that when it invokes the Rscript, the R run is not subject to the limitations placed on the java job. And I don't think we've ever had a reason to build in an option to disable plotting. If you want to do it yourself and make a patch, we'd be happy to look at it.

  • mike_lyonsmike_lyons Member
    edited November 2012

    Actually -- you already have an option to disable the plotting in the BaseRecalibrator.
    in package org.broadinstitute.sting.gatk.walkers.bqsr

    public class RecalibrationArgumentCollection {
     //line 173 from most recent github update
      @Hidden
      @Argument(fullName = "no_plots", shortName = "np", required = false, doc = "does not generate any plots -- useful for queue    scatter/gathering")
    public boolean NO_PLOTS = false;
     // ...
    }
    
    public class BaseRecalibrator extends LocusWalker<Long, Long> implements TreeReducible<Long> {
    // line 278in most recent github update
    @Override
        public void onTraversalDone(Long result) {
            logger.info("Calculating quantized quality scores...");
            quantizeQualityScores();
    
            logger.info("Writing recalibration report...");
            generateReport();
            logger.info("...done!");
    
            if (!RAC.NO_PLOTS) {
                logger.info("Generating recalibration plots...");
                generatePlots();
            }
    
            logger.info("Processed: " + result + " sites");
        }
     //...
    }
    
  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Oh yeah, there you go -- I hadn't run into that one yet. Thanks for pointing it out!

Sign In or Register to comment.