Large Batch - "Unable to create BasicFeatureReader"

james_lawlorjames_lawlor Huntsville, ALMember

Hi,
I've been running into an error using GATK 3.8-0, " Unable to create BasicFeatureReader using feature file , for input source: " when I try to do a somewhat-large batch call of bgzipped and tabix indexed single-chromosome GVCFs. (Ultimately I need to batch call ~1000 samples together.) I can't find an intrinsic problem with the input files--I am able to batch-call the entire set in groups of ten or groups of 100, but if I try 200 files or more, I get this error. However, I don't understand why this particular error would be related to a lack of resources, and I can't otherwise tell from our job queuing system (LSF) that our resource limits were exceeded.

Any advice would be appreciated. Thank you!

I'm giving the jobs:
32 processors
~256GB RAM
Javascript heap size: 212GB
HTSLIB version 1.4.1

calling GATK with this command (after setting some ENV variables which I have excluded here):

CMD="java -Xmx212g -jar GenomeAnalysisTK.jar \
    -T GenotypeGVCFs \
    -R ${BWA_GENOME_DIR}/${GENOME}.fa \
    -nt 32 \
    -L $CHR \
    ${inputFileList} \
    -o ${CHR}_${1}_$(date +%s%N).vcf"

Here is the GATK output of one of the failures:

Sender: LSF System <[email protected]>
Subject: Job 71822: <sh run_big_tabix_input.sh 2hundredset00.txt> in cluster <helion-poc> Exited

Job <sh run_big_tabix_input.sh 2hundredset00.txt> was submitted from host <login01> by user <jlawlor> in cluster <helion-poc>.
Job was executed on host(s) <32*hpc0006>, in queue <c7normal>, as user <jlawlor> in cluster <helion-poc>.
</gpfs/gpfs1/home/jlawlor> was used as the home directory.
</gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call> was used as the working directory.
Started at Wed Aug 16 06:55:28 2017
Results reported on Wed Aug 16 06:56:06 2017

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
sh run_big_tabix_input.sh 2hundredset00.txt
------------------------------------------------------------

Exited with exit code 1.

Resource usage summary:

    CPU time :                                   560.00 sec.
    Max Memory :                                 25028 MB
    Average Memory :                             11828.50 MB
    Total Requested Memory :                     260000.00 MB
    Delta Memory :                               234972.00 MB
    Max Processes :                              4
    Max Threads :                                107

The output (if any) follows:

INFO  06:55:30,757 HelpFormatter - ---------------------------------------------------------------------------------- 
INFO  06:55:30,759 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-0-ge9d806836, Compiled 2017/07/28 21:26:50 
INFO  06:55:30,759 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute 
INFO  06:55:30,759 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk 
INFO  06:55:30,766 HelpFormatter - [Wed Aug 16 06:55:30 CDT 2017] Executing on Linux 3.10.0-327.3.1.el7.x86_64 amd64 
INFO  06:55:30,767 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_65-b17 
INFO  06:55:30,770 HelpFormatter - Program Args: -T GenotypeGVCFs -R /gpfs/gpfs1/myerslab/reference/genomes/bwa-0.7.8/GRCh37.fa -nt 32 -L 1 -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115394_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115395_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115396_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115397_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115398_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115399_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115400_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115401_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115402_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115403_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115404_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115405_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115415_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115416_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115417_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115418_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115419_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115420_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115421_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115422_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115423_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115424_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115425_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115426_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115427_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115428_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115429_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115430_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115431_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115432_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115433_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115434_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115435_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115436_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115437_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115438_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115439_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115440_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115441_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115442_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115443_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115444_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115445_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115446_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115447_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115448_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115449_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115450_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115451_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115452_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115453_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115454_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115455_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115456_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115457_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115458_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115459_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115460_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115461_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115462_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115463_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115464_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115465_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115466_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115467_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115468_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115469_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115470_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115471_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115472_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115473_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115474_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115475_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115476_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115477_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115478_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115479_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115480_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115481_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115482_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115483_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115484_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115485_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115486_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115487_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115488_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115489_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115490_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115491_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115492_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115493_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115494_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115495_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115496_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115497_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115498_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115499_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115500_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115501_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115502_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115503_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115504_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115505_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115506_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115507_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115508_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115509_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115510_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122119_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122120_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122121_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122122_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122123_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122124_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122125_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122126_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122127_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122128_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122129_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122130_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122131_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122132_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122133_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122134_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122135_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122136_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122137_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122138_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122139_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122140_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122141_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122142_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122143_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122144_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122145_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122146_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122147_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122148_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122149_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122150_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122151_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122152_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122153_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122154_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122155_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122156_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122157_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122158_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122159_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122160_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122161_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122162_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122163_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122164_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122165_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122166_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122167_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122168_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122169_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122170_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122171_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122172_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122173_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122174_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122175_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122176_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122177_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126155_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126156_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126157_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126158_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126159_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126160_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126161_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126162_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126163_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126164_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126165_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126166_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126167_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126168_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126169_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126170_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126171_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126172_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126173_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126174_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126175_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126176_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126177_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126178_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126179_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126180_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126181_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126182_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126183_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126184_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126185_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126186_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126187_1.g.vcf.gz -o 1_2hundredset00.txt_1502884529140524240.vcf 
INFO  06:55:30,777 HelpFormatter - Executing as [email protected] on Linux 3.10.0-327.3.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_65-b17. 
INFO  06:55:30,777 HelpFormatter - Date/Time: 2017/08/16 06:55:30 
INFO  06:55:30,778 HelpFormatter - ---------------------------------------------------------------------------------- 
INFO  06:55:30,778 HelpFormatter - ---------------------------------------------------------------------------------- 
ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
INFO  06:55:36,737 GenomeAnalysisEngine - Deflater: IntelDeflater 
INFO  06:55:36,737 GenomeAnalysisEngine - Inflater: IntelInflater 
INFO  06:55:36,738 GenomeAnalysisEngine - Strictness is SILENT 
INFO  06:55:36,894 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 
INFO  06:55:45,190 IntervalUtils - Processing 249250621 bp from intervals 
WARN  06:55:45,191 IndexDictionaryUtils - Track variant doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  06:55:45,191 IndexDictionaryUtils - Track variant2 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  06:55:45,192 IndexDictionaryUtils - Track variant3 doesn't have a sequence dictionary built in, skipping dictionary validation 
[ Removed for length ]
WARN  06:55:45,223 IndexDictionaryUtils - Track variant199 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  06:55:45,223 IndexDictionaryUtils - Track variant200 doesn't have a sequence dictionary built in, skipping dictionary validation 
INFO  06:55:45,232 MicroScheduler - Running the GATK in parallel mode with 32 total threads, 1 CPU thread(s) for each of 32 data thread(s), of 64 processors available on this machine 
INFO  06:55:45,281 GenomeAnalysisEngine - Preparing for traversal 
INFO  06:55:45,283 GenomeAnalysisEngine - Done preparing for traversal 
INFO  06:55:45,284 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] 
INFO  06:55:45,285 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining 
INFO  06:55:45,285 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime 
WARN  06:55:45,928 StrandBiasTest - StrandBiasBySample annotation exists in input VCF header. Attempting to use StrandBiasBySample values to calculate strand bias annotation values. If no sample has the SB genotype annotation, annotation may still fail. 
WARN  06:55:45,929 StrandBiasTest - StrandBiasBySample annotation exists in input VCF header. Attempting to use StrandBiasBySample values to calculate strand bias annotation values. If no sample has the SB genotype annotation, annotation may still fail. 
INFO  06:55:45,929 GenotypeGVCFs - Notice that the -ploidy parameter is ignored in GenotypeGVCFs tool as this is automatically determined by the input variant files 
WARN  06:55:51,116 HaplotypeScore - Annotation will not be calculated, must be called from UnifiedGenotyper, not GenotypeGVCFs 
WARN  06:55:53,111 ExactAFCalculator - This tool is currently set to genotype at most 6 alternate alleles in a given context, but the context at 1: 10445 has 7 alternate alleles so only the top alleles will be used; see the --max_alternate_alleles argument. Unless the DEBUG logging level is used, this warning message is output just once per run and further warnings are suppressed. 
##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 3.8-0-ge9d806836): 
##### ERROR
##### ERROR This means that one or more arguments or inputs in your command are incorrect.
##### ERROR The error message below tells you what is the problem.
##### ERROR
##### ERROR If the problem is an invalid argument, please check the online documentation guide
##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
##### ERROR
##### ERROR Visit our website and forum for extensive documentation and answers to 
##### ERROR commonly asked questions https://software.broadinstitute.org/gatk
##### ERROR
##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
##### ERROR
##### ERROR MESSAGE: Unable to create BasicFeatureReader using feature file , for input source: /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115506_1.g.vcf.gz
##### ERROR ------------------------------------------------------------------------------------------

In contrast, here is the output of a successful batch call with fewer samples:

Sender: LSF System <[email protected]>
Subject: Job 71700: <sh run_big_tabix_input.sh x00.txt> in cluster <helion-poc> Done

Job <sh run_big_tabix_input.sh x00.txt> was submitted from host <login01> by user <jlawlor> in cluster <helion-poc>.
Job was executed on host(s) <32*hpc0006>, in queue <c7normal>, as user <jlawlor> in cluster <helion-poc>.
</gpfs/gpfs1/home/jlawlor> was used as the home directory.
</gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call> was used as the working directory.
Started at Tue Aug 15 15:45:26 2017
Results reported on Tue Aug 15 15:51:16 2017

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
sh run_big_tabix_input.sh x00.txt
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   10318.77 sec.
    Max Memory :                                 72817 MB
    Average Memory :                             64894.37 MB
    Total Requested Memory :                     260000.00 MB
    Delta Memory :                               187183.00 MB
    Max Processes :                              4
    Max Threads :                                107

The output (if any) follows:

INFO  15:45:27,917 HelpFormatter - ---------------------------------------------------------------------------------- 
INFO  15:45:27,919 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-0-ge9d806836, Compiled 2017/07/28 21:26:50 
INFO  15:45:27,919 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute 
INFO  15:45:27,919 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk 
INFO  15:45:27,921 HelpFormatter - [Tue Aug 15 15:45:27 CDT 2017] Executing on Linux 3.10.0-327.3.1.el7.x86_64 amd64 
INFO  15:45:27,921 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_65-b17 
INFO  15:45:27,924 HelpFormatter - Program Args: -T GenotypeGVCFs -R /gpfs/gpfs1/myerslab/reference/genomes/bwa-0.7.8/GRCh37.fa -nt 32 -L 1 -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115394_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115395_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115396_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115397_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115398_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115399_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115400_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115401_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115402_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115403_1.g.vcf.gz -o 1_x00.txt_1502829926656725768.vcf 
INFO  15:45:27,929 HelpFormatter - Executing as [email protected] on Linux 3.10.0-327.3.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_65-b17. 
INFO  15:45:27,930 HelpFormatter - Date/Time: 2017/08/15 15:45:27 
INFO  15:45:27,930 HelpFormatter - ---------------------------------------------------------------------------------- 
INFO  15:45:27,930 HelpFormatter - ---------------------------------------------------------------------------------- 
ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
INFO  15:45:28,461 GenomeAnalysisEngine - Deflater: IntelDeflater 
INFO  15:45:28,461 GenomeAnalysisEngine - Inflater: IntelInflater 
INFO  15:45:28,462 GenomeAnalysisEngine - Strictness is SILENT 
INFO  15:45:28,621 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 
INFO  15:45:28,940 IntervalUtils - Processing 249250621 bp from intervals 
WARN  15:45:28,941 IndexDictionaryUtils - Track variant doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,941 IndexDictionaryUtils - Track variant2 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,941 IndexDictionaryUtils - Track variant3 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant4 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant5 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant6 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant7 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant8 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant9 doesn't have a sequence dictionary built in, skipping dictionary validation 
WARN  15:45:28,942 IndexDictionaryUtils - Track variant10 doesn't have a sequence dictionary built in, skipping dictionary validation 
INFO  15:45:28,948 MicroScheduler - Running the GATK in parallel mode with 32 total threads, 1 CPU thread(s) for each of 32 data thread(s), of 64 processors available on this machine 
INFO  15:45:28,992 GenomeAnalysisEngine - Preparing for traversal 
INFO  15:45:28,994 GenomeAnalysisEngine - Done preparing for traversal 
INFO  15:45:28,995 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] 
INFO  15:45:28,996 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining 
INFO  15:45:28,996 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime 
WARN  15:45:29,095 StrandBiasTest - StrandBiasBySample annotation exists in input VCF header. Attempting to use StrandBiasBySample values to calculate strand bias annotation values. If no sample has the SB genotype annotation, annotation may still fail. 
WARN  15:45:29,096 StrandBiasTest - StrandBiasBySample annotation exists in input VCF header. Attempting to use StrandBiasBySample values to calculate strand bias annotation values. If no sample has the SB genotype annotation, annotation may still fail. 
INFO  15:45:29,096 GenotypeGVCFs - Notice that the -ploidy parameter is ignored in GenotypeGVCFs tool as this is automatically determined by the input variant files 
WARN  15:45:29,675 HaplotypeScore - Annotation will not be calculated, must be called from UnifiedGenotyper, not GenotypeGVCFs 
WARN  15:45:31,082 ExactAFCalculator - This tool is currently set to genotype at most 6 alternate alleles in a given context, but the context at 1: 12004479 has 8 alternate alleles so only the top alleles will be used; see the --max_alternate_alleles argument. Unless the DEBUG logging level is used, this warning message is output just once per run and further warnings are suppressed. 
INFO  15:45:59,000 ProgressMeter -      1:32054701   1000000.0    30.0 s      30.0 s       12.9%     3.9 m       3.4 m 
[Removed for length]
INFO  15:51:15,369 ProgressMeter - Total runtime 346.37 secs, 5.77 min, 0.10 hours 
------------------------------------------------------------------------------------------
Done. There were 14 WARN messages, the first 10 are repeated below.
...

Best Answer

  • james_lawlorjames_lawlor Huntsville, ALMember
    Accepted Answer

    Thanks Sheila and Geraldine!

    It looks like I was able to alleviate my problem by increasing the Java heap size to 385 GB (-Xmx385g -Xms385g) and reserving ~450GB of total memory for the job (via LSF). This allowed me to use up to 64 data threads in GATK and successfully genotype a batch of ~900 GVCFs.

    I'm still not entirely sure why that ends up triggering a Cannot Create Basic FeatureReader error as opposed to the out of memory / increase java heap size error; however, I'm glad to have a path forward. I'll go ahead and mark my question as answered.
    Thanks again!

Answers

  • james_lawlorjames_lawlor Huntsville, ALMember

    Also, I have experimented with reducing the Java heap size (-Xmx64g) with the same results.

  • james_lawlorjames_lawlor Huntsville, ALMember

    Sorry, I inadvertently excluded the GATK arguments in trying to trim that output to fit in the post length.
    Here are the arguments for the failed run:

    INFO  23:48:05,051 HelpFormatter - Program Args: -T GenotypeGVCFs -R /gpfs/gpfs1/myerslab/reference/genomes/bwa-0.7.8/GRCh37.fa -nt 32 -L 1 -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115394_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115395_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115396_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115397_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115398_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115399_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115400_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115401_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115402_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115403_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115404_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115405_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115415_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115416_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115417_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115418_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115419_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115420_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115421_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115422_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115423_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115424_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115425_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115426_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115427_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115428_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115429_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115430_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115431_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115432_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115433_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115434_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115435_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115436_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115437_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115438_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115439_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115440_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115441_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115442_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115443_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115444_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115445_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115446_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115447_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115448_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115449_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115450_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115451_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115452_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115453_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115454_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115455_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115456_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115457_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115458_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115459_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115460_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115461_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115462_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115463_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115464_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115465_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115466_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115467_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115468_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115469_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115470_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115471_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115472_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115473_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115474_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115475_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115476_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115477_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115478_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115479_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115480_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115481_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115482_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115483_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115484_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115485_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115486_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115487_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115488_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115489_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115490_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115491_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115492_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115493_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115494_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115495_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115496_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115497_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115498_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115499_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115500_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115501_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115502_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115503_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115504_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115505_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115506_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115507_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115508_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115509_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL115510_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122119_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122120_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122121_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122122_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122123_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122124_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122125_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122126_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122127_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122128_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122129_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122130_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122131_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122132_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122133_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122134_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122135_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122136_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122137_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122138_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122139_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122140_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122141_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122142_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122143_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122144_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122145_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122146_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122147_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122148_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122149_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122150_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122151_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122152_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122153_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122154_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122155_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122156_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122157_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122158_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122159_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122160_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122161_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122162_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122163_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122164_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122165_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122166_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122167_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122168_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122169_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122170_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122171_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122172_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122173_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122174_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122175_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122176_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL122177_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126155_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126156_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126157_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126158_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126159_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126160_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126161_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126162_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126163_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126164_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126165_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126166_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126167_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126168_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126169_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126170_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126171_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126172_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126173_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126174_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126175_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126176_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126177_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126178_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126179_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126180_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126181_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126182_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126183_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126184_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126185_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126186_1.g.vcf.gz -V /gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/bgzipped/SL126187_1.g.vcf.gz -o 1_2hundredset00.txt_1502858883453006665.vcf 
    INFO  23:48:05,057 HelpFormatter - Executing as [email protected] on Linux 3.10.0-327.3.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_65-b17. 
    INFO  23:48:05,058 HelpFormatter - Date/Time: 2017/08/15 23:48:05 
    INFO  23:48:05,058 HelpFormatter - ---------------------------------------------------------------------------------- 
    INFO  23:48:05,058 HelpFormatter - ---------------------------------------------------------------------------------- 
    ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/gpfs2/cooperlab/test_batch/inadvisable_batch_call/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties
    ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
    INFO  23:48:10,918 GenomeAnalysisEngine - Deflater: IntelDeflater 
    INFO  23:48:10,919 GenomeAnalysisEngine - Inflater: IntelInflater 
    INFO  23:48:10,920 GenomeAnalysisEngine - Strictness is SILENT 
    INFO  23:48:11,124 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 
    INFO  23:48:19,242 IntervalUtils - Processing 249250621 bp from intervals 
    WARN  23:48:19,245 IndexDictionaryUtils - Track variant doesn't have a...
    
  • SheilaSheila Broad InstituteMember, Broadie, Moderator
    edited August 2017

    @james_lawlor
    Hi,

    This looks like it could be a corrupt GVCF index issue. Can you try running CombineGVCFs on 10-20 GVCFs at a time? That way you can combine the GVCFs before inputting them to GenotypeGVCFs, and also you can figure out which GVCFs may have corrupt indices.

    I hope that helps.

    -Sheila

    EDIT: On second look, it looks like this issue is similar to the one in this thread. Perhaps try the very latest nightly build first, and see if that fixes your issue :smile:

  • james_lawlorjames_lawlor Huntsville, ALMember
    edited August 2017

    I'll give that a try--however, using the 3.8 release build I am successfully able to genotype the entire set in groups of 10 or 100, but if I try groups of 200 it fails with the Unable to create BasicFeatureReader error, hence my confusion. (I am using CentOS 7, FYI, I noticed it mentioned when I skimmed the thread you linked.)

  • james_lawlorjames_lawlor Huntsville, ALMember

    Hi @Sheila,
    The plot thickens: I think I'm running into some kind of incompatibility with our compute architecture. Using the nightly build (vnightly-2017-08-21-g34e1f1e) I had a successful run of 200 samples on one particular compute node; however, submitting the exact same command to a different node resulted in the Unable to create BasicFeatureReader failure as before. Right now, I have ~915 samples genotyping about ~12% through chromosome 1, so it appears to be working fine on the good node.

    I'm going to try a couple more test cases (3.8 release) and get with our IT admin to find out if there are any obvious architecture differences, and then I'll report back with more information. Thanks!

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @james_lawlor
    Hi,

    Interesting. Good luck, and please do post about your findings.

    Thanks,
    Sheila

  • james_lawlorjames_lawlor Huntsville, ALMember

    Hi @Sheila,

    Update and a question: After working with our IT folks, it seemed like an architecture problem was less likely--we couldn't find a pattern that made sense. However, after some more test runs and looking back at the documentation, I don't think I was accounting for the fact that the -nt multithreading option multiplies the memory requirements.

    For example, using 4 data threads, ~256GB RAM, and 16GB JVM heap size (-Xmx16g) was successful*, however, increasing to 8 threads returned to the Unable to create BasicFeatureReader error.

    My question for continuing to optimize the memory allocation: Is the JVM heap size specified shared between all of the data threads, or is it duplicated for each thread? (I.e., do I have 16GB of max heap per thread, or does each thread then effectively have 4G of heap?).

    Thanks!
    James

    Issue · Github
    by Sheila

    Issue Number
    2475
    State
    closed
    Last Updated
    Assignee
    Array
    Milestone
    Array
    Closed By
    vdauwera
  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @james_lawlor
    Hi James,

    Let me confirm with the team before responding. We will get back to you soon.

    -Sheila

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie

    You're correct that using nt multiplies the memory requirements -- or conversely, divides available memory among the threads you requested. (-> each effectively has 4)

  • james_lawlorjames_lawlor Huntsville, ALMember
    Accepted Answer

    Thanks Sheila and Geraldine!

    It looks like I was able to alleviate my problem by increasing the Java heap size to 385 GB (-Xmx385g -Xms385g) and reserving ~450GB of total memory for the job (via LSF). This allowed me to use up to 64 data threads in GATK and successfully genotype a batch of ~900 GVCFs.

    I'm still not entirely sure why that ends up triggering a Cannot Create Basic FeatureReader error as opposed to the out of memory / increase java heap size error; however, I'm glad to have a path forward. I'll go ahead and mark my question as answered.
    Thanks again!

  • init_jsinit_js Member
    edited April 25

    I'm not sure this will help or not... I'm getting BasicFeatureReader errors on a different tool (GenomicsDBImport), an issue which I will post separately to this and link here.

    I believe the BasicFeatureReader error can have multiple causes. I would suggest including

    -DGATK_STACKTRACE_ON_USER_EXCEPTION=true

    as part of your java options (i.e. along with -Xmx385g), and then you'll have more information wrt the context of the error.

    In my case, with GenomicsDBImport, the error occurs due to an underlying missing file error, and I wouldn't have known without the stack trace:

    htsjdk.tribble.TribbleException$MalformedFeatureFile: Unable to create BasicFeatureReader using feature file , for input source: file:///home/jslegare/src/genomics/snake4/data/gwas/variants/Hidatsa%231.bam.g.vcf.gz
            at htsjdk.tribble.AbstractFeatureReader.getFeatureReader(AbstractFeatureReader.java:113)
            at org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport.getReaderFromPath(GenomicsDBImport.java:615)
            at org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport.getHeaderFromPath(GenomicsDBImport.java:356)
            at org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport.initializeHeaderAndSampleMappings(GenomicsDBImport.java:342)
            at org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport.onStartup(GenomicsDBImport.java:297)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:134)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
            at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:153)
            at org.broadinstitute.hellbender.Main.mainEntry(Main.java:195)
            at org.broadinstitute.hellbender.Main.main(Main.java:277)
    Caused by: java.io.FileNotFoundException: /home/jslegare/src/genomics/snake4/data/gwas/variants/Hidatsa%231.bam.g.vcf.gz (No such file or directory)
            at java.io.RandomAccessFile.open0(Native Method)
            at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
            at java.io.RandomAccessFile.(RandomAccessFile.java:243)
            at htsjdk.samtools.seekablestream.SeekableFileStream.(SeekableFileStream.java:47)
            at htsjdk.samtools.seekablestream.SeekableStreamFactory$DefaultSeekableStreamFactory.getStreamFor(SeekableStreamFactory.java:99)
            at htsjdk.tribble.readers.TabixReader.(TabixReader.java:129)
            at htsjdk.tribble.TabixFeatureReader.(TabixFeatureReader.java:80)
            at htsjdk.tribble.AbstractFeatureReader.getFeatureReader(AbstractFeatureReader.java:106)
            ... 10 more
    
    
Sign In or Register to comment.