Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.
Attention:
We will be out of the office on October 14, 2019, due to the U.S. holiday. We will return to monitoring the forum on October 15.

Errors when I run the BaseRecalibrator

wulywuly Member
edited May 29 in Ask the GATK team

Hi,I got some errors when I used BaseRecalibrator. It can run at first ,but it got some errors after a while.
The command is :

/data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk --java-options "-Xmx20G -Djava.io.tmpdir=./" BaseRecalibrator \
-R /data/home/wuly/source/Homo_sapiens_assembly38.fasta \
-I /data/home/wuly/breast_cancer/clean_data/M1/markup/CL100066321_L02_547.sorted.marked.bam \
--known-sites /data/home/wuly/source/hapmap_3.3.hg38.vcf.gz \
--known-sites /data/home/wuly/source/dbsnp_146.hg38.vcf.gz \
--known-sites /data/home/wuly/source/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \
--known-sites /data/home/wuly/source/1000G_phase1.snps.high_confidence.hg38.vcf.gz \
-O M1_recal.table

This is the log:

Using GATK jar /data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx20G -Djava.io.tmpdir=./
 -jar /data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar BaseRecalibrator -R /data/home/wuly/source/Homo_sapiens_assembly38.fasta -I /data/home/wuly/breast_cancer/clean_data/M1.bam --known-sites /data/home/wuly/source/hapmap_3.3.hg38.vcf.gz --known-sites /data/home/wuly/source/dbsnp_146.hg38.vcf.gz --known-sites /data/home/wuly/source/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz --known-sites /data/home/wuly/source/1000G_phase1.snps.high_confidence.hg38.vcf.gz -O M1_recal.table23:26:23.393 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar!/com/intel/gkl/native/libgkl_compre
ssion.soMay 28, 2019 11:26:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
23:26:25.147 INFO  BaseRecalibrator - ------------------------------------------------------------
23:26:25.147 INFO  BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.1.2.0
23:26:25.147 INFO  BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/
23:26:25.147 INFO  BaseRecalibrator - Executing as [email protected] on Linux v3.10.0-957.10.1.el7.x86_64 amd64
23:26:25.147 INFO  BaseRecalibrator - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
23:26:25.148 INFO  BaseRecalibrator - Start Date/Time: May 28, 2019 11:26:23 PM EDT
23:26:25.148 INFO  BaseRecalibrator - ------------------------------------------------------------
23:26:25.148 INFO  BaseRecalibrator - ------------------------------------------------------------
23:26:25.148 INFO  BaseRecalibrator - HTSJDK Version: 2.19.0
23:26:25.148 INFO  BaseRecalibrator - Picard Version: 2.19.0
23:26:25.148 INFO  BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
23:26:25.148 INFO  BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
23:26:25.148 INFO  BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
23:26:25.148 INFO  BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
23:26:25.148 INFO  BaseRecalibrator - Deflater: IntelDeflater
23:26:25.148 INFO  BaseRecalibrator - Inflater: IntelInflater
23:26:25.148 INFO  BaseRecalibrator - GCS max retries/reopens: 20
23:26:25.148 INFO  BaseRecalibrator - Requester pays: disabled
23:26:25.148 INFO  BaseRecalibrator - Initializing engine
23:26:25.686 INFO  FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/hapmap_3.3.hg38.vcf.gz
23:26:25.821 INFO  FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/dbsnp_146.hg38.vcf.gz
23:26:25.906 INFO  FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz
23:26:25.984 INFO  FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/1000G_phase1.snps.high_confidence.hg38.vcf.gz
23:26:26.082 WARN  IndexUtils - Feature file "/data/home/wuly/source/dbsnp_146.hg38.vcf.gz" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the as
sociated index file23:26:26.156 WARN  IndexUtils - Index file /data/home/wuly/source/dbsnp_146.hg38.vcf.gz.tbi is out of date (index older than input file). Use IndexFeatureFile to make a new index.
23:26:26.221 INFO  BaseRecalibrator - Done initializing engine
23:26:26.225 INFO  BaseRecalibrationEngine - The covariates being used here: 
23:26:26.225 INFO  BaseRecalibrationEngine -    ReadGroupCovariate
23:26:26.225 INFO  BaseRecalibrationEngine -    QualityScoreCovariate
23:26:26.225 INFO  BaseRecalibrationEngine -    ContextCovariate
23:26:26.225 INFO  BaseRecalibrationEngine -    CycleCovariate
23:26:26.298 INFO  ProgressMeter - Starting traversal
23:26:26.299 INFO  ProgressMeter -        Current Locus  Elapsed Minutes       Reads Processed     Reads/Minute
23:26:36.311 INFO  ProgressMeter -         chr1:3022190              0.2                292000        1750074.9
23:26:46.314 INFO  ProgressMeter -         chr1:6578963              0.3                495000        1483961.2
23:26:56.318 INFO  ProgressMeter -         chr1:9975774              0.5                691000        1381171.3
23:27:06.372 INFO  ProgressMeter -        chr1:12234125              0.7                953000        1426931.5
23:27:16.382 INFO  ProgressMeter -        chr1:15383910              0.8               1139000        1364562.1
23:27:26.420 INFO  ProgressMeter -        chr1:17587500              1.0               1452000        1449101.8
23:27:36.446 INFO  ProgressMeter -        chr1:20483560              1.2               1658000        1418164.7
....
23:41:58.683 INFO  BaseRecalibrator - Shutting down engine
[May 28, 2019 11:41:58 PM EDT] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 15.59 minutes.
Runtime.totalMemory()=4510973952
htsjdk.samtools.SAMFormatException: Did not inflate expected amount
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:147)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
    at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
    at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
    at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
    at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:241)
    at htsjdk.tribble.readers.TabixReader.readLine(TabixReader.java:215)
    at htsjdk.tribble.readers.TabixReader.access$300(TabixReader.java:48)
    at htsjdk.tribble.readers.TabixReader$IteratorImpl.next(TabixReader.java:434)
    at htsjdk.tribble.readers.TabixIteratorLineReader.readLine(TabixIteratorLineReader.java:46)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:205)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:149)
    at org.broadinstitute.hellbender.engine.FeatureCache.fill(FeatureCache.java:141)
    at org.broadinstitute.hellbender.engine.FeatureDataSource.refillQueryCache(FeatureDataSource.java:534)
    at org.broadinstitute.hellbender.engine.FeatureDataSource.queryAndPrefetch(FeatureDataSource.java:502)
    at org.broadinstitute.hellbender.engine.FeatureManager.getFeatures(FeatureManager.java:340)
    at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:172)
    at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:124)
    at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:262)
    at org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator.apply(BaseRecalibrator.java:185)
    at org.broadinstitute.hellbender.engine.ReadWalker.lambda$traverse$0(ReadWalker.java:96)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at org.broadinstitute.hellbender.engine.ReadWalker.traverse(ReadWalker.java:94)
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1039)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
    at org.broadinstitute.hellbender.Main.main(Main.java:291)

I also run the ValidateSamFile, but it reported:no error found.

Post edited by bshifaw on

Best Answers

Answers

  • wulywuly Member
    edited May 30

    Thanks!@bshifaw ,
    I checked my disk space :

    (base) [[email protected] ~]$ df -lh
    Filesystem               Size  Used Avail Use% Mounted on
    /dev/mapper/centos-root   50G  6.3G   44G  13% /
    devtmpfs                 472G     0  472G   0% /dev
    tmpfs                    473G     0  473G   0% /dev/shm
    tmpfs                    473G   37M  472G   1% /run
    tmpfs                    473G     0  473G   0% /sys/fs/cgroup
    /dev/sda2               1014M  234M  781M  23% /boot
    /dev/mapper/centos-home  2.2T   37M  2.2T   1% /home
    tmpfs                     95G   12K   95G   1% /run/user/42
    /dev/mapper/vg1-lv1      8.7T  1.8T  6.5T  22% /data
    tmpfs                     95G     0   95G   0% /run/user/1003
    

    I think that there is still a lot of disk space left .I also tried to change the path for "-Djava.io.tmpdir=" ,but it still can't work.

    Post edited by bshifaw on
  • wulywuly Member
Sign In or Register to comment.