Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

GATK GenomicsDBImport error

jodybkjodybk LondonMember

Hi, I am trying to run gatk GenomicsDBI import and it is giving an error (see below). I seems like the tools runs fine and I can even use it DB with GenotypeGVCFs. I have validated all VCF files using ValidateVariants.

Any suggestions?

Thanks

gatk GenomicsDBImport --genomicsdb-workspace-path my_database --batch-size 500 -L Chromosome:3343176-3343820 --sample-name-map samples.map --reader-threads 40 --tmp-dir=. --java-options '-Xmx50g'

Using GATK jar /mnt/storage/jody/miniconda3/envs/dev/share/gatk4-4.1.3.0-0/gatk-package-4.1.3.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx50g -jar /mnt/storage/jody/miniconda3/envs/dev/share/gatk4-4.1.3.0-0/gatk-package-4.1.3.0-local.jar GenomicsDBImport --genomicsdb-workspace-path my_database --batch-size 500 -L Chromosome:3343176-3343820 --sample-name-map samples.map --reader-threads 40 --tmp-dir=.
01:44:53.437 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/storage/jody/miniconda3/envs/dev/share/gatk4-4.1.3.0-0/gatk-package-4.1.3.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Sep 26, 2019 1:44:55 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
01:44:55.301 INFO GenomicsDBImport - ------------------------------------------------------------
01:44:55.302 INFO GenomicsDBImport - The Genome Analysis Toolkit (GATK) v4.1.3.0
01:44:55.302 INFO GenomicsDBImport - For support and documentation go to https://software.broadinstitute.org/gatk/
01:44:55.302 INFO GenomicsDBImport - Executing as [email protected] on Linux v4.2.0-27-generic amd64
01:44:55.302 INFO GenomicsDBImport - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
01:44:55.303 INFO GenomicsDBImport - Start Date/Time: 26 September 2019 01:44:53 BST
01:44:55.303 INFO GenomicsDBImport - ------------------------------------------------------------
01:44:55.303 INFO GenomicsDBImport - ------------------------------------------------------------
01:44:55.304 INFO GenomicsDBImport - HTSJDK Version: 2.20.1
01:44:55.304 INFO GenomicsDBImport - Picard Version: 2.20.5
01:44:55.304 INFO GenomicsDBImport - HTSJDK Defaults.COMPRESSION_LEVEL : 2
01:44:55.304 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
01:44:55.304 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
01:44:55.304 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
01:44:55.304 INFO GenomicsDBImport - Deflater: IntelDeflater
01:44:55.304 INFO GenomicsDBImport - Inflater: IntelInflater
01:44:55.305 INFO GenomicsDBImport - GCS max retries/reopens: 20
01:44:55.305 INFO GenomicsDBImport - Requester pays: disabled
01:44:55.305 INFO GenomicsDBImport - Initializing engine
01:44:55.779 INFO IntervalArgumentCollection - Processing 645 bp from intervals
01:44:55.780 INFO GenomicsDBImport - Done initializing engine
01:44:55.976 INFO GenomicsDBImport - Vid Map JSON file will be written to /mnt/storage/jody/ena/leopold/my_database/vidmap.json
01:44:55.976 INFO GenomicsDBImport - Callset Map JSON file will be written to /mnt/storage/jody/ena/leopold/my_database/callset.json
01:44:55.976 INFO GenomicsDBImport - Complete VCF Header will be written to /mnt/storage/jody/ena/leopold/my_database/vcfheader.vcf
01:44:55.976 INFO GenomicsDBImport - Importing to array - /mnt/storage/jody/ena/leopold/my_database/genomicsdb_array
01:44:55.977 INFO ProgressMeter - Starting traversal
01:44:55.977 INFO ProgressMeter - Current Locus Elapsed Minutes Batches Processed Batches/Minute
01:44:56.289 INFO GenomicsDBImport - Starting batch input file preload
01:44:58.037 INFO GenomicsDBImport - Finished batch preload
01:44:58.037 INFO GenomicsDBImport - Importing batch 1 with 500 samples
01:44:59.890 INFO GenomicsDBImport - Done importing batch 1/36
01:44:59.890 INFO GenomicsDBImport - Starting batch input file preload
01:45:00.863 INFO GenomicsDBImport - Finished batch preload
01:45:00.863 INFO GenomicsDBImport - Importing batch 2 with 500 samples
01:45:04.317 INFO GenomicsDBImport - Done importing batch 2/36
01:45:04.317 INFO GenomicsDBImport - Starting batch input file preload
01:45:05.236 INFO GenomicsDBImport - Finished batch preload
01:45:05.236 INFO GenomicsDBImport - Importing batch 3 with 500 samples
01:45:06.552 INFO ProgressMeter - Chromosome:3343176 0.2 3 17.0
01:45:06.552 INFO GenomicsDBImport - Done importing batch 3/36
01:45:06.553 INFO GenomicsDBImport - Starting batch input file preload
01:45:06.846 INFO GenomicsDBImport - Finished batch preload
01:45:06.846 INFO GenomicsDBImport - Importing batch 4 with 500 samples
01:45:08.196 INFO GenomicsDBImport - Done importing batch 4/36
01:45:08.196 INFO GenomicsDBImport - Starting batch input file preload
01:45:09.411 INFO GenomicsDBImport - Finished batch preload
01:45:09.411 INFO GenomicsDBImport - Importing batch 5 with 500 samples
01:45:10.575 INFO GenomicsDBImport - Done importing batch 5/36
01:45:10.575 INFO GenomicsDBImport - Starting batch input file preload
01:45:10.934 INFO GenomicsDBImport - Finished batch preload
01:45:10.934 INFO GenomicsDBImport - Importing batch 6 with 500 samples
01:45:12.244 INFO GenomicsDBImport - Done importing batch 6/36
01:45:12.244 INFO GenomicsDBImport - Starting batch input file preload
01:45:12.915 INFO GenomicsDBImport - Finished batch preload
01:45:12.915 INFO GenomicsDBImport - Importing batch 7 with 500 samples
01:45:13.777 INFO GenomicsDBImport - Done importing batch 7/36
01:45:13.777 INFO GenomicsDBImport - Starting batch input file preload
01:45:14.132 INFO GenomicsDBImport - Finished batch preload
01:45:14.132 INFO GenomicsDBImport - Importing batch 8 with 500 samples
01:45:17.730 INFO ProgressMeter - Chromosome:3343176 0.4 8 22.1
01:45:17.731 INFO GenomicsDBImport - Done importing batch 8/36
01:45:17.732 INFO GenomicsDBImport - Starting batch input file preload
01:45:19.248 INFO GenomicsDBImport - Finished batch preload
01:45:19.248 INFO GenomicsDBImport - Importing batch 9 with 500 samples
01:45:20.292 INFO GenomicsDBImport - Done importing batch 9/36
01:45:20.293 INFO GenomicsDBImport - Starting batch input file preload
01:45:20.512 INFO GenomicsDBImport - Finished batch preload
01:45:20.513 INFO GenomicsDBImport - Importing batch 10 with 500 samples
01:45:22.159 INFO GenomicsDBImport - Done importing batch 10/36
01:45:22.159 INFO GenomicsDBImport - Starting batch input file preload
01:45:23.519 INFO GenomicsDBImport - Finished batch preload
01:45:23.519 INFO GenomicsDBImport - Importing batch 11 with 500 samples
01:45:25.032 INFO GenomicsDBImport - Done importing batch 11/36
01:45:25.033 INFO GenomicsDBImport - Starting batch input file preload
01:45:25.360 INFO GenomicsDBImport - Finished batch preload
01:45:25.360 INFO GenomicsDBImport - Importing batch 12 with 500 samples
01:45:26.533 INFO GenomicsDBImport - Done importing batch 12/36
01:45:26.534 INFO GenomicsDBImport - Starting batch input file preload
01:45:27.308 INFO GenomicsDBImport - Finished batch preload
01:45:27.308 INFO GenomicsDBImport - Importing batch 13 with 500 samples
01:45:31.119 INFO ProgressMeter - Chromosome:3343176 0.6 13 22.2
01:45:31.119 INFO GenomicsDBImport - Done importing batch 13/36
01:45:31.119 INFO GenomicsDBImport - Starting batch input file preload
01:45:32.688 INFO GenomicsDBImport - Finished batch preload
01:45:32.689 INFO GenomicsDBImport - Importing batch 14 with 500 samples
01:45:34.246 INFO GenomicsDBImport - Done importing batch 14/36
01:45:34.246 INFO GenomicsDBImport - Starting batch input file preload
01:45:34.589 INFO GenomicsDBImport - Finished batch preload
01:45:34.589 INFO GenomicsDBImport - Importing batch 15 with 500 samples
01:45:36.108 INFO GenomicsDBImport - Done importing batch 15/36
01:45:36.109 INFO GenomicsDBImport - Starting batch input file preload
01:45:36.613 INFO GenomicsDBImport - Finished batch preload
01:45:36.613 INFO GenomicsDBImport - Importing batch 16 with 500 samples
01:45:38.998 INFO GenomicsDBImport - Done importing batch 16/36
01:45:38.998 INFO GenomicsDBImport - Starting batch input file preload
01:45:39.362 INFO GenomicsDBImport - Finished batch preload
01:45:39.362 INFO GenomicsDBImport - Importing batch 17 with 500 samples
01:45:40.850 INFO GenomicsDBImport - Done importing batch 17/36
01:45:40.850 INFO GenomicsDBImport - Starting batch input file preload
01:45:41.249 INFO GenomicsDBImport - Finished batch preload
01:45:41.249 INFO GenomicsDBImport - Importing batch 18 with 500 samples
01:45:42.568 INFO ProgressMeter - Chromosome:3343176 0.8 18 23.2
01:45:42.568 INFO GenomicsDBImport - Done importing batch 18/36
01:45:42.568 INFO GenomicsDBImport - Starting batch input file preload
01:45:43.812 INFO GenomicsDBImport - Finished batch preload
01:45:43.812 INFO GenomicsDBImport - Importing batch 19 with 500 samples
01:45:45.008 INFO GenomicsDBImport - Done importing batch 19/36
01:45:45.008 INFO GenomicsDBImport - Starting batch input file preload
01:45:45.224 INFO GenomicsDBImport - Finished batch preload
01:45:45.224 INFO GenomicsDBImport - Importing batch 20 with 500 samples
01:45:46.446 INFO GenomicsDBImport - Done importing batch 20/36
01:45:46.447 INFO GenomicsDBImport - Starting batch input file preload
01:45:46.754 INFO GenomicsDBImport - Finished batch preload
01:45:46.754 INFO GenomicsDBImport - Importing batch 21 with 500 samples
01:45:47.967 INFO GenomicsDBImport - Done importing batch 21/36
01:45:47.967 INFO GenomicsDBImport - Starting batch input file preload
01:45:48.685 INFO GenomicsDBImport - Finished batch preload
01:45:48.685 INFO GenomicsDBImport - Importing batch 22 with 500 samples
01:45:49.785 INFO GenomicsDBImport - Done importing batch 22/36
01:45:49.785 INFO GenomicsDBImport - Starting batch input file preload
01:45:49.906 INFO GenomicsDBImport - Finished batch preload
01:45:49.906 INFO GenomicsDBImport - Importing batch 23 with 500 samples
01:45:51.206 INFO GenomicsDBImport - Done importing batch 23/36
01:45:51.207 INFO GenomicsDBImport - Starting batch input file preload
01:45:51.383 INFO GenomicsDBImport - Finished batch preload
01:45:51.384 INFO GenomicsDBImport - Importing batch 24 with 500 samples
01:45:52.789 INFO ProgressMeter - Chromosome:3343176 0.9 24 25.3
01:45:52.789 INFO GenomicsDBImport - Done importing batch 24/36
01:45:52.789 INFO GenomicsDBImport - Starting batch input file preload
01:45:53.052 INFO GenomicsDBImport - Finished batch preload
01:45:53.052 INFO GenomicsDBImport - Importing batch 25 with 500 samples
01:45:54.187 INFO GenomicsDBImport - Done importing batch 25/36
01:45:54.187 INFO GenomicsDBImport - Starting batch input file preload
01:45:54.516 INFO GenomicsDBImport - Finished batch preload
01:45:54.516 INFO GenomicsDBImport - Importing batch 26 with 500 samples
01:45:55.719 INFO GenomicsDBImport - Done importing batch 26/36
01:45:55.720 INFO GenomicsDBImport - Starting batch input file preload
01:45:56.041 INFO GenomicsDBImport - Finished batch preload
01:45:56.041 INFO GenomicsDBImport - Importing batch 27 with 500 samples
01:45:57.172 INFO GenomicsDBImport - Done importing batch 27/36
01:45:57.172 INFO GenomicsDBImport - Starting batch input file preload
01:45:57.828 INFO GenomicsDBImport - Finished batch preload
01:45:57.828 INFO GenomicsDBImport - Importing batch 28 with 500 samples
01:45:58.832 INFO GenomicsDBImport - Done importing batch 28/36
01:45:58.832 INFO GenomicsDBImport - Starting batch input file preload
01:45:59.038 INFO GenomicsDBImport - Finished batch preload
01:45:59.038 INFO GenomicsDBImport - Importing batch 29 with 500 samples
01:46:00.158 INFO GenomicsDBImport - Done importing batch 29/36
01:46:00.159 INFO GenomicsDBImport - Starting batch input file preload
01:46:00.387 INFO GenomicsDBImport - Finished batch preload
01:46:00.388 INFO GenomicsDBImport - Importing batch 30 with 500 samples
01:46:01.252 INFO GenomicsDBImport - Done importing batch 30/36
01:46:01.253 INFO GenomicsDBImport - Starting batch input file preload
01:46:01.884 INFO GenomicsDBImport - Finished batch preload
01:46:01.884 INFO GenomicsDBImport - Importing batch 31 with 500 samples
01:46:03.322 INFO ProgressMeter - Chromosome:3343176 1.1 31 27.6
01:46:03.323 INFO GenomicsDBImport - Done importing batch 31/36
01:46:03.323 INFO GenomicsDBImport - Starting batch input file preload
01:46:03.688 INFO GenomicsDBImport - Finished batch preload
01:46:03.688 INFO GenomicsDBImport - Importing batch 32 with 500 samples
01:46:05.148 INFO GenomicsDBImport - Done importing batch 32/36
01:46:05.148 INFO GenomicsDBImport - Starting batch input file preload
01:46:05.706 INFO GenomicsDBImport - Finished batch preload
01:46:05.706 INFO GenomicsDBImport - Importing batch 33 with 500 samples
01:46:07.155 INFO GenomicsDBImport - Done importing batch 33/36
01:46:07.156 INFO GenomicsDBImport - Starting batch input file preload
01:46:07.456 INFO GenomicsDBImport - Finished batch preload
01:46:07.456 INFO GenomicsDBImport - Importing batch 34 with 500 samples
01:46:09.047 INFO GenomicsDBImport - Done importing batch 34/36
01:46:09.047 INFO GenomicsDBImport - Starting batch input file preload
01:46:15.085 INFO GenomicsDBImport - Shutting down engine
[26 September 2019 01:46:15 BST] org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport done. Elapsed time: 1.36 minutes.
Runtime.totalMemory()=28423225344


A USER ERROR has occurred: Couldn't read file. Error was: Failure while waiting for FeatureReader to initialize with exception: htsjdk.samtools.SAMFormatException: Invalid GZIP header


Answers

Sign In or Register to comment.