We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

BwaMemIndexImageCreator fails to generate reference.img file

Hi every one,
I am trying to create a BWA-MEM index image file for use with GATK BWA tools StructuralVariationDiscoveryPipelineSpark.

I am indexed the human reference genome with BWA with following command:

  $bwa index NGS/GRef/bwa-GRef/gr.fa
  $samtools faidx NGS/GRef/bwa-GRef/gr.fa

then, run the BwaMemIndexImageCreator tool for creation of .img file with following command:

$java -DGATK_STACKTRACE_ON_USER_EXCEPTION=’true’ -Xms10g -jar NGS/soft/gatk/gatk-package-4.0.8.1-local.jar BwaMemIndexImageCreator -I NGS/GRef/bwa-GRef/gr.fa -O GRef/bwa-GRef/gr.fa.img

but it fails to generate .img file with following error:

20:38:34.953 INFO NativeLibraryLoader - Loading libgkl_compression.so
from
jar:file:/home/novingenetic/NGS/soft/gatk/gatk-package-4.0.8.1-local.jar!/com/intel/gkl/native/libgkl_compression.so 20:38:36.291 INFO BwaMemIndexImageCreator -
------------------------------------------------------------ 20:38:36.291 INFO BwaMemIndexImageCreator - The Genome Analysis
Toolkit (GATK) v4.0.8.1 20:38:36.292 INFO BwaMemIndexImageCreator -
For support and documentation go to
https://software.broadinstitute.org/gatk/ 20:38:36.292 INFO
BwaMemIndexImageCreator - Executing as [email protected] on
Linux v4.15.0-58-generic amd64 20:38:36.293 INFO
BwaMemIndexImageCreator - Java runtime: OpenJDK 64-Bit Server VM
v1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 20:38:36.293 INFO
BwaMemIndexImageCreator - Start Date/Time: August 26, 2019 8:38:34 PM
IRDT 20:38:36.294 INFO BwaMemIndexImageCreator -
------------------------------------------------------------ 20:38:36.294 INFO BwaMemIndexImageCreator -
------------------------------------------------------------ 20:38:36.295 INFO BwaMemIndexImageCreator - HTSJDK Version: 2.16.0
20:38:36.296 INFO BwaMemIndexImageCreator - Picard Version: 2.18.7
20:38:36.296 INFO BwaMemIndexImageCreator - HTSJDK
Defaults.COMPRESSION_LEVEL : 2 20:38:36.296 INFO
BwaMemIndexImageCreator - HTSJDK
Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 20:38:36.297 INFO
BwaMemIndexImageCreator - HTSJDK
Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 20:38:36.297 INFO
BwaMemIndexImageCreator - HTSJDK
Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 20:38:36.297 INFO
BwaMemIndexImageCreator - Deflater: IntelDeflater 20:38:36.298 INFO
BwaMemIndexImageCreator - Inflater: IntelInflater 20:38:36.298 INFO
BwaMemIndexImageCreator - GCS max retries/reopens: 20 20:38:36.298
INFO BwaMemIndexImageCreator - Using google-cloud-java fork
https://github.com/broadinstitute/google-cloud-java/releases/tag/0.20.5-alpha-GCS-RETRY-FIX
20:38:36.298 INFO BwaMemIndexImageCreator - Initializing engine
20:38:36.299 INFO BwaMemIndexImageCreator - Done initializing engine
[bwa_index] Pack FASTA... 21.34 sec [bwa_index] Construct BWT for the
packed sequence... Killed

I was wondering if any one knows how I can generate the .img file?

  • operation system: linux ubuntu v18
  • java version: openjdk version "1.8.0_222"
  • bwa version: 0.7.17-r1188
  • gatk version: gatk-4.0.8.1

Answers

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    Hi @faramarzg

    Can you please use the latest gatk 4.1.3.0 version and if the error persists can you please post the entire error log.

  • faramarzgfaramarzg IranMember

    @bhanuGandham said:
    Hi @faramarzg

    Can you please use the latest gatk 4.1.3.0 version and if the error persists can you please post the entire error log.

    Hi @bhanuGandham
    I try it by gatk 4.1.3.0 version and the same result was obtained.

    19:00:36.089 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/novingenetic/NGS/soft/gatk/gatk-package-4.1.3.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    Aug 27, 2019 7:00:44 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    19:00:44.212 INFO BwaMemIndexImageCreator - ------------------------------------------------------------
    19:00:44.213 INFO BwaMemIndexImageCreator - The Genome Analysis Toolkit (GATK) v4.1.3.0
    19:00:44.214 INFO BwaMemIndexImageCreator - For support and documentation go to https://software.broadinstitute.org/gatk/
    19:00:44.215 INFO BwaMemIndexImageCreator - Executing as [email protected] on Linux v4.15.0-58-generic amd64
    19:00:44.216 INFO BwaMemIndexImageCreator - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10
    19:00:44.217 INFO BwaMemIndexImageCreator - Start Date/Time: August 27, 2019 7:00:36 PM IRDT
    19:00:44.217 INFO BwaMemIndexImageCreator - ------------------------------------------------------------
    19:00:44.218 INFO BwaMemIndexImageCreator - ------------------------------------------------------------
    19:00:44.218 INFO BwaMemIndexImageCreator - HTSJDK Version: 2.20.1
    19:00:44.219 INFO BwaMemIndexImageCreator - Picard Version: 2.20.5
    19:00:44.219 INFO BwaMemIndexImageCreator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    19:00:44.219 INFO BwaMemIndexImageCreator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    19:00:44.219 INFO BwaMemIndexImageCreator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    19:00:44.219 INFO BwaMemIndexImageCreator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    19:00:44.219 INFO BwaMemIndexImageCreator - Deflater: IntelDeflater
    19:00:44.219 INFO BwaMemIndexImageCreator - Inflater: IntelInflater
    19:00:44.219 INFO BwaMemIndexImageCreator - GCS max retries/reopens: 20
    19:00:44.220 INFO BwaMemIndexImageCreator - Requester pays: disabled
    19:00:44.220 INFO BwaMemIndexImageCreator - Initializing engine
    19:00:44.220 INFO BwaMemIndexImageCreator - Done initializing engine
    [bwa_index] Pack FASTA... 84.37 sec
    [bwa_index] Construct BWT for the packed sequence...
    Killed

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin
    edited September 2019

    HI @faramarzg

    You have set the initial memory allocation pool(Xms) as 10G. Maybe that is more than the available memory on your machine. What are the specs of the machine you are using?

  • faramarzgfaramarzg IranMember

    @bhanuGandham said:
    HI @faramarzg

    You have set the initial memory allocation pool(Xms) as 10G. Maybe that is more than the available memory on your machine. What are the specs of the machine you are using?

    I did test it by 4, 6, 8 and 10 initial memory, but same result was obtained.
    My system configure is CPU i7, 16GB RAM. I useing Ubuntu Linux operation system v18 by wmvare with initial configure 2 core of CPU and 8GB RAM.

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    HI @faramarzg

    Can you please share your Input fasta file so that I can try to recreate the issue on my end. Follow the directions here to send us your file.

  • faramarzgfaramarzg IranMember

    @bhanuGandham said:
    HI @faramarzg

    Can you please share your Input fasta file so that I can try to recreate the issue on my end. Follow the directions here to send us your file.

    Hi dear bhanuGandham
    I uploaded a zip file containing .fa .dict .fai files and two text files including common line and log output. zip file name is Novin.

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    HI @faramarzg

    I was not able to recreate the error on my end using GATK4.1.3.0. The tool ran without an error. Can you please post the entire error log?

  • faramarzgfaramarzg IranMember

    @bhanuGandham said:
    HI @faramarzg

    I was not able to recreate the error on my end using GATK4.1.3.0. The tool ran without an error. Can you please post the entire error log?

    $ java -DGATK_STACKTRACE_ON_USER_EXCEPTION=’true’ -jar soft/gatk/gatk-package-4.1.3.0-local.jar BwaMemIndexImageCreator -I GRef/bwa-GRef/gr.fa -O GRef/bwa-GRef/gr.fa.img
    12:35:38.511 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/novingenetic/NGS/soft/gatk/gatk-package-4.1.3.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    Sep 22, 2019 12:35:40 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    12:35:41.008 INFO BwaMemIndexImageCreator - ------------------------------------------------------------
    12:35:41.009 INFO BwaMemIndexImageCreator - The Genome Analysis Toolkit (GATK) v4.1.3.0
    12:35:41.009 INFO BwaMemIndexImageCreator - For support and documentation go to https://software.broadinstitute.org/gatk/
    12:35:41.010 INFO BwaMemIndexImageCreator - Executing as [email protected] on Linux v4.15.0-64-generic amd64
    12:35:41.011 INFO BwaMemIndexImageCreator - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10
    12:35:41.011 INFO BwaMemIndexImageCreator - Start Date/Time: September 22, 2019 12:35:38 PM IRST
    12:35:41.012 INFO BwaMemIndexImageCreator - ------------------------------------------------------------
    12:35:41.012 INFO BwaMemIndexImageCreator - ------------------------------------------------------------
    12:35:41.014 INFO BwaMemIndexImageCreator - HTSJDK Version: 2.20.1
    12:35:41.014 INFO BwaMemIndexImageCreator - Picard Version: 2.20.5
    12:35:41.014 INFO BwaMemIndexImageCreator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    12:35:41.015 INFO BwaMemIndexImageCreator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    12:35:41.015 INFO BwaMemIndexImageCreator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    12:35:41.015 INFO BwaMemIndexImageCreator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    12:35:41.015 INFO BwaMemIndexImageCreator - Deflater: IntelDeflater
    12:35:41.016 INFO BwaMemIndexImageCreator - Inflater: IntelInflater
    12:35:41.016 INFO BwaMemIndexImageCreator - GCS max retries/reopens: 20
    12:35:41.016 INFO BwaMemIndexImageCreator - Requester pays: disabled
    12:35:41.016 INFO BwaMemIndexImageCreator - Initializing engine
    12:35:41.016 INFO BwaMemIndexImageCreator - Done initializing engine
    [bwa_index] Pack FASTA... 21.99 sec
    [bwa_index] Construct BWT for the packed sequence...
    Killed

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    HI @faramarzg

    How long did BwaMemIndexImageCreator run before it got killed?

  • faramarzgfaramarzg IranMember

    @bhanuGandham said:
    HI @faramarzg

    How long did BwaMemIndexImageCreator run before it got killed?

    It's different. Usually occurs after 30min.

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    Hi @faramarzg

    Ah looks like you are setting the xmx value, can you please try to adjust your -Xmx values to 4, 6, 8 and 10 and see if that helps.
    Note: With GATK tools we recommend setting the xmx value such that there are a 2-3GBs of space for the native code.

Sign In or Register to comment.