Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

Error while running GATK4 CNV workflow (something with the HDF5 lib)

xuejiaxuejia CrownbioMember

Hi the GATK team,
I am a new user of GATK4 CNV workflow. I always had an error while running the 3rd step NormalizeSomaticReadCounts:

16:08:11.670 INFO IntelGKLUtils - Trying to load Intel GKL library from:
jar:file:/home/jxue/softwares/GATK4_CNV/gatk4.jar!/com/intel/gkl/native/libIntelGKL.so
16:08:11.716 INFO IntelGKLUtils - Intel GKL library loaded from classpath.
[February 14, 2017 4:08:11 PM CST] org.broadinstitute.hellbender.tools.exome.NormalizeSomaticReadCounts --input E08055T.pcov.txt --panelOfNormals combined-normals.tsv --tangentNormalized E08055T.tn.tsv --preTangentNormalized E08055T.ptn.tsv --help false --version false --verbosity INFO --QUIET false --use_jdk_deflater false
[February 14, 2017 4:08:11 PM CST] Executing as [email protected] on Linux 2.6.32-431.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_112-b15; Version: Version:0288cff-SNAPSHOT
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.BUFFER_SIZE : 131072
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.COMPRESSION_LEVEL : 5
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.CREATE_INDEX : false
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.CREATE_MD5 : false
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.CUSTOM_READER_FACTORY :
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.EBI_REFERENCE_SERVICE_URL_MASK : http://www.ebi.ac.uk/ena/cram/md5/%s
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.NON_ZERO_BUFFER_SIZE : 131072
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.REFERENCE_FASTA : null
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
16:08:11.719 INFO NormalizeSomaticReadCounts - Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : false
16:08:11.720 INFO NormalizeSomaticReadCounts - Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
16:08:11.720 INFO NormalizeSomaticReadCounts - Defaults.USE_CRAM_REF_DOWNLOAD : false
16:08:11.720 INFO NormalizeSomaticReadCounts - Deflater IntelDeflater
16:08:11.720 INFO NormalizeSomaticReadCounts - Initializing engine
16:08:11.720 INFO NormalizeSomaticReadCounts - Done initializing engine
log4j:WARN No appenders could be found for logger (org.broadinstitute.hdf5.HDF5Library).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
#000: /mnt/scr1/abyrne/HDFJava-platypus-2.11/native/HDF5-prefix/src/HDF5/src/H5F.c line 604 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: /mnt/scr1/abyrne/HDFJava-platypus-2.11/native/HDF5-prefix/src/HDF5/src/H5Fint.c line 1085 in H5F_open(): unable to read superblock
major: File accessibilty
minor: Read failed
#002: /mnt/scr1/abyrne/HDFJava-platypus-2.11/native/HDF5-prefix/src/HDF5/src/H5Fsuper.c line 277 in H5F_super_read(): file signature not found
major: File accessibilty
minor: Not an HDF5 file
16:08:11.775 INFO NormalizeSomaticReadCounts - Shutting down engine
[February 14, 2017 4:08:11 PM CST] org.broadinstitute.hellbender.tools.exome.NormalizeSomaticReadCounts done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=1465384960
org.broadinstitute.hdf5.HDF5LibException: exception when opening '/home/jxue/xue/CopywriteR/79WES_TandN/combined-normals.tsv' with READ_ONLY mode: Not an HDF5 file
at org.broadinstitute.hdf5.HDF5File.open(HDF5File.java:490)
at org.broadinstitute.hdf5.HDF5File.(HDF5File.java:82)
at org.broadinstitute.hdf5.HDF5File.(HDF5File.java:66)
at org.broadinstitute.hellbender.tools.exome.NormalizeSomaticReadCounts.doWork(NormalizeSomaticReadCounts.java:105)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:108)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:166)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:185)
at org.broadinstitute.hellbender.Main.instanceMain(Main.java:76)
at org.broadinstitute.hellbender.Main.main(Main.java:92)
Caused by: ncsa.hdf.hdf5lib.exceptions.HDF5FileInterfaceException: Not an HDF5 file
at ncsa.hdf.hdf5lib.H5._H5Fopen(Native Method)
at ncsa.hdf.hdf5lib.H5.H5Fopen(H5.java:2795)
at org.broadinstitute.hdf5.HDF5File.open(HDF5File.java:487)
... 8 more

I guess it should be the problem with HDF5 library.
Please reply as soon as possible. Thank you very much in advance!

Cheers,
Jia

Tagged:

Best Answer

Answers

  • EADGEADG KielMember ✭✭✭

    Hi @xuejia,

    Iam not an expert for gatk4 CNV but after a short glance at your stack-trace I would say either your hdf5 file is corrupt or is not on the position you specified or there is a right/permission problem.

    major: File accessibilty
    minor: Unable to open file
    
    major: File accessibilty
    minor: Not an HDF5 file
    

    Can you check this ?

  • xuejiaxuejia CrownbioMember

    @EADG said:
    Hi @xuejia,

    Iam not an expert for gatk4 CNV but after a short glance at your stack-trace I would say either your hdf5 file is corrupt or is not on the position you specified or there is a right/permission problem.

    major: File accessibilty
    minor: Unable to open file
    
    major: File accessibilty
    minor: Not an HDF5 file
    

    Can you check this ?

    Thank you for your answer! I have no idea. I followed the instruction (http://depts.washington.edu/cssuwb/wiki/linux_hdf5_installation) to install HDF5 1.8.13 for my Redhat system. And then put HDF5-Java JNI Libraries Release 2.9 under "/usr/lib/jni/" as the tutorial (http://gatkforums.broadinstitute.org/gatk/discussion/6791/description-and-examples-of-the-steps-in-the-cnv-case-and-cnv-pon-creation-workflows#) described. Then gave the command to run the analysis:
    java -Djava.library.path=/usr/lib/jni/lib/linux/ -jar /home/jxue/softwares/GATK4_CNV/gatk4.jar NormalizeSomaticReadCounts -I E08055T.pcov.txt -PON combined-normals.tsv -PTN E08055T.ptn.tsv -TN E08055T.tn.tsv

    It always gives this error...
    Do you have any idea about this? It is more likely that I did not install HDF5 or HDF5-Java correctly.
    Thanks again!

    Jia

  • xuejiaxuejia CrownbioMember

    @EADG said:
    hm...crossreading over the tutorial i would say something went wrong with the creation of the PON.

    --panelOfNormals combined-normals.tsv
    

    The combined-normals.tsv is the file which you need in Step 2 (see the pdf in GoogleDrive) to create your PON.

    java -jar gatk4.jar CreatePanelOfNormals \
    
    -I sandbox/combined-normals.tsv \
    
    -O sandbox/normals.pon \
    
    -noQC \
    
    --disableSpark \
    
    --minimumTargetFactorPercentileThreshold 5
    

    Can you run the CreatePanelOfNormals again and put the result normals.pon into NormalizeSomaticReadCounts instead of combined-normals.tsv .

    java -jar gatk4.jar NormalizeSomaticReadCounts \
    
    -I cov/tumor.tsv \
    
    -PON sandbox/normals.pon \
    
    -PTN sandbox/tumor.ptn.tsv \
    
    -TN sandbox/tumor.tn.tsv
    

    By the way E08055T.pcov.txt should have the suffix tsv :)

    Hope this helps.

    Exactly! I missed one step before computing normalized read counts. Thank you very much for the help!
    But now I have another problem while plotting the results. I can get all the output files from Step 3,4 and 6 except the genome plots. All the read dots are missing in resulting PNG files. I can only see the axes and labels.

    I don't know if you can help.

    Thanks again!

    Jia
    imageimageimageimageimageimageimageimageimageimageimage

  • EADGEADG KielMember ✭✭✭

    hm can you check if all your outputfiles from the different steps have data in it? (open with nano/vim and check if more then the header is present)...or something went wrong with the installation of the r-packages (Rscript install_R_packages.R)... if this dont help, can you open a new thread ? Maybe other people can help...

  • xuejiaxuejia CrownbioMember

    @EADG said:
    hm can you check if all your outputfiles from the different steps have data in it? (open with nano/vim and check if more then the header is present)...or something went wrong with the installation of the r-packages (Rscript install_R_packages.R)... if this dont help, can you open a new thread ? Maybe other people can help...

    I check the outputs files. They all look fine. If it is someting with the R packages, I did not see any error message during this step.
    Thank you very much anyway. I will post a new question to others.

    Cheers,
    Jia

  • shleeshlee CambridgeMember, Broadie ✭✭✭✭✭

    Hi @xuejia,

    Please see if your issue is the same as that described here.

  • pdupdu Member

    Sorry to revive this thread, but I'm getting the same error using DenoiseReadCounts in GATK-4.0.1.2. I checked my pon.hdf5 file, so I don't think it's corrupt. Running DenoiseReadCounts gives me the following error, but still runs to completion and outputs the correct files. I'm not sure if it is actually using the PoN since it seems like it is not opening the file correctly.

    log4j:WARN No appenders could be found for logger (org.broadinstitute.hdf5.HDF5Library).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    15:47:20.728 INFO  DenoiseReadCounts - Reading read-counts file (./tumor.clean.counts.1.tsv)...
    HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
      #000: /mnt/scr1/abyrne/HDFJava-platypus-2.11/native/HDF5-prefix/src/HDF5/src/H5F.c line 604 in H5Fopen(): unable to open file
        major: File accessibilty
        minor: Unable to open file
      #001: /mnt/scr1/abyrne/HDFJava-platypus-2.11/native/HDF5-prefix/src/HDF5/src/H5Fint.c line 1085 in H5F_open(): unable to read superblock
        major: File accessibilty
        minor: Read failed
      #002: /mnt/scr1/abyrne/HDFJava-platypus-2.11/native/HDF5-prefix/src/HDF5/src/H5Fsuper.c line 277 in H5F_super_read(): file signature not found
        major: File accessibilty
        minor: Not an HDF5 file
    15:47:21.590 INFO  SVDDenoisingUtils - Validating sample intervals against original intervals used to build panel of normals...
    15:47:21.766 INFO  SVDDenoisingUtils - Preprocessing and standardizing sample read counts...
    15:47:22.066 INFO  SVDDenoisingUtils - Preprocessing read counts...
    15:47:22.066 INFO  SVDDenoisingUtils - Transforming read counts to fractional coverage...
    15:47:22.119 INFO  SVDDenoisingUtils - Subsetting sample intervals to post-filter panel intervals...
    15:47:22.211 INFO  SVDDenoisingUtils - Dividing by interval medians from the panel of normals...
    15:47:22.217 INFO  SVDDenoisingUtils - Sample read counts preprocessed.
    15:47:22.217 INFO  SVDDenoisingUtils - Standardizing read counts...
    15:47:22.217 INFO  SVDDenoisingUtils - Dividing by sample medians and transforming to log2 space...
    15:47:22.276 INFO  SVDDenoisingUtils - Subtracting sample median...
    15:47:22.297 INFO  SVDDenoisingUtils - Sample read counts standardized.
    15:47:22.297 INFO  SVDDenoisingUtils - Using 5 out of 5 eigensamples to denoise...
    15:47:22.311 INFO  SVDDenoisingUtils - Subtracting projection onto space spanned by eigensamples...
    15:47:22.366 INFO  SVDDenoisingUtils - Distributing the standardized read counts...
    15:47:22.366 INFO  SVDDenoisingUtils - Composing eigensample matrix for the requested number of eigensamples and transposing them...
    15:47:22.369 INFO  SVDDenoisingUtils - Computing projection...
    15:47:22.411 INFO  SVDDenoisingUtils - Subtracting projection...
    15:47:22.424 INFO  SVDDenoisingUtils - Sample denoised.
    15:47:22.631 INFO  DenoiseReadCounts - Writing standardized and denoised copy ratios...
    15:47:24.190 INFO  DenoiseReadCounts - Read counts successfully denoised.
    15:47:24.190 INFO  DenoiseReadCounts - Shutting down engine
    [August 27, 2019 3:47:24 PM UTC] org.broadinstitute.hellbender.tools.copynumber.DenoiseReadCounts done. Elapsed time: 0.06 minutes.
    Runtime.totalMemory()=2270167040
    
  • sleeslee Member, Broadie, Dev ✭✭✭

    @pdu this was fixed in https://github.com/broadinstitute/gatk/pull/5082 (and in any case, the warning is harmless).

Sign In or Register to comment.