Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

CreateReadCountPanelOfNormals Errors:java.lang.IllegalArgumentException: TableColumnCollection must

ahdaahda ChinaMember

I met an error when i used CreateReadCountPanelOfNormals, I had the S*.count.hdf5 by CollectReadCounts and everything seemed ok.

gatk --java-options "-Xmx5g" CreateReadCountPanelOfNormals --input gatk_commonCnv/S07.count.hdf5 --input gatk_commonCnv/S08.count.hdf5 --input gatk_commonCnv/S09.count.hdf5 --input gatk_commonCnv/S10.count.hdf5 --input gatk_commonCnv/S11.count.hdf5 --input gatk_commonCnv/S12.count.hdf5 --input gatk_commonCnv/S13.count.hdf5 --input gatk_commonCnv/S14.count.hdf5 --input gatk_commonCnv/S15.count.hdf5 --input gatk_commonCnv/S16.count.hdf5 --input gatk_commonCnv/S17.count.hdf5 --input gatk_commonCnv/S18.count.hdf5 --input gatk_commonCnv/S19.count.hdf5 --input gatk_commonCnv/S20.count.hdf5 --input gatk_commonCnv/S21.count.hdf5 --input gatk_commonCnv/S22.count.hdf5 --input gatk_commonCnv/S23.count.hdf5 --input gatk_commonCnv/S24.count.hdf5 --input gatk_commonCnv/S25.count.hdf5 --input gatk_commonCnv/S26.count.hdf5 --input gatk_commonCnv/S49.count.hdf5 --input gatk_commonCnv/S50.count.hdf5 --input gatk_commonCnv/S51.count.hdf5 --input gatk_commonCnv/S52.count.hdf5 --input gatk_commonCnv/S53.count.hdf5 --input gatk_commonCnv/S54.count.hdf5 --input gatk_commonCnv/S55.count.hdf5 --input gatk_commonCnv/S56.count.hdf5 --annotated-intervals call_region/preprocessed.filter.interval_list --output gatk_somaticCnv/normal.pon.hdf5
Using GATK jar /home/my/anaconda2/share/gatk4-4.1.2.0-1/gatk-package-4.1.2.0-local.jar
Running:
/home/my/anaconda2/bin/java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx5g -jar /home/my/anaconda2/share/gatk4-4.1.2.0-1/gatk-package-4.1.2.0-local.jar CreateReadCountPanelOfNormals --input gatk_commonCnv/S07.count.hdf5 --input gatk_commonCnv/S08.count.hdf5 --input gatk_commonCnv/S09.count.hdf5 --input gatk_commonCnv/S10.count.hdf5 --input gatk_commonCnv/S11.count.hdf5 --input gatk_commonCnv/S12.count.hdf5 --input gatk_commonCnv/S13.count.hdf5 --input gatk_commonCnv/S14.count.hdf5 --input gatk_commonCnv/S15.count.hdf5 --input gatk_commonCnv/S16.count.hdf5 --input gatk_commonCnv/S17.count.hdf5 --input gatk_commonCnv/S18.count.hdf5 --input gatk_commonCnv/S19.count.hdf5 --input gatk_commonCnv/S20.count.hdf5 --input gatk_commonCnv/S21.count.hdf5 --input gatk_commonCnv/S22.count.hdf5 --input gatk_commonCnv/S23.count.hdf5 --input gatk_commonCnv/S24.count.hdf5 --input gatk_commonCnv/S25.count.hdf5 --input gatk_commonCnv/S26.count.hdf5 --input gatk_commonCnv/S49.count.hdf5 --input gatk_commonCnv/S50.count.hdf5 --input gatk_commonCnv/S51.count.hdf5 --input gatk_commonCnv/S52.count.hdf5 --input gatk_commonCnv/S53.count.hdf5 --input gatk_commonCnv/S54.count.hdf5 --input gatk_commonCnv/S55.count.hdf5 --input gatk_commonCnv/S56.count.hdf5 --annotated-intervals call_region/preprocessed.filter.interval_list --output gatk_somaticCnv/normal.pon.hdf5
20:02:17.211 WARN SparkContextFactory - Environment variables HELLBENDER_TEST_PROJECT and HELLBENDER_JSON_SERVICE_ACCOUNT_KEY must be set or the GCS hadoop connector will not be configured properly
20:02:32.828 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/chendan/anaconda2/share/gatk4-4.1.2.0-1/gatk-package-4.1.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Jul 01, 2019 8:02:35 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
20:02:35.455 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
20:02:35.455 INFO CreateReadCountPanelOfNormals - The Genome Analysis Toolkit (GATK) v4.1.2.0
20:02:35.456 INFO CreateReadCountPanelOfNormals - For support and documentation go to https://software.broadinstitute.org/gatk/
20:02:35.456 INFO CreateReadCountPanelOfNormals - Executing as [email protected] on Linux v3.10.0-693.5.2.el7.x86_64 amd64
20:02:35.456 INFO CreateReadCountPanelOfNormals - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
20:02:35.457 INFO CreateReadCountPanelOfNormals - Start Date/Time: July 1, 2019 8:02:27 PM CST
20:02:35.457 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
20:02:35.457 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
20:02:35.458 INFO CreateReadCountPanelOfNormals - HTSJDK Version: 2.19.0
20:02:35.458 INFO CreateReadCountPanelOfNormals - Picard Version: 2.19.0
20:02:35.458 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.COMPRESSION_LEVEL : 2
20:02:35.458 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
20:02:35.458 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
20:02:35.458 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
20:02:35.458 INFO CreateReadCountPanelOfNormals - Deflater: IntelDeflater
20:02:35.459 INFO CreateReadCountPanelOfNormals - Inflater: IntelInflater
20:02:35.459 INFO CreateReadCountPanelOfNormals - GCS max retries/reopens: 20
20:02:35.459 INFO CreateReadCountPanelOfNormals - Requester pays: disabled
20:02:35.459 INFO CreateReadCountPanelOfNormals - Initializing engine
20:02:35.459 INFO CreateReadCountPanelOfNormals - Done initializing engine
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/07/01 20:03:21 INFO SparkContext: Running Spark version 2.2.0
19/07/01 20:03:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/01 20:03:45 INFO SparkContext: Submitted application: CreateReadCountPanelOfNormals
19/07/01 20:03:52 INFO SecurityManager: Changing view acls to: chendan
19/07/01 20:03:52 INFO SecurityManager: Changing modify acls to: chendan
19/07/01 20:03:52 INFO SecurityManager: Changing view acls groups to:
19/07/01 20:03:52 INFO SecurityManager: Changing modify acls groups to:
19/07/01 20:03:52 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(chendan); groups with view permissions: Set(); users with modify permissions: Set(chendan); groups with modify permissions: Set()
19/07/01 20:04:14 INFO Utils: Successfully started service 'sparkDriver' on port 35819.
19/07/01 20:04:15 INFO SparkEnv: Registering MapOutputTracker
19/07/01 20:04:15 INFO SparkEnv: Registering BlockManagerMaster
19/07/01 20:04:15 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/07/01 20:04:15 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/07/01 20:04:15 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-f5c9bd81-f8bf-4775-ad1e-6f2213f7e7f0
19/07/01 20:04:15 INFO MemoryStore: MemoryStore started with capacity 2.5 GB
19/07/01 20:04:19 INFO SparkEnv: Registering OutputCommitCoordinator
19/07/01 20:04:31 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/07/01 20:04:31 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.0.100:4040
19/07/01 20:04:33 INFO Executor: Starting executor ID driver on host localhost
19/07/01 20:04:33 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46344.
19/07/01 20:04:33 INFO NettyBlockTransferService: Server created on 10.0.0.100:46344
19/07/01 20:04:33 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/07/01 20:04:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.0.100, 46344, None)
19/07/01 20:04:33 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.100:46344 with 2.5 GB RAM, BlockManagerId(driver, 10.0.0.100, 46344, None)
19/07/01 20:04:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.0.100, 46344, None)
19/07/01 20:04:33 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.0.0.100, 46344, None)
20:04:39.309 INFO CreateReadCountPanelOfNormals - Spark verbosity set to INFO (see --spark-verbosity argument)
19/07/01 20:04:39 INFO HDF5Library: Trying to load HDF5 library from:
jar:file:/home/my/anaconda2/share/gatk4-4.1.2.0-1/gatk-package-4.1.2.0-local.jar!/org/broadinstitute/hdf5/libjhdf5.2.11.0.so
19/07/01 20:04:44 INFO H5: HDF5 library:
19/07/01 20:04:44 INFO H5: successfully loaded.
20:04:46.606 INFO CreateReadCountPanelOfNormals - Retrieving intervals from first read-counts file (gatk_commonCnv/S07.count.hdf5)...
20:04:59.226 INFO CreateReadCountPanelOfNormals - Reading and validating annotated intervals...
19/07/01 20:05:05 INFO SparkUI: Stopped Spark web UI at http://10.0.0.100:4040
19/07/01 20:05:05 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/07/01 20:05:05 INFO MemoryStore: MemoryStore cleared
19/07/01 20:05:05 INFO BlockManager: BlockManager stopped
19/07/01 20:05:05 INFO BlockManagerMaster: BlockManagerMaster stopped
19/07/01 20:05:05 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/07/01 20:05:05 INFO SparkContext: Successfully stopped SparkContext
20:05:05.721 INFO CreateReadCountPanelOfNormals - Shutting down engine
[July 1, 2019 8:05:05 PM CST] org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals done. Elapsed time: 2.64 minutes.
Runtime.totalMemory()=623902720
java.lang.IllegalArgumentException: TableColumnCollection must contain standard columns: [CONTIG, START, END].
at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:724)

at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AnnotatedIntervalCollection.getAnnotationKeys(AnnotatedIntervalCollection.java:111)
at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AnnotatedIntervalCollection.(AnnotatedIntervalCollection.java:79)
at org.broadinstitute.hellbender.tools.copynumber.arguments.CopyNumberArgumentValidationUtils.validateAnnotatedIntervals(CopyNumberArgumentValidationUtils.java:130)
at org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals.runPipeline(CreateReadCountPanelOfNormals.java:276)
at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:31)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
at org.broadinstitute.hellbender.Main.main(Main.java:291)
19/07/01 20:05:05 INFO ShutdownHookManager: Shutdown hook called
19/07/01 20:05:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-1b686777-b88f-4b62-bcd6-dd79490c9359

Answers

  • sleeslee Member, Broadie, Dev ✭✭✭

    @ahda It looks like you might be incorrectly passing the output of PreprocessIntervals to the --annotated-intervals argument; if so, you should instead pass the output of AnnotatedIntervals.

  • ahdaahda ChinaMember

    @slee Thank you for the reply. this problem is because: I did CollectReadCounts and CreatePoN by the output of FilterIntervals. now I redo the CollectReadCounts by specified interval file from output of PreprocessIntervals, and CreateReadCountPanelOfNormals by the output of AnnotatedIntervals. then everthing seems fine.
    Then I just want to know why I can not use the output of FilterIntervals in step of CollectReadCounts (I tried but some errors happened, I‘m sorry I forget to save this error message,which is noised as that, the hdf5 regions are not matched with AnnotatedIntervals)

  • sleeslee Member, Broadie, Dev ✭✭✭

    @ahda FilterIntervals is intended primarily for use in the germline CNV workflow, since CreateReadCountPanelOfNormals in the somatic CNV workflow already does some filtering steps. However, if you'd like to use intervals produced by FilterIntervals to re-collect coverage, you should be sure to re-annotate the filtered intervals with AnnotateIntervals before passing everything to CreateReadCountPanelOfNormals. The key is to make sure that all intervals (both in read count files and the annotated intervals) match at that step.

  • ahdaahda ChinaMember

    @slee Thank you very much for the quickly reply. Now I am all clear.

Sign In or Register to comment.