Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

GermlineCNVCaller --interval-merging-rule error.

SkyWarriorSkyWarrior TurkeyMember ✭✭✭

Hi
I was testing the brand new GermlineCNVCaller in 4.0.3.0 however I met a very strange error.

All my read collections was made with the following command

gatk CollectReadCounts -R $HG19FULL --interval-merging-rule OVERLAPPING_ONLY -L $TSOREG -I samplename_final.bam -O samplename_counts.hdf5

And there was no problem with DetermineGermlineContigPloidy step. All files were generated using GATK 4.0.3.0 docker image fresh from docker repo.

DetermineGermlineContigPloidy command was according to the doc files within the gatk folder. I have 32 samples and I was working in COHORT mode.

Here is the error message

Where is the problem here?

Issue · Github
by Sheila

Issue Number
3035
State
closed
Last Updated
Assignee
Array
Closed By
chandrans

Best Answers

  • SkyWarriorSkyWarrior Turkey ✭✭✭
    edited April 2018 Accepted Answer

    Yes finally I was able to get this thing to work and I have good and bad news. But more of good news and I am hopefull for the future of this tool in general.

    I am mainly interested in recalling regions that were already called by XHMM and confirmed by MLPA or similar methods.

    I got this tool to work on a 32 Trusight One Clinical Exome panels sequenced on average 60-80x on NextSeq 500. This panel consists of 4800 genes and approximately 63000 non-overlapping intervals.

    Out of this 32 sample cohort I had 2 samples that had a long deletion in different spots.
    Sample 780 had a multi exon deletion in COL7A1 gene with a EBD phenotype and Sample 784 had a multi exon deletion in DMD gene with muscular dystrophy phenotype.

    Sample 780

    Intervals were exactly the same as XHMM predicted and we confirmed this sample with PCR due to evident breakpoints on the bam file. We were lucky about those breakpoints since almost the entire family was tested for carrier status with a very cheap experiment. Every carrier had the same deletion and every affected individual had the same homozygous deletion. Interesting case indeed.

    Sample 784

    Intervals were exactly the same as XHMM predicted and this sample was confirmed with MLPA analysis.

    Good news.

    • We now have another robust tool to detect copy number variations especially from WES data.
    • Tools prediction levels seem to be similar to my long time favorite XHMM for confirmed samples. I will be doing more testing on some of my known duplication samples. I sense that there needs to be more data to train this tool to eliminate some of the false positives or maybe a post processing option to eliminate intervals known to have CNV in human population. (I filter some of XHMM predictions based on that. I will try to filter GATK output and make a more fair comparison)

    Bad news

    • Tool still takes to much time to analyze cohort data. 32 Trusight samples in XHMM takes about 5 to 10 minutes of execution time with fairly decent call rate. (Though XHMM needs some data mining afterwards for CN status especially for 1 and 3 or more copies. ) GATK takes 450 minutes to process 32 Trusight samples. (XHMM can process 150 WES samples in that time period.)

    Overall
    I am happy that GATK CNV works as expected even in the beta status. I will be posting more results sooner. Now back to my dissertation :)

  • SkyWarriorSkyWarrior Turkey ✭✭✭
    edited February 4 Accepted Answer

    @AdelaideR said:
    @SkyWarrior - Could you please post an update hear once you try?

    Hi

    Here are the updates from my latest trial.

    • Good news is that I was able to capture the event that was missed by many others just by using default parameters for all the rest except for the -imr OVERLAPPING_ONLY. Now I need to optimize other parameters especially to reduce the inference time for GermlineCNVCalls step. Training epochs reach 38 cycles if I leave the max number of epochs at 50. I am wondering if 20 would suffice???

    • Bad news is I am receiving a Not implemented error for the MT contig that was included in the bed file since it was my default bed file to use. Using -XL MT does not stop the algorithm to use MT and generate the segmented vcf.

    Below is the DEBUG output for PostprocessGermlineCNVCalls.

    (gatk) [[email protected] GATKCNVtesting_V2]$ gatk PostprocessGermlineCNVCalls --calls-shard-path CNVCalls/GATK_CNV-calls --model-shard-path CNVCalls/GATK_CNV-model --sample-index 22 --autosomal-ref-copy-number 2 --allosomal-contig X --allosomal-contig Y --output-genotyped-intervals Wes492_intervals.vcf --output-genotyped-segments Wes492_segments.vcf --contig-ploidy-calls CNV/GATKCNVtesting_V2_CNV-calls -imr OVERLAPPING_ONLY -XL MT -R $B37 -verbosity DEBUG
    Using GATK jar /home/exome/scripts/gatk-package-4.1.0.0-local.jar
    Running:
        java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/exome/scripts/gatk-package-4.1.0.0-local.jar PostprocessGermlineCNVCalls --calls-shard-path CNVCalls/GATK_CNV-calls --model-shard-path CNVCalls/GATK_CNV-model --sample-index 22 --autosomal-ref-copy-number 2 --allosomal-contig X --allosomal-contig Y --output-genotyped-intervals Wes492_intervals.vcf --output-genotyped-segments Wes492_segments.vcf --contig-ploidy-calls CNV/GATKCNVtesting_V2_CNV-calls -imr OVERLAPPING_ONLY -XL MT -R /home/exome/GenomeData/B37/hs37d5.fa -verbosity DEBUG
    09:45:21.030 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/exome/scripts/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    09:45:21.040 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /tmp/libgkl_compression2296465957449338279.so
    09:45:22.712 INFO  PostprocessGermlineCNVCalls - ------------------------------------------------------------
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - The Genome Analysis Toolkit (GATK) v4.1.0.0
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - For support and documentation go to https://software.broadinstitute.org/gatk/
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - Executing as [email protected] on Linux v4.19.16-1-MANJARO amd64
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b26
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - Start Date/Time: February 4, 2019 9:45:21 AM EET
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - ------------------------------------------------------------
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - ------------------------------------------------------------
    09:45:22.714 INFO  PostprocessGermlineCNVCalls - HTSJDK Version: 2.18.2
    09:45:22.714 INFO  PostprocessGermlineCNVCalls - Picard Version: 2.18.25
    09:45:22.715 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.BUFFER_SIZE : 131072
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.CREATE_INDEX : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.CREATE_MD5 : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.CUSTOM_READER_FACTORY :
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.REFERENCE_FASTA : null
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false
    09:45:22.716 DEBUG ConfigFactory - Configuration file values:
    09:45:22.720 DEBUG ConfigFactory -      gcsMaxRetries = 20
    09:45:22.721 DEBUG ConfigFactory -      gcsProjectForRequesterPays =
    09:45:22.721 DEBUG ConfigFactory -      gatk_stacktrace_on_user_exception = false
    09:45:22.721 DEBUG ConfigFactory -      samjdk.use_async_io_read_samtools = false
    09:45:22.721 DEBUG ConfigFactory -      samjdk.use_async_io_write_samtools = true
    09:45:22.721 DEBUG ConfigFactory -      samjdk.use_async_io_write_tribble = false
    09:45:22.721 DEBUG ConfigFactory -      samjdk.compression_level = 2
    09:45:22.721 DEBUG ConfigFactory -      spark.kryoserializer.buffer.max = 512m
    09:45:22.721 DEBUG ConfigFactory -      spark.driver.maxResultSize = 0
    09:45:22.721 DEBUG ConfigFactory -      spark.driver.userClassPathFirst = true
    09:45:22.721 DEBUG ConfigFactory -      spark.io.compression.codec = lzf
    09:45:22.721 DEBUG ConfigFactory -      spark.yarn.executor.memoryOverhead = 600
    09:45:22.721 DEBUG ConfigFactory -      spark.driver.extraJavaOptions =
    09:45:22.722 DEBUG ConfigFactory -      codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]
    09:45:22.722 DEBUG ConfigFactory -      spark.executor.extraJavaOptions =
    09:45:22.722 DEBUG ConfigFactory -      read_filter_packages = [org.broadinstitute.hellbender.engine.filters]
    09:45:22.722 DEBUG ConfigFactory -      annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]
    09:45:22.722 DEBUG ConfigFactory -      cloudPrefetchBuffer = 40
    09:45:22.722 DEBUG ConfigFactory -      cloudIndexPrefetchBuffer = -1
    09:45:22.722 DEBUG ConfigFactory -      createOutputBamIndex = true
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Deflater: IntelDeflater
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Inflater: IntelInflater
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - GCS max retries/reopens: 20
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Requester pays: disabled
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Initializing engine
    09:45:22.878 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
    09:45:22.878 DEBUG GenomeLocParser -  1 (249250621 bp)
    09:45:22.878 DEBUG GenomeLocParser -  2 (243199373 bp)
    09:45:22.878 DEBUG GenomeLocParser -  3 (198022430 bp)
    09:45:22.878 DEBUG GenomeLocParser -  4 (191154276 bp)
    09:45:22.878 DEBUG GenomeLocParser -  5 (180915260 bp)
    09:45:22.878 DEBUG GenomeLocParser -  6 (171115067 bp)
    09:45:22.878 DEBUG GenomeLocParser -  7 (159138663 bp)
    09:45:22.878 DEBUG GenomeLocParser -  8 (146364022 bp)
    09:45:22.878 DEBUG GenomeLocParser -  9 (141213431 bp)
    09:45:22.879 DEBUG GenomeLocParser -  10 (135534747 bp)
    09:45:22.879 DEBUG GenomeLocParser -  11 (135006516 bp)
    09:45:22.879 DEBUG GenomeLocParser -  12 (133851895 bp)
    09:45:22.879 DEBUG GenomeLocParser -  13 (115169878 bp)
    09:45:22.879 DEBUG GenomeLocParser -  14 (107349540 bp)
    09:45:22.879 DEBUG GenomeLocParser -  15 (102531392 bp)
    09:45:22.879 DEBUG GenomeLocParser -  16 (90354753 bp)
    09:45:22.879 DEBUG GenomeLocParser -  17 (81195210 bp)
    09:45:22.879 DEBUG GenomeLocParser -  18 (78077248 bp)
    09:45:22.879 DEBUG GenomeLocParser -  19 (59128983 bp)
    09:45:22.879 DEBUG GenomeLocParser -  20 (63025520 bp)
    09:45:22.879 DEBUG GenomeLocParser -  21 (48129895 bp)
    09:45:22.879 DEBUG GenomeLocParser -  22 (51304566 bp)
    09:45:22.879 DEBUG GenomeLocParser -  X (155270560 bp)
    09:45:22.879 DEBUG GenomeLocParser -  Y (59373566 bp)
    09:45:22.879 DEBUG GenomeLocParser -  MT (16569 bp)
    09:45:22.879 DEBUG GenomeLocParser -  GL000207.1 (4262 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000226.1 (15008 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000229.1 (19913 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000231.1 (27386 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000210.1 (27682 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000239.1 (33824 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000235.1 (34474 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000201.1 (36148 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000247.1 (36422 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000245.1 (36651 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000197.1 (37175 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000203.1 (37498 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000246.1 (38154 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000249.1 (38502 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000196.1 (38914 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000248.1 (39786 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000244.1 (39929 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000238.1 (39939 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000202.1 (40103 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000234.1 (40531 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000232.1 (40652 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000206.1 (41001 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000240.1 (41933 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000236.1 (41934 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000241.1 (42152 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000243.1 (43341 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000242.1 (43523 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000230.1 (43691 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000237.1 (45867 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000233.1 (45941 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000204.1 (81310 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000198.1 (90085 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000208.1 (92689 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000191.1 (106433 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000227.1 (128374 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000228.1 (129120 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000214.1 (137718 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000221.1 (155397 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000209.1 (159169 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000218.1 (161147 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000220.1 (161802 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000213.1 (164239 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000211.1 (166566 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000199.1 (169874 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000217.1 (172149 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000216.1 (172294 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000215.1 (172545 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000205.1 (174588 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000219.1 (179198 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000224.1 (179693 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000223.1 (180455 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000195.1 (182896 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000212.1 (186858 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000222.1 (186861 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000200.1 (187035 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000193.1 (189789 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000194.1 (191469 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000225.1 (211173 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000192.1 (547496 bp)
    09:45:22.883 DEBUG GenomeLocParser -  NC_007605 (171823 bp)
    09:45:22.883 DEBUG GenomeLocParser -  hs37d5 (35477943 bp)
    09:45:22.883 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
    09:45:22.883 DEBUG GenomeLocParser -  1 (249250621 bp)
    09:45:22.883 DEBUG GenomeLocParser -  2 (243199373 bp)
    09:45:22.883 DEBUG GenomeLocParser -  3 (198022430 bp)
    09:45:22.883 DEBUG GenomeLocParser -  4 (191154276 bp)
    09:45:22.883 DEBUG GenomeLocParser -  5 (180915260 bp)
    09:45:22.883 DEBUG GenomeLocParser -  6 (171115067 bp)
    09:45:22.883 DEBUG GenomeLocParser -  7 (159138663 bp)
    09:45:22.883 DEBUG GenomeLocParser -  8 (146364022 bp)
    09:45:22.883 DEBUG GenomeLocParser -  9 (141213431 bp)
    09:45:22.884 DEBUG GenomeLocParser -  10 (135534747 bp)
    09:45:22.884 DEBUG GenomeLocParser -  11 (135006516 bp)
    09:45:22.884 DEBUG GenomeLocParser -  12 (133851895 bp)
    09:45:22.884 DEBUG GenomeLocParser -  13 (115169878 bp)
    09:45:22.884 DEBUG GenomeLocParser -  14 (107349540 bp)
    09:45:22.884 DEBUG GenomeLocParser -  15 (102531392 bp)
    09:45:22.884 DEBUG GenomeLocParser -  16 (90354753 bp)
    09:45:22.884 DEBUG GenomeLocParser -  17 (81195210 bp)
    09:45:22.884 DEBUG GenomeLocParser -  18 (78077248 bp)
    09:45:22.884 DEBUG GenomeLocParser -  19 (59128983 bp)
    09:45:22.884 DEBUG GenomeLocParser -  20 (63025520 bp)
    09:45:22.884 DEBUG GenomeLocParser -  21 (48129895 bp)
    09:45:22.884 DEBUG GenomeLocParser -  22 (51304566 bp)
    09:45:22.884 DEBUG GenomeLocParser -  X (155270560 bp)
    09:45:22.884 DEBUG GenomeLocParser -  Y (59373566 bp)
    09:45:22.884 DEBUG GenomeLocParser -  MT (16569 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000207.1 (4262 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000226.1 (15008 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000229.1 (19913 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000231.1 (27386 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000210.1 (27682 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000239.1 (33824 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000235.1 (34474 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000201.1 (36148 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000247.1 (36422 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000245.1 (36651 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000197.1 (37175 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000203.1 (37498 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000246.1 (38154 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000249.1 (38502 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000196.1 (38914 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000248.1 (39786 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000244.1 (39929 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000238.1 (39939 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000202.1 (40103 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000234.1 (40531 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000232.1 (40652 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000206.1 (41001 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000240.1 (41933 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000236.1 (41934 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000241.1 (42152 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000243.1 (43341 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000242.1 (43523 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000230.1 (43691 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000237.1 (45867 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000233.1 (45941 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000204.1 (81310 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000198.1 (90085 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000208.1 (92689 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000191.1 (106433 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000227.1 (128374 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000228.1 (129120 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000214.1 (137718 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000221.1 (155397 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000209.1 (159169 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000218.1 (161147 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000220.1 (161802 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000213.1 (164239 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000211.1 (166566 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000199.1 (169874 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000217.1 (172149 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000216.1 (172294 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000215.1 (172545 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000205.1 (174588 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000219.1 (179198 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000224.1 (179693 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000223.1 (180455 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000195.1 (182896 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000212.1 (186858 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000222.1 (186861 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000200.1 (187035 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000193.1 (189789 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000194.1 (191469 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000225.1 (211173 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000192.1 (547496 bp)
    09:45:22.886 DEBUG GenomeLocParser -  NC_007605 (171823 bp)
    09:45:22.886 DEBUG GenomeLocParser -  hs37d5 (35477943 bp)
    09:45:22.891 DEBUG GenomeLocSortedSet - removeRegions operation: i = 1
    09:45:22.891 INFO  IntervalArgumentCollection - Initial include intervals span 3137454505 loci; exclude intervals span 16569 loci
    09:45:22.892 INFO  IntervalArgumentCollection - Excluding 16569 loci from original intervals (0.00% reduction)
    09:45:22.892 INFO  IntervalArgumentCollection - Processing 3137437936 bp from intervals
    09:45:22.894 DEBUG ScriptExecutor - Executing:
    09:45:22.894 DEBUG ScriptExecutor -   python
    09:45:22.894 DEBUG ScriptExecutor -   -c
    09:45:22.894 DEBUG ScriptExecutor -   import gcnvkernel
    
    09:45:24.795 DEBUG ScriptExecutor - Result: 0
    09:45:24.795 INFO  PostprocessGermlineCNVCalls - Done initializing engine
    09:45:25.464 INFO  ProgressMeter - Starting traversal
    09:45:25.464 INFO  ProgressMeter -        Current Locus  Elapsed Minutes     Records Processed   Records/Minute
    09:45:25.464 INFO  PostprocessGermlineCNVCalls - Generating intervals VCF file...
    09:45:25.477 INFO  PostprocessGermlineCNVCalls - Analyzing shard 0...
    09:45:28.253 INFO  PostprocessGermlineCNVCalls - Generating segments VCF file...
    09:45:28.256 DEBUG ScriptExecutor - Executing:
    09:45:28.256 DEBUG ScriptExecutor -   python
    09:45:28.256 DEBUG ScriptExecutor -   /tmp/segment_gcnv_calls.4286460784218017052.py
    09:45:28.256 DEBUG ScriptExecutor -   --ploidy_calls_path
    09:45:28.256 DEBUG ScriptExecutor -   /mnt/E/Exomes/GATKCNVtesting_V2/CNV/GATKCNVtesting_V2_CNV-calls
    09:45:28.256 DEBUG ScriptExecutor -   --model_shards
    09:45:28.256 DEBUG ScriptExecutor -   /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-model
    09:45:28.256 DEBUG ScriptExecutor -   --calls_shards
    09:45:28.256 DEBUG ScriptExecutor -   /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-calls
    09:45:28.256 DEBUG ScriptExecutor -   --output_path
    09:45:28.256 DEBUG ScriptExecutor -   /tmp/gcnv-segmented-calls2678318989462462101
    09:45:28.256 DEBUG ScriptExecutor -   --sample_index
    09:45:28.256 DEBUG ScriptExecutor -   22
    09:45:29.991 INFO segment_gcnv_calls - Loading ploidy calls...
    09:45:29.991 INFO gcnvkernel.io.io_metadata - Loading germline contig ploidy and global read depth metadata...
    09:45:29.994 WARNING gcnvkernel.structs.metadata - Sample Wes491 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:29.996 WARNING gcnvkernel.structs.metadata - Sample Wes494 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:29.998 WARNING gcnvkernel.structs.metadata - Sample Wes500 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.000 WARNING gcnvkernel.structs.metadata - Sample Wes492 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.001 WARNING gcnvkernel.structs.metadata - Sample Wes484 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.003 WARNING gcnvkernel.structs.metadata - Sample Wes497 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.005 WARNING gcnvkernel.structs.metadata - Sample Wes478 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.007 WARNING gcnvkernel.structs.metadata - Sample Wes473 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.009 WARNING gcnvkernel.structs.metadata - Sample Wes495 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.011 WARNING gcnvkernel.structs.metadata - Sample Wes479 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.012 WARNING gcnvkernel.structs.metadata - Sample Wes498 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.014 WARNING gcnvkernel.structs.metadata - Sample Wes493 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.016 WARNING gcnvkernel.structs.metadata - Sample Wes499 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.018 WARNING gcnvkernel.structs.metadata - Sample Wes482 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.019 WARNING gcnvkernel.structs.metadata - Sample Wes476 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.021 WARNING gcnvkernel.structs.metadata - Sample Wes470 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.023 WARNING gcnvkernel.structs.metadata - Sample Wes496 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.025 WARNING gcnvkernel.structs.metadata - Sample Wes472 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.026 WARNING gcnvkernel.structs.metadata - Sample Wes486 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.028 WARNING gcnvkernel.structs.metadata - Sample Wes481 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.030 WARNING gcnvkernel.structs.metadata - Sample Wes480 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.032 WARNING gcnvkernel.structs.metadata - Sample Wes483 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.033 WARNING gcnvkernel.structs.metadata - Sample Wes485 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.035 WARNING gcnvkernel.structs.metadata - Sample Wes471 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.037 WARNING gcnvkernel.structs.metadata - Sample Wes477 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.039 WARNING gcnvkernel.structs.metadata - Sample Wes475 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.041 WARNING gcnvkernel.structs.metadata - Sample Wes490 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.042 WARNING gcnvkernel.structs.metadata - Sample Wes487 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.044 WARNING gcnvkernel.structs.metadata - Sample Wes474 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.046 WARNING gcnvkernel.structs.metadata - Sample Wes488 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.048 WARNING gcnvkernel.structs.metadata - Sample Wes489 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.048 INFO segment_gcnv_calls - Instantiating the Viterbi segmentation engine...
    09:45:30.055 INFO gcnvkernel.postprocess.viterbi_segmentation - Assembling interval list and copy-number class posterior from model shards...
    09:45:30.852 INFO gcnvkernel.io.io_intervals_and_counts - The given interval list provides the following interval annotations: {'GC_CONTENT'}
    09:45:32.320 INFO gcnvkernel.structs.metadata - Generating intervals metadata...
    09:45:33.059 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano forward-backward function...
    09:45:34.373 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano Viterbi function...
    09:45:36.048 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano variational HHMM...
    09:45:36.210 INFO gcnvkernel.postprocess.viterbi_segmentation - Processing sample index: 22, sample name: Wes492...
    09:45:36.422 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (1/25) (contig name: 1)...
    09:45:37.757 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (2/25) (contig name: 2)...
    09:45:38.751 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (3/25) (contig name: 3)...
    09:45:39.517 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (4/25) (contig name: 4)...
    09:45:40.065 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (5/25) (contig name: 5)...
    09:45:40.669 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (6/25) (contig name: 6)...
    09:45:41.319 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (7/25) (contig name: 7)...
    09:45:41.989 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (8/25) (contig name: 8)...
    09:45:42.453 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (9/25) (contig name: 9)...
    09:45:42.987 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (10/25) (contig name: 10)...
    09:45:43.561 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (11/25) (contig name: 11)...
    09:45:44.252 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (12/25) (contig name: 12)...
    09:45:44.968 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (13/25) (contig name: 13)...
    09:45:45.236 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (14/25) (contig name: 14)...
    09:45:45.627 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (15/25) (contig name: 15)...
    09:45:46.156 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (16/25) (contig name: 16)...
    09:45:46.716 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (17/25) (contig name: 17)...
    09:45:47.424 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (18/25) (contig name: 18)...
    09:45:47.633 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (19/25) (contig name: 19)...
    09:45:48.308 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (20/25) (contig name: 20)...
    09:45:48.629 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (21/25) (contig name: 21)...
    09:45:48.777 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (22/25) (contig name: 22)...
    09:45:49.069 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (23/25) (contig name: X)...
    09:45:49.534 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (24/25) (contig name: Y)...
    09:45:49.593 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (25/25) (contig name: MT)...
    Traceback (most recent call last):
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 884, in __call__
        self.fn() if output_subset is None else\
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 989, in rval
        r = p(n, [x[0] for x in i], o)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 978, in p
        self, node)
      File "theano/scan_module/scan_perform.pyx", line 215, in theano.scan_module.scan_perform.perform (/home/exome/.theano/compiledir_Linux-4.19--MANJARO-x86_64-with-arch-Manjaro-Linux--3.6.2-64/scan_perform/mod.cpp:2628)
    NotImplementedError: We didn't implemented yet the case where scan do 0 iteration
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/tmp/segment_gcnv_calls.4286460784218017052.py", line 73, in <module>
        viterbi_engine.write_copy_number_segments()
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 234, in write_copy_number_segments
        for segment in self._viterbi_segments_generator():
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 160, in _viterbi_segments_generator
        log_prior_c, log_trans_contig_tcc, copy_number_log_emission_contig_tc)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/models/theano_hmm.py", line 88, in perform_forward_backward
        prev_log_posterior_tc, admixing_rate, temperature)))
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 898, in __call__
        storage_map=getattr(self.fn, 'storage_map', None))
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/link.py", line 325, in raise_with_op
        reraise(exc_type, exc_value, exc_trace)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/six.py", line 692, in reraise
        raise value.with_traceback(tb)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 884, in __call__
        self.fn() if output_subset is None else\
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 989, in rval
        r = p(n, [x[0] for x in i], o)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 978, in p
        self, node)
      File "theano/scan_module/scan_perform.pyx", line 215, in theano.scan_module.scan_perform.perform (/home/exome/.theano/compiledir_Linux-4.19--MANJARO-x86_64-with-arch-Manjaro-Linux--3.6.2-64/scan_perform/mod.cpp:2628)
    NotImplementedError: We didn't implemented yet the case where scan do 0 iteration
    Apply node that caused the error: forall_inplace,cpu,scan_fn}(Elemwise{minimum,no_inplace}.0, InplaceDimShuffle{0,2,1}.0, Subtensor{int64:int64:int64}.0, IncSubtensor{InplaceSet;:int64:}.0, Shape_i{0}.0)
    Toposort index: 97
    Inputs types: [TensorType(int64, scalar), TensorType(float64, 3D), TensorType(float64, matrix), TensorType(float64, matrix), TensorType(int64, scalar)]
    Inputs shapes: [(), (0, 6, 6), (0, 6), (2, 6), ()]
    Inputs strides: [(), (288, 8, 48), (48, 8), (48, 8), ()]
    Inputs values: [array(0), array([], shape=(0, 6, 6), dtype=float64), array([], shape=(0, 6), dtype=float64), 'not shown', array(6)]
    Outputs clients: [[Subtensor{int64:int64:int8}(forall_inplace,cpu,scan_fn}.0, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1})]]
    
    HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
    HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
    09:45:50.363 DEBUG ScriptExecutor - Result: 1
    09:45:50.364 INFO  PostprocessGermlineCNVCalls - Shutting down engine
    [February 4, 2019 9:45:50 AM EET] org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls done. Elapsed time: 0.49 minutes.
    Runtime.totalMemory()=3004170240
    org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
    python exited with 1
    Command Line: python /tmp/segment_gcnv_calls.4286460784218017052.py --ploidy_calls_path /mnt/E/Exomes/GATKCNVtesting_V2/CNV/GATKCNVtesting_V2_CNV-calls --model_shards /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-model --calls_shards /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-calls --output_path /tmp/gcnv-segmented-calls2678318989462462101 --sample_index 22
            at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
            at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
            at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
            at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:151)
            at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:121)
            at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.executeSegmentGermlineCNVCallsPythonScript(PostprocessGermlineCNVCalls.java:499)
            at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.generateSegmentsVCFFileFromAllShards(PostprocessGermlineCNVCalls.java:435)
            at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.traverse(PostprocessGermlineCNVCalls.java:296)
            at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:966)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
            at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
            at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
            at org.broadinstitute.hellbender.Main.main(Main.java:291)
    (gatk) [[email protected] GATKCNVtesting_V2]$
    

    Looks like a warning for omission of MT segments is needed in the documentation or There must be some other way to implement a function to skip segments that will require 0 iterations.

    For the time being I will modify my bed file for CNV calls to omit MT.

  • SkyWarriorSkyWarrior Turkey ✭✭✭
    Accepted Answer

    Thanks @slee for the detailed response.

    My default bed file for analysis includes the MT contig as a whole and I used the very same bed file inadvertently during CollectReadCounts step. After that I realized the issue and tried excluding MT in DetermineContigPloidy Step using -XL MT but it did not work. I let the DetermineContigPloidy run with MT included. GermlineCNVCaller also ran without issues except for the warning for MT but PostProcess step generated this error and refused to output segmented VCF. Intervals vcf was generated without any issues.

    For the output folder creation. GermlineCNVCaller tries to write all the files in the folder with the prefix given. Although the root folder for the prefix named folder is present in the path and write permissions are all set GermlineCNVCaller throws an error message indicating that the prefix named folder is not available. When that folder is generated manually before running GermlineCNVCaller that error is gone and run completes properly.

  • SkyWarriorSkyWarrior Turkey ✭✭✭
    Accepted Answer

    If my memory serves me well it was the same message. I can reproduce it very quickly tomorrow and return here.

    The PostProcess error was related to an unimplemented case where iteration number equals 0.

  • SkyWarriorSkyWarrior Turkey ✭✭✭
    edited February 16 Accepted Answer

    Hi

    Sorry that the error message was not the same actually.

    Here is the command line

    gatk GermlineCNVCaller --java-options "-Djava.io.tmpdir=tmp" --run-mode COHORT --contig-ploidy-calls ./CNV/${PWD##*/}_CNV-calls --output CNVCalls --output-prefix GATK_CNV --annotated-intervals b37_annotated.txt --verbosity DEBUG -L $CNVREGB37 -imr OVERLAPPING_ONLY -I Wes470_counts.hdf5 -I Wes471_counts.hdf5 -I Wes472_counts.hdf5 -I Wes473_counts.hdf5 -I Wes474_counts.hdf5 -I Wes475_counts.hdf5 -I Wes476_counts.hdf5 -I Wes477_counts.hdf5 -I Wes478_counts.hdf5 -I Wes479_counts.hdf5 -I Wes480_counts.hdf5 -I Wes481_counts.hdf5 -I Wes482_counts.hdf5 -I Wes483_counts.hdf5 -I Wes484_counts.hdf5 -I Wes485_counts.hdf5 -I Wes486_counts.hdf5 -I Wes487_counts.hdf5 -I Wes488_counts.hdf5 -I Wes489_counts.hdf5 -I Wes490_counts.hdf5 -I Wes491_counts.hdf5 -I Wes492_counts.hdf5 -I Wes493_counts.hdf5 -I Wes494_counts.hdf5 -I Wes495_counts.hdf5 -I Wes496_counts.hdf5 -I Wes497_counts.hdf5 -I Wes498_counts.hdf5 -I Wes499_counts.hdf5 -I Wes500_counts.hdf5
    

    I expect the output should be in the directory named CNVCalls but when directory is not created ahead of this command I get the error message below.

    08:19:04.104 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/exome/scripts/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    08:19:04.379 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /mnt/E/Exomes/GATKCNVtesting_V2/tmp/libgkl_compression1008933078259947848.so
    08:19:06.094 INFO  GermlineCNVCaller - ------------------------------------------------------------
    08:19:06.095 INFO  GermlineCNVCaller - The Genome Analysis Toolkit (GATK) v4.1.0.0
    08:19:06.095 INFO  GermlineCNVCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
    08:19:06.095 INFO  GermlineCNVCaller - Executing as [email protected] on Linux v4.19.20-1-MANJARO amd64
    08:19:06.095 INFO  GermlineCNVCaller - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_202-b26
    08:19:06.095 INFO  GermlineCNVCaller - Start Date/Time: February 16, 2019 8:19:04 AM EET
    08:19:06.095 INFO  GermlineCNVCaller - ------------------------------------------------------------
    08:19:06.095 INFO  GermlineCNVCaller - ------------------------------------------------------------
    08:19:06.096 INFO  GermlineCNVCaller - HTSJDK Version: 2.18.2
    08:19:06.096 INFO  GermlineCNVCaller - Picard Version: 2.18.25
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.BUFFER_SIZE : 131072
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.CREATE_INDEX : false
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.CREATE_MD5 : false
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.CUSTOM_READER_FACTORY :
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.REFERENCE_FASTA : null
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false
    08:19:06.099 DEBUG ConfigFactory - Configuration file values:
    08:19:06.104 DEBUG ConfigFactory -      gcsMaxRetries = 20
    08:19:06.104 DEBUG ConfigFactory -      gcsProjectForRequesterPays =
    08:19:06.104 DEBUG ConfigFactory -      gatk_stacktrace_on_user_exception = false
    08:19:06.104 DEBUG ConfigFactory -      samjdk.use_async_io_read_samtools = false
    08:19:06.104 DEBUG ConfigFactory -      samjdk.use_async_io_write_samtools = true
    08:19:06.104 DEBUG ConfigFactory -      samjdk.use_async_io_write_tribble = false
    08:19:06.104 DEBUG ConfigFactory -      samjdk.compression_level = 2
    08:19:06.104 DEBUG ConfigFactory -      spark.kryoserializer.buffer.max = 512m
    08:19:06.104 DEBUG ConfigFactory -      spark.driver.maxResultSize = 0
    08:19:06.104 DEBUG ConfigFactory -      spark.driver.userClassPathFirst = true
    08:19:06.104 DEBUG ConfigFactory -      spark.io.compression.codec = lzf
    08:19:06.104 DEBUG ConfigFactory -      spark.yarn.executor.memoryOverhead = 600
    08:19:06.104 DEBUG ConfigFactory -      spark.driver.extraJavaOptions =
    08:19:06.104 DEBUG ConfigFactory -      codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]
    08:19:06.104 DEBUG ConfigFactory -      spark.executor.extraJavaOptions =
    08:19:06.104 DEBUG ConfigFactory -      read_filter_packages = [org.broadinstitute.hellbender.engine.filters]
    08:19:06.104 DEBUG ConfigFactory -      annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]
    08:19:06.104 DEBUG ConfigFactory -      cloudPrefetchBuffer = 40
    08:19:06.105 DEBUG ConfigFactory -      cloudIndexPrefetchBuffer = -1
    08:19:06.105 DEBUG ConfigFactory -      createOutputBamIndex = true
    08:19:06.105 INFO  GermlineCNVCaller - Deflater: IntelDeflater
    08:19:06.105 INFO  GermlineCNVCaller - Inflater: IntelInflater
    08:19:06.105 INFO  GermlineCNVCaller - GCS max retries/reopens: 20
    08:19:06.105 INFO  GermlineCNVCaller - Requester pays: disabled
    08:19:06.105 INFO  GermlineCNVCaller - Initializing engine
    08:19:06.110 DEBUG ScriptExecutor - Executing:
    08:19:06.110 DEBUG ScriptExecutor -   python
    08:19:06.111 DEBUG ScriptExecutor -   -c
    08:19:06.111 DEBUG ScriptExecutor -   import gcnvkernel
    
    08:19:08.932 DEBUG ScriptExecutor - Result: 0
    08:19:08.932 INFO  GermlineCNVCaller - Done initializing engine
    log4j:WARN No appenders could be found for logger (org.broadinstitute.hdf5.HDF5Library).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    08:19:09.470 INFO  GermlineCNVCaller - Intervals specified...
    08:19:09.471 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
    08:19:09.471 DEBUG GenomeLocParser -  1 (249250621 bp)
    08:19:09.471 DEBUG GenomeLocParser -  2 (243199373 bp)
    08:19:09.471 DEBUG GenomeLocParser -  3 (198022430 bp)
    08:19:09.471 DEBUG GenomeLocParser -  4 (191154276 bp)
    08:19:09.471 DEBUG GenomeLocParser -  5 (180915260 bp)
    08:19:09.471 DEBUG GenomeLocParser -  6 (171115067 bp)
    08:19:09.471 DEBUG GenomeLocParser -  7 (159138663 bp)
    08:19:09.471 DEBUG GenomeLocParser -  8 (146364022 bp)
    08:19:09.471 DEBUG GenomeLocParser -  9 (141213431 bp)
    08:19:09.471 DEBUG GenomeLocParser -  10 (135534747 bp)
    08:19:09.471 DEBUG GenomeLocParser -  11 (135006516 bp)
    08:19:09.472 DEBUG GenomeLocParser -  12 (133851895 bp)
    08:19:09.472 DEBUG GenomeLocParser -  13 (115169878 bp)
    08:19:09.472 DEBUG GenomeLocParser -  14 (107349540 bp)
    08:19:09.472 DEBUG GenomeLocParser -  15 (102531392 bp)
    08:19:09.472 DEBUG GenomeLocParser -  16 (90354753 bp)
    08:19:09.472 DEBUG GenomeLocParser -  17 (81195210 bp)
    08:19:09.472 DEBUG GenomeLocParser -  18 (78077248 bp)
    08:19:09.472 DEBUG GenomeLocParser -  19 (59128983 bp)
    08:19:09.472 DEBUG GenomeLocParser -  20 (63025520 bp)
    08:19:09.472 DEBUG GenomeLocParser -  21 (48129895 bp)
    08:19:09.472 DEBUG GenomeLocParser -  22 (51304566 bp)
    08:19:09.472 DEBUG GenomeLocParser -  X (155270560 bp)
    08:19:09.472 DEBUG GenomeLocParser -  Y (59373566 bp)
    08:19:09.472 DEBUG GenomeLocParser -  MT (16569 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000207.1 (4262 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000226.1 (15008 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000229.1 (19913 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000231.1 (27386 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000210.1 (27682 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000239.1 (33824 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000235.1 (34474 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000201.1 (36148 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000247.1 (36422 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000245.1 (36651 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000197.1 (37175 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000203.1 (37498 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000246.1 (38154 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000249.1 (38502 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000196.1 (38914 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000248.1 (39786 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000244.1 (39929 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000238.1 (39939 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000202.1 (40103 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000234.1 (40531 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000232.1 (40652 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000206.1 (41001 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000240.1 (41933 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000236.1 (41934 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000241.1 (42152 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000243.1 (43341 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000242.1 (43523 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000230.1 (43691 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000237.1 (45867 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000233.1 (45941 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000204.1 (81310 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000198.1 (90085 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000208.1 (92689 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000191.1 (106433 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000227.1 (128374 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000228.1 (129120 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000214.1 (137718 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000221.1 (155397 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000209.1 (159169 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000218.1 (161147 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000220.1 (161802 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000213.1 (164239 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000211.1 (166566 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000199.1 (169874 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000217.1 (172149 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000216.1 (172294 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000215.1 (172545 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000205.1 (174588 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000219.1 (179198 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000224.1 (179693 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000223.1 (180455 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000195.1 (182896 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000212.1 (186858 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000222.1 (186861 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000200.1 (187035 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000193.1 (189789 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000194.1 (191469 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000225.1 (211173 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000192.1 (547496 bp)
    08:19:09.474 DEBUG GenomeLocParser -  NC_007605 (171823 bp)
    08:19:09.474 DEBUG GenomeLocParser -  hs37d5 (35477943 bp)
    08:19:09.609 INFO  FeatureManager - Using codec BEDCodec to read file file:///home/exome/GenomeData/ExomeReg_b37_CNV.bed
    08:19:10.008 DEBUG FeatureDataSource - Cache statistics for FeatureInput /home/exome/GenomeData/ExomeReg_b37_CNV.bed:/home/exome/GenomeData/ExomeReg_b37_CNV.bed:
    08:19:10.009 DEBUG FeatureCache - Cache hit rate  was 0.00% (0 out of 0 total queries)
    08:19:10.097 INFO  IntervalArgumentCollection - Processing 103604700 bp from intervals
    08:19:10.223 INFO  GermlineCNVCaller - Reading and validating annotated intervals...
    08:19:11.051 INFO  GermlineCNVCaller - GC-content annotations for intervals found; explicit GC-bias correction will be performed...
    08:19:11.418 INFO  GermlineCNVCaller - Running the tool in the COHORT mode...
    08:19:11.419 INFO  GermlineCNVCaller - Shutting down engine
    [February 16, 2019 8:19:11 AM EET] org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller done. Elapsed time: 0.12 minutes.
    Runtime.totalMemory()=2352480256
    java.lang.IllegalArgumentException: Output directory CNVCalls does not exist.
            at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:724)
            at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.validateArguments(GermlineCNVCaller.java:361)
            at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.doWork(GermlineCNVCaller.java:281)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
            at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
            at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
            at org.broadinstitute.hellbender.Main.main(Main.java:291)
    

Answers

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited March 2018

    Update: I might have fixed this issue myself. The docs does not mention about using
    --interval-merging-rule OVERLAPPING_ONLY for GermlineCNVCaller in the example. Looks like it has to be used. Also output folder should be created before running the tool otherwise tool throws a folder not found error and exits. This does not happen with DetermineGermlineContigPloidy tool. I am waiting for the results now. I will be comparing them against XHMM :smile:

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited March 2018

    More problems incoming.

    I am not sure what I am seeing here but it may be a incompatibility between GATK java runtime and python code execution.

    my contig ploidy calls from DetermineGermlineContigPloidy tool is under folder X. Calls and model are under folder X/prefix-calls and X/prefix-model.

    When GermlineCNVCaller is run with parameter

    --contig-ploidy-calls X/prefix-calls

    execution stalls at aggregating readcount data from the cohort.

    When GermlineCNVCaller is run with parameter

    --contig-ploidy-calls X

    execution ends on the java part. Aggregation completes but when python script executes I get an error

    org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
    python exited with 1
    Command Line: python /tmp/root/cohort_denoising_calling.1528767346035786763.py --ploidy_calls_path=denemegatkcnv --output_calls_path=outputCNVGATKdeneme/Tso767-798-calls --modeling_interval_list=/tmp/root/intervals750549210195499360.tsv --output_model_path=outputCNVGATKdeneme/Tso767-798-model --enable_explicit_gc_bias_modeling=False --read_count_tsv_files /tmp/root/sample-01940138986000960332.tsv /tmp/root/sample-11184885287348925315.tsv /tmp/root/sample-22341592388899717607.tsv /tmp/root/sample-38273872274678620239.tsv /tmp/root/sample-45759464707102858063.tsv /tmp/root/sample-51283071696536011758.tsv /tmp/root/sample-62966553904181704103.tsv /tmp/root/sample-7559559318400303548.tsv /tmp/root/sample-86683319526280676870.tsv /tmp/root/sample-91668779139838160127.tsv /tmp/root/sample-104482778452187069957.tsv /tmp/root/sample-113597173648161702713.tsv /tmp/root/sample-127546794789954842464.tsv /tmp/root/sample-137642142007797929063.tsv /tmp/root/sample-143527057916965428805.tsv /tmp/root/sample-158469706267472743349.tsv /tmp/root/sample-16401684942404822092.tsv /tmp/root/sample-171770141381599819674.tsv /tmp/root/sample-186606637729027906937.tsv /tmp/root/sample-1943161770256717040.tsv /tmp/root/sample-208797709777816010529.tsv /tmp/root/sample-215368508338211273189.tsv /tmp/root/sample-226662812717299724795.tsv /tmp/root/sample-237635055203790381113.tsv /tmp/root/sample-247924211662815535266.tsv /tmp/root/sample-251148770430045512728.tsv /tmp/root/sample-267961662072929101076.tsv /tmp/root/sample-276550572036705660903.tsv /tmp/root/sample-281911595193353318388.tsv /tmp/root/sample-292923691606900091159.tsv /tmp/root/sample-307220651188459829117.tsv /tmp/root/sample-317045364463317205775.tsv --psi_s_scale=1.000000e-04 --mapping_error_rate=1.000000e-02 --depth_correction_tau=1.000000e+04 --q_c_expectation_mode=hybrid --max_bias_factors=5 --psi_t_scale=1.000000e-03 --log_mean_bias_std=1.000000e-01 --init_ard_rel_unexplained_variance=1.000000e-01 --num_gc_bins=20 --gc_curve_sd=1.000000e+00 --active_class_padding_hybrid_mode=50000 --enable_bias_factors=True --disable_bias_factors_in_active_class=False --p_alt=1.000000e-06 --cnv_coherence_length=1.000000e+04 --max_copy_number=5 --p_active=0.010000 --class_coherence_length=10000.000000 --learning_rate=5.000000e-02 --adamax_beta1=9.000000e-01 --adamax_beta2=9.900000e-01 --log_emission_samples_per_round=50 --log_emission_sampling_rounds=10 --log_emission_sampling_median_rel_error=5.000000e-03 --max_advi_iter_first_epoch=100 --max_advi_iter_subsequent_epochs=100 --min_training_epochs=10 --max_training_epochs=50 --initial_temperature=2.000000e+00 --num_thermal_epochs=20 --convergence_snr_averaging_window=500 --convergence_snr_trigger_threshold=1.000000e-01 --convergence_snr_countdown_window=10 --max_calling_iters=10 --caller_update_convergence_threshold=1.000000e-03 --caller_admixing_rate=7.500000e-01 --disable_caller=false --disable_sampler=false --disable_annealing=false
    Stdout: 08:49:01.446 INFO cohort_denoising_calling - Loading 32 read counts file(s)...
    08:49:06.968 INFO gcnvkernel.io.io_metadata - Loading germline contig ploidy and global read depth metadata...
    
    Stderr: Traceback (most recent call last):
      File "/tmp/root/cohort_denoising_calling.1528767346035786763.py", line 117, in <module>
        n_st, sample_names, sample_metadata_collection)
      File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/models/model_denoising_calling.py", line 390, in __init__
        sample_metadata_collection, sample_names, self.contig_list)
      File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/models/model_denoising_calling.py", line 630, in _get_baseline_copy_number_and_read_depth
        "Some samples do not have read depth metadata"
    AssertionError: Some samples do not have read depth metadata
    
        at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
        at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
        at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
        at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:151)
        at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:121)
        at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.executeGermlineCNVCallerPythonScript(GermlineCNVCaller.java:450)
        at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.doWork(GermlineCNVCaller.java:288)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:134)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
        at org.broadinstitute.hellbender.Main.main(Main.java:289)
    

    Something is off here. Can someone help me?

    I am seeing this problem with the fresh gatk 4.0.3.0 docker image under mac os 10.13.3 and Ubuntu 17.10. No difference.

    Issue · Github
    by Sheila

    Issue Number
    3030
    State
    closed
    Last Updated
    Assignee
    Array
    Closed By
    chandrans
  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Thanks I will be waiting.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @SkyWarrior
    Hi,

    I have some information from the developer:

    The correct argument is the first, i.e. --contig-ploidy-calls X/prefix-calls (GermlineCNVCaller expects the ploidy calls folder). It is possible the tool is running but not logging the progress. Can you try running GermlineCNVCaller with the additional argument --verbosity DEBUG?

    With regards to comparing to XHMM, a new beta with tuned default arguments will be rolled out for Bio-IT (May 15th). The current beta needs manual interval pre-processing and some argument tuning to yield reliable calls. We can give you a development branch to try out in two weeks. Let us know if you are interested. If you are interested, please share your findings with us :smile:

    -Sheila

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Thanks for the info. I will try with debug parameter. For the comparison yes I can try that. I have a set of exomes with known copy number variations that I use for workflow testing. These samples were all confirmed via array, MLPA etc.

    Thank you.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited April 2018

    Hi
    I have updates on this one. Looks like the docker image is not very happy with using the default tmp folder so -Djava.io.tmpdir must be specified at the working directory to be on the safeside for this tool. Now it looks like it is working. I am curious though some of my samples X and Y ploidies were calculated wrong. Y:1 X:2. I am hundred percent sure that these samples are either Y:1 X:1 or Y:0 X:2. I will check some details about this one and write them here. Warning message says unmasked PAR may result in this but my X and Y chromosomes are from the analysis bundle so PARs should not be a problem?

    Lets see what happens.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    A new question?

    How long does this thing need to run in the gcnvkernel?

    It has been 13 epochs of denoising sampling and CNV calling (Sort of). This is really taking too long. I am checking the memory usage but It does not seem to be too high.

    I am curious of the result...

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited April 2018 Accepted Answer

    Yes finally I was able to get this thing to work and I have good and bad news. But more of good news and I am hopefull for the future of this tool in general.

    I am mainly interested in recalling regions that were already called by XHMM and confirmed by MLPA or similar methods.

    I got this tool to work on a 32 Trusight One Clinical Exome panels sequenced on average 60-80x on NextSeq 500. This panel consists of 4800 genes and approximately 63000 non-overlapping intervals.

    Out of this 32 sample cohort I had 2 samples that had a long deletion in different spots.
    Sample 780 had a multi exon deletion in COL7A1 gene with a EBD phenotype and Sample 784 had a multi exon deletion in DMD gene with muscular dystrophy phenotype.

    Sample 780

    Intervals were exactly the same as XHMM predicted and we confirmed this sample with PCR due to evident breakpoints on the bam file. We were lucky about those breakpoints since almost the entire family was tested for carrier status with a very cheap experiment. Every carrier had the same deletion and every affected individual had the same homozygous deletion. Interesting case indeed.

    Sample 784

    Intervals were exactly the same as XHMM predicted and this sample was confirmed with MLPA analysis.

    Good news.

    • We now have another robust tool to detect copy number variations especially from WES data.
    • Tools prediction levels seem to be similar to my long time favorite XHMM for confirmed samples. I will be doing more testing on some of my known duplication samples. I sense that there needs to be more data to train this tool to eliminate some of the false positives or maybe a post processing option to eliminate intervals known to have CNV in human population. (I filter some of XHMM predictions based on that. I will try to filter GATK output and make a more fair comparison)

    Bad news

    • Tool still takes to much time to analyze cohort data. 32 Trusight samples in XHMM takes about 5 to 10 minutes of execution time with fairly decent call rate. (Though XHMM needs some data mining afterwards for CN status especially for 1 and 3 or more copies. ) GATK takes 450 minutes to process 32 Trusight samples. (XHMM can process 150 WES samples in that time period.)

    Overall
    I am happy that GATK CNV works as expected even in the beta status. I will be posting more results sooner. Now back to my dissertation :)

  • dayzcooldayzcool Member

    Thank you for sharing the interesting results, @SkyWarrior!
    @slee, @Sheila, and @SkyWarrior, I will try to run germline CNV caller for WGS and Exome in the next week. I plan to use the wdl pipeline in the git repo. and build master, since there seems to be some updates since the last release. I would appreciate if you would give any suggestions. Especially, I wonder if there are any differences between WGS and Exome in parametrization (other than target region).
    Just FYI, we are currently using GenomeStrip and will evaluate two other commercial tools to decide what to use in our operations. Thanks!

  • sleeslee Member, Broadie, Dev ✭✭✭

    Looping in @Mehrtash, who is the main developer of GermlineCNVCaller.

    As mentioned above, we're currently running evaluations and have made a few tweaks to master in the process. These tweaks are still in a dev branch (mb_gcnvkernel_tuning) and have not been merged yet.

    As for WGS vs. WES, there are a few significant differences in some of the hyperparameter choices that control how systematic biases (which exhibit a higher degree of variation in WES) are modeled. After this first round of evaluations, we'll share some reasonable parameter choices for typical WES and WGS data. We've also done some scans over the hyperparameter space to explore how performance varies---fortunately, we still seem to perform relatively well even if the hyperparameters are not optimally tuned!

  • dayzcooldayzcool Member

    Thank you so much for your advice, @slee! I will use the mb_gcnvkernel_tuning branch for evaluation in the meantime, unless you/your team don't recommend to use it. I really want to wait for the your recommendations, but the first iteration of evaluation needs to be done soon, unfortunately.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @dayzcool
    Hi,

    I just emailed the developer to confirm, but I think you can go ahead and work with that branch. Let us know what you find :smiley:

    -Sheila

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin
    edited April 2018

    @dayzcool
    Hi again,

    Here is what the developer says:

    "Here's what we do in the evaluations, though, we have not made it to the branch yet:

    For exomes, I use the attached set of hyper-parameters. There's still quite a lot of room for improvement, but it already beats XHMM and CODEX by a wide margin.

    {
    "CNVGermlineCohortWorkflow.allosomal_contigs": [
    "X",
    "Y"
    ],
    "CNVGermlineCohortWorkflow.bin_length": 0,
    "CNVGermlineCohortWorkflow.cohort_entity_id": "XXX",
    "CNVGermlineCohortWorkflow.contig_ploidy_priors": "gs://broad-dsde-methods/mehrtash/data/contig_ploidy_priors_homo_sapiens.tsv",
    "CNVGermlineCohortWorkflow.cpu_for_determine_germline_contig_ploidy": 4,
    "CNVGermlineCohortWorkflow.cpu_for_germline_cnv_caller": 8,
    "CNVGermlineCohortWorkflow.do_explicit_gc_correction": true,
    "CNVGermlineCohortWorkflow.gatk_docker": "us.gcr.io/broad-dsde-methods/broad-gatk-snapshots:mb_gCNV_eval_bbecc0ce4b86cf32e9ab795746596218caf3280c",
    "CNVGermlineCohortWorkflow.gcnv_adamax_beta_1": 0.9,
    "CNVGermlineCohortWorkflow.gcnv_adamax_beta_2": 0.97,
    "CNVGermlineCohortWorkflow.gcnv_caller_external_admixing_rate": 1.0,
    "CNVGermlineCohortWorkflow.gcnv_caller_internal_admixing_rate": 0.5,
    "CNVGermlineCohortWorkflow.gcnv_caller_update_convergence_threshold": 1e-06,
    "CNVGermlineCohortWorkflow.gcnv_class_coherence_length": 100000,
    "CNVGermlineCohortWorkflow.gcnv_cnv_coherence_length": 100000,
    "CNVGermlineCohortWorkflow.gcnv_convergence_snr_trigger_threshold": 0.2,
    "CNVGermlineCohortWorkflow.gcnv_copy_number_posterior_expectation_mode": "EXACT",
    "CNVGermlineCohortWorkflow.gcnv_depth_correction_tau": 100.0,
    "CNVGermlineCohortWorkflow.gcnv_disable_annealing": false,
    "CNVGermlineCohortWorkflow.gcnv_initial_temperature": 2.0,
    "CNVGermlineCohortWorkflow.gcnv_interval_psi_scale": 0.002,
    "CNVGermlineCohortWorkflow.gcnv_learning_rate": 0.03,
    "CNVGermlineCohortWorkflow.gcnv_log_emission_sampling_median_rel_error": 0.001,
    "CNVGermlineCohortWorkflow.gcnv_log_emission_sampling_rounds": 20,
    "CNVGermlineCohortWorkflow.gcnv_log_mean_bias_standard_deviation": 10.0,
    "CNVGermlineCohortWorkflow.gcnv_max_advi_iter_first_epoch": 5000,
    "CNVGermlineCohortWorkflow.gcnv_max_advi_iter_subsequent_epochs": 200,
    "CNVGermlineCohortWorkflow.gcnv_max_bias_factors": 16,
    "CNVGermlineCohortWorkflow.gcnv_max_calling_iters": 20,
    "CNVGermlineCohortWorkflow.gcnv_max_copy_number": 5,
    "CNVGermlineCohortWorkflow.gcnv_max_training_epochs": 20,
    "CNVGermlineCohortWorkflow.gcnv_min_training_epochs": 5,
    "CNVGermlineCohortWorkflow.gcnv_num_thermal_advi_iters": 4000,
    "CNVGermlineCohortWorkflow.gcnv_p_active": 0.01,
    "CNVGermlineCohortWorkflow.gcnv_p_alt": 0.001,
    "CNVGermlineCohortWorkflow.gcnv_sample_psi_scale": 1e-07,
    "CNVGermlineCohortWorkflow.intervals": "XXX",
    "CNVGermlineCohortWorkflow.mem_gb_for_determine_germline_contig_ploidy": 16,
    "CNVGermlineCohortWorkflow.mem_gb_for_germline_cnv_caller": 24,
    "CNVGermlineCohortWorkflow.normal_bais": [XXX],
    "CNVGermlineCohortWorkflow.normal_bams": [XXX],
    "CNVGermlineCohortWorkflow.num_intervals_per_scatter": 5000,
    "CNVGermlineCohortWorkflow.padding": 0,
    "CNVGermlineCohortWorkflow.ploidy_global_psi_scale": 0.05,
    "CNVGermlineCohortWorkflow.ploidy_mean_bias_standard_deviation": 1.0,
    "CNVGermlineCohortWorkflow.ploidy_sample_psi_scale": 0.001,
    "CNVGermlineCohortWorkflow.preemptible_attempts": 0,
    "CNVGermlineCohortWorkflow.ref_copy_number_autosomal_contigs": 2,
    "CNVGermlineCohortWorkflow.ref_fasta": "gs://broad-dsde-methods/mehrtash/data/Homo_sapiens_assembly19.fasta",
    "CNVGermlineCohortWorkflow.ref_fasta_dict": "gs://broad-dsde-methods/mehrtash/data/Homo_sapiens_assembly19.dict",
    "CNVGermlineCohortWorkflow.ref_fasta_fai": "gs://broad-dsde-methods/mehrtash/data/Homo_sapiens_assembly19.fasta.fai"
    }

    I manually filter targets based on mappability and counts. I recommend you do the same and not include failing targets (e.g. extremely low counts)."

    -Sheila

  • dayzcooldayzcool Member

    How kind of you! I am glad I asked and I really appreciate it.
    Just for the clarification, do I understand correctly that I can use this set of hyper-parameters for both cohort and case workflows for Exome, and default parameters for WGS?

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin
    edited April 2018

    @dayzcool
    Hi,

    Yes, that sounds right. They may have additional tweaks for WGS later.

    -Sheila

  • dayzcooldayzcool Member

    Could you help me get thecontig_ploidy_priors file? I couldn't access gs://broad-dsde-methods/mehrtash/data/contig_ploidy_priors_homo_sapiens.tsv

  • sleeslee Member, Broadie, Dev ✭✭✭

    Hi @dayzcool,

    That file is a TSV (make sure that you use tabs, not spaces) that looks as follows:

    CONTIG_NAME PLOIDY_PRIOR_0  PLOIDY_PRIOR_1  PLOIDY_PRIOR_2  PLOIDY_PRIOR_3
    1   0.0 0.01    0.98    0.01
    2   0.0 0.01    0.98    0.01
    3   0.0 0.01    0.98    0.01
    4   0.0 0.01    0.98    0.01
    5   0.0 0.01    0.98    0.01
    6   0.0 0.01    0.98    0.01
    7   0.0 0.01    0.98    0.01
    8   0.0 0.01    0.98    0.01
    9   0.0 0.01    0.98    0.01
    10  0.0 0.01    0.98    0.01
    11  0.0 0.01    0.98    0.01
    12  0.0 0.01    0.98    0.01
    13  0.0 0.01    0.98    0.01
    14  0.0 0.01    0.98    0.01
    15  0.0 0.01    0.98    0.01
    16  0.0 0.01    0.98    0.01
    17  0.0 0.01    0.98    0.01
    18  0.0 0.01    0.98    0.01
    19  0.0 0.01    0.98    0.01
    20  0.0 0.01    0.98    0.01
    21  0.0 0.01    0.98    0.01
    22  0.0 0.01    0.98    0.01
    X   0.01    0.49    0.49    0.01
    Y   0.495   0.495   0.01    0.0
    

    You should feel free to adjust the priors if you like.

  • dayzcooldayzcool Member

    Thanks again for your kind help! I could successfully call Germline CNVs using GATK4. I ended up using the provided set of parameters for both WGS and WES. Default parameters didn't always work on WGS samples.

    So it was part of a small, sensitivity-focused evaluation on tens of known CNVs. I learned from the colleague who conduct the evaluation that GATK performed excellent in germline CNV calling.
    Anecdotal issues we had were that gcnvkernel (0.6.6dev) hangs/fails randomly (may not be a GATK issue) and a long CNV was sometimes segmented as smaller CNVs.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @dayzcool
    Hi,

    Thank you for updating us! I will forward this on to the team :smiley:

    -Sheila

  • sleeslee Member, Broadie, Dev ✭✭✭
    edited May 2018

    (accidentally posted twice---editing to remove the duplicate post!)

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Hi @slee

    Thanks for the heads up. I have been using a very simple algoritm to detect Y ploidy on my samples using samtools view to count the reads on Y chromosome and calculate the ratio towards autosomal chromosomes and X robustly and my numbers were pretty solid since then. Total count may still work properly however it is impossible to mask some of the regions as they are in the canonical analysis (CD24) My reference is PAR masked Y so it is should not be a problem. I will run my samples with any updates that you may push soon. It is good to hear that I was not the only one :smile:
    Thanks

  • TintestTintest FranceMember

    @Sheila said:

    I manually filter targets based on mappability and counts. I recommend you do the same and not include failing targets (e.g. extremely low counts)."

    Hello,

    I got several questions about the germline CNV calling, I'm posting here, because it seems @SkyWarrior did some of the most detailed work on the germline CNV caller here on the forum so it's seems a good place to gather the questions :)

    Is there any new about the recommended parameters ? Could you show us the correct syntax to use in a GATKconfig file (I do not use the WDL language) if possible with updated parameters values ? Or ask the developers for implementing specific parameters "mode" suited either if you're doing WES or WGS ?

    You said you were filtering by counting targets. Do you know if there is a tool which can give me the count of redundant segments / intervals within a batch of VCFs obtained with the PostprocessGermlineCNVCalls. By filtering the very common segments (and with the different quality scores), I could filter out some background noise, because I got way to much segments in each VCF file.

    Thank you.

    Issue · Github
    by Sheila

    Issue Number
    3117
    State
    open
    Last Updated
  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @Tintest
    Hi,

    I will check with the developers and get back to you.

    -Sheila

  • sleeslee Member, Broadie, Dev ✭✭✭
    edited June 2018

    Hi @Tintest,

    You should be able to infer the parameters for the GermlineCNVCaller command line from the JSON config above. The relevant entries will start with CNVGermlineCohortWorkflow.gcnv_. For example, CNVGermlineCohortWorkflow.gcnv_adamax_beta_1": 0.9 corresponds to --adamax-beta-1 0.9, and so on. You'll find that most of these parameters are set to the tool defaults, but there are a few (e.g., --interval-psi-scale that will need to be adjusted depending on if your data is WES or WGS. (You may find the discussion at https://github.com/broadinstitute/gatk/issues/4719 helpful---adding the ability to set "parameter modes" is on our list.)

    You said you were filtering by counting targets. Do you know if there is a tool which can give me the count of redundant segments / intervals within a batch of VCFs obtained with the PostprocessGermlineCNVCalls. By filtering the very common segments (and with the different quality scores), I could filter out some background noise, because I got way to much segments in each VCF file.

    Note that @Mehrtash (the developer quoted by @Sheila) was referring to filtering the initial set of targets used for coverage collection. We have some plans to implement a tool to do this step (see https://github.com/broadinstitute/gatk/issues/2992).

    As for filtering calls, we typically filter on QS. I'm not sure what you mean by "redundant segments / intervals"---do you mean filtering calls by allele frequency, etc.? If so, you will probably want to implement your own scripts to do so based on the specific overlap criteria, etc. that you desire. However, you may be able to use some GATK tools to combine the per-sample VCFs, which might help you in this process.

    Thanks,
    Samuel

    Post edited by slee on
  • stefstefstefstef Member

    Hi @Sheila,

    I'm looking to carry out CNV calling on targeted capture (single gene panels), from what I can tell these settings are for WES. What would a reasonable number for num_intervals_per_scatter be? I assume if WES is 5000 then maybe it's 1, or just to have some splitting of intervals, 10?

    Thanks

    Stef

    @Sheila said:
    @dayzcool
    Hi again,

    Here is what the developer says:

    "Here's what we do in the evaluations, though, we have not made it to the branch yet:

    For exomes, I use the attached set of hyper-parameters. There's still quite a lot of room for improvement, but it already beats XHMM and CODEX by a wide margin.

    {
    "CNVGermlineCohortWorkflow.allosomal_contigs": [
    "X",
    "Y"
    ],
    "CNVGermlineCohortWorkflow.bin_length": 0,
    "CNVGermlineCohortWorkflow.cohort_entity_id": "XXX",
    ...
    "CNVGermlineCohortWorkflow.num_intervals_per_scatter": 5000,
    ...

    I manually filter targets based on mappability and counts. I recommend you do the same and not include failing targets (e.g. extremely low counts)."

    -Sheila

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Hi

    I am revisiting this workflow again in 4.0.12.0 for a particular event that was confirmed by MLPA but missed by XHMM and some others.

    It seems the -imr OVERLAPPING_ONLY parameter still needs to be input manually and GermlineCNVCaller cannot create an output folder so it has to be present before the tool starts crunching numbers. This was present way back in 4.0.3.0 as well.

    Lets see if GATK can capture this event.

    Regards.

  • AdelaideRAdelaideR Unconfirmed, Member, Broadie, Moderator admin

    @SkyWarrior - Could you please post an update hear once you try?

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Sure I will. Once I get the hard drive fixed.

  • pawankrpawankr Member
    Hello
    I start using GATK 4.0.12.0 for germline cnv discovery, i used contig_ploidy_priors (tsv file given this thread above) its work ,how I generate own contig_ploidy_priors .tsv file ? .

    I also get error in GermlineCNVCaller case mode : Some samples do not have read depth metadata.

    any workflow for Germline copy number variant discovery ???

    Thanks
  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Just copy and paste the file given in the sample. It should work. That file is an input for HMM probability.

  • bhanuGandhambhanuGandham Cambridge MAMember, Administrator, Broadie, Moderator admin

    HI @pawankr

    Let me know if you were able to resolve this issue with @SkyWarrior's suggestion.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited February 4 Accepted Answer

    @AdelaideR said:
    @SkyWarrior - Could you please post an update hear once you try?

    Hi

    Here are the updates from my latest trial.

    • Good news is that I was able to capture the event that was missed by many others just by using default parameters for all the rest except for the -imr OVERLAPPING_ONLY. Now I need to optimize other parameters especially to reduce the inference time for GermlineCNVCalls step. Training epochs reach 38 cycles if I leave the max number of epochs at 50. I am wondering if 20 would suffice???

    • Bad news is I am receiving a Not implemented error for the MT contig that was included in the bed file since it was my default bed file to use. Using -XL MT does not stop the algorithm to use MT and generate the segmented vcf.

    Below is the DEBUG output for PostprocessGermlineCNVCalls.

    (gatk) [[email protected] GATKCNVtesting_V2]$ gatk PostprocessGermlineCNVCalls --calls-shard-path CNVCalls/GATK_CNV-calls --model-shard-path CNVCalls/GATK_CNV-model --sample-index 22 --autosomal-ref-copy-number 2 --allosomal-contig X --allosomal-contig Y --output-genotyped-intervals Wes492_intervals.vcf --output-genotyped-segments Wes492_segments.vcf --contig-ploidy-calls CNV/GATKCNVtesting_V2_CNV-calls -imr OVERLAPPING_ONLY -XL MT -R $B37 -verbosity DEBUG
    Using GATK jar /home/exome/scripts/gatk-package-4.1.0.0-local.jar
    Running:
        java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/exome/scripts/gatk-package-4.1.0.0-local.jar PostprocessGermlineCNVCalls --calls-shard-path CNVCalls/GATK_CNV-calls --model-shard-path CNVCalls/GATK_CNV-model --sample-index 22 --autosomal-ref-copy-number 2 --allosomal-contig X --allosomal-contig Y --output-genotyped-intervals Wes492_intervals.vcf --output-genotyped-segments Wes492_segments.vcf --contig-ploidy-calls CNV/GATKCNVtesting_V2_CNV-calls -imr OVERLAPPING_ONLY -XL MT -R /home/exome/GenomeData/B37/hs37d5.fa -verbosity DEBUG
    09:45:21.030 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/exome/scripts/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    09:45:21.040 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /tmp/libgkl_compression2296465957449338279.so
    09:45:22.712 INFO  PostprocessGermlineCNVCalls - ------------------------------------------------------------
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - The Genome Analysis Toolkit (GATK) v4.1.0.0
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - For support and documentation go to https://software.broadinstitute.org/gatk/
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - Executing as [email protected] on Linux v4.19.16-1-MANJARO amd64
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b26
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - Start Date/Time: February 4, 2019 9:45:21 AM EET
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - ------------------------------------------------------------
    09:45:22.713 INFO  PostprocessGermlineCNVCalls - ------------------------------------------------------------
    09:45:22.714 INFO  PostprocessGermlineCNVCalls - HTSJDK Version: 2.18.2
    09:45:22.714 INFO  PostprocessGermlineCNVCalls - Picard Version: 2.18.25
    09:45:22.715 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.BUFFER_SIZE : 131072
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.CREATE_INDEX : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.CREATE_MD5 : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.CUSTOM_READER_FACTORY :
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.REFERENCE_FASTA : null
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    09:45:22.716 INFO  PostprocessGermlineCNVCalls - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false
    09:45:22.716 DEBUG ConfigFactory - Configuration file values:
    09:45:22.720 DEBUG ConfigFactory -      gcsMaxRetries = 20
    09:45:22.721 DEBUG ConfigFactory -      gcsProjectForRequesterPays =
    09:45:22.721 DEBUG ConfigFactory -      gatk_stacktrace_on_user_exception = false
    09:45:22.721 DEBUG ConfigFactory -      samjdk.use_async_io_read_samtools = false
    09:45:22.721 DEBUG ConfigFactory -      samjdk.use_async_io_write_samtools = true
    09:45:22.721 DEBUG ConfigFactory -      samjdk.use_async_io_write_tribble = false
    09:45:22.721 DEBUG ConfigFactory -      samjdk.compression_level = 2
    09:45:22.721 DEBUG ConfigFactory -      spark.kryoserializer.buffer.max = 512m
    09:45:22.721 DEBUG ConfigFactory -      spark.driver.maxResultSize = 0
    09:45:22.721 DEBUG ConfigFactory -      spark.driver.userClassPathFirst = true
    09:45:22.721 DEBUG ConfigFactory -      spark.io.compression.codec = lzf
    09:45:22.721 DEBUG ConfigFactory -      spark.yarn.executor.memoryOverhead = 600
    09:45:22.721 DEBUG ConfigFactory -      spark.driver.extraJavaOptions =
    09:45:22.722 DEBUG ConfigFactory -      codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]
    09:45:22.722 DEBUG ConfigFactory -      spark.executor.extraJavaOptions =
    09:45:22.722 DEBUG ConfigFactory -      read_filter_packages = [org.broadinstitute.hellbender.engine.filters]
    09:45:22.722 DEBUG ConfigFactory -      annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]
    09:45:22.722 DEBUG ConfigFactory -      cloudPrefetchBuffer = 40
    09:45:22.722 DEBUG ConfigFactory -      cloudIndexPrefetchBuffer = -1
    09:45:22.722 DEBUG ConfigFactory -      createOutputBamIndex = true
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Deflater: IntelDeflater
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Inflater: IntelInflater
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - GCS max retries/reopens: 20
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Requester pays: disabled
    09:45:22.722 INFO  PostprocessGermlineCNVCalls - Initializing engine
    09:45:22.878 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
    09:45:22.878 DEBUG GenomeLocParser -  1 (249250621 bp)
    09:45:22.878 DEBUG GenomeLocParser -  2 (243199373 bp)
    09:45:22.878 DEBUG GenomeLocParser -  3 (198022430 bp)
    09:45:22.878 DEBUG GenomeLocParser -  4 (191154276 bp)
    09:45:22.878 DEBUG GenomeLocParser -  5 (180915260 bp)
    09:45:22.878 DEBUG GenomeLocParser -  6 (171115067 bp)
    09:45:22.878 DEBUG GenomeLocParser -  7 (159138663 bp)
    09:45:22.878 DEBUG GenomeLocParser -  8 (146364022 bp)
    09:45:22.878 DEBUG GenomeLocParser -  9 (141213431 bp)
    09:45:22.879 DEBUG GenomeLocParser -  10 (135534747 bp)
    09:45:22.879 DEBUG GenomeLocParser -  11 (135006516 bp)
    09:45:22.879 DEBUG GenomeLocParser -  12 (133851895 bp)
    09:45:22.879 DEBUG GenomeLocParser -  13 (115169878 bp)
    09:45:22.879 DEBUG GenomeLocParser -  14 (107349540 bp)
    09:45:22.879 DEBUG GenomeLocParser -  15 (102531392 bp)
    09:45:22.879 DEBUG GenomeLocParser -  16 (90354753 bp)
    09:45:22.879 DEBUG GenomeLocParser -  17 (81195210 bp)
    09:45:22.879 DEBUG GenomeLocParser -  18 (78077248 bp)
    09:45:22.879 DEBUG GenomeLocParser -  19 (59128983 bp)
    09:45:22.879 DEBUG GenomeLocParser -  20 (63025520 bp)
    09:45:22.879 DEBUG GenomeLocParser -  21 (48129895 bp)
    09:45:22.879 DEBUG GenomeLocParser -  22 (51304566 bp)
    09:45:22.879 DEBUG GenomeLocParser -  X (155270560 bp)
    09:45:22.879 DEBUG GenomeLocParser -  Y (59373566 bp)
    09:45:22.879 DEBUG GenomeLocParser -  MT (16569 bp)
    09:45:22.879 DEBUG GenomeLocParser -  GL000207.1 (4262 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000226.1 (15008 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000229.1 (19913 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000231.1 (27386 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000210.1 (27682 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000239.1 (33824 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000235.1 (34474 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000201.1 (36148 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000247.1 (36422 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000245.1 (36651 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000197.1 (37175 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000203.1 (37498 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000246.1 (38154 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000249.1 (38502 bp)
    09:45:22.880 DEBUG GenomeLocParser -  GL000196.1 (38914 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000248.1 (39786 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000244.1 (39929 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000238.1 (39939 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000202.1 (40103 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000234.1 (40531 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000232.1 (40652 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000206.1 (41001 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000240.1 (41933 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000236.1 (41934 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000241.1 (42152 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000243.1 (43341 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000242.1 (43523 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000230.1 (43691 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000237.1 (45867 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000233.1 (45941 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000204.1 (81310 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000198.1 (90085 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000208.1 (92689 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000191.1 (106433 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000227.1 (128374 bp)
    09:45:22.881 DEBUG GenomeLocParser -  GL000228.1 (129120 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000214.1 (137718 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000221.1 (155397 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000209.1 (159169 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000218.1 (161147 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000220.1 (161802 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000213.1 (164239 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000211.1 (166566 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000199.1 (169874 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000217.1 (172149 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000216.1 (172294 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000215.1 (172545 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000205.1 (174588 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000219.1 (179198 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000224.1 (179693 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000223.1 (180455 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000195.1 (182896 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000212.1 (186858 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000222.1 (186861 bp)
    09:45:22.882 DEBUG GenomeLocParser -  GL000200.1 (187035 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000193.1 (189789 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000194.1 (191469 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000225.1 (211173 bp)
    09:45:22.883 DEBUG GenomeLocParser -  GL000192.1 (547496 bp)
    09:45:22.883 DEBUG GenomeLocParser -  NC_007605 (171823 bp)
    09:45:22.883 DEBUG GenomeLocParser -  hs37d5 (35477943 bp)
    09:45:22.883 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
    09:45:22.883 DEBUG GenomeLocParser -  1 (249250621 bp)
    09:45:22.883 DEBUG GenomeLocParser -  2 (243199373 bp)
    09:45:22.883 DEBUG GenomeLocParser -  3 (198022430 bp)
    09:45:22.883 DEBUG GenomeLocParser -  4 (191154276 bp)
    09:45:22.883 DEBUG GenomeLocParser -  5 (180915260 bp)
    09:45:22.883 DEBUG GenomeLocParser -  6 (171115067 bp)
    09:45:22.883 DEBUG GenomeLocParser -  7 (159138663 bp)
    09:45:22.883 DEBUG GenomeLocParser -  8 (146364022 bp)
    09:45:22.883 DEBUG GenomeLocParser -  9 (141213431 bp)
    09:45:22.884 DEBUG GenomeLocParser -  10 (135534747 bp)
    09:45:22.884 DEBUG GenomeLocParser -  11 (135006516 bp)
    09:45:22.884 DEBUG GenomeLocParser -  12 (133851895 bp)
    09:45:22.884 DEBUG GenomeLocParser -  13 (115169878 bp)
    09:45:22.884 DEBUG GenomeLocParser -  14 (107349540 bp)
    09:45:22.884 DEBUG GenomeLocParser -  15 (102531392 bp)
    09:45:22.884 DEBUG GenomeLocParser -  16 (90354753 bp)
    09:45:22.884 DEBUG GenomeLocParser -  17 (81195210 bp)
    09:45:22.884 DEBUG GenomeLocParser -  18 (78077248 bp)
    09:45:22.884 DEBUG GenomeLocParser -  19 (59128983 bp)
    09:45:22.884 DEBUG GenomeLocParser -  20 (63025520 bp)
    09:45:22.884 DEBUG GenomeLocParser -  21 (48129895 bp)
    09:45:22.884 DEBUG GenomeLocParser -  22 (51304566 bp)
    09:45:22.884 DEBUG GenomeLocParser -  X (155270560 bp)
    09:45:22.884 DEBUG GenomeLocParser -  Y (59373566 bp)
    09:45:22.884 DEBUG GenomeLocParser -  MT (16569 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000207.1 (4262 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000226.1 (15008 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000229.1 (19913 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000231.1 (27386 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000210.1 (27682 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000239.1 (33824 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000235.1 (34474 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000201.1 (36148 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000247.1 (36422 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000245.1 (36651 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000197.1 (37175 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000203.1 (37498 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000246.1 (38154 bp)
    09:45:22.884 DEBUG GenomeLocParser -  GL000249.1 (38502 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000196.1 (38914 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000248.1 (39786 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000244.1 (39929 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000238.1 (39939 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000202.1 (40103 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000234.1 (40531 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000232.1 (40652 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000206.1 (41001 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000240.1 (41933 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000236.1 (41934 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000241.1 (42152 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000243.1 (43341 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000242.1 (43523 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000230.1 (43691 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000237.1 (45867 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000233.1 (45941 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000204.1 (81310 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000198.1 (90085 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000208.1 (92689 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000191.1 (106433 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000227.1 (128374 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000228.1 (129120 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000214.1 (137718 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000221.1 (155397 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000209.1 (159169 bp)
    09:45:22.885 DEBUG GenomeLocParser -  GL000218.1 (161147 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000220.1 (161802 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000213.1 (164239 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000211.1 (166566 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000199.1 (169874 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000217.1 (172149 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000216.1 (172294 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000215.1 (172545 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000205.1 (174588 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000219.1 (179198 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000224.1 (179693 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000223.1 (180455 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000195.1 (182896 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000212.1 (186858 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000222.1 (186861 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000200.1 (187035 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000193.1 (189789 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000194.1 (191469 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000225.1 (211173 bp)
    09:45:22.886 DEBUG GenomeLocParser -  GL000192.1 (547496 bp)
    09:45:22.886 DEBUG GenomeLocParser -  NC_007605 (171823 bp)
    09:45:22.886 DEBUG GenomeLocParser -  hs37d5 (35477943 bp)
    09:45:22.891 DEBUG GenomeLocSortedSet - removeRegions operation: i = 1
    09:45:22.891 INFO  IntervalArgumentCollection - Initial include intervals span 3137454505 loci; exclude intervals span 16569 loci
    09:45:22.892 INFO  IntervalArgumentCollection - Excluding 16569 loci from original intervals (0.00% reduction)
    09:45:22.892 INFO  IntervalArgumentCollection - Processing 3137437936 bp from intervals
    09:45:22.894 DEBUG ScriptExecutor - Executing:
    09:45:22.894 DEBUG ScriptExecutor -   python
    09:45:22.894 DEBUG ScriptExecutor -   -c
    09:45:22.894 DEBUG ScriptExecutor -   import gcnvkernel
    
    09:45:24.795 DEBUG ScriptExecutor - Result: 0
    09:45:24.795 INFO  PostprocessGermlineCNVCalls - Done initializing engine
    09:45:25.464 INFO  ProgressMeter - Starting traversal
    09:45:25.464 INFO  ProgressMeter -        Current Locus  Elapsed Minutes     Records Processed   Records/Minute
    09:45:25.464 INFO  PostprocessGermlineCNVCalls - Generating intervals VCF file...
    09:45:25.477 INFO  PostprocessGermlineCNVCalls - Analyzing shard 0...
    09:45:28.253 INFO  PostprocessGermlineCNVCalls - Generating segments VCF file...
    09:45:28.256 DEBUG ScriptExecutor - Executing:
    09:45:28.256 DEBUG ScriptExecutor -   python
    09:45:28.256 DEBUG ScriptExecutor -   /tmp/segment_gcnv_calls.4286460784218017052.py
    09:45:28.256 DEBUG ScriptExecutor -   --ploidy_calls_path
    09:45:28.256 DEBUG ScriptExecutor -   /mnt/E/Exomes/GATKCNVtesting_V2/CNV/GATKCNVtesting_V2_CNV-calls
    09:45:28.256 DEBUG ScriptExecutor -   --model_shards
    09:45:28.256 DEBUG ScriptExecutor -   /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-model
    09:45:28.256 DEBUG ScriptExecutor -   --calls_shards
    09:45:28.256 DEBUG ScriptExecutor -   /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-calls
    09:45:28.256 DEBUG ScriptExecutor -   --output_path
    09:45:28.256 DEBUG ScriptExecutor -   /tmp/gcnv-segmented-calls2678318989462462101
    09:45:28.256 DEBUG ScriptExecutor -   --sample_index
    09:45:28.256 DEBUG ScriptExecutor -   22
    09:45:29.991 INFO segment_gcnv_calls - Loading ploidy calls...
    09:45:29.991 INFO gcnvkernel.io.io_metadata - Loading germline contig ploidy and global read depth metadata...
    09:45:29.994 WARNING gcnvkernel.structs.metadata - Sample Wes491 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:29.996 WARNING gcnvkernel.structs.metadata - Sample Wes494 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:29.998 WARNING gcnvkernel.structs.metadata - Sample Wes500 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.000 WARNING gcnvkernel.structs.metadata - Sample Wes492 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.001 WARNING gcnvkernel.structs.metadata - Sample Wes484 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.003 WARNING gcnvkernel.structs.metadata - Sample Wes497 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.005 WARNING gcnvkernel.structs.metadata - Sample Wes478 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.007 WARNING gcnvkernel.structs.metadata - Sample Wes473 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.009 WARNING gcnvkernel.structs.metadata - Sample Wes495 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.011 WARNING gcnvkernel.structs.metadata - Sample Wes479 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.012 WARNING gcnvkernel.structs.metadata - Sample Wes498 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.014 WARNING gcnvkernel.structs.metadata - Sample Wes493 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.016 WARNING gcnvkernel.structs.metadata - Sample Wes499 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.018 WARNING gcnvkernel.structs.metadata - Sample Wes482 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.019 WARNING gcnvkernel.structs.metadata - Sample Wes476 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.021 WARNING gcnvkernel.structs.metadata - Sample Wes470 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.023 WARNING gcnvkernel.structs.metadata - Sample Wes496 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.025 WARNING gcnvkernel.structs.metadata - Sample Wes472 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.026 WARNING gcnvkernel.structs.metadata - Sample Wes486 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.028 WARNING gcnvkernel.structs.metadata - Sample Wes481 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.030 WARNING gcnvkernel.structs.metadata - Sample Wes480 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.032 WARNING gcnvkernel.structs.metadata - Sample Wes483 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.033 WARNING gcnvkernel.structs.metadata - Sample Wes485 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.035 WARNING gcnvkernel.structs.metadata - Sample Wes471 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.037 WARNING gcnvkernel.structs.metadata - Sample Wes477 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.039 WARNING gcnvkernel.structs.metadata - Sample Wes475 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.041 WARNING gcnvkernel.structs.metadata - Sample Wes490 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.042 WARNING gcnvkernel.structs.metadata - Sample Wes487 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.044 WARNING gcnvkernel.structs.metadata - Sample Wes474 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.046 WARNING gcnvkernel.structs.metadata - Sample Wes488 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.048 WARNING gcnvkernel.structs.metadata - Sample Wes489 has an unrecognized contig (MT). Germline contig ploidy determination may not be reliable for decoy/non-standard contigs.
    09:45:30.048 INFO segment_gcnv_calls - Instantiating the Viterbi segmentation engine...
    09:45:30.055 INFO gcnvkernel.postprocess.viterbi_segmentation - Assembling interval list and copy-number class posterior from model shards...
    09:45:30.852 INFO gcnvkernel.io.io_intervals_and_counts - The given interval list provides the following interval annotations: {'GC_CONTENT'}
    09:45:32.320 INFO gcnvkernel.structs.metadata - Generating intervals metadata...
    09:45:33.059 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano forward-backward function...
    09:45:34.373 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano Viterbi function...
    09:45:36.048 INFO gcnvkernel.postprocess.viterbi_segmentation - Compiling theano variational HHMM...
    09:45:36.210 INFO gcnvkernel.postprocess.viterbi_segmentation - Processing sample index: 22, sample name: Wes492...
    09:45:36.422 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (1/25) (contig name: 1)...
    09:45:37.757 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (2/25) (contig name: 2)...
    09:45:38.751 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (3/25) (contig name: 3)...
    09:45:39.517 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (4/25) (contig name: 4)...
    09:45:40.065 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (5/25) (contig name: 5)...
    09:45:40.669 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (6/25) (contig name: 6)...
    09:45:41.319 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (7/25) (contig name: 7)...
    09:45:41.989 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (8/25) (contig name: 8)...
    09:45:42.453 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (9/25) (contig name: 9)...
    09:45:42.987 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (10/25) (contig name: 10)...
    09:45:43.561 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (11/25) (contig name: 11)...
    09:45:44.252 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (12/25) (contig name: 12)...
    09:45:44.968 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (13/25) (contig name: 13)...
    09:45:45.236 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (14/25) (contig name: 14)...
    09:45:45.627 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (15/25) (contig name: 15)...
    09:45:46.156 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (16/25) (contig name: 16)...
    09:45:46.716 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (17/25) (contig name: 17)...
    09:45:47.424 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (18/25) (contig name: 18)...
    09:45:47.633 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (19/25) (contig name: 19)...
    09:45:48.308 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (20/25) (contig name: 20)...
    09:45:48.629 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (21/25) (contig name: 21)...
    09:45:48.777 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (22/25) (contig name: 22)...
    09:45:49.069 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (23/25) (contig name: X)...
    09:45:49.534 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (24/25) (contig name: Y)...
    09:45:49.593 INFO gcnvkernel.postprocess.viterbi_segmentation - Segmenting contig (25/25) (contig name: MT)...
    Traceback (most recent call last):
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 884, in __call__
        self.fn() if output_subset is None else\
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 989, in rval
        r = p(n, [x[0] for x in i], o)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 978, in p
        self, node)
      File "theano/scan_module/scan_perform.pyx", line 215, in theano.scan_module.scan_perform.perform (/home/exome/.theano/compiledir_Linux-4.19--MANJARO-x86_64-with-arch-Manjaro-Linux--3.6.2-64/scan_perform/mod.cpp:2628)
    NotImplementedError: We didn't implemented yet the case where scan do 0 iteration
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/tmp/segment_gcnv_calls.4286460784218017052.py", line 73, in <module>
        viterbi_engine.write_copy_number_segments()
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 234, in write_copy_number_segments
        for segment in self._viterbi_segments_generator():
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 160, in _viterbi_segments_generator
        log_prior_c, log_trans_contig_tcc, copy_number_log_emission_contig_tc)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/gcnvkernel/models/theano_hmm.py", line 88, in perform_forward_backward
        prev_log_posterior_tc, admixing_rate, temperature)))
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 898, in __call__
        storage_map=getattr(self.fn, 'storage_map', None))
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/gof/link.py", line 325, in raise_with_op
        reraise(exc_type, exc_value, exc_trace)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/six.py", line 692, in reraise
        raise value.with_traceback(tb)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 884, in __call__
        self.fn() if output_subset is None else\
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 989, in rval
        r = p(n, [x[0] for x in i], o)
      File "/home/exome/miniconda2/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 978, in p
        self, node)
      File "theano/scan_module/scan_perform.pyx", line 215, in theano.scan_module.scan_perform.perform (/home/exome/.theano/compiledir_Linux-4.19--MANJARO-x86_64-with-arch-Manjaro-Linux--3.6.2-64/scan_perform/mod.cpp:2628)
    NotImplementedError: We didn't implemented yet the case where scan do 0 iteration
    Apply node that caused the error: forall_inplace,cpu,scan_fn}(Elemwise{minimum,no_inplace}.0, InplaceDimShuffle{0,2,1}.0, Subtensor{int64:int64:int64}.0, IncSubtensor{InplaceSet;:int64:}.0, Shape_i{0}.0)
    Toposort index: 97
    Inputs types: [TensorType(int64, scalar), TensorType(float64, 3D), TensorType(float64, matrix), TensorType(float64, matrix), TensorType(int64, scalar)]
    Inputs shapes: [(), (0, 6, 6), (0, 6), (2, 6), ()]
    Inputs strides: [(), (288, 8, 48), (48, 8), (48, 8), ()]
    Inputs values: [array(0), array([], shape=(0, 6, 6), dtype=float64), array([], shape=(0, 6), dtype=float64), 'not shown', array(6)]
    Outputs clients: [[Subtensor{int64:int64:int8}(forall_inplace,cpu,scan_fn}.0, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1})]]
    
    HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
    HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
    09:45:50.363 DEBUG ScriptExecutor - Result: 1
    09:45:50.364 INFO  PostprocessGermlineCNVCalls - Shutting down engine
    [February 4, 2019 9:45:50 AM EET] org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls done. Elapsed time: 0.49 minutes.
    Runtime.totalMemory()=3004170240
    org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
    python exited with 1
    Command Line: python /tmp/segment_gcnv_calls.4286460784218017052.py --ploidy_calls_path /mnt/E/Exomes/GATKCNVtesting_V2/CNV/GATKCNVtesting_V2_CNV-calls --model_shards /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-model --calls_shards /mnt/E/Exomes/GATKCNVtesting_V2/CNVCalls/GATK_CNV-calls --output_path /tmp/gcnv-segmented-calls2678318989462462101 --sample_index 22
            at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
            at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
            at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
            at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:151)
            at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:121)
            at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.executeSegmentGermlineCNVCallsPythonScript(PostprocessGermlineCNVCalls.java:499)
            at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.generateSegmentsVCFFileFromAllShards(PostprocessGermlineCNVCalls.java:435)
            at org.broadinstitute.hellbender.tools.copynumber.PostprocessGermlineCNVCalls.traverse(PostprocessGermlineCNVCalls.java:296)
            at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:966)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
            at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
            at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
            at org.broadinstitute.hellbender.Main.main(Main.java:291)
    (gatk) [[email protected] GATKCNVtesting_V2]$
    

    Looks like a warning for omission of MT segments is needed in the documentation or There must be some other way to implement a function to skip segments that will require 0 iterations.

    For the time being I will modify my bed file for CNV calls to omit MT.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Thanks for the heads up. Also I noticed that enrichment kit seems to affect ploidy detection for allosomal contigs. Current brand that we use almost exclusively gives the best ploidy estimation for X and Y however another brand that we used in the past (one of the most expensive ones in the market) generates a bias towards counting one extra X for all the samples we have. Luckily I already have other means to check X and Y ploidy and I can adjust contig-priors table to avoid 3rd X for females however I could not find a way to avoid extra X for male samples for the moment.

    I am using the very same bed file and same reference genome (1000g b37) for CNV calling for both sample sets but I am not sure what causes this discrepancy. I can provide more details and may provide snippets of data if the team wants to work on them.

    Thanks.

  • AdelaideRAdelaideR Unconfirmed, Member, Broadie, Moderator admin

    @SkyWarrior Feel free to send that along and I can ask the development team for more feedback on the discrepancy.

  • AdelaideRAdelaideR Unconfirmed, Member, Broadie, Moderator admin

    I am just going to close the ticket at the moment, but we can always follow up on this later.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    Accepted Answer

    Thanks @slee for the detailed response.

    My default bed file for analysis includes the MT contig as a whole and I used the very same bed file inadvertently during CollectReadCounts step. After that I realized the issue and tried excluding MT in DetermineContigPloidy Step using -XL MT but it did not work. I let the DetermineContigPloidy run with MT included. GermlineCNVCaller also ran without issues except for the warning for MT but PostProcess step generated this error and refused to output segmented VCF. Intervals vcf was generated without any issues.

    For the output folder creation. GermlineCNVCaller tries to write all the files in the folder with the prefix given. Although the root folder for the prefix named folder is present in the path and write permissions are all set GermlineCNVCaller throws an error message indicating that the prefix named folder is not available. When that folder is generated manually before running GermlineCNVCaller that error is gone and run completes properly.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    Accepted Answer

    If my memory serves me well it was the same message. I can reproduce it very quickly tomorrow and return here.

    The PostProcess error was related to an unimplemented case where iteration number equals 0.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭
    edited February 16 Accepted Answer

    Hi

    Sorry that the error message was not the same actually.

    Here is the command line

    gatk GermlineCNVCaller --java-options "-Djava.io.tmpdir=tmp" --run-mode COHORT --contig-ploidy-calls ./CNV/${PWD##*/}_CNV-calls --output CNVCalls --output-prefix GATK_CNV --annotated-intervals b37_annotated.txt --verbosity DEBUG -L $CNVREGB37 -imr OVERLAPPING_ONLY -I Wes470_counts.hdf5 -I Wes471_counts.hdf5 -I Wes472_counts.hdf5 -I Wes473_counts.hdf5 -I Wes474_counts.hdf5 -I Wes475_counts.hdf5 -I Wes476_counts.hdf5 -I Wes477_counts.hdf5 -I Wes478_counts.hdf5 -I Wes479_counts.hdf5 -I Wes480_counts.hdf5 -I Wes481_counts.hdf5 -I Wes482_counts.hdf5 -I Wes483_counts.hdf5 -I Wes484_counts.hdf5 -I Wes485_counts.hdf5 -I Wes486_counts.hdf5 -I Wes487_counts.hdf5 -I Wes488_counts.hdf5 -I Wes489_counts.hdf5 -I Wes490_counts.hdf5 -I Wes491_counts.hdf5 -I Wes492_counts.hdf5 -I Wes493_counts.hdf5 -I Wes494_counts.hdf5 -I Wes495_counts.hdf5 -I Wes496_counts.hdf5 -I Wes497_counts.hdf5 -I Wes498_counts.hdf5 -I Wes499_counts.hdf5 -I Wes500_counts.hdf5
    

    I expect the output should be in the directory named CNVCalls but when directory is not created ahead of this command I get the error message below.

    08:19:04.104 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/exome/scripts/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    08:19:04.379 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /mnt/E/Exomes/GATKCNVtesting_V2/tmp/libgkl_compression1008933078259947848.so
    08:19:06.094 INFO  GermlineCNVCaller - ------------------------------------------------------------
    08:19:06.095 INFO  GermlineCNVCaller - The Genome Analysis Toolkit (GATK) v4.1.0.0
    08:19:06.095 INFO  GermlineCNVCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
    08:19:06.095 INFO  GermlineCNVCaller - Executing as [email protected] on Linux v4.19.20-1-MANJARO amd64
    08:19:06.095 INFO  GermlineCNVCaller - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_202-b26
    08:19:06.095 INFO  GermlineCNVCaller - Start Date/Time: February 16, 2019 8:19:04 AM EET
    08:19:06.095 INFO  GermlineCNVCaller - ------------------------------------------------------------
    08:19:06.095 INFO  GermlineCNVCaller - ------------------------------------------------------------
    08:19:06.096 INFO  GermlineCNVCaller - HTSJDK Version: 2.18.2
    08:19:06.096 INFO  GermlineCNVCaller - Picard Version: 2.18.25
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.BUFFER_SIZE : 131072
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.CREATE_INDEX : false
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.CREATE_MD5 : false
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.CUSTOM_READER_FACTORY :
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072
    08:19:06.098 INFO  GermlineCNVCaller - HTSJDK Defaults.REFERENCE_FASTA : null
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    08:19:06.099 INFO  GermlineCNVCaller - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false
    08:19:06.099 DEBUG ConfigFactory - Configuration file values:
    08:19:06.104 DEBUG ConfigFactory -      gcsMaxRetries = 20
    08:19:06.104 DEBUG ConfigFactory -      gcsProjectForRequesterPays =
    08:19:06.104 DEBUG ConfigFactory -      gatk_stacktrace_on_user_exception = false
    08:19:06.104 DEBUG ConfigFactory -      samjdk.use_async_io_read_samtools = false
    08:19:06.104 DEBUG ConfigFactory -      samjdk.use_async_io_write_samtools = true
    08:19:06.104 DEBUG ConfigFactory -      samjdk.use_async_io_write_tribble = false
    08:19:06.104 DEBUG ConfigFactory -      samjdk.compression_level = 2
    08:19:06.104 DEBUG ConfigFactory -      spark.kryoserializer.buffer.max = 512m
    08:19:06.104 DEBUG ConfigFactory -      spark.driver.maxResultSize = 0
    08:19:06.104 DEBUG ConfigFactory -      spark.driver.userClassPathFirst = true
    08:19:06.104 DEBUG ConfigFactory -      spark.io.compression.codec = lzf
    08:19:06.104 DEBUG ConfigFactory -      spark.yarn.executor.memoryOverhead = 600
    08:19:06.104 DEBUG ConfigFactory -      spark.driver.extraJavaOptions =
    08:19:06.104 DEBUG ConfigFactory -      codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]
    08:19:06.104 DEBUG ConfigFactory -      spark.executor.extraJavaOptions =
    08:19:06.104 DEBUG ConfigFactory -      read_filter_packages = [org.broadinstitute.hellbender.engine.filters]
    08:19:06.104 DEBUG ConfigFactory -      annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]
    08:19:06.104 DEBUG ConfigFactory -      cloudPrefetchBuffer = 40
    08:19:06.105 DEBUG ConfigFactory -      cloudIndexPrefetchBuffer = -1
    08:19:06.105 DEBUG ConfigFactory -      createOutputBamIndex = true
    08:19:06.105 INFO  GermlineCNVCaller - Deflater: IntelDeflater
    08:19:06.105 INFO  GermlineCNVCaller - Inflater: IntelInflater
    08:19:06.105 INFO  GermlineCNVCaller - GCS max retries/reopens: 20
    08:19:06.105 INFO  GermlineCNVCaller - Requester pays: disabled
    08:19:06.105 INFO  GermlineCNVCaller - Initializing engine
    08:19:06.110 DEBUG ScriptExecutor - Executing:
    08:19:06.110 DEBUG ScriptExecutor -   python
    08:19:06.111 DEBUG ScriptExecutor -   -c
    08:19:06.111 DEBUG ScriptExecutor -   import gcnvkernel
    
    08:19:08.932 DEBUG ScriptExecutor - Result: 0
    08:19:08.932 INFO  GermlineCNVCaller - Done initializing engine
    log4j:WARN No appenders could be found for logger (org.broadinstitute.hdf5.HDF5Library).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    08:19:09.470 INFO  GermlineCNVCaller - Intervals specified...
    08:19:09.471 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
    08:19:09.471 DEBUG GenomeLocParser -  1 (249250621 bp)
    08:19:09.471 DEBUG GenomeLocParser -  2 (243199373 bp)
    08:19:09.471 DEBUG GenomeLocParser -  3 (198022430 bp)
    08:19:09.471 DEBUG GenomeLocParser -  4 (191154276 bp)
    08:19:09.471 DEBUG GenomeLocParser -  5 (180915260 bp)
    08:19:09.471 DEBUG GenomeLocParser -  6 (171115067 bp)
    08:19:09.471 DEBUG GenomeLocParser -  7 (159138663 bp)
    08:19:09.471 DEBUG GenomeLocParser -  8 (146364022 bp)
    08:19:09.471 DEBUG GenomeLocParser -  9 (141213431 bp)
    08:19:09.471 DEBUG GenomeLocParser -  10 (135534747 bp)
    08:19:09.471 DEBUG GenomeLocParser -  11 (135006516 bp)
    08:19:09.472 DEBUG GenomeLocParser -  12 (133851895 bp)
    08:19:09.472 DEBUG GenomeLocParser -  13 (115169878 bp)
    08:19:09.472 DEBUG GenomeLocParser -  14 (107349540 bp)
    08:19:09.472 DEBUG GenomeLocParser -  15 (102531392 bp)
    08:19:09.472 DEBUG GenomeLocParser -  16 (90354753 bp)
    08:19:09.472 DEBUG GenomeLocParser -  17 (81195210 bp)
    08:19:09.472 DEBUG GenomeLocParser -  18 (78077248 bp)
    08:19:09.472 DEBUG GenomeLocParser -  19 (59128983 bp)
    08:19:09.472 DEBUG GenomeLocParser -  20 (63025520 bp)
    08:19:09.472 DEBUG GenomeLocParser -  21 (48129895 bp)
    08:19:09.472 DEBUG GenomeLocParser -  22 (51304566 bp)
    08:19:09.472 DEBUG GenomeLocParser -  X (155270560 bp)
    08:19:09.472 DEBUG GenomeLocParser -  Y (59373566 bp)
    08:19:09.472 DEBUG GenomeLocParser -  MT (16569 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000207.1 (4262 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000226.1 (15008 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000229.1 (19913 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000231.1 (27386 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000210.1 (27682 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000239.1 (33824 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000235.1 (34474 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000201.1 (36148 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000247.1 (36422 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000245.1 (36651 bp)
    08:19:09.472 DEBUG GenomeLocParser -  GL000197.1 (37175 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000203.1 (37498 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000246.1 (38154 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000249.1 (38502 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000196.1 (38914 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000248.1 (39786 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000244.1 (39929 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000238.1 (39939 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000202.1 (40103 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000234.1 (40531 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000232.1 (40652 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000206.1 (41001 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000240.1 (41933 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000236.1 (41934 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000241.1 (42152 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000243.1 (43341 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000242.1 (43523 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000230.1 (43691 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000237.1 (45867 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000233.1 (45941 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000204.1 (81310 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000198.1 (90085 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000208.1 (92689 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000191.1 (106433 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000227.1 (128374 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000228.1 (129120 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000214.1 (137718 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000221.1 (155397 bp)
    08:19:09.473 DEBUG GenomeLocParser -  GL000209.1 (159169 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000218.1 (161147 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000220.1 (161802 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000213.1 (164239 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000211.1 (166566 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000199.1 (169874 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000217.1 (172149 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000216.1 (172294 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000215.1 (172545 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000205.1 (174588 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000219.1 (179198 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000224.1 (179693 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000223.1 (180455 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000195.1 (182896 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000212.1 (186858 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000222.1 (186861 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000200.1 (187035 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000193.1 (189789 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000194.1 (191469 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000225.1 (211173 bp)
    08:19:09.474 DEBUG GenomeLocParser -  GL000192.1 (547496 bp)
    08:19:09.474 DEBUG GenomeLocParser -  NC_007605 (171823 bp)
    08:19:09.474 DEBUG GenomeLocParser -  hs37d5 (35477943 bp)
    08:19:09.609 INFO  FeatureManager - Using codec BEDCodec to read file file:///home/exome/GenomeData/ExomeReg_b37_CNV.bed
    08:19:10.008 DEBUG FeatureDataSource - Cache statistics for FeatureInput /home/exome/GenomeData/ExomeReg_b37_CNV.bed:/home/exome/GenomeData/ExomeReg_b37_CNV.bed:
    08:19:10.009 DEBUG FeatureCache - Cache hit rate  was 0.00% (0 out of 0 total queries)
    08:19:10.097 INFO  IntervalArgumentCollection - Processing 103604700 bp from intervals
    08:19:10.223 INFO  GermlineCNVCaller - Reading and validating annotated intervals...
    08:19:11.051 INFO  GermlineCNVCaller - GC-content annotations for intervals found; explicit GC-bias correction will be performed...
    08:19:11.418 INFO  GermlineCNVCaller - Running the tool in the COHORT mode...
    08:19:11.419 INFO  GermlineCNVCaller - Shutting down engine
    [February 16, 2019 8:19:11 AM EET] org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller done. Elapsed time: 0.12 minutes.
    Runtime.totalMemory()=2352480256
    java.lang.IllegalArgumentException: Output directory CNVCalls does not exist.
            at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:724)
            at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.validateArguments(GermlineCNVCaller.java:361)
            at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.doWork(GermlineCNVCaller.java:281)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
            at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
            at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
            at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
            at org.broadinstitute.hellbender.Main.main(Main.java:291)
    
  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Oh I see. So if that's the intended behavior then I should make sure that my scripts are ready.

    Thanks.

  • SkyWarriorSkyWarrior TurkeyMember ✭✭✭

    Awesome news! You guys are all great!

Sign In or Register to comment.