slee ✭✭✭
About
- Username
- slee
- Joined
- Visits
- 1,218
- Last Active
- Roles
- Member, Broadie, Dev
- Points
- 204
- Badges
- 14
- Full Name
- Samuel Lee
Reactions
Comments
-
@sarawasl Unfortunately, calling germline CNVs is typically difficult without a reference cohort of samples (sequenced with similar protocols as your case samples) from which a model of systematic noise/bias can be learned. As @SkyWarrior pointed o…
-
Thanks for your suggestions, @SkyWarrior. @Yangyxt if you'd like more detailed descriptions of the quality scores, you may find the code comments (and perhaps the code itself) at https://github.com/broadinstitute/gatk/blob/master/src/main/python/or…
-
@Yangyxt the *intervals.vcf is generated by concatenating the results of running the forward-backward algorithm in each shard, while the *segments.vcf gives the single-sample Viterbi segmentation across all shards. So differences such as that you o…
-
Hi @Yangyxt, just a minimal set of the input files and the command line for the PostprocessGermlineCNVCalls step should be fine. For example, if you can reproduce the issue with just the HDF5 file from a single sample, you don't need to include fil…
-
Hi @Yangyxt, thanks for raising this issue. I don't see any obvious reason in the underlying code (which simply uses HTSJDK) that might cause this to happen. Did you merge/modify the files, as @SkyWarrior conjectured? In any case, could you provi…
-
@Yangyxt this is indeed normal. You may notice that many GATK tools perform traversals over various types of records---intervals, reads, variants, etc.---in which case the ProgressMeter provides updates on the status of the traversal. The code to …
-
@pdu this was fixed in https://github.com/broadinstitute/gatk/pull/5082 (and in any case, the warning is harmless).
-
Hi @Matthieu_M, It's probably safe to replicate the autosomal values for chr3-chr22. Hopefully your data should be discerning enough that your results are not overly sensitive on the prior. Ideally, you'd have a set of representative samples for …
-
@Yangyxt GermlineCNVCaller is designed to be scattered over the genome in multiple shards. See the tutorial posted above by @shlee and the WDLs referenced there to see how this works.
-
Thanks as always @SkyWarrior for sharing useful info with the community!
-
Hi @SkyWarrior, yes, your assessment is correct. We changed the output format of some of the files to add output of concatenated denoised copy ratios in https://github.com/broadinstitute/gatk/pull/5823, but neglected to mention that this breaks bac…
-
@jiehuang001 Those warnings do not affect the results. See https://github.com/broadinstitute/gatk/issues/3763.
-
@jonathanYu I would not recommend using gatk-protected or ACNV; these have been superseded by GATK4 and ModelSegments, which introduce many improvements. The GATK4-equivalent of the internal scripts mentioned above can be found at https://github.co…
-
Thanks for the question, @jasonbwarner! Although the current GATK somatic CNV pipeline uses relatively sophisticated methods for segmentation/modeling of CR/AF, its calling capability is somewhat limited (and is more on par with standard methods fo…
-
Oops, sorry @Tintest, didn't realize the age of the original post---but thanks for your quick response! @lakhujanivijay if you are getting an actual tool failure (as opposed to just emitted warnings), please let us know.
-
@Tintest I believe that those warnings should not cause the tool to fail. Was there any additional output in the log (it looks like the text you copied might've gotten cut off)? Any chance you would be able to share your coverage files so that we …
-
@Unguilla, a few things: 1) This workflow is not designed to detect CNVs in such small panels---it doesn't really make sense to perform PCA denoising or segmentation when you have such a small number of probes (<50) to work with. Typically we a…
-
@Elliothui it looks like you might be missing a space in your command line---perhaps -O/PANEL/hg19-try/$tsvname.tsv should be -O /PANEL/hg19-try/$tsvname.tsv?
-
@Begali The probabilities in this file should reflect your prior belief for the copy-number state of each contig, given the prevalence of aneuploidies and sex genotypes in the population. For example, the table used in the tutorial indicates that w…
-
@lakhujanivijay @Begali You should construct this file manually. You can use the file provided for the tutorial as a starting point. You may need to change contig names (e.g., if you are using a difference reference) or adjust the values for the p…
-
@lzhan140 I would use the same number of eigensamples to denoise all samples. Using more eigensamples to denoise the tumor might indeed explain the discrepancy in chr19. Your data looks relatively clean, so you might even want to try using zero ei…
-
Hi @lzhan140, When you don't observe an elbow in the scree plot, it typically means that your data is relatively isotropic or spherical in data space (i.e., standardized-coverage space). This means you should be able to get a good result with only…
-
Hi @lzhan140, I think you meant to tag me, instead of @shlee (who no longer works at the Broad). You can use a PoN to denoise a normal sample that was included in it, but you need to be careful not to use too many eigensamples---otherwise you will…
-
@lakhujanivijay CollectFragmentCounts has been replaced by CollectReadCounts since that post was written. Note the link to the other thread above if you are interested in the priors-table file; you don't need to collect counts to construct this fil…
-
@jml96 As I suspected, that is a counts HDF5 file. It's created by the WDL task CollectCounts, which runs the GATK tool CollectReadCounts on a single BAM and produces a corresponding HDF5 file that represents the counts in each genomic bin. What y…
-
@jml96, can you verify that you are passing the panel of normals HDF5 file created by cnv_somatic_panel_workflow.wdl as input to the --count-panel-of-normals? If you are passing an HDF5 file that does not have the fields expected for a PoN (includi…
-
@lakhujanivijay looks like you are following the gCNV tutorial. I think the command you want is: gatk GermlineCNVCaller --run-mode COHORT -L scatter-sm/twelve_1of2.interval_list -I cvg/HG00096.tsv -I cvg/HG00268.tsv -I cvg/HG00419.tsv -I cvg/HG007…
-
@lzhan140 you should not mix WES and WGS samples, as this is not likely to yield good PCA denoising results. Samples used for the PoN should ideally be representative of the same sequencing protocol, as should your case samples; i.e., all samples s…
-
Hi @lzhan140, You are right that the workflow does not yet have a step to remove germline CNVs detected in the matched normal. You might be interested in the unsupported WDLs at https://github.com/broadinstitute/gatk/tree/master/scripts/unsupporte…
-
@ahda FilterIntervals is intended primarily for use in the germline CNV workflow, since CreateReadCountPanelOfNormals in the somatic CNV workflow already does some filtering steps. However, if you'd like to use intervals produced by FilterIntervals…
-
@ahda It looks like you might be incorrectly passing the output of PreprocessIntervals to the --annotated-intervals argument; if so, you should instead pass the output of AnnotatedIntervals.
-
@dcampo looks like you are passing a denoisedCR.tsv file to --normal-allelic-counts. You should be passing the result of running CollectAllelicCounts on the matched normal to this argument. Hope that resolves the issue!
-
@dcampo I didn't run into any issue with parsing your snippet when running GATK 4.1.2.0 ModelSegments. Can you post your command line and version number? EDIT: It occurs to me that one possible error is that you are inadvertently passing some othe…
-
@mtkk94 In order to parse the copy-number states represented in the table, the code looks for the text PLOIDY_PRIOR_ and then reads the integer that follows in each tab-separated column header. However, if your column headers are not tab separated,…
-
@mtkk94 it looks like your ploidy table might not be a properly formatted TSV file. Can you make sure that all columns are separated by tabs (in particular, between PLOIDY_PRIOR_0, PLOIDY_PRIOR_1, etc.)?
-
Thanks for reporting back, @johnma. It's been some time since I looked at the old beta code, but I don't believe that the target names were ever used (although there is probably a check that they are all unique). So inserting dummy target names fo…
-
Yes, the sample name is taken from the SM tag in the BAM header.
-
From the error message, it looks like you have multiple samples with the sample name "unknown"; unique sample names are required when running in cohort mode.
-
@johnma, the output denoised log2 copy-ratio values are essentially identical to those output by NormalizeSomaticReadCounts (down to levels of ~1E-16) if appropriate parameters are selected during the coverage collection step (recall that the --tran…
-
Hi @ngerald, I think if you specify --contig-ploidy-calls contig_ploidy_out_rerun/201to209-calls/ rather than --contig-ploidy-calls contig_ploidy_out_rerun/201to209-calls/SAMPLE_0, you should be in business. I realize that it can be a little confu…
-
Hi @ngerald, on a recent benchmarking run of 50 WES samples over ~220k target intervals, I ran with 45 shards of ~5k intervals each. Each shard was run on a GCE n1-standard-1 VM with 3.75GB memory, yielding a cost of ~0.5 cents per sample. With 20…
-
Hi @ngerald, @bhanuGandham's suggestion about sharding should solve your memory issues, which is most likeliy causing the GermlineCNVCaller step to crash. However, your anomalous ploidy calls need to be addressed upstream in the DetermineGermlineC…
-
See the PR at https://github.com/broadinstitute/gatk/pull/5976.
-
Hi @MatthewP, * NUM_POINTS_COPY_RATIO gives the number of bins that contribute to each segment. This is analogous to the Num_Probes field for segment files generated from arrays. For the IGV-compatible *.cr.igv.seg files output by ModelSegments, …
-
@MatthewP I'd suggest taking a look at some of the other posts in this thread, if you haven't already. You can use SelectVariants to subset common SNPs from gnomAD that lie in your intervals of interest, further filtering by desired allele frequenc…
-
I suspect this may be related to https://github.com/broadinstitute/gatk/issues/5893, which was fixed after the 4.1.2.0 release (however, see https://github.com/broadinstitute/gatk/issues/5945). @mullinyu can you try running PostprocessGermlineCNVCa…
-
@jml96 CreateReadCountPanelOfNormals takes as input read-count files generated by CollectReadCounts, not BAMs. You may find the tutorial at https://software.broadinstitute.org/gatk/documentation/article?id=11682 helpful.
-
@obigbando CreateReadCountPanelOfNormals uses Spark MLlib to perform PCA. However, it's probably fine to simply run the tool in local mode, as the computational requirements are modest even for building high resolution WGS PoNs (with say, ~30M bins…
-
OK, thanks @mtkk94. Assuming that you're using a properly formatted TSV (perhaps check for missing tabs or accidental use of other whitespace) and that your contig names are all included correctly, I'm not sure why you should be getting that error.…
-
@mtkk94 looks like there might be a problem with your contig-ploidy priors (/mnt/data/smb_share/Mandal_project/Aim1_10PCa10Con_Data_Mayo/gatk/contigPloidyPriorsTable.tsv). Do you mind posting the contents of that file?
-
@pateln13 @UniCorn please take a look at the documentation for DenoiseReadCounts (https://software.broadinstitute.org/gatk/documentation/tooldocs/4.1.2.0/org_broadinstitute_hellbender_tools_copynumber_DenoiseReadCounts.php), which includes the funct…
-
@rsinghania see above---we list all possible ALT alleles, but the call made is specified by GT. So GT=0 is REF, GT=1 is DEL, and GT=2 is DUP. The CN annotation gives the absolute copy number and the qualities give various posterior probabilities a…
-
@dislek you might find the main gCNV tutorial at https://gatkforums.broadinstitute.org/gatk/discussion/11684 useful, in addition to the supplementary notebooks that @bhanuGandham linked. Hopefully, this main tutorial makes the intended workflow mor…
-
Thanks for fielding this question, @SkyWarrior! I've filed an issue to fix this here: https://github.com/broadinstitute/gatk/issues/5852
-
Great to hear that, @WimS, thanks for the feedback!
-
@SkyWarrior note that I changed the behavior of the CNV tools in the 4.1.1.0 release so that they attempt to create output directories if they don't exist. Should be backwards compatible with scripts expecting the previous behavior, but just though…
-
Hi @dislek, Yes, as alluded to in the GermlineCNVCaller tool documentation (https://software.broadinstitute.org/gatk/documentation/tooldocs/current/org_broadinstitute_hellbender_tools_copynumber_GermlineCNVCaller.php), the tool is intended to be ru…
-
Hi @WimS, I see---note that the hierarchical HMM used in gCNV does try to encourage shared regions of common/rare CNV activity, but it doesn't guarantee shared breakpoints in the way you'd seem to want. You may have to deal with statistical noise …
-
@WimS, thanks for bringing that to our attention. I've filed an issue to amend this at https://github.com/broadinstitute/gatk/issues/5809. I don't believe you should get any errors when merging the *segments.vcf files, but let me know if this is t…
-
Hi @Chip, That tool is quite outdated, but if you're curious about the model used, see Sec. IIK at https://github.com/broadinstitute/gatk/blob/4.1.0.0/docs/CNVs/CNV-methods.pdf Note that these notes will be moved to an archived location in the rep…
-
If you don't want to reheader your BAMs, you could insert the correct sample names into the count files produced by the CollectReadCounts step. There are python libraries that you can use to edit HDF5 files (e.g., h5py), but you might find it easie…
-
Hi @DavidNix, Apologies, just now seeing your comment regarding the presence of germline events from the matched normal in the tumor segments. We are indeed aware of this issue and plan to address it with downstream tools that either implement a f…
-
Hi @lishiyong, That error message suggests that either 1) the sample name, and/or 2) the sequence dictionary does not match in the header of your 18060701T.10K.denoisedCR.tsv and 18060701T.allelicCounts.tsv files. Can you confirm? If the sequenc…
-
@dislek, it looks like you have multiple samples with the name "none": gcnvkernel.structs.metadata.SampleAlreadyInCollectionException: Sample "none" already has coverage metadata annotations. The gCNV tools currently expect that…
-
@alongalor please see the instructions for installing the conda environment in the "Python dependencies" section at: https://github.com/broadinstitute/gatk/blob/master/README.md#requirements The file gatkcondaenv.yml.template is simply a …
-
@JiantaoShi thanks again for sharing those files. I think that your PoN creation failed due to incorrect linking of the BLAS libraries (see https://gatkforums.broadinstitute.org/gatk/discussion/12537/get-error-when-using-createreadcountpanelofnorma…
-
Hi @JiantaoShi, Thanks for sharing your files. I was able to successfully build a PoN from your samples and use it with DenoiseReadCounts to denoise them. I could also successfully view the PoN with hdfview. Could you share the log generated fro…
-
Hi @manolis, I think in your original command line for GermlineCNVCaller, you needed to pass the -calls directory to the --contig-ploidy-calls argument (i.e., ${fol8}/"Karyo"/"Karyo_cohort-calls", rather than just ${fol8}/"…
-
@NawarDalila glad you were able to resolve the issue and thanks for sharing your findings!
-
@dislek I would recommend that you double check that your TSV is formatted correctly (with tab-separated values), that your contig names match those used in your reference (e.g., 1 vs. chr1), and that your read-count files do not cover contigs that …
-
I'd recommend just taking >50 blood normals from relatively quiet tumor types (we typically use THCA) and creating a PoN with your desired bin length. Typically only a few (<3) principal components are needed to achieve a good denoising resul…
-
@kimy I responded to you over in the issue thread.
-
@xinzheng it sounds like you might not have enough samples to train a good model. Ideally, all of your samples would be from the same batch, and you would have at least several tens, if not 50-100. If you know your samples are from multiple batche…
-
@xinzheng I think you are looking at the *intervals.vcf, which reports per-bin quantities that may be noisy. You will want to look at the *segments.vcf, which gives Viterbi-smoothed segment-level calls along with various quality scores that may be …
-
Sorry, just to be clear---I was referring to scattering across shards of intervals, not samples, to bring down memory requirements. The number of bins in each shard x the number of samples is what determines the memory requirement for each shard. …
-
@crackiee that looks like an out-of-memory error to me. It looks like you are running a decent number (162) of samples. Depending on the number of bins in /home/wd256/references/capture_coverage/xGen_preprocessed.interval_list, you may need to sca…
-
@xinzheng @SkyWarrior along with absolute copy-number calls (CN), gCNV reports the symbolic ALT alleles <DEL>, <DUP>. So GT=1 represents any deletion, which may be either CN=0 or 1. @xinzheng if you are getting a lot of questionable de…
-
Thanks for reproducing that, @SkyWarrior. That's actually intended behavior---many of the CNV tools which take an output directory as one of the parameters perform a check to make sure that the directory exists beforehand. The fact that this check…
-
OK, thanks! I think you should be able to exclude MT at the DetermineGermlineContigPloidy step as you describe, but I can look into this to see if I can reproduce your issue. In the meantime, you might have more success excluding it at the FilterI…
-
Actually, @SkyWarrior, if you could provide more details on the output folder creation issue, that would be helpful. You mentioned previously that this issue only affected GermlineCNVCaller and not DetermineGermlineContigPloidy. Is that still the …
-
The model is generated by running the tool in COHORT mode on a set of training samples. This will produce calls for those training samples as well as a model that can later be used for calling additional samples in CASE mode. The calls produced fo…
-
@SkyWarrior Thanks for your detailed feedback, and for raising various issues in this thread! A few responses: -Requiring that -imr OVERLAPPING_ONLY be input manually is a limitation of the GATK engine. See https://github.com/broadinstitute/gatk/…
-
Hi @stefan_stojanovic, Glad you were able to find a workaround. My guess is that you may have had a lot of zero-count bins with the smaller bin size, which may not have been filtered out correctly by the panel of normals, resulting in funky arithm…
-
Hi @FPBarthel, Just dropping in on the forum---you might be interested in this recent PR: https://github.com/broadinstitute/gatk/pull/5556 Thanks, Samuel
-
Hi @iremsarihan, See above---you do not want to pass the list generated by PreprocessIntervals to CollectAllelicCounts. Rather, you want to pass a list of sites of common variation in your exome target regions, which can be constructed using e.g. …
-
@dannykwells, great to hear that you're finding the CNV pipeline useful! The current pipeline includes a relatively naive caller (CallCopyRatioSegments). This caller uses only the segmented copy-ratio estimates from ModelSegments to perform simple…
-
Hi @ehodis, @ehodis, and @registered_user, For running the Somatic-CNVs-GATK4 workflow on WGS data, you can start by simply passing a .list or .intervals file containing all of the autosomes (1, 2, 3, etc.) This will run the CNV workflow on the au…
-
@xiongssg great to hear that the script worked for your sample. I would recommend experimenting with your coverage-collection bin size or increasing number-of-changepoints-penalty-factor in ModelSegments to smooth out your noisy samples. As I comm…
-
@alphahmed if you do not scatter across genomic chunks, then you will only have a single model tar file covering the entire genome, which you should be able to use as input to the case-mode WDL. gcnv_model_tars will then be an array with only a sin…
-
Hi @alphahmed, We typically scatter across genomic chunks, not chunks of samples. If you study the WDL, you'll see that this is accomplished by using the ScatterIntervals task to break the intervals for coverage collection into chunks containing a…
-
@Muyiyuan, Can you give us some more information about your runtime environment and how you installed the BLAS libraries? Are you sure you pointed to the appropriate path using the export LD_PRELOAD=... statement? Also, on an unrelated topic, loo…
-
There is a PR open to address this at https://github.com/broadinstitute/gatk/pull/5040.
-
Hi @jreeve, See this thread: https://gatkforums.broadinstitute.org/gatk/discussion/12078/gatk-4-4-docker-image-missing-dependancies This package was not installed in the latest Docker image. There is a PR to fix this open at https://github.com/br…
-
Hi @booby35 and @Crisovilo, The plot thickens---see the comments in the PR here: https://github.com/broadinstitute/gatk/pull/5026 Thanks, Samuel
-
Hi @booby35 and @Crisovilo, Looks like I'm to blame---I inadvertently removed this dependency some time ago. I've filed an issue to add it back: https://github.com/broadinstitute/gatk/issues/5022 Thanks for catching this! Thanks, Samuel
-
@sutturka If all of your normal and tumor samples were sequenced together or with the same protocol, I would rather use all of your normal samples to build a single PoN to denoise all of your tumor samples. The idea of the PoN in the CNV workflow i…
-
Hi @ekofman, The ModelSegments pipeline replaces the old CNV + AllelicCNV pipeline from 4.beta.6 and previous versions. The output of the DenoiseReadCounts step is analogous to the "tangent-normalized" output of the beta pipeline. Note …
-
Apologies for the confusion, @sutturka. To clarify, the tool (CalculateTargetCoverage) that @ekofman was originally asking about was used to collect coverage for WES CNV analyses in versions of the pipeline prior to and including 4.beta.6. These o…
-
@ekofman, You should not use CNV workflows from 4.beta.6 or previous releases if you wish to run on WGS; those workflows are more suited for WES. The CNV workflow has been significantly reworked in the current release to be scalable and performant…
-
Hi @Tintest, You should be able to infer the parameters for the GermlineCNVCaller command line from the JSON config above. The relevant entries will start with CNVGermlineCohortWorkflow.gcnv_. For example, CNVGermlineCohortWorkflow.gcnv_adamax_be…