Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

Stage2 and Stage3 error in CNVDiscoveryPipeline.

SunhyeSunhye KoreaMember
edited June 2015 in GenomeSTRiP

Hi,
Thanks to Bob, I have solved the prior problem about SGE.
http://gatkforums.broadinstitute.org/discussion/5631/genomestrip-2-0-dont-have-anything-result-about-cnvdiscoverypipeline#latest
However, I encountered another error.
my script ,
java -cp ${classpath} ${mx} \
org.broadinstitute.gatk.queue.QCommandLine \
-jobNative "-v SV_DIR" \
-S ${SV_DIR}/qscript/discovery/cnv/CNVDiscoveryPipeline.q \
-S ${SV_DIR}/qscript/SVQScript.q \
-gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
-cp ${classpath} \
-configFile ${SV_DIR}/conf/genstrip_parameters.txt \
-tempDir ${SV_TMPDIR} \
-R ${bundle}/human_g1k_v37.chr20.fasta \
-runDirectory ${runDir} \
-md ${runDir}/metadata \
-jobLogDir ${runDir}/logs \
-ploidyMapFile ${bundle}/human_g1k_v37.ploidymap.txt \
-genderMapFile ${bundle}/KPGP-00009_gender.map \
-genomeMaskFile ${bundle}/human_g1k_v37.svmask.chr20.fasta \
-tilingWindowSize 1000 \
-tilingWindowOverlap 500 \
-maximumReferenceGapLength 1000 \
-boundaryPrecision 100 \
-minimumRefinedLength 500 \
-produceAuxiliaryFiles \
-I ${bamList} \
-jobRunner Drmaa \
-gatkJobRunner Drmaa \
-run \
|| exit 1
Below error message,

INFO  13:13:44,182 FunctionEdge - Starting:  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage3.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVQScript.q'  '-gatk' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs'  '-memLimit' '2.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '-v SV_DIR'  -run  '-sequenceName' '20'  '-runDirectory' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20'  '-sentinelFile' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_sentinel_files/stage_3_seq_20.sent'  --disableJobReport  '-configFile' '/BiO/psh/tools/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  '-ploidyMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt'  '-genomeMaskFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta'  '-genderMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/sample_gender.map'  '-md' '/BiO/psh/Project/SVtest/test2_sample_chr20/metadata'  -disableGATKTraversal  '-I' '/BiO/psh/Project/SVtest/test2_sample_chr20/bam_headers/merged_headers.bam'  '-vcf' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf'  '-mergedVcfFile' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.sites.vcf'  '-duplicateScoreThresholdMax' '0'  
INFO  13:13:44,183 FunctionEdge - Output written to /BiO/psh/Project/SVtest/test2_sample_chr20/logs/CNVDiscoveryPipeline-5.out 
INFO  13:13:44,194 DrmaaJobRunner - Submitted job id: 21217 
INFO  13:13:44,194 QGraph - 9 Pend, 1 Run, 0 Fail, 4 Done 
ERROR 13:18:43,916 FunctionEdge - Error:  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage3.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVQScript.q'  '-gatk' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs'  '-memLimit' '2.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '-v SV_DIR'  -run  '-sequenceName' '20'  '-runDirectory' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20'  '-sentinelFile' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_sentinel_files/stage_3_seq_20.sent'  --disableJobReport  '-configFile' '/BiO/psh/tools/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  '-ploidyMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt'  '-genomeMaskFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta'  '-genderMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/sample_gender.map'  '-md' '/BiO/psh/Project/SVtest/test2_sample_chr20/metadata'  -disableGATKTraversal  '-I' '/BiO/psh/Project/SVtest/test2_sample_chr20/bam_headers/merged_headers.bam'  '-vcf' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf'  '-mergedVcfFile' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.sites.vcf'  '-duplicateScoreThresholdMax' '0'  
ERROR 13:18:43,985 FunctionEdge - Contents of /BiO/psh/Project/SVtest/test2_sample_chr20/logs/CNVDiscoveryPipeline-5.out:
INFO  13:15:53,479 QScriptManager - Compiling 4 QScripts 
INFO  13:18:11,506 QScriptManager - Compilation complete 
INFO  13:18:11,783 HelpFormatter - ---------------------------------------------------------------------- 
INFO  13:18:11,784 HelpFormatter - Queue v3.3.GS2-0-g7ad6c61, Compiled 2015/04/22 10:24:33 
INFO  13:18:11,784 HelpFormatter - Copyright (c) 2012 The Broad Institute 
INFO  13:18:11,784 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
INFO  13:18:11,786 HelpFormatter - Program Args: -cp /BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar -S /BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage3.q -S /BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q -S /BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q -S /BiO/psh/tools/svtoolkit/qscript/SVQScript.q -gatk /BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar -jobLogDir /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs -memLimit 2.0 -jobRunner Drmaa -gatkJobRunner Drmaa -jobNative -v SV_DIR -run -sequenceName 20 -runDirectory /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20 -sentinelFile /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_sentinel_files/stage_3_seq_20.sent --disableJobReport -configFile /BiO/psh/tools/svtoolkit/conf/genstrip_parameters.txt -R /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta -ploidyMapFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt -genomeMaskFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta -genderMapFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/sample_gender.map -md /BiO/psh/Project/SVtest/test2_sample_chr20/metadata -disableGATKTraversal -I /BiO/psh/Project/SVtest/test2_sample_chr20/bam_headers/merged_headers.bam -vcf /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf -mergedVcfFile /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.sites.vcf -duplicateScoreThresholdMax 0 
INFO  13:18:11,787 HelpFormatter - Executing as [email protected] on Linux 3.16.0-031600-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17. 
INFO  13:18:11,787 HelpFormatter - Date/Time: 2015/06/05 13:18:11 
INFO  13:18:11,788 HelpFormatter - ---------------------------------------------------------------------- 
INFO  13:18:11,788 HelpFormatter - ---------------------------------------------------------------------- 
INFO  13:18:11,803 QCommandLine - Scripting CNVDiscoveryStage3 
INFO  13:18:12,055 QCommandLine - Added 3 functions 
INFO  13:18:12,056 QGraph - Generating graph. 
INFO  13:18:12,086 QGraph - Running jobs. 
INFO  13:18:12,271 FunctionEdge - Starting:  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-jar' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-T' 'VariantFiltration'  '-V' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf'  '-o' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.input.vcf'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  -filterName ALIGNLENGTH -filter "GSELENGTH < 200" -filterName CLUSTERSEP -filter "GSCLUSTERSEP == \"NA\" || GSCLUSTERSEP <= 2.0" -filterName GTDEPTH -filter "GSM1 == \"NA\" || GSM1 <= 0.5 || GSM1 >= 2.0" -filterName NONVARIANT -filter "GSNONVARSCORE != \"NA\" && GSNONVARSCORE >= 13.0" -filterName INBREEDINGCOEFF -filter "GLINBREEDINGCOEFF != \"NA\" && GLINBREEDINGCOEFF < -0.15" 
INFO  13:18:12,273 FunctionEdge - Output written to /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs/CNVDiscoveryStage3-1.out 
INFO  13:18:12,310 DrmaaJobRunner - Submitted job id: 21218 
INFO  13:18:12,312 QGraph - 2 Pend, 1 Run, 0 Fail, 0 Done 
ERROR 13:19:15,474 FunctionEdge - Error:  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-jar' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-T' 'VariantFiltration'  '-V' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf'  '-o' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.input.vcf'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  -filterName ALIGNLENGTH -filter "GSELENGTH < 200" -filterName CLUSTERSEP -filter "GSCLUSTERSEP == \"NA\" || GSCLUSTERSEP <= 2.0" -filterName GTDEPTH -filter "GSM1 == \"NA\" || GSM1 <= 0.5 || GSM1 >= 2.0" -filterName NONVARIANT -filter "GSNONVARSCORE != \"NA\" && GSNONVARSCORE >= 13.0" -filterName INBREEDINGCOEFF -filter "GLINBREEDINGCOEFF != \"NA\" && GLINBREEDINGCOEFF < -0.15" 
ERROR 13:19:16,175 FunctionEdge - Contents of /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs/CNVDiscoveryStage3-1.out:
INFO  13:18:51,065 HelpFormatter - --------------------------------------------------------------------------- 
INFO  13:18:51,253 HelpFormatter - The Genome Analysis Toolkit (GATK) v<unknown>, Compiled 2015/04/22 10:24:33 
INFO  13:18:51,254 HelpFormatter - Copyright (c) 2010 The Broad Institute 
INFO  13:18:51,254 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
INFO  13:18:51,265 HelpFormatter - Program Args: -T VariantFiltration -V /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf -o /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.input.vcf -R /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta -filterName ALIGNLENGTH -filter GSELENGTH < 200 -filterName CLUSTERSEP -filter GSCLUSTERSEP == "NA" || GSCLUSTERSEP <= 2.0 -filterName GTDEPTH -filter GSM1 == "NA" || GSM1 <= 0.5 || GSM1 >= 2.0 -filterName NONVARIANT -filter GSNONVARSCORE != "NA" && GSNONVARSCORE >= 13.0 -filterName INBREEDINGCOEFF -filter GLINBREEDINGCOEFF != "NA" && GLINBREEDINGCOEFF < -0.15 
INFO  13:18:51,270 HelpFormatter - Executing as [email protected] on Linux 3.16.0-031600-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17. 
INFO  13:18:51,271 HelpFormatter - Date/Time: 2015/06/05 13:18:51 
INFO  13:18:51,271 HelpFormatter - --------------------------------------------------------------------------- 
INFO  13:18:51,271 HelpFormatter - --------------------------------------------------------------------------- 
INFO  13:18:55,940 GATKRunReport - Uploaded run statistics report to AWS S3 
##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version <unknown>): 
##### ERROR
##### ERROR This means that one or more arguments or inputs in your command are incorrect.
##### ERROR The error message below tells you what is the problem.
##### ERROR
##### ERROR If the problem is an invalid argument, please check the online documentation guide
##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
##### ERROR
##### ERROR Visit our website and forum for extensive documentation and answers to 
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
##### ERROR
##### ERROR MESSAGE: Couldn't read file /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf because file does not exist
##### ERROR ------------------------------------------------------------------------------------------ 
INFO  13:19:16,176 QGraph - Writing incremental jobs reports... 
INFO  13:19:16,177 QGraph - 2 Pend, 0 Run, 1 Fail, 0 Done 
INFO  13:19:16,180 QCommandLine - Writing final jobs report... 
INFO  13:19:16,180 QCommandLine - Done with errors 
INFO  13:19:16,183 QGraph - ------- 
INFO  13:19:16,185 QGraph - Failed:   'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-jar' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-T' 'VariantFiltration'  '-V' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf'  '-o' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.input.vcf'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  -filterName ALIGNLENGTH -filter "GSELENGTH < 200" -filterName CLUSTERSEP -filter "GSCLUSTERSEP == \"NA\" || GSCLUSTERSEP <= 2.0" -filterName GTDEPTH -filter "GSM1 == \"NA\" || GSM1 <= 0.5 || GSM1 >= 2.0" -filterName NONVARIANT -filter "GSNONVARSCORE != \"NA\" && GSNONVARSCORE >= 13.0" -filterName INBREEDINGCOEFF -filter "GLINBREEDINGCOEFF != \"NA\" && GLINBREEDINGCOEFF < -0.15" 
INFO  13:19:16,185 QGraph - Log:     /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs/CNVDiscoveryStage3-1.out 
INFO  13:19:16,186 QCommandLine - Script failed: 2 Pend, 0 Run, 1 Fail, 0 Done  
INFO  13:18:43,987 QGraph - Writing incremental jobs reports... 
INFO  13:18:43,987 QJobsReporter - Writing JobLogging GATKReport to file /BiO/psh/Project/SVtest/CNVDiscoveryPipeline.jobreport.txt 
INFO  13:18:44,000 QGraph - 9 Pend, 0 Run, 1 Fail, 4 Done 
INFO  13:18:44,002 QCommandLine - Writing final jobs report... 
INFO  13:18:44,002 QJobsReporter - Writing JobLogging GATKReport to file /BiO/psh/Project/SVtest/CNVDiscoveryPipeline.jobreport.txt 
INFO  13:18:44,015 QJobsReporter - Plotting JobLogging GATKReport to file /BiO/psh/Project/SVtest/CNVDiscoveryPipeline.jobreport.pdf 
WARN  13:18:46,025 RScriptExecutor - RScript exited with 1. Run with -l DEBUG for more info. 
INFO  13:18:46,136 QCommandLine - Done with errors 
INFO  13:18:46,138 QGraph - ------- 
INFO  13:18:46,542 QGraph - Failed:   'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage3.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVQScript.q'  '-gatk' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/logs'  '-memLimit' '2.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '-v SV_DIR'  -run  '-sequenceName' '20'  '-runDirectory' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20'  '-sentinelFile' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_sentinel_files/stage_3_seq_20.sent'  --disableJobReport  '-configFile' '/BiO/psh/tools/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  '-ploidyMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt'  '-genomeMaskFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta'  '-genderMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/sample_gender.map'  '-md' '/BiO/psh/Project/SVtest/test2_sample_chr20/metadata'  -disableGATKTraversal  '-I' '/BiO/psh/Project/SVtest/test2_sample_chr20/bam_headers/merged_headers.bam'  '-vcf' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.annotated.vcf'  '-mergedVcfFile' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage3/seq_20/seq_20.merged.sites.vcf'  '-duplicateScoreThresholdMax' '0'  
INFO  13:18:46,543 QGraph - Log:     /BiO/psh/Project/SVtest/test2_sample_chr20/logs/CNVDiscoveryPipeline-5.out 

In Stage2 , CNVDiscoveryStage2.q didn't create seq_20.genotypes.annotated.vcf and it did not create any files.

ls test2_sample_chr20/cnv*/*/

test2_sample_chr20/cnv_stage1/seq_20/:
logs  seq_20.sites.vcf  seq_20.sites.vcf.idx

test2_sample_chr20/cnv_stage2/seq_20/:
eval  logs

test2_sample_chr20/cnv_stage3/seq_20/:
logs

Why don't the results create anyfiles in Stage2?
What can I do for resolving this problem ?
and What is the difference between "FunctionEdge - Error" and "FunctionEdge - Failed" ?

Comments

  • bhandsakerbhandsaker Member, Broadie ✭✭✭✭

    You are right that you need to back up and investigate what went wrong in stage 2.
    You should ensure that cnv_stage1/seq_20/seq_20.sites.vcf contains a large list of sites (windows).
    Then I would consult the log files in cnv_stage2/seq_20/logs.

    You should be aware that for performance reasons the pipeline creates sentinel files in cnv_sentinel_files/stage_n_seq_X.sent, normally only when the stage completes successfully.
    If you want to force a stage to rerun, you need to delete both the sentinel file and .stage_n_seq_X.sent.done.

  • SunhyeSunhye KoreaMember
    edited June 2015

    Hi Bob,
    I confirmed seq_20.sites.vcf file.
    In my cnv_stage1/seq_20/seq20.sites.vcf file,

    ##fileformat=VCFv4.1
    ##ALT=<ID=CNV,Description="Copy number polymorphism">
    ##ALT=<ID=DEL,Description="Deletion">
    ##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
    ##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
    ##INFO=<ID=END,Number=1,Type=Integer,Description="End coordinate of this variant">
    ##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
    ##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
    ##INFO=<ID=IMPRECISE,Number=0,Type=Flag,Description="Imprecise structural variation">
    ##INFO=<ID=NOVEL,Number=0,Type=Flag,Description="Indicates a novel structural variation">
    ##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
    ##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
    ##fileDate=20150605
    ##source=GenomeSTRiP_v2.00
    #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
    20      60000   CNV_20_60000_61000      N       <CNV>   .       .       END=61000;SVTYPE=CNV
    20      60500   CNV_20_60500_61760      C       <CNV>   .       .       END=61760;SVTYPE=CNV
    20      61000   CNV_20_61000_62268      T       <CNV>   .       .       END=62268;SVTYPE=CNV
    20      61760   CNV_20_61760_62862      G       <CNV>   .       .       END=62862;SVTYPE=CNV
    20      62268   CNV_20_62268_63375      C       <CNV>   .       .       END=63375;SVTYPE=CNV
    

    I think that the seq20_sites.vcf of the cnv_stage1's result don't have any problem.
    Then, I tried to confirem the log file to cnv_stage2 / seq_20 / log but there was no file.
    So, I checked different log file about Stage2 in test2_sample_chr20/logs.

    In CNVDiscoveryPipeline-4.out

    INFO  14:05:43,395 QScriptManager - Compiling 4 QScripts 
    INFO  14:06:13,801 QScriptManager - Compilation complete 
    INFO  14:06:24,402 HelpFormatter - ---------------------------------------------------------------------- 
    INFO  14:06:24,403 HelpFormatter - Queue v3.3.GS2-0-g7ad6c61, Compiled 2015/04/22 10:24:33 
    INFO  14:06:24,404 HelpFormatter - Copyright (c) 2012 The Broad Institute 
    INFO  14:06:24,404 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
    INFO  14:06:24,405 HelpFormatter - Program Args: -cp /BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar -S /
    BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q -S /BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q -S /BiO/psh/tools/svtoolkit/qscript/discovery/cnv/CNVDisco
    veryGenotyper.q -S /BiO/psh/tools/svtoolkit/qscript/SVQScript.q -gatk /BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar -jobLogDir /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_2
    0/logs -memLimit 2.0 -jobRunner Drmaa -gatkJobRunner Drmaa -jobNative -v SV_DIR -run -sequenceName 20 -runDirectory /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20 -sentinelFile /BiO
    /psh/Project/SVtest/test2_sample_chr20/cnv_sentinel_files/stage_2_seq_20.sent --disableJobReport -configFile /BiO/psh/tools/svtoolkit/conf/genstrip_parameters.txt -R /BiO/psh/tools/svtoolkit/bund
    le/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta -ploidyMapFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt -ge
    nomeMaskFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta -genderMapFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1
    000G_phase1/Chr20/sample_gender.map -md /BiO/psh/Project/SVtest/test2_sample_chr20/metadata -disableGATKTraversal -I /BiO/psh/Project/SVtest/test2_sample_chr20/bam_headers/merged_headers.
    bam -vcf /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage1/seq_20/seq_20.sites.vcf -genotypingParallelRecords 1000 
    INFO  14:06:24,406 HelpFormatter - Executing as [email protected] on Linux 3.16.0-031600-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17. 
    INFO  14:06:24,406 HelpFormatter - Date/Time: 2015/06/05 14:06:24 
    INFO  14:06:24,407 HelpFormatter - ---------------------------------------------------------------------- 
    INFO  14:06:24,407 HelpFormatter - ---------------------------------------------------------------------- 
    INFO  14:06:24,421 QCommandLine - Scripting CNVDiscoveryStage2 
    INFO  14:06:24,599 QCommandLine - Added 4 functions 
    INFO  14:06:24,600 QGraph - Generating graph. 
    INFO  14:06:24,629 QGraph - Running jobs. 
    INFO  14:06:28,379 FunctionEdge - Starting:  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Projec
    t/SVtest/tmpdir'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/BiO/psh/tools/svtoolki
    t/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-skipAnnotator' 'CNQuality'  '-
    cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVGenotyper
    2.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVQScript.q'  '-gatk' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/logs'  '-memLimit' '2.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '-v SV_DIR'  -run  --disableJobReport  '-configFile' '/BiO/psh/tools/svtoolkit/conf/genstrip_parameters.txt'  '-P' 'depth.readCountCacheIgnoreGenomeMask:true'  '-P' 'genotyping.modules:depth'  '-R' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta'  '-genomeMaskFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta'  '-ploidyMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt'  '-genderMapFile' '/BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/sample_gender.map'  '-md' '/BiO/psh/Project/SVtest/test2_sample_chr20/metadata'  -disableGATKTraversal  '-vcf' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage1/seq_20/seq_20.sites.vcf'  '-parallelRecords' '1000'  '-I' '/BiO/psh/Project/SVtest/test2_sample_chr20/bam_headers/merged_headers.bam'  '-O' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/seq_20.genotypes.vcf'  '-runDirectory' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20'  '-duplicateScoreThreshold' '0.0'  
    INFO  14:06:28,379 FunctionEdge - Output written to /BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/logs/CNVDiscoveryStage2-1.out 
    INFO  14:06:33,091 DrmaaJobRunner - Submitted job id: 21225 
    INFO  14:06:33,094 QGraph - 3 Pend, 1 Run, 0 Fail, 0 Done 
    INFO  14:17:57,039 FunctionEdge - Done:  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/BiO/psh/Project/SVtest/tmpdir'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-skipAnnotator' 'CNQuality'  '-cp' '/BiO/psh/tools/svtoolkit/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit/lib/gatk/Queue.jar'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVGenotyper2.q'  '-S' '/BiO/psh/tools/svtoolkit/qscript/SVQScript.q'  '-gatk' '/BiO/psh/tools/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/BiO/psh/Project/SVtest/test2_sample_chr20/cnv_stage2/seq_20/logs'  '-memLimit' '2.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobN
    

    And I check in cnv_sentinel_files.

    cnv_sentinel_files$ la
    stage_1_seq_20.sent        stage_2_seq_20.sent        .stage_3_seq_20.sent.fail
    .stage_1_seq_20.sent.done  .stage_2_seq_20.sent.done
    

    Although Stage 2 was successfully performed, it did not exist any files in cnv_stage2/seq20 even in both cnv_stage2/seq20/eval and cnv_stage2/seq20/logs.

    According to your adivce, after I deleted both stage_2_seq_20.sent and .stage_2_seq_20.sent.done, I re-ran CNVDiscoveryPipeline.
    I encountered another error which did not occur previously.

        SVToolkit version 2.00 (build 1572)
        Build date: 2015/05/12 16:43:13
        Web site: http://www.broadinstitute.org/software/genomestrip
        INFO  12:48:30,863 QScriptManager - Compiling 2 QScripts 
        INFO  12:49:16,949 QScriptManager - Compilation complete 
        INFO  12:49:19,245 HelpFormatter - ---------------------------------------------------------------------- 
        INFO  12:49:19,249 HelpFormatter - Queue v3.3-0-g3b67b44, Compiled 2014/11/26 23:42:58 
        INFO  12:49:19,250 HelpFormatter - Copyright (c) 2012 The Broad Institute 
        INFO  12:49:19,250 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
        INFO  12:49:19,251 HelpFormatter - Program Args: -jobNative -v SV_DIR -S /BiO/psh/tools/svtoolkit2/qscript/discovery/cnv/CNVDiscoveryPipeline.q -S /BiO/psh/tools/svtoolkit2/qscript/SVQScript.q -gatk /BiO/psh/tools/svtoolkit2/lib/gatk/GenomeAnalysisTK.jar -cp /BiO/psh/tools/svtoolkit2/lib/SVToolkit.jar:/BiO/psh/tools/svtoolkit2/lib/gatk/GenomeAnalysisTK.jar:/BiO/psh/tools/svtoolkit2/lib/gatk/Queue.jar -configFile /BiO/psh/tools/svtoolkit2/conf/genstrip_parameters.txt -tempDir ./tmpdir -R /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.chr20.fasta -runDirectory /BiO/psh/Project/SVtest/test2_sample_chr20 -md /BiO/psh/Project/SVtest/test2_sample_chr20/metadata -jobLogDir /BiO/psh/Project/SVtest/test2_sample_chr20/logs -ploidyMapFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt -genderMapFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/sample_gender.map -genomeMaskFile /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.svmask.chr20.fasta -tilingWindowSize 1000 -tilingWindowOverlap 500 -maximumReferenceGapLength 1000 -boundaryPrecision 100 -minimumRefinedLength 500 -produceAuxiliaryFiles -I sample.sickle.sort.addRG.dup.chr20.bam -jobRunner Drmaa -gatkJobRunner Drmaa -run 
        INFO  12:49:19,252 HelpFormatter - Executing as [email protected] on Linux 3.16.0-031600-generic amd64; Java HotSpot(TM) 64-Bit Server VM 1.7.0_72-b14. 
        INFO  12:49:19,253 HelpFormatter - Date/Time: 2015/06/08 12:49:19 
        INFO  12:49:19,253 HelpFormatter - ---------------------------------------------------------------------- 
        INFO  12:49:19,253 HelpFormatter - ---------------------------------------------------------------------- 
        INFO  12:49:19,294 QCommandLine - Scripting CNVDiscoveryPipeline 
        INFO  12:49:20,118 QCommandLine - Done with errors 
        ##### ERROR ------------------------------------------------------------------------------------------
        ##### ERROR stack trace 
        org.broadinstitute.gatk.utils.exceptions.UserException$CannotExecuteQScript: Unable to execute QScript: CNVDiscoveryPipeline.script() threw the following exception: java.lang.RuntimeException: /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt is an invalid sequence
            at org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5.apply(QCommandLine.scala:159)
            at org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5.apply(QCommandLine.scala:147)
            at scala.collection.Iterator$class.foreach(Iterator.scala:727)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
            at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
            at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
            at org.broadinstitute.gatk.queue.QCommandLine.execute(QCommandLine.scala:147)
            at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:248)
            at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:155)
            at org.broadinstitute.gatk.queue.QCommandLine$.main(QCommandLine.scala:62)
            at org.broadinstitute.gatk.queue.QCommandLine.main(QCommandLine.scala)
        Caused by: java.lang.RuntimeException: /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt is an invalid sequence
            at CNVDiscoveryPipeline$$anonfun$sequenceIntervalMap$2.apply(CNVDiscoveryPipeline.q:71)
            at CNVDiscoveryPipeline$$anonfun$sequenceIntervalMap$2.apply(CNVDiscoveryPipeline.q:67)
            at scala.collection.immutable.List.foreach(List.scala:318)
            at CNVDiscoveryPipeline.sequenceIntervalMap$lzycompute(CNVDiscoveryPipeline.q:67)
            at CNVDiscoveryPipeline.sequenceIntervalMap(CNVDiscoveryPipeline.q:55)
            at CNVDiscoveryPipeline.createStages(CNVDiscoveryPipeline.q:213)
            at CNVDiscoveryPipeline.script(CNVDiscoveryPipeline.q:97)
            at org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5.apply(QCommandLine.scala:156)
            ... 10 more
        ##### ERROR ------------------------------------------------------------------------------------------
        ##### ERROR A GATK RUNTIME ERROR has occurred (version <unknown>):
        ##### ERROR
        ##### ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.
        ##### ERROR If not, please post the error message, with stack trace, to the GATK forum.
        ##### ERROR Visit our website and forum for extensive documentation and answers to 
        ##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
        ##### ERROR
        ##### ERROR MESSAGE: Unable to execute QScript: CNVDiscoveryPipeline.script() threw the following exception: java.lang.RuntimeException: /BiO/psh/tools/svtoolkit/bundle/reference_metadata_bundles/1000G_phase1/Chr20/human_g1k_v37.ploidymap.txt is an invalid sequence
        ##### ERROR ------------------------------------------------------------------------------------------
        INFO  12:49:20,179 QCommandLine - Shutting down jobs. Please wait... 
    

    In human_g1k_v37.ploidymap.txt,
    * * * * 2

    What do I revise something about CNV_Stage2 ?
    What should I do for this situation?

Sign In or Register to comment.