We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

Cohort mode of GermlineCNVCaller: FileNotFoundError

EmiliamwEmiliamw HongkongMember
Running GermlineCNVCaller in the cohort mode---FileNotFoundError: [Errno 2] No such file or directory: '/tmp/intervals2009046725159223001.tsv'

My code for calling CNV:

cgsdir=/paedwy/disk1/wma/old_samples/raw_data/read_cgs
gatk=/home/yangyxt/software/gatk-4.1.3.0/gatk
ref_gen=/paedwy/disk1/yangyxt/indexed_genome
hdfsamples=/paedwy/disk1/wma/old_samples/raw_data/read_cgs

gCNV_model=${hdfsamples}/cgsnoB9_gCNV_model
valid_ploidy_call=${hdfsamples}/cgsNOB9_ploidy_model/cgs_normal_cohort-calls


module load miniconda3
module load gcc/4.9.1
module load GenomeAnalysisTK/4.1.3.0

source activate /software/GenomeAnalysisTK/4.1.3.0
$gatk GermlineCNVCaller \
--run-mode COHORT \
-L ${hdfsamples}/cgs.cohort.gc.filtered.interval_list \
--class-coherence-length 1000 \
--cnv-coherence-length 1000 \
--interval-merging-rule OVERLAPPING_ONLY \
--contig-ploidy-calls ${valid_ploidy_call} \
--verbosity DEBUG \
--annotated-intervals ${cgsdir}/cgs.annotated.tsv \
-I ${hdfsamples}/A100175.counts.hdf5 \
-I ${hdfsamples}/A100288B.counts.hdf5 \
-I ${hdfsamples}/A100308.counts.hdf5 \
-I ${hdfsamples}/A130042.counts.hdf5 \
-I ${hdfsamples}/A140135A.counts.hdf5 \
-I ${hdfsamples}/A140136A.counts.hdf5 \
-I ${hdfsamples}/A140347.counts.hdf5 \
-I ${hdfsamples}/A140348.counts.hdf5 \
-I ${hdfsamples}/A140417A.counts.hdf5 \
-I ${hdfsamples}/A140418A.counts.hdf5 \
-I ${hdfsamples}/A150123.counts.hdf5 \
-I ${hdfsamples}/A160134A.counts.hdf5 \
-I ${hdfsamples}/A160135A.counts.hdf5 \
-I ${hdfsamples}/A160149A.counts.hdf5 \
-I ${hdfsamples}/A160352A.counts.hdf5 \
-I ${hdfsamples}/A160353.counts.hdf5 \
-I ${hdfsamples}/A160354.counts.hdf5 \
-I ${hdfsamples}/A160355.counts.hdf5 \
-I ${hdfsamples}/A160627.counts.hdf5 \
-I ${hdfsamples}/A160788A.counts.hdf5 \
-I ${hdfsamples}/A160790B.counts.hdf5 \
-I ${hdfsamples}/A160792B.counts.hdf5 \
-I ${hdfsamples}/A170001.counts.hdf5 \
-I ${hdfsamples}/A170007.counts.hdf5 \
-I ${hdfsamples}/PID11-210.counts.hdf5 \
-I ${hdfsamples}/PID12-027.counts.hdf5 \
-I ${hdfsamples}/PID12-028.counts.hdf5 \
-I ${hdfsamples}/PID12-029.counts.hdf5 \
-I ${hdfsamples}/PID12-102.counts.hdf5 \
-I ${hdfsamples}/PID12-103.counts.hdf5 \
-I ${hdfsamples}/PID13-020.counts.hdf5 \
-I ${hdfsamples}/PID13-100.counts.hdf5 \
-I ${hdfsamples}/PID13-101.counts.hdf5 \
-I ${hdfsamples}/PID13-119.counts.hdf5 \
-I ${hdfsamples}/PID13-120.counts.hdf5 \
-I ${hdfsamples}/PID13-129.counts.hdf5 \
-I ${hdfsamples}/PID13-130.counts.hdf5 \
-I ${hdfsamples}/PID13-215.counts.hdf5 \
-I ${hdfsamples}/PID13-216.counts.hdf5 \
-I ${hdfsamples}/PID13-217.counts.hdf5 \
-I ${hdfsamples}/PID13-218.counts.hdf5 \
-I ${hdfsamples}/PID13-223.counts.hdf5 \
-I ${hdfsamples}/PID13-224.counts.hdf5 \
-I ${hdfsamples}/PID13-264.counts.hdf5 \
-I ${hdfsamples}/PID13-272.counts.hdf5 \
-I ${hdfsamples}/PID13-273.counts.hdf5 \
-I ${hdfsamples}/PID13-286.counts.hdf5 \
-I ${hdfsamples}/PID13-313.counts.hdf5 \
-I ${hdfsamples}/PID13-376.counts.hdf5 \
-I ${hdfsamples}/PID14-097.counts.hdf5 \
-I ${hdfsamples}/PID14-101.counts.hdf5 \
-I ${hdfsamples}/PID14-152A.counts.hdf5 \
-I ${hdfsamples}/PID14-191.counts.hdf5 \
-I ${hdfsamples}/PID14-192.counts.hdf5 \
-I ${hdfsamples}/PID14-200.counts.hdf5 \
-I ${hdfsamples}/PID14-203.counts.hdf5 \
-I ${hdfsamples}/PID14-205.counts.hdf5 \
-I ${hdfsamples}/PID14-208.counts.hdf5 \
-I ${hdfsamples}/PID14-209.counts.hdf5 \
-I ${hdfsamples}/PID14-210.counts.hdf5 \
-I ${hdfsamples}/PID14-211.counts.hdf5 \
-I ${hdfsamples}/PID14-229.counts.hdf5 \
-I ${hdfsamples}/PID14-230.counts.hdf5 \
-I ${hdfsamples}/PID14-325.counts.hdf5 \
-I ${hdfsamples}/PID14-360.counts.hdf5 \
-I ${hdfsamples}/PID14-361.counts.hdf5 \
-I ${hdfsamples}/PID14-362.counts.hdf5 \
-I ${hdfsamples}/PID15-017.counts.hdf5 \
-I ${hdfsamples}/PID15-018.counts.hdf5 \
-I ${hdfsamples}/PID15-105.counts.hdf5 \
-I ${hdfsamples}/PID15-106.counts.hdf5 \
-I ${hdfsamples}/PID15-116A.counts.hdf5 \
-I ${hdfsamples}/PID15-119A.counts.hdf5 \
-I ${hdfsamples}/PID16-101.counts.hdf5 \
-I ${hdfsamples}/PID16-102.counts.hdf5 \
-I ${hdfsamples}/PID16-124B.counts.hdf5 \
-I ${hdfsamples}/PID16-125A.counts.hdf5 \
-I ${hdfsamples}/PID16-177.counts.hdf5 \
-I ${hdfsamples}/PID16-178.counts.hdf5 \
-I ${hdfsamples}/PID16-179.counts.hdf5 \
-I ${hdfsamples}/PID16-202A.counts.hdf5 \
-I ${hdfsamples}/PID16-203A.counts.hdf5 \
-I ${hdfsamples}/PID16-204A.counts.hdf5 \
-I ${hdfsamples}/PID16-205A.counts.hdf5 \
-I ${hdfsamples}/PID16-211.counts.hdf5 \
-I ${hdfsamples}/PID16-212.counts.hdf5 \
-I ${hdfsamples}/PID16-213.counts.hdf5 \
-I ${hdfsamples}/XLA-016A.counts.hdf5 \
--output ${gCNV_model} \
--output-prefix cgsnoB9_gCNV_normal_cohort

source deactivate

Answers

  • EmiliamwEmiliamw HongkongMember
    Can anybody please help me to fix this? When I used around 30 samples for calling the CNVs, there is no such a kind error. Thank u very much!
  • sleeslee Member, Broadie, Dev ✭✭✭

    @Emiliamw can you provide the stacktrace, or even better, the complete log for your run? It looks like the python gCNV module is having trouble finding the temporary intervals file that GATK creates and passes to the module. If there's any reason you wouldn't have access to /tmp, you might want to set the temporary directory by running the GATK launcher as gatk --java-options "-Djava.io.tmpdir=..."

    However, since you've successfully run with 30 samples in the past, I'm guessing that's probably not the case. It's possible you might be running into an out-of-memory error, in which case you would want to run in multiple shards across your intervals as the gCNV WDL does. But the log might reveal some other error, so it would be good to see it.

  • EmiliamwEmiliamw HongkongMember
    > @slee I only extracted the last err part in my err log of run since the whole log is too long, the other content of the log is common without any err, thanks.

    18:06:52.306 DEBUG ScriptExecutor - --enable_explicit_gc_bias_modeling=True
    18:06:52.306 DEBUG ScriptExecutor - --read_count_tsv_files
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-0887541712011179213.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-16920891310541704341.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-22778816665190221119.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-36255903483855227886.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-47829010146622385107.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-55170178568454848607.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-65880195217817698373.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-76712267830464203129.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-87759256754081192203.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-95312934577168946909.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-106632198333112909776.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-113036280094121045072.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-128482459343540853440.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-136969402472505517087.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-142860916711352996076.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-152928436164970464416.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-162440316154191488960.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-173300499013456611687.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-188487435118843933521.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-197071809676439631272.tsv
    18:06:52.306 DEBUG ScriptExecutor - /tmp/sample-20314961897747324562.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-215910997934407071763.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-225530780288529557543.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-23416845355068765726.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-245966761647313885236.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-255053648563142699464.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-266450616841762384506.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-276146038227081505881.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-288231378022665843303.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-292977517878871804227.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-302351963557288869808.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-316614465499593104136.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-326713893314282155174.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-331953744022033188858.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-343604202960130207932.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-351309634376419170605.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-364180196238572598212.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-372846273688180371723.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-386025768931052149130.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-392779745978571133550.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-402056988723565994381.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-417804606767873298845.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-428192350774198406876.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-433443214420272665941.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-446020516378359268814.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-45812566793821679171.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-462079945167807062674.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-478899048776796101455.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-486622231026753964428.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-496379707016002705463.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-501994441510510390921.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-517342287949826629085.tsv
    18:06:52.307 DEBUG ScriptExecutor - /tmp/sample-52387177403424586616.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-533475796800621932398.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-545175011929768127798.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-554786624704419989905.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-564598194339098229882.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-571976232440168057023.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-588824506082659602091.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-59107658921914843054.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-602646379164832077653.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-612011552777155083299.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-627150316532973998882.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-636022363009590843472.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-645840991762674224878.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-657385020299040259269.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-666535263116783533826.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-674092929862132187094.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-681425478786265334580.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-698449173392144854423.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-707726207396925880629.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-717742020378267614099.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-724681717165036099688.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-735595670513848910637.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-748806049347862023476.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-75623441509098097211.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-761855193731965119471.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-776999185127464489054.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-782097135223965473624.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-795776883788126532494.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-807909239818741034591.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-818934014606692202890.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-822828362072477876391.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-838187068383130303658.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-843544931384815094404.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-851141420251899053107.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-86500455462929148574.tsv
    18:06:52.308 DEBUG ScriptExecutor - /tmp/sample-874114135112710905799.tsv
    18:06:52.308 DEBUG ScriptExecutor - --psi_s_scale=1.000000e-04
    18:06:52.309 DEBUG ScriptExecutor - --mapping_error_rate=1.000000e-02
    18:06:52.309 DEBUG ScriptExecutor - --depth_correction_tau=1.000000e+04
    18:06:52.309 DEBUG ScriptExecutor - --q_c_expectation_mode=hybrid
    18:06:52.309 DEBUG ScriptExecutor - --max_bias_factors=5
    18:06:52.309 DEBUG ScriptExecutor - --psi_t_scale=1.000000e-03
    18:06:52.309 DEBUG ScriptExecutor - --log_mean_bias_std=1.000000e-01
    18:06:52.309 DEBUG ScriptExecutor - --init_ard_rel_unexplained_variance=1.000000e-01
    18:06:52.309 DEBUG ScriptExecutor - --num_gc_bins=20
    18:06:52.309 DEBUG ScriptExecutor - --gc_curve_sd=1.000000e+00
    18:06:52.309 DEBUG ScriptExecutor - --active_class_padding_hybrid_mode=50000
    18:06:52.309 DEBUG ScriptExecutor - --enable_bias_factors=True
    18:06:52.309 DEBUG ScriptExecutor - --disable_bias_factors_in_active_class=False
    18:06:52.309 DEBUG ScriptExecutor - --p_alt=1.000000e-06
    18:06:52.309 DEBUG ScriptExecutor - --cnv_coherence_length=1.000000e+03
    18:06:52.309 DEBUG ScriptExecutor - --max_copy_number=5
    18:06:52.309 DEBUG ScriptExecutor - --p_active=0.010000
    18:06:52.309 DEBUG ScriptExecutor - --class_coherence_length=1000.000000
    18:06:52.309 DEBUG ScriptExecutor - --learning_rate=1.000000e-02
    18:06:52.309 DEBUG ScriptExecutor - --adamax_beta1=9.000000e-01
    18:06:52.309 DEBUG ScriptExecutor - --adamax_beta2=9.900000e-01
    18:06:52.309 DEBUG ScriptExecutor - --log_emission_samples_per_round=50
    18:06:52.309 DEBUG ScriptExecutor - --log_emission_sampling_rounds=10
    18:06:52.309 DEBUG ScriptExecutor - --log_emission_sampling_median_rel_error=5.000000e-03
    18:06:52.309 DEBUG ScriptExecutor - --max_advi_iter_first_epoch=5000
    18:06:52.309 DEBUG ScriptExecutor - --max_advi_iter_subsequent_epochs=200
    18:06:52.309 DEBUG ScriptExecutor - --min_training_epochs=10
    18:06:52.309 DEBUG ScriptExecutor - --max_training_epochs=50
    18:06:52.309 DEBUG ScriptExecutor - --initial_temperature=1.500000e+00
    18:06:52.309 DEBUG ScriptExecutor - --num_thermal_advi_iters=2500
    18:06:52.309 DEBUG ScriptExecutor - --convergence_snr_averaging_window=500
    18:06:52.309 DEBUG ScriptExecutor - --convergence_snr_trigger_threshold=1.000000e-01
    18:06:52.309 DEBUG ScriptExecutor - --convergence_snr_countdown_window=10
    18:06:52.309 DEBUG ScriptExecutor - --max_calling_iters=10
    18:06:52.309 DEBUG ScriptExecutor - --caller_update_convergence_threshold=1.000000e-03
    18:06:52.309 DEBUG ScriptExecutor - --caller_internal_admixing_rate=7.500000e-01
    18:06:52.309 DEBUG ScriptExecutor - --caller_external_admixing_rate=1.000000e+00
    18:06:52.310 DEBUG ScriptExecutor - --disable_caller=false
    18:06:52.310 DEBUG ScriptExecutor - --disable_sampler=false
    18:06:52.310 DEBUG ScriptExecutor - --disable_annealing=false

    Traceback (most recent call last):
    File "/tmp/cohort_denoising_calling.631031049823448254.py", line 183, in <module>
    File "/software/GenomeAnalysisTK/4.1.3.0/lib/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/software/GenomeAnalysisTK/4.1.3.0/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/intervals2009046725159223001.tsv'
    01:29:11.256 DEBUG ScriptExecutor - Result: 1
    01:29:11.518 INFO GermlineCNVCaller - Shutting down engine
    [November 29, 2019 1:29:11 AM HKT] org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller done. Elapsed time: 3,324.34 minutes.
    Runtime.totalMemory()=4711776256
    org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
    python exited with 1
    Command Line: python /tmp/cohort_denoising_calling.631031049823448254.py --ploidy_calls_path=/paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgsNOB9_ploidy_model/cgs_normal_cohort-calls --output_calls_path=/paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgsnoB9_gCNV_model/cgsnoB9_gCNV_normal_cohort-calls --output_tracking_path=/paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgsnoB9_gCNV_model/cgsnoB9_gCNV_normal_cohort-tracking --modeling_interval_list=/tmp/intervals2009046725159223001.tsv --output_model_path=/paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgsnoB9_gCNV_model/cgsnoB9_gCNV_normal_cohort-model --enable_explicit_gc_bias_modeling=True --read_count_tsv_files /tmp/sample-0887541712011179213.tsv /tmp/sample-16920891310541704341.tsv /tmp/sample-22778816665190221119.tsv /tmp/sample-36255903483855227886.tsv /tmp/sample-47829010146622385107.tsv /tmp/sample-55170178568454848607.tsv /tmp/sample-65880195217817698373.tsv /tmp/sample-76712267830464203129.tsv /tmp/sample-87759256754081192203.tsv /tmp/sample-95312934577168946909.tsv /tmp/sample-106632198333112909776.tsv /tmp/sample-113036280094121045072.tsv /tmp/sample-128482459343540853440.tsv /tmp/sample-136969402472505517087.tsv /tmp/sample-142860916711352996076.tsv /tmp/sample-152928436164970464416.tsv /tmp/sample-162440316154191488960.tsv /tmp/sample-173300499013456611687.tsv /tmp/sample-188487435118843933521.tsv /tmp/sample-197071809676439631272.tsv /tmp/sample-20314961897747324562.tsv /tmp/sample-215910997934407071763.tsv /tmp/sample-225530780288529557543.tsv /tmp/sample-23416845355068765726.tsv /tmp/sample-245966761647313885236.tsv /tmp/sample-255053648563142699464.tsv /tmp/sample-266450616841762384506.tsv /tmp/sample-276146038227081505881.tsv /tmp/sample-288231378022665843303.tsv /tmp/sample-292977517878871804227.tsv /tmp/sample-302351963557288869808.tsv /tmp/sample-316614465499593104136.tsv /tmp/sample-326713893314282155174.tsv /tmp/sample-331953744022033188858.tsv /tmp/sample-343604202960130207932.tsv /tmp/sample-351309634376419170605.tsv /tmp/sample-364180196238572598212.tsv /tmp/sample-372846273688180371723.tsv /tmp/sample-386025768931052149130.tsv /tmp/sample-392779745978571133550.tsv /tmp/sample-402056988723565994381.tsv /tmp/sample-417804606767873298845.tsv /tmp/sample-428192350774198406876.tsv /tmp/sample-433443214420272665941.tsv /tmp/sample-446020516378359268814.tsv /tmp/sample-45812566793821679171.tsv /tmp/sample-462079945167807062674.tsv /tmp/sample-478899048776796101455.tsv /tmp/sample-486622231026753964428.tsv /tmp/sample-496379707016002705463.tsv /tmp/sample-501994441510510390921.tsv /tmp/sample-517342287949826629085.tsv /tmp/sample-52387177403424586616.tsv /tmp/sample-533475796800621932398.tsv /tmp/sample-545175011929768127798.tsv /tmp/sample-554786624704419989905.tsv /tmp/sample-564598194339098229882.tsv /tmp/sample-571976232440168057023.tsv /tmp/sample-588824506082659602091.tsv /tmp/sample-59107658921914843054.tsv /tmp/sample-602646379164832077653.tsv /tmp/sample-612011552777155083299.tsv /tmp/sample-627150316532973998882.tsv /tmp/sample-636022363009590843472.tsv /tmp/sample-645840991762674224878.tsv /tmp/sample-657385020299040259269.tsv /tmp/sample-666535263116783533826.tsv /tmp/sample-674092929862132187094.tsv /tmp/sample-681425478786265334580.tsv /tmp/sample-698449173392144854423.tsv /tmp/sample-707726207396925880629.tsv /tmp/sample-717742020378267614099.tsv /tmp/sample-724681717165036099688.tsv /tmp/sample-735595670513848910637.tsv /tmp/sample-748806049347862023476.tsv /tmp/sample-75623441509098097211.tsv /tmp/sample-761855193731965119471.tsv /tmp/sample-776999185127464489054.tsv /tmp/sample-782097135223965473624.tsv /tmp/sample-795776883788126532494.tsv /tmp/sample-807909239818741034591.tsv /tmp/sample-818934014606692202890.tsv /tmp/sample-822828362072477876391.tsv /tmp/sample-838187068383130303658.tsv /tmp/sample-843544931384815094404.tsv /tmp/sample-851141420251899053107.tsv /tmp/sample-86500455462929148574.tsv /tmp/sample-874114135112710905799.tsv --psi_s_scale=1.000000e-04 --mapping_error_rate=1.000000e-02 --depth_correction_tau=1.000000e+04 --q_c_expectation_mode=hybrid --max_bias_factors=5 --psi_t_scale=1.000000e-03 --log_mean_bias_std=1.000000e-01 --init_ard_rel_unexplained_variance=1.000000e-01 --num_gc_bins=20 --gc_curve_sd=1.000000e+00 --active_class_padding_hybrid_mode=50000 --enable_bias_factors=True --disable_bias_factors_in_active_class=False --p_alt=1.000000e-06 --cnv_coherence_length=1.000000e+03 --max_copy_number=5 --p_active=0.010000 --class_coherence_length=1000.000000 --learning_rate=1.000000e-02 --adamax_beta1=9.000000e-01 --adamax_beta2=9.900000e-01 --log_emission_samples_per_round=50 --log_emission_sampling_rounds=10 --log_emission_sampling_median_rel_error=5.000000e-03 --max_advi_iter_first_epoch=5000 --max_advi_iter_subsequent_epochs=200 --min_training_epochs=10 --max_training_epochs=50 --initial_temperature=1.500000e+00 --num_thermal_advi_iters=2500 --convergence_snr_averaging_window=500 --convergence_snr_trigger_threshold=1.000000e-01 --convergence_snr_countdown_window=10 --max_calling_iters=10 --caller_update_convergence_threshold=1.000000e-03 --caller_internal_admixing_rate=7.500000e-01 --caller_external_admixing_rate=1.000000e+00 --disable_caller=false --disable_sampler=false --disable_annealing=false
    at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
    at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
    at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
    at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:151)
    at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeScript(PythonScriptExecutor.java:121)
    at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.executeGermlineCNVCallerPythonScript(GermlineCNVCaller.java:441)
    at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.doWork(GermlineCNVCaller.java:292)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
    at org.broadinstitute.hellbender.Main.main(Main.java:291)
    Using GATK jar /home/yangyxt/software/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/yangyxt/software/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar GermlineCNVCaller --run-mode COHORT -L /paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgs.cohort.gc.filtered.interval_list --class-coherence-length 1000 --cnv-coherence-length 1000 --interval-merging-rule OVERLAPPING_ONLY --contig-ploidy-calls /paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgsNOB9_ploidy_model/cgs_normal_cohort-calls --verbosity DEBUG --annotated-intervals /paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgs.annotated.tsv -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A100175.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A100288B.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A100308.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A130042.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A140135A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A140136A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A140347.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A140348.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A140417A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A140418A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A150123.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160134A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160135A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160149A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160352A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160353.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160354.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160355.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160627.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160788A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160790B.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A160792B.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A170001.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/A170007.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID11-210.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID12-027.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID12-028.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID12-029.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID12-102.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID12-103.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-020.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-100.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-101.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-119.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-120.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-129.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-130.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-215.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-216.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-217.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-218.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-223.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-224.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-264.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-272.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-273.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-286.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-313.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID13-376.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-097.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-101.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-152A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-191.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-192.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-200.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-203.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-205.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-208.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-209.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-210.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-211.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-229.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-230.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-325.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-360.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-361.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID14-362.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID15-017.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID15-018.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID15-105.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID15-106.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID15-116A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID15-119A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-101.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-102.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-124B.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-125A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-177.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-178.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-179.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-202A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-203A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-204A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-205A.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-211.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-212.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/PID16-213.counts.hdf5 -I /paedwy/disk1/wma/old_samples/raw_data/read_cgs/XLA-016A.counts.hdf5 --output /paedwy/disk1/wma/old_samples/raw_data/read_cgs/cgsnoB9_gCNV_model --output-prefix cgsnoB9_gCNV_normal_cohort
  • sleeslee Member, Broadie, Dev ✭✭✭

    Hi @emiliamw, is it possible that your /tmp directory is getting cleaned up between the start and the end of your relatively long run? After the gCNV python module performs model inference and calling, there is a simple step in which some temporary files are simply copied over to the output model directory. But if those files have been inadvertently cleaned up during inference, then you would get an error like the one you observe.

    I can issue a PR to move this copying step before inference to avoid this in the future. However, I'd also suggest that you shard your gCNV runs across your intervals (as is done in the WDL), which will most likely bring the runtime down significantly.

  • EmiliamwEmiliamw HongkongMember
    Hi @slee,

    "I'd also suggest that you shard your gCNV runs across your intervals (as is done in the WDL), which will most likely bring the runtime down significantly." ----> I didn't quite understand.

    What do you mean by "shard gCNV runs across intervals"? And do u mean to achieve this through using WDL?

    I saw a similar post at https://gatkforums.broadinstitute.org/gatk/discussion/comment/57237#Comment_57237. But I still did not understand how to use WDL? Thanks
  • sleeslee Member, Broadie, Dev ✭✭✭

    @Emiliamw you can use the -L option to specify which intervals GermlineCNVCaller is run over. By splitting up your intervals into subsets and running separate instances of GermlineCNVCaller over them in parallel, you can bring runtime and memory requirements for each instance down. See descriptions in the tool documentation and the gCNV tutorial.

    The WDL script runs the gCNV workflow by chaining together tools in the intended way, including sharding as described above---you can think of it as a sort of Best Practices workflow. You may still need to adjust workflow/tool parameters to achieve optimal results, depending on your data; because the quality of read-depth data can be very platform specific, we can only make rough recommendations for specific parameter values, some of which are described in the documentation.

    That said, you don't need to use the WDL if you can run the workflow in an equivalent way without it, but you can use it as a guide to structure your own workflow.

  • EmiliamwEmiliamw HongkongMember

    Hi @slee: For using the -L option to specify which intervals GermlineCNVCaller is run over, I used FilterIntervals to create the interval list upon all samples. But, even though utilizing the interval list, I still got the same error. Can u please help me figure out this? Is there anything wrong I did for getting the interval list? Thanks,

    #

    The following is the code for creating the interval list:

    ${gatk} FilterIntervals \
    -L ${cgsdir}/cgs.hg19.preprocessed.interval_list \
    -imr OVERLAPPING_ONLY \
    --annotated-intervals ${cgsdir}/cgs.annotated.tsv \
    -I ${hdfsamples}/A100175.counts.hdf5 \
    -I ${hdfsamples}/A100288B.counts.hdf5 \
    -I ${hdfsamples}/A100308.counts.hdf5 \
    -I ${hdfsamples}/A130042.counts.hdf5 \
    ...
    -I ${hdfsamples}/PID16-213.counts.hdf5 \
    -I ${hdfsamples}/XLA-016A.counts.hdf5 \
    -O ${hdfsamples}/cgsNOB9.cohort.gc.filtered.interval_list

    #

    The following is the code for running the GermlineCNVcaller:

    $gatk GermlineCNVCaller \
    --run-mode COHORT \
    -L ${hdfsamples}/cgsNOB9.cohort.gc.filtered.interval_list \
    --class-coherence-length 1000 \
    --cnv-coherence-length 1000 \
    --interval-merging-rule OVERLAPPING_ONLY \
    --contig-ploidy-calls ${valid_ploidy_call} \
    --verbosity DEBUG \
    --annotated-intervals ${cgsdir}/cgs.annotated.tsv \
    -I ${hdfsamples}/A100175.counts.hdf5 \
    -I ${hdfsamples}/A100288B.counts.hdf5 \
    -I ${hdfsamples}/A100308.counts.hdf5 \
    -I ${hdfsamples}/A130042.counts.hdf5 \
    ...
    -I ${hdfsamples}/PID16-213.counts.hdf5 \
    -I ${hdfsamples}/XLA-016A.counts.hdf5 \
    --output ${gCNV_model} \
    --output-prefix cgsnoB9_gCNV_normal_cohort

    #

    The error of running the GermlineCNVcaller is:

    Traceback (most recent call last):
    File "/tmp/cohort_denoising_calling.8345882579244784621.py", line 183, in
    File "/software/GenomeAnalysisTK/4.1.3.0/lib/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/software/GenomeAnalysisTK/4.1.3.0/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/intervals7735559344426051448.tsv'
    17:06:54.638 DEBUG ScriptExecutor - Result: 1
    17:06:54.996 INFO GermlineCNVCaller - Shutting down engine
    [December 7, 2019 5:06:55 PM HKT] org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller done. Elapsed time: 3,351.55 minutes.
    Runtime.totalMemory()=4433903616
    org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
    python exited with 1

  • sleeslee Member, Broadie, Dev ✭✭✭
    edited December 2019

    Hi @Emiliamw,

    I'd suggest you study my responses above and perhaps also the tutorial at https://software.broadinstitute.org/gatk/documentation/article?id=11684.

    As I said above, I'm guessing that the reason you are getting the error is most likely because your tmp files are somehow getting deleted during the run. If this is indeed the case, then the tools are working as intended and there is no bug---the PR fix I mentioned above is just preventative. You should try to confirm on your end whether or not this is the case and either fix it on your own (by making sure your tmp files are not deleted during the run, or by using a jar which includes the fix). If this is not the case, then please let us know and we'll try to debug further.

    If you want to make your runs shorter by sharding the interval list (as we do in the tutorial and the Best Practices WDL), then you should use IntervalListTools on the result of FilterIntervals to scatter the filtered intervals into multiple interval lists, each of which is passed to a separate instance of GermlineCNVCaller.

  • EmiliamwEmiliamw HongkongMember

    Hi, @slee. I see. Let me try to fix this on my own. Thank you very much!

Sign In or Register to comment.