Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Gather proceeded without awaiting completion of scatter job

Dear all,

I'm trying to incorporate the CNN filtering in my Germline variant discovery work flow but noted some abnormal behaviour of Cromwell. My scatter is followed by a gather task (MergeVCF_HC4) with an array of input files supplied by the output of the previous task(CNNScoreVariants).

However, Cromwell is going straight to MergeVCF_HC4 without going through the scatter task and hence producing an error that the input of MergeVCF_HC4 is not found.

My code as follows

#CNN filtering
call SplitIntervals {
input:
intervals = calling_interval_list,
ref_fasta = ref_fasta_extracted,
ref_dict = ref_dict,
ref_fai = ref_fasta_index
}

scatter (calling_interval in SplitIntervals.interval_files) {
call RunHC4 {
input:
input_bam = GatherSortedBamFiles.output_bam,
input_bam_index = GatherSortedBamFiles.output_bam_index,
reference_fasta = ref_fasta_extracted,
reference_dict = ref_dict,
reference_fasta_index = ref_fasta_index,
output_prefix = batch_base_file_name,
interval_list = calling_interval
}

call CNNScoreVariants {
input:
input_vcf = RunHC4.raw_vcf,
input_vcf_index = RunHC4.raw_vcf_index,
bam_file = RunHC4.bamout,
bam_file_index = RunHC4.bamout_index,
reference_fasta = ref_fasta_extracted,
reference_dict = ref_dict,
reference_fasta_index = ref_fasta_index,
output_prefix = batch_base_file_name,
interval_list = calling_interval
}
}

call MergeVCF_HC4 {
input:
input_vcfs = CNNScoreVariants.cnn_annotated_vcf,
input_vcfs_indexes = CNNScoreVariants.cnn_annotated_vcf_index,
output_prefix = batch_base_file_name
}

call FilterVariantTranches {
input:
input_vcf = MergeVCF_HC4.merged_vcf,
input_vcf_index = MergeVCF_HC4.merged_vcf_index,
hapmap = hapmap,
hapmap_idx = hapmap_idx,
mills = mills,
mills_idx = mills_idx,
output_prefix = batch_base_file_name
}

task CNNScoreVariants {
File input_vcf
File input_vcf_index
File reference_fasta
File reference_dict
File reference_fasta_index
String output_prefix
File bam_file
File bam_file_index
File? architecture_json
File? architecture_hd5
File interval_list

command {

gatk --java-options -Xmx60G \
CNNScoreVariants \
${"-I " + bam_file} \
-R ${reference_fasta} \
-V ${input_vcf} \
-O ${output_prefix}_cnn_annotated.vcf.gz \
-L ${interval_list} \
--tensor-type read_tensor \
--inference-batch-size 8 \
--transfer-batch-size 32

}
output {
Array[File] log = glob("gatkStreamingProcessJournal*")
File cnn_annotated_vcf = "${output_prefix}_cnn_annotated.vcf.gz"
File cnn_annotated_vcf_index = "${output_prefix}_cnn_annotated.vcf.gz.tbi"
}
}

task MergeVCF_HC4 {
Array[File] input_vcfs
Array[File] input_vcfs_indexes
String output_prefix
String output_vcf = "${output_prefix}_cnn_scored.vcf.gz"

command {
gatk --java-options -Xmx60 MergeVcfs \
-I ${sep=' -I ' input_vcfs} -O "${output_vcf}"
}

output {
File merged_vcf = "${output_vcf}"
File merged_vcf_index = "${output_vcf}.tbi"
}
}

My error as follows.

[2019-02-27 12:20:53,67] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.SplitIntervals:NA:1]: job id: 13360
[2019-02-27 12:20:53,67] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.CreateSequenceGroupingTSV:NA:1]: job id: 13366
[2019-02-27 12:20:53,67] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.CollectQualityYieldMetrics:0:1]: job id: 13512
[2019-02-27 12:20:53,68] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.SamToFastqAndBwaMemAndMba:0:1]: job id: 13570
[2019-02-27 12:20:53,68] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.ScatterIntervalList:NA:1]: job id: 13362
[2019-02-27 12:20:53,68] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.CollectQualityYieldMetrics:0:1]: Status change from - to WaitingForReturnCode
[2019-02-27 12:20:53,69] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.SamToFastqAndBwaMemAndMba:0:1]: Status change from - to WaitingForReturnCode
[2019-02-27 12:20:53,69] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.CreateSequenceGroupingTSV:NA:1]: Status change from - to Done
[2019-02-27 12:20:53,69] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.ScatterIntervalList:NA:1]: Status change from - to Done
[2019-02-27 12:20:53,69] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.SplitIntervals:NA:1]: Status change from - to WaitingForReturnCode
[2019-02-27 12:20:55,12] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.CollectQualityYieldMetrics:0:1]: Status change from WaitingForReturnCode to Done
[2019-02-27 12:20:55,33] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.SplitIntervals:NA:1]: Status change from WaitingForReturnCode to Done
[2019-02-27 12:21:00,43] [info] WorkflowExecutionActor-180317e4-cbfd-4e94-97ab-0aeb4bbb4943 [180317e4]: Starting RBCWorkflow.MergeVCF_HC4
[2019-02-27 12:21:00,72] [info] Assigned new job execution tokens to the following groups: 180317e4: 1
[2019-02-27 12:21:00,74] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.MergeVCF_HC4:NA:1]: gatk --java-options -Xmx60 MergeVcfs \
-I -O "RBC1_BATCH1_cnn_scored.vcf.gz"
[2019-02-27 12:21:00,75] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.MergeVCF_HC4:NA:1]: executing: /bin/bash /mnt/operation/RedCellNGS/cromwell-executions/RBCWorkflow/180317e4-cbfd-4e94-97ab-0aeb4bbb4943/call-MergeVCF_HC4/execution/script
[2019-02-27 12:21:03,66] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.MergeVCF_HC4:NA:1]: job id: 13780
[2019-02-27 12:21:03,67] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.MergeVCF_HC4:NA:1]: Status change from - to Done
[2019-02-27 12:21:06,10] [info] BackgroundConfigAsyncJobExecutionActor [180317e4RBCWorkflow.SamToFastqAndBwaMemAndMba:0:1]: Status change from WaitingForReturnCode to Done
[2019-02-27 12:21:06,59] [error] WorkflowManagerActor Workflow 180317e4-cbfd-4e94-97ab-0aeb4bbb4943 failed (during ExecutingWorkflowState): Job RBCWorkflow.MergeVCF_HC4:NA:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /mnt/operation/RedCellNGS/cromwell-executions/RBCWorkflow/180317e4-cbfd-4e94-97ab-0aeb4bbb4943/call-MergeVCF_HC4/execution/stderr.
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt/operation/RedCellNGS/cromwell-executions/RBCWorkflow/180317e4-cbfd-4e94-97ab-0aeb4bbb4943/call-MergeVCF_HC4/tmp.e07d0773
Using GATK jar /home/nelson/miniconda3/envs/bioinfo/share/gatk4-4.1.0.0-0/gatk-package-4.1.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx60 -jar /home/nelson/miniconda3/envs/bioinfo/share/gatk4-4.1.0.0-0/gatk-package-4.1.0.0-local.jar MergeVcfs -I -O RBC1_BATCH1_cnn_scored.vcf.gz

[2019-02-27 12:21:06,59] [info] WorkflowManagerActor WorkflowActor-180317e4-cbfd-4e94-97ab-0aeb4bbb4943 is in a terminal state: WorkflowFailedState
[2019-02-27 12:21:12,03] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2019-02-27 12:21:13,66] [info] Workflow polling stopped
[2019-02-27 12:21:13,68] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2019-02-27 12:21:13,69] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2019-02-27 12:21:13,69] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2019-02-27 12:21:13,69] [info] JobExecutionTokenDispenser stopped
[2019-02-27 12:21:13,70] [info] Aborting all running workflows.
[2019-02-27 12:21:13,70] [info] WorkflowStoreActor stopped
[2019-02-27 12:21:13,70] [info] WorkflowLogCopyRouter stopped
[2019-02-27 12:21:13,70] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2019-02-27 12:21:13,71] [info] WorkflowManagerActor All workflows finished
[2019-02-27 12:21:13,71] [info] WorkflowManagerActor stopped
[2019-02-27 12:21:13,94] [info] Connection pools shut down
[2019-02-27 12:21:13,95] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2019-02-27 12:21:13,95] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2019-02-27 12:21:13,95] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2019-02-27 12:21:13,95] [info] SubWorkflowStoreActor stopped
[2019-02-27 12:21:13,95] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2019-02-27 12:21:13,95] [info] JobStoreActor stopped
[2019-02-27 12:21:13,95] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2019-02-27 12:21:13,95] [info] KvWriteActor Shutting down: 0 queued messages to process
[2019-02-27 12:21:13,95] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2019-02-27 12:21:13,95] [info] CallCacheWriteActor stopped
[2019-02-27 12:21:13,95] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2019-02-27 12:21:13,95] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2019-02-27 12:21:13,96] [info] ServiceRegistryActor stopped
[2019-02-27 12:21:13,96] [info] IoProxy stopped
[2019-02-27 12:21:13,96] [info] DockerHashActor stopped
[2019-02-27 12:21:13,99] [info] Database closed
[2019-02-27 12:21:13,99] [info] Stream materializer shut down
[2019-02-27 12:21:13,99] [info] WDL HTTP import resolver closed
[2019-02-27 12:21:14,00] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2019-02-27 12:21:14,00] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2019-02-27 12:21:14,00] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
Workflow 180317e4-cbfd-4e94-97ab-0aeb4bbb4943 transitioned to state Failed


Thank you!
Sign In or Register to comment.