Async Connection Dropped Suddenly During Cromwell + WDL Mutect2 Pipeline

Hello. I recently tried running a mutect2 pipeline with cromwell + wdl. I am using GATK4 and cromwell 33.1. I am not using a docker. I have successfully run my M2 pipeline with cromwell + wdl using a .bed file as the -L argument. However, I ultimately want to run M2 on the whole genome. To that end, I removed the .bed file from my list of JSON inputs and reran the script. The run initialized properly and seemed like it would go to completion. However, in the middle of the run, the async connection drops and the workflow aborts.

Below is a log of output I received during the run. Any help would be much appreciated.

[2018-07-29 19:38:01,56] [info] Running with database db.url = jdbc:hsqldb:mem:95ad6550-bc35-4ec3-aa4f-8f29708c9ecc;shutdown=false;hsqldb.tx=mvcc
[2018-07-29 19:38:11,98] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2018-07-29 19:38:12,01] [info] [RenameWorkflowOptionsInMetadata] 100%
[2018-07-29 19:38:12,36] [info] Running with database db.url = jdbc:hsqldb:mem:3ca7ab2b-b2aa-46d4-898c-55bd11502bcb;shutdown=false;hsqldb.tx=mvcc
[2018-07-29 19:38:13,23] [info] Slf4jLogger started
[2018-07-29 19:38:13,70] [info] Workflow heartbeat configuration:
{
"cromwellId" : "cromid-1499a71",
"heartbeatInterval" : "2 minutes",
"ttl" : "10 minutes",
"writeBatchSize" : 10000,
"writeThreshold" : 10000
}
[2018-07-29 19:38:13,96] [info] Metadata summary refreshing every 2 seconds.
[2018-07-29 19:38:14,23] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-07-29 19:38:14,33] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-07-29 19:38:14,33] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-07-29 19:38:15,69] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-07-29 19:38:15,71] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-07-29 19:38:15,77] [info] Unspecified type (Unspecified version) workflow 045183d0-3ee4-4df7-90a6-671609a0112e submitted
[2018-07-29 19:38:15,82] [info] SingleWorkflowRunnerActor: Workflow submitted [38;5;2m045183d0-3ee4-4df7-90a6-671609a0112e[0m
[2018-07-29 19:38:15,83] [info] 1 new workflows fetched
[2018-07-29 19:38:15,83] [info] WorkflowManagerActor Starting workflow [38;5;2m045183d0-3ee4-4df7-90a6-671609a0112e[0m
[2018-07-29 19:38:15,84] [[38;5;220mwarn[0m] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-07-29 19:38:15,84] [[38;5;220mwarn[0m] Couldn't find a suitable DSN, defaulting to a Noop one.
[2018-07-29 19:38:15,85] [info] Using noop to send events.
[2018-07-29 19:38:15,90] [info] WorkflowManagerActor Successfully started WorkflowActor-045183d0-3ee4-4df7-90a6-671609a0112e
[2018-07-29 19:38:15,90] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-07-29 19:38:15,90] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-07-29 19:38:15,94] [info] MaterializeWorkflowDescriptorActor [[38;5;2m045183d0[0m]: Parsing workflow as WDL draft-2
[2018-07-29 19:38:17,59] [info] MaterializeWorkflowDescriptorActor [[38;5;2m045183d0[0m]: Call-to-Backend assignments: Mutect2.SplitIntervals -> Local, Mutect2.MergeVCFs -> Local, Mutect2.CalculateContamination -> Local, Mutect2.M2 -> Local, Mutect2.MergeBamOuts -> Local, Mutect2.Filter -> Local
[2018-07-29 19:38:17,71] [[38;5;220mwarn[0m] Local [[38;5;2m045183d0[0m]: Key/s [preemptible, bootDiskSizeGb, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-07-29 19:38:17,71] [[38;5;220mwarn[0m] Local [[38;5;2m045183d0[0m]: Key/s [preemptible, bootDiskSizeGb, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-07-29 19:38:17,72] [[38;5;220mwarn[0m] Local [[38;5;2m045183d0[0m]: Key/s [bootDiskSizeGb, memory, disks, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-07-29 19:38:17,72] [[38;5;220mwarn[0m] Local [[38;5;2m045183d0[0m]: Key/s [preemptible, bootDiskSizeGb, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-07-29 19:38:17,72] [[38;5;220mwarn[0m] Local [[38;5;2m045183d0[0m]: Key/s [preemptible, bootDiskSizeGb, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-07-29 19:38:17,74] [[38;5;220mwarn[0m] Local [[38;5;2m045183d0[0m]: Key/s [preemptible, bootDiskSizeGb, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-07-29 19:38:20,17] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: Condition met: 'defined(variants_for_contamination)'. Running conditional section
[2018-07-29 19:38:21,22] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: Condition met: 'make_bamout_or_default'. Running conditional section
[2018-07-29 19:38:22,28] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: Starting Mutect2.CalculateContamination, Mutect2.SplitIntervals
[2018-07-29 19:38:22,82] [[38;5;220mwarn[0m] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.SplitIntervals:NA:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2018-07-29 19:38:22,83] [[38;5;220mwarn[0m] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, memory
[2018-07-29 19:38:22,89] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-SplitIntervals/inputs/-1999123578/Homo_sapiens_assembly38.fasta.fai -> /data/Genomes/Homo_sapiens/gatk/Sequence/Homo_sapiens_assembly38.fasta.fai: Operation not permitted
[2018-07-29 19:38:22,89] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/-1999123578/Homo_sapiens_assembly38.fasta.fai -> /data/Genomes/Homo_sapiens/gatk/Sequence/Homo_sapiens_assembly38.fasta.fai: Operation not permitted
[2018-07-29 19:38:22,90] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-SplitIntervals/inputs/-1999123578/Homo_sapiens_assembly38.fasta -> /data/Genomes/Homo_sapiens/gatk/Sequence/Homo_sapiens_assembly38.fasta: Operation not permitted
[2018-07-29 19:38:22,90] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/-1273940790/small_exac_common_3.hg38.vcf.gz -> /data/Genomes/Homo_sapiens/gatk/small_exac_common_3.hg38.vcf.gz: Operation not permitted
[2018-07-29 19:38:22,91] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/-1999123578/Homo_sapiens_assembly38.fasta -> /data/Genomes/Homo_sapiens/gatk/Sequence/Homo_sapiens_assembly38.fasta: Operation not permitted
[2018-07-29 19:38:22,91] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-SplitIntervals/inputs/-1999123578/Homo_sapiens_assembly38.dict -> /data/Genomes/Homo_sapiens/gatk/Sequence/Homo_sapiens_assembly38.dict: Operation not permitted
[2018-07-29 19:38:22,91] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/-1999123578/Homo_sapiens_assembly38.dict -> /data/Genomes/Homo_sapiens/gatk/Sequence/Homo_sapiens_assembly38.dict: Operation not permitted
[2018-07-29 19:38:22,91] [[38;5;220mwarn[0m] Localization via hard link has failed: /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/-1273940790/small_exac_common_3.hg38.vcf.gz.tbi -> /data/Genomes/Homo_sapiens/gatk/small_exac_common_3.hg38.vcf.gz.tbi: Operation not permitted
[2018-07-29 19:38:24,23] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: [38;5;5mset -e

export PATH=$PATH:/data/Tools/gatk-4.0.2.1
gatk GetPileupSummaries -I /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/124921695/7316-6-T.markduplicates.bam -V /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/inputs/-1273940790/small_exac_common_3.hg38.vcf.gz -O pileups.table
gatk CalculateContamination -I pileups.table -O contamination.table[0m
[2018-07-29 19:38:24,23] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.SplitIntervals:NA:1]: [38;5;5mset -e
export PATH=$PATH:/data/Tools/gatk-4.0.2.1

mkdir interval-files
gatk --java-options "-Xmx20g" SplitIntervals \
-R /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-SplitIntervals/inputs/-1999123578/Homo_sapiens_assembly38.fasta \
\
-scatter 20 \
-O interval-files \

cp interval-files/*.intervals .[0m
[2018-07-29 19:38:24,42] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-CalculateContamination/execution/script
[2018-07-29 19:38:24,45] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.SplitIntervals:NA:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-SplitIntervals/execution/script
[2018-07-29 19:38:29,45] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.SplitIntervals:NA:1]: job id: 14953
[2018-07-29 19:38:29,46] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: job id: 14955
[2018-07-29 19:38:29,47] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.SplitIntervals:NA:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:29,47] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:44,14] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.SplitIntervals:NA:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:38:52,02] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: Starting Mutect2.M2 (20 shards)

....

[2018-07-29 19:38:53,35] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:7:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-7/execution/script
[2018-07-29 19:38:53,36] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:11:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-11/execution/script
[2018-07-29 19:38:53,46] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:1:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-1/execution/script
[2018-07-29 19:38:53,49] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:9:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-9/execution/script
[2018-07-29 19:38:53,51] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:16:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-16/execution/script
[2018-07-29 19:38:53,53] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:3:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-3/execution/script
[2018-07-29 19:38:53,58] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:8:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-8/execution/script
[2018-07-29 19:38:53,58] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:5:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-5/execution/script
[2018-07-29 19:38:53,59] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:14:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-14/execution/script
[2018-07-29 19:38:53,59] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:17:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-17/execution/script
[2018-07-29 19:38:53,59] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:2:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-2/execution/script
[2018-07-29 19:38:53,59] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:13:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-13/execution/script
[2018-07-29 19:38:53,62] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:18:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-18/execution/script
[2018-07-29 19:38:53,62] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:4:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-4/execution/script
[2018-07-29 19:38:53,63] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:15:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-15/execution/script
[2018-07-29 19:38:53,63] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:0:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-0/execution/script
[2018-07-29 19:38:53,64] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:19:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-19/execution/script
[2018-07-29 19:38:53,67] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:6:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-6/execution/script
[2018-07-29 19:38:53,69] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:12:1]: executing: /bin/bash /data/CBTTC_Proteogenomic/WGS/7316-6/cromwell-executions/Mutect2/045183d0-3ee4-4df7-90a6-671609a0112e/call-M2/shard-12/execution/script
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:7:1]: job id: 15085
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:10:1]: job id: 15083
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:11:1]: job id: 15087
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:1:1]: job id: 15099
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:16:1]: job id: 15103
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:2:1]: job id: 15149
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:18:1]: job id: 15156
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:9:1]: job id: 15101
[2018-07-29 19:38:54,38] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:6:1]: job id: 15173
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:12:1]: job id: 15188
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:8:1]: job id: 15141
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:3:1]: job id: 15119
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:5:1]: job id: 15143
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:14:1]: job id: 15145
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:17:1]: job id: 15147
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:13:1]: job id: 15151
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:4:1]: job id: 15159
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:8:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:17:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:6:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,39] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:1:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,40] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:16:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,40] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:9:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,40] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:10:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:19:1]: job id: 15170
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:15:1]: job id: 15161
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:0:1]: job id: 15163
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:19:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:11:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:12:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:4:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:15:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:18:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:14:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,41] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:5:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,42] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:7:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,42] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:13:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,42] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:3:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,42] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:2:1]: Status change from - to WaitingForReturnCodeFile
[2018-07-29 19:38:54,42] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:0:1]: Status change from - to WaitingForReturnCodeFile

//Problem starts here

[2018-07-29 19:45:24,06] [info] Automatic shutdown of the async connection
[2018-07-29 19:45:24,08] [info] Starting coordinated shutdown from JVM shutdown hook
[2018-07-29 19:45:24,09] [info] Workflow polling stopped
[2018-07-29 19:45:24,06] [info] Gracefully shutdown sentry threads.
[2018-07-29 19:45:24,12] [info] Shutdown finished.
[2018-07-29 19:45:24,16] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-07-29 19:45:24,17] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-07-29 19:45:24,20] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-07-29 19:45:24,26] [info] JobExecutionTokenDispenser stopped
[2018-07-29 19:45:24,31] [info] WorkflowLogCopyRouter stopped
[2018-07-29 19:45:24,32] [info] Aborting all running workflows.
[2018-07-29 19:45:24,32] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-07-29 19:45:24,33] [info] WorkflowManagerActor Aborting all workflows
[2018-07-29 19:45:24,33] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: Aborting workflow
[2018-07-29 19:45:24,33] [info] WorkflowStoreActor stopped
[2018-07-29 19:45:24,93] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:19:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:19:1] Aborted StandardAsyncJob(15170)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:7:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:7:1] Aborted StandardAsyncJob(15085)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:15:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:15:1] Aborted StandardAsyncJob(15161)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:9:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:9:1] Aborted StandardAsyncJob(15101)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:8:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:8:1] Aborted StandardAsyncJob(15141)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:12:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:12:1] Aborted StandardAsyncJob(15188)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:16:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:16:1] Aborted StandardAsyncJob(15103)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:2:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:2:1] Aborted StandardAsyncJob(15149)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.CalculateContamination:NA:1] Aborted StandardAsyncJob(14955)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:10:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:10:1] Aborted StandardAsyncJob(15083)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:1:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:1:1] Aborted StandardAsyncJob(15099)
[2018-07-29 19:45:24,94] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:18:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:18:1] Aborted StandardAsyncJob(15156)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:3:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:3:1] Aborted StandardAsyncJob(15119)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:5:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:5:1] Aborted StandardAsyncJob(15143)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:17:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:17:1] Aborted StandardAsyncJob(15147)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:6:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:6:1] Aborted StandardAsyncJob(15173)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:11:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:11:1] Aborted StandardAsyncJob(15087)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:4:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:4:1] Aborted StandardAsyncJob(15159)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:0:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:0:1] Aborted StandardAsyncJob(15163)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:13:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:13:1] Aborted StandardAsyncJob(15151)
[2018-07-29 19:45:24,95] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:14:1]: BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0m:Mutect2.M2:14:1] Aborted StandardAsyncJob(15145)
[2018-07-29 19:45:28,30] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.CalculateContamination:NA:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:28,31] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.CalculateContamination:NA:1
[2018-07-29 19:45:38,44] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:2:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:38,46] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:2:1
[2018-07-29 19:45:43,96] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:14:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:43,97] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:14:1
[2018-07-29 19:45:45,91] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:12:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:45,92] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:12:1
[2018-07-29 19:45:46,17] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:7:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:46,18] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:7:1
[2018-07-29 19:45:46,18] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:4:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:46,19] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:4:1
[2018-07-29 19:45:48,53] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:19:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:48,54] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:19:1
[2018-07-29 19:45:49,84] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:10:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:49,84] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:10:1
[2018-07-29 19:45:53,13] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:9:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:53,14] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:9:1
[2018-07-29 19:45:53,67] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:5:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:53,67] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:5:1
[2018-07-29 19:45:58,51] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:11:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:45:58,51] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:11:1
[2018-07-29 19:46:12,21] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:3:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:12,21] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:3:1
[2018-07-29 19:46:14,12] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:1:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:14,12] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:1:1
[2018-07-29 19:46:16,46] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:17:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:16,46] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:17:1
[2018-07-29 19:46:21,27] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:0:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:21,29] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:0:1
[2018-07-29 19:46:21,66] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:18:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:21,67] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:18:1
[2018-07-29 19:46:23,54] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:16:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:23,55] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:16:1
[2018-07-29 19:46:25,36] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:8:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:25,36] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:8:1
[2018-07-29 19:46:38,78] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:15:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:38,78] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:15:1
[2018-07-29 19:46:44,20] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:13:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:44,22] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:13:1
[2018-07-29 19:46:46,92] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2m045183d0[0mMutect2.M2:6:1]: Status change from WaitingForReturnCodeFile to Done
[2018-07-29 19:46:46,93] [info] WorkflowExecutionActor-045183d0-3ee4-4df7-90a6-671609a0112e [[38;5;2m045183d0[0m]: WorkflowExecutionActor [[38;5;2m045183d0[0m] aborted: Mutect2.M2:6:1
[2018-07-29 19:46:47,29] [info] WorkflowManagerActor All workflows are aborted
[2018-07-29 19:46:47,29] [info] WorkflowManagerActor All workflows finished
[2018-07-29 19:46:47,29] [info] WorkflowManagerActor stopped
[2018-07-29 19:46:47,29] [info] Connection pools shut down
[2018-07-29 19:46:47,29] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-07-29 19:46:47,29] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-07-29 19:46:47,29] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-07-29 19:46:47,29] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-07-29 19:46:47,29] [info] SubWorkflowStoreActor stopped
[2018-07-29 19:46:47,29] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-07-29 19:46:47,29] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-07-29 19:46:47,29] [info] JobStoreActor stopped
[2018-07-29 19:46:47,29] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-07-29 19:46:47,29] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-07-29 19:46:47,29] [info] CallCacheWriteActor stopped
[2018-07-29 19:46:47,29] [info] IoProxy stopped
[2018-07-29 19:46:47,29] [info] DockerHashActor stopped
[2018-07-29 19:46:47,29] [info] WriteMetadataActor Shutting down: 21 queued messages to process
[2018-07-29 19:46:47,30] [info] WriteMetadataActor Shutting down: processing 0 queued messages
[2018-07-29 19:46:47,30] [info] ServiceRegistryActor stopped
[2018-07-29 19:46:47,32] [info] Database closed
[2018-07-29 19:46:47,32] [info] Stream materializer shut down

Any help would be much appreciated.

Comments

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @CWiseman,

    Where are you running this Cromwell from? I wonder if there is a scheduler aborting jobs for some strange reason...

  • Hi @Ruchi,

    I am running Cromwell on a local VM. I don't think there is a scheduler aborting my jobs because I have had successful Cromwell runs before using the same wdl and json scripts (with slight input variations in the json script between different samples). This is usually not a problem but I would rather it not happen at all. I didn't find anyone with a similar issue, so I'm not quite sure what steps to take or if there is a way to log more detailed messages when a shutdown occurs in the middle of a workflow.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @CWiseman,

    Is this issue reproducible for you on the same VM and running the same workflow/inputs?
    AFAIK, the only thing that should trigger a shutdown is a SIGINT. Do you see a return code file for the aborted jobs? Anything in the tail of the stderr/stdout logs that is unusual? Might be worth running this exact dataset & workflows on a larger VM if possible.

    Thanks

  • Hi @Ruchi,

    The issue is not reproducible. Usually just rerunning the workflow without changing anything fixes the issue. When the issue occurs, the stderr file just stops midway through with no error message. The stdout file however has a kill command that always says something like "killing pid 30142". I'm not sure why. Unfortunately I can't run the workflows on a larger VM.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @CWiseman,

    If you can't raise the memory, it may be helpful to add a max-concurrent-job-limit to reduce the load on the system. Assuming the failure is based on the system being overloaded, this configuration should help reduce the likelihood of hitting a load point.

Sign In or Register to comment.