We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

Errors from local implementation of gatk4-data-processing

OjoOjo UKMember
edited September 2019 in Ask the GATK team

Hi,

I have some issues while trying to implement the localgatk4-data-processing workflow locally, and none of my troubleshooting attempts has been successful.

OS: Ubuntu
RAM: 250GB
wdl :https://github.com/gatk-workflows/gatk4-data-processing/blob/master/processing-for-variant-discovery-gatk4.wdl
json: https://github.com/gatk-workflows/gatk4-data-processing/blob/master/processing-for-variant-discovery-gatk4.hg38.wgs.inputs.json

2 input files:
53G file1.bam
46G file2.bam

cmd:
java -jar cromwell-45.1.jar run ./gatk4-data-processing/processing-for-variant-discovery-gatk4.wdl --inputs ./gatk4-data-processing/processing-for-variant-discovery-gatk4.hg38.wgs.inputs.json

My runs generated a lot of errors - please see below:

java -jar cromwell-45.1.jar run ./gatk4-data-processing/processing-for-variant-discovery-gatk4.wdl --inputs ./gatk4-data-processing/processing-for-variant-discovery-gatk4.hg38.wgs.inputs.json
[2019-09-12 21:44:54,31] [info] Running with database db.url = jdbc:hsqldb:mem:b35fe689-8d58-4549-bb37-11b13e3b321a;shutdown=false;hsqldb.tx=mvcc
[2019-09-12 21:45:10,46] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2019-09-12 21:45:10,48] [info] [RenameWorkflowOptionsInMetadata] 100%
[2019-09-12 21:45:10,68] [info] Running with database db.url = jdbc:hsqldb:mem:26c5b3a9-fc2e-4ee6-8bda-aa69107d7690;shutdown=false;hsqldb.tx=mvcc
[2019-09-12 21:45:11,40] [info] Slf4jLogger started
[2019-09-12 21:45:12,00] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-f7a4c01",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "failureShutdownDuration" : "5 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2019-09-12 21:45:12,13] [info] Metadata summary refreshing every 1 second.
[2019-09-12 21:45:12,28] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2019-09-12 21:45:12,28] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2019-09-12 21:45:12,29] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2019-09-12 21:45:12,31] [warn] 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf)
[2019-09-12 21:45:12,49] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2019-09-12 21:45:13,10] [info] SingleWorkflowRunnerActor: Version 45.1
[2019-09-12 21:45:13,11] [info] SingleWorkflowRunnerActor: Submitting workflow
[2019-09-12 21:45:13,19] [info] Unspecified type (Unspecified version) workflow e490ef62-0435-4992-9008-c1b1a03006cf submitted
[2019-09-12 21:45:13,22] [info] SingleWorkflowRunnerActor: Workflow submitted e490ef62-0435-4992-9008-c1b1a03006cf
[2019-09-12 21:45:13,24] [info] 1 new workflows fetched by cromid-f7a4c01: e490ef62-0435-4992-9008-c1b1a03006cf
[2019-09-12 21:45:13,26] [info] WorkflowManagerActor Starting workflow e490ef62-0435-4992-9008-c1b1a03006cf
[2019-09-12 21:45:13,28] [info] WorkflowManagerActor Successfully started WorkflowActor-e490ef62-0435-4992-9008-c1b1a03006cf
[2019-09-12 21:45:13,28] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2019-09-12 21:45:13,32] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2019-09-12 21:45:13,52] [info] MaterializeWorkflowDescriptorActor [e490ef62]: Parsing workflow as WDL draft-2
[2019-09-12 21:45:15,28] [info] MaterializeWorkflowDescriptorActor [e490ef62]: Call-to-Backend assignments: PreProcessingForVariantDiscovery_GATK4.SortAndFixTags -> Local, PreProcessingForVariantDiscovery_GATK4.GatherBqsrReports -> Local, PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV -> Local, PreProcessingForVariantDiscovery_GATK4.GetBwaVersion -> Local, PreProcessingForVariantDiscovery_GATK4.MarkDuplicates -> Local, PreProcessingForVariantDiscovery_GATK4.GatherBamFiles -> Local, PreProcessingForVariantDiscovery_GATK4.BaseRecalibrator -> Local, PreProcessingForVariantDiscovery_GATK4.MergeBamAlignment -> Local, PreProcessingForVariantDiscovery_GATK4.ApplyBQSR -> Local, PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem -> Local
[2019-09-12 21:45:15,51] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,51] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,51] [warn] Local [e490ef62]: Key/s [preemptible, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,51] [warn] Local [e490ef62]: Key/s [preemptible, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,52] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,52] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,52] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,52] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,52] [warn] Local [e490ef62]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:15,52] [warn] Local [e490ef62]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2019-09-12 21:45:17,50] [info] Not triggering log of token queue status. Effective log interval = None
[2019-09-12 21:45:19,63] [info] WorkflowExecutionActor-e490ef62-0435-4992-9008-c1b1a03006cf [e490ef62]: Starting PreProcessingForVariantDiscovery_GATK4.GetBwaVersion, PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV
[2019-09-12 21:45:20,52] [info] Assigned new job execution tokens to the following groups: e490ef62: 2
[2019-09-12 21:45:22,87] [info] WorkflowExecutionActor-e490ef62-0435-4992-9008-c1b1a03006cf [e490ef62]: Starting PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem (2 shards)
[2019-09-12 21:45:23,50] [warn] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.GetBwaVersion:NA:1]: Unrecognized runtime attribute keys: preemptible, memory
[2019-09-12 21:45:23,50] [warn] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV:NA:1]: Unrecognized runtime attribute keys: preemptible, memory
[2019-09-12 21:45:23,51] [info] Assigned new job execution tokens to the following groups: e490ef62: 2
[2019-09-12 21:45:23,55] [warn] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:0:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, memory
[2019-09-12 21:45:23,56] [warn] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:1:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, memory
[2019-09-12 21:45:23,75] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.GetBwaVersion:NA:1]: # Not setting "set -o pipefail" here because /bwa has a rc=1 and we don't want to allow rc=1 to succeed 
# because the sed may also fail with that error and that is something we actually want to fail on.
/usr/gitc/bwa 2>&1 | \
grep -e '^Version' | \
sed 's/Version: //'
[2019-09-12 21:45:23,76] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV:NA:1]: python <<CODE
with open("/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/inputs/695027235/Homo_sapiens_assembly38.dict", "r") as ref_dict_file:
    sequence_tuple_list = []
    longest_sequence = 0
    for line in ref_dict_file:
        if line.startswith("@SQ"):
            line_split = line.split("\t")
            # (Sequence_Name, Sequence_Length)
            sequence_tuple_list.append((line_split[1].split("SN:")[1], int(line_split[2].split("LN:")[1])))
    longest_sequence = sorted(sequence_tuple_list, key=lambda x: x[1], reverse=True)[0][1]
# We are adding this to the intervals because hg38 has contigs named with embedded colons (:) and a bug in 
# some versions of GATK strips off the last element after a colon, so we add this as a sacrificial element.
hg38_protection_tag = ":1+"
# initialize the tsv string with the first sequence
tsv_string = sequence_tuple_list[0][0] + hg38_protection_tag
temp_size = sequence_tuple_list[0][1]
for sequence_tuple in sequence_tuple_list[1:]:
    if temp_size + sequence_tuple[1] <= longest_sequence:
        temp_size += sequence_tuple[1]
        tsv_string += "\t" + sequence_tuple[0] + hg38_protection_tag
    else:
        tsv_string += "\n" + sequence_tuple[0] + hg38_protection_tag
        temp_size = sequence_tuple[1]
# add the unmapped sequences as a separate line to ensure that they are recalibrated as well
with open("sequence_grouping.txt","w") as tsv_file:
  tsv_file.write(tsv_string)
  tsv_file.close()

tsv_string += '\n' + "unmapped"

with open("sequence_grouping_with_unmapped.txt","w") as tsv_file_with_unmapped:
  tsv_file_with_unmapped.write(tsv_string)
  tsv_file_with_unmapped.close()
CODE
[2019-09-12 21:45:23,76] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:0:1]:   set -o pipefail
  set -e

  # set the bash variable needed for the command-line
  bash_ref_fasta=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/inputs/695027235/Homo_sapiens_assembly38.fasta

java -Dsamjdk.compression_level=5 -Xms10000m -jar /usr/gitc/picard.jar \
    SamToFastq \
    INPUT=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/inputs/-2017269263/SRR7665096_fastqtosam.unmapped.bam \
    FASTQ=/dev/stdout \
    INTERLEAVE=true \
    NON_PF=true \
  | \
/usr/gitc/bwa mem -K 100000000 -p -v 3 -t 16 -Y $bash_ref_fasta /dev/stdin -  2> >(tee SRR7665096_fastqtosam.unmapped.unmerged.bwa.stderr.log >&2) \
  | \
samtools view -1 - > SRR7665096_fastqtosam.unmapped.unmerged.bam
[2019-09-12 21:45:23,77] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:1:1]:   set -o pipefail
  set -e

  # set the bash variable needed for the command-line
  bash_ref_fasta=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/inputs/695027235/Homo_sapiens_assembly38.fasta

java -Dsamjdk.compression_level=5 -Xms10000m -jar /usr/gitc/picard.jar \
    SamToFastq \
    INPUT=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/inputs/-2017269263/SRR7665097_fastqtosam.unmapped.bam \
    FASTQ=/dev/stdout \
    INTERLEAVE=true \
    NON_PF=true \
  | \
/usr/gitc/bwa mem -K 100000000 -p -v 3 -t 16 -Y $bash_ref_fasta /dev/stdin -  2> >(tee SRR7665097_fastqtosam.unmapped.unmerged.bwa.stderr.log >&2) \
  | \
samtools view -1 - > SRR7665097_fastqtosam.unmapped.unmerged.bam
[2019-09-12 21:45:23,88] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.GetBwaVersion:NA:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion:/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion:delegated \
  broadinstitute/[email protected]:4fca8ca945c17fd86e31eeef1c02983e091d4f2cb437199e74b164d177d5b2d1 /cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/docker_cid)

# remove the container after waiting
docker rm cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/docker_cid

# return exit code
exit $rc
[2019-09-12 21:45:23,88] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:1:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1:/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1:delegated \
  broadinstitute/[email protected]:4fca8ca945c17fd86e31eeef1c02983e091d4f2cb437199e74b164d177d5b2d1 /cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/docker_cid)

# remove the container after waiting
docker rm cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/docker_cid

# return exit code
exit $rc
[2019-09-12 21:45:23,88] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:0:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0:/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0:delegated \
  broadinstitute/[email protected]:4fca8ca945c17fd86e31eeef1c02983e091d4f2cb437199e74b164d177d5b2d1 /cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/docker_cid)

# remove the container after waiting
docker rm cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/docker_cid

# return exit code
exit $rc
[2019-09-12 21:45:23,89] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV:NA:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV:/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV:delegated \
  [email protected]:f8ddb17441f991ebe54b6cec809ecfef0d23bc996a9874fa892734c6567d7a7e /cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/docker_cid)

# remove the container after waiting
docker rm cat /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/docker_cid

# return exit code
exit $rc
[2019-09-12 21:45:27,36] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV:NA:1]: job id: 1069
[2019-09-12 21:45:27,36] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:1:1]: job id: 1070
[2019-09-12 21:45:27,36] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:0:1]: job id: 1078
[2019-09-12 21:45:27,36] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.GetBwaVersion:NA:1]: job id: 1075
[2019-09-12 21:45:27,37] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:0:1]: Status change from - to Done
[2019-09-12 21:45:27,38] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.GetBwaVersion:NA:1]: Status change from - to Done
[2019-09-12 21:45:27,38] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV:NA:1]: Status change from - to Done
[2019-09-12 21:45:27,38] [info] BackgroundConfigAsyncJobExecutionActor [e490ef62PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:1:1]: Status change from - to Done
[2019-09-12 21:45:28,04] [info] WorkflowManagerActor Workflow e490ef62-0435-4992-9008-c1b1a03006cf failed (during ExecutingWorkflowState): Job PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:1:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/stderr.
 Could not retrieve content: Could not read from /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/stderr: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-1/execution/stderr
Job PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem:0:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/stderr.
 Could not retrieve content: Could not read from /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/stderr: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-SamToFastqAndBwaMem/shard-0/execution/stderr
Job PreProcessingForVariantDiscovery_GATK4.GetBwaVersion:NA:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/stderr.
 Could not retrieve content: Could not read from /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/stderr: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-GetBwaVersion/execution/stderr
Job PreProcessingForVariantDiscovery_GATK4.CreateSequenceGroupingTSV:NA:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/stderr.
 Could not retrieve content: Could not read from /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/stderr: /gatk/gatk-workflows/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/e490ef62-0435-4992-9008-c1b1a03006cf/call-CreateSequenceGroupingTSV/execution/stderr
[2019-09-12 21:45:28,04] [info] WorkflowManagerActor WorkflowActor-e490ef62-0435-4992-9008-c1b1a03006cf is in a terminal state: WorkflowFailedState
[2019-09-12 21:45:35,44] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2019-09-12 21:45:37,31] [info] Workflow polling stopped
[2019-09-12 21:45:37,34] [info] 0 workflows released by cromid-f7a4c01
[2019-09-12 21:45:37,35] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2019-09-12 21:45:37,36] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2019-09-12 21:45:37,36] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2019-09-12 21:45:37,37] [info] Aborting all running workflows.
[2019-09-12 21:45:37,37] [info] JobExecutionTokenDispenser stopped
[2019-09-12 21:45:37,38] [info] WorkflowStoreActor stopped
[2019-09-12 21:45:37,38] [info] WorkflowLogCopyRouter stopped
[2019-09-12 21:45:37,38] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2019-09-12 21:45:37,38] [info] WorkflowManagerActor All workflows finished
[2019-09-12 21:45:37,38] [info] WorkflowManagerActor stopped
[2019-09-12 21:45:37,91] [info] Connection pools shut down
[2019-09-12 21:45:37,91] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2019-09-12 21:45:37,91] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2019-09-12 21:45:37,91] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2019-09-12 21:45:37,91] [info] SubWorkflowStoreActor stopped
[2019-09-12 21:45:37,91] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2019-09-12 21:45:37,91] [info] JobStoreActor stopped
[2019-09-12 21:45:37,91] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2019-09-12 21:45:37,91] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2019-09-12 21:45:37,92] [info] CallCacheWriteActor stopped
[2019-09-12 21:45:37,92] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2019-09-12 21:45:37,92] [info] DockerHashActor stopped
[2019-09-12 21:45:37,92] [info] KvWriteActor Shutting down: 0 queued messages to process
[2019-09-12 21:45:37,91] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2019-09-12 21:45:37,93] [info] ServiceRegistryActor stopped
[2019-09-12 21:45:37,93] [info] IoProxy stopped
[2019-09-12 21:45:37,98] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2019-09-12 21:45:37,98] [info] Shutting down connection pool: curAllocated=2 idleQueues.size=1 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2019-09-12 21:45:37,98] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2019-09-12 21:45:37,98] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2019-09-12 21:45:38,06] [info] Database closed
[2019-09-12 21:45:38,06] [info] Stream materializer shut down
[2019-09-12 21:45:38,06] [info] WDL HTTP import resolver closed
Workflow e490ef62-0435-4992-9008-c1b1a03006cf transitioned to state Failed

Please help find the source of the problem.

Tagged:
Sign In or Register to comment.