Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/17/19
Release Notes can be found here.

why would a Workflow be marked as Failed?

esalinasesalinas BroadMember, Broadie ✭✭✭
edited February 2017 in Ask the FireCloud Team

I am reviewing a workflow. The workflow is marked as Failed (see attached PDF).

The workflow ran 3 tasks. Two are done (with green check marks) and one is marked "Running". Of the one that is marked "Running" if I ask gcloud alpha genomics operations describe about it, then it says done: true.

wm8b1-75c:~ esalinas$ gcloud alpha  genomics operations describe operations/EJOTlISoKxjaw-zrh9eokKkBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
done: true
metadata:
  '@type': type.googleapis.com/google.genomics.v1.OperationMetadata
  clientId: ''
  createTime: '2017-02-27T17:01:26Z'
  endTime: '2017-02-27T17:27:07Z'
  events:
  - description: start
    startTime: '2017-02-27T17:02:10.496569246Z'
  - description: pulling-image
    startTime: '2017-02-27T17:02:12.154200674Z'
  - description: localizing-files
    startTime: '2017-02-27T17:03:26.833744916Z'
  - description: running-docker
    startTime: '2017-02-27T17:11:43.078594022Z'
  - description: delocalizing-files
    startTime: '2017-02-27T17:26:58.038856218Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt"
    startTime: '2017-02-27T17:26:59.216433922Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/wigs.coverage.tgz"
    startTime: '2017-02-27T17:27:00.452218832Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/dstat.log.txt"
    startTime: '2017-02-27T17:27:01.672082122Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/df.log.txt"
    startTime: '2017-02-27T17:27:02.928070326Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/call_stats.tgz"
    startTime: '2017-02-27T17:27:04.333975179Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/power.coverage.tgz"
    startTime: '2017-02-27T17:27:05.908141602Z'
  - description: ok
    startTime: '2017-02-27T17:27:07.217623964Z'
  labels: {}
  projectId: broad-firecloud-benchmark
  request:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
    ephemeralPipeline:
      docker:
        cmd: /bin/bash /cromwell_root/exec.sh
        imageName: broadinstitute/broadmutationcalling_beta:benchmark_3
      inputParameters:
      - name: __extra_config_gcs_path
      - localCopy:
          disk: local-disk
          path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam.bai
        name: Mutect1_Task.tumorBamIdx-0
      - localCopy:
          disk: local-disk
          path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam.bai
        name: Mutect1_Task.normalBamIdx-0
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
        name: Mutect1_Task.refFastaIdx-0
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
        name: Mutect1_Task.refFastaDict-0
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
        name: Mutect1_Task.normalPanel-0
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
        name: Mutect1_Task.cosmicVCF-0
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
        name: Mutect1_Task.dbSNPVCF-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-CallSomaticMutations_131_Prepare_Task/split_intervals.tar.gz
        name: Mutect1_Task.mutectIntervals-0
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
        name: Mutect1_Task.readGroupBlackList-0
      - localCopy:
          disk: local-disk
          path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam
        name: Mutect1_Task.normalBam-0
      - localCopy:
          disk: local-disk
          path: exec.sh
        name: exec
      - localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
        name: Mutect1_Task.refFasta-0
      - localCopy:
          disk: local-disk
          path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam
        name: Mutect1_Task.tumorBam-0
      name: CallingGroup_Workflow
      outputParameters:
      - localCopy:
          disk: local-disk
          path: Mutect1_Task-rc.txt
        name: Mutect1_Task-rc.txt
      - localCopy:
          disk: local-disk
          path: wigs.coverage.tgz
        name: wigs.coverage.tgz
      - localCopy:
          disk: local-disk
          path: dstat.log.txt
        name: dstat.log.txt
      - localCopy:
          disk: local-disk
          path: df.log.txt
        name: df.log.txt
      - localCopy:
          disk: local-disk
          path: call_stats.tgz
        name: call_stats.tgz
      - localCopy:
          disk: local-disk
          path: power.coverage.tgz
        name: power.coverage.tgz
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - mountPoint: /cromwell_root
          name: local-disk
          sizeGb: 103
          type: PERSISTENT_HDD
        minimumCpuCores: 32
        minimumRamGb: 100
        preemptible: true
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
    pipelineArgs:
      clientId: ''
      inputs:
        Mutect1_Task.cosmicVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
        Mutect1_Task.dbSNPVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
        Mutect1_Task.mutectIntervals-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-CallSomaticMutations_131_Prepare_Task/split_intervals.tar.gz
        Mutect1_Task.normalBam-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam
        Mutect1_Task.normalBamIdx-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam.bai
        Mutect1_Task.normalPanel-0: gs://firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
        Mutect1_Task.readGroupBlackList-0: gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
        Mutect1_Task.refFasta-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
        Mutect1_Task.refFastaDict-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
        Mutect1_Task.refFastaIdx-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
        Mutect1_Task.tumorBam-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam
        Mutect1_Task.tumorBamIdx-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam.bai
        __extra_config_gcs_path: gs://cromwell-auth-broad-firecloud-benchmark/52641b4a-006f-4421-81a9-77443b420970_auth.json
        exec: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/exec.sh
      labels: {}
      logging:
        gcsPath: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task.log
      outputs:
        Mutect1_Task-rc.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt
        call_stats.tgz: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/call_stats.tgz
        df.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/df.log.txt
        dstat.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/dstat.log.txt
        power.coverage.tgz: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/power.coverage.tgz
        wigs.coverage.tgz: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/wigs.coverage.tgz
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: ''
          name: local-disk
          sizeGb: 103
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 32
        minimumRamGb: 100
        noAddress: false
        preemptible: true
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
      serviceAccount:
        email: default
        scopes:
        - https://www.googleapis.com/auth/genomics
        - https://www.googleapis.com/auth/compute
  runtimeMetadata:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
    computeEngine:
      diskNames:
      - local-disk-12186919504927465946
      instanceName: ggp-12186919504927465946
      machineType: us-central1-b/n1-standard-32
      zone: us-central1-b
  startTime: '2017-02-27T17:01:32Z'
name: operations/EJOTlISoKxjaw-zrh9eokKkBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
wm8b1-75c:~ esalinas$ gsutil gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/
CommandException: Invalid command "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/".
wm8b1-75c:~ esalinas$ gsutil ls gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/ 
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-stderr.log
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-stdout.log
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task.log
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/call_stats.tgz
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/df.log.txt
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/dstat.log.txt
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/exec.sh
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/power.coverage.tgz
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/wigs.coverage.tgz

Now, when I review the return code for the "Running" task and I see zero:

wm8b1-75c:~ esalinas$ gsutil cat  gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt
0

My question is what causes a workflow to be marked as Failed? In the case here, I would presume that the non-done task (without a green-checkmark and marked as "Running" would be marked "Failed" or have a non-zero return code. I reviewed the JES log and saw the all files were successfully delocalized.

fwf.pdf 70.8K
Sign In or Register to comment.