Latest Release: 03/12/19
Release Notes can be found here.

task is done but Cromwell marks it as "Running" prevents entire submission from being marked Done

esalinasesalinas BroadMember, Broadie ✭✭✭

I have a task that is the last task in the workflow. Its gcloud alpha genomics describe output is:

[email protected]:~/benchmarking$ gcloud alpha genomics operations describe operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
done: true
metadata:
  '@type': type.googleapis.com/google.genomics.v1.OperationMetadata
  clientId: ''
  createTime: '2017-02-25T11:32:33Z'
  endTime: '2017-02-25T11:34:10Z'
  events:
  - description: start
    startTime: '2017-02-25T11:33:08.421318432Z'
  - description: pulling-image
    startTime: '2017-02-25T11:33:10.946862092Z'
  - description: localizing-files
    startTime: '2017-02-25T11:33:45.985306414Z'
  - description: running-docker
    startTime: '2017-02-25T11:33:53.274120769Z'
  - description: delocalizing-files
    startTime: '2017-02-25T11:33:54.498750004Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.html"
    startTime: '2017-02-25T11:33:55.901959069Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.RData"
    startTime: '2017-02-25T11:33:57.060829757Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variants.vcf"
    startTime: '2017-02-25T11:33:58.718975202Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_effect_output.txt"
    startTime: '2017-02-25T11:34:00.694531228Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/vepReport.html"
    startTime: '2017-02-25T11:34:02.030846479Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/force_called.call_stats.txt"
    startTime: '2017-02-25T11:34:03.722244615Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/df.log.txt"
    startTime: '2017-02-25T11:34:05.004386006Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/dstat.log.txt"
    startTime: '2017-02-25T11:34:06.319586373Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/mutect_nozzle_task-rc.txt"
    startTime: '2017-02-25T11:34:07.393543466Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_nozzle_table.dat"
    startTime: '2017-02-25T11:34:08.572725025Z'
  - description: ok
    startTime: '2017-02-25T11:34:09.729690266Z'
  labels: {}
  projectId: broad-firecloud-benchmark
  request:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
    ephemeralPipeline:
      docker:
        cmd: /bin/bash /cromwell_root/exec.sh
        imageName: broadinstitute/broadmutationcalling_beta:benchmark_nozzle_1
      inputParameters:
      - name: __extra_config_gcs_path
      - localCopy:
          disk: local-disk
          path: exec.sh
        name: exec
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt
        name: mutect_nozzle_task.vepVCFFile-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-MutectFC_Task/MuTectFC.call_stats.txt
        name: mutect_nozzle_task.fcCallStats-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-GatherAndOncotate_Task/MuTect1.call_stats.vcf.filtered.vcf.annotated.vcf
        name: mutect_nozzle_task.vcfFile-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt_summary.html
        name: mutect_nozzle_task.vepReportHTML-0
      name: CallingGroup_Workflow
      outputParameters:
      - localCopy:
          disk: local-disk
          path: mutect_nozzle_task-rc.txt
        name: mutect_nozzle_task-rc.txt
      - localCopy:
          disk: local-disk
          path: df.log.txt
        name: df.log.txt
      - localCopy:
          disk: local-disk
          path: force_called.call_stats.txt
        name: force_called.call_stats.txt
      - localCopy:
          disk: local-disk
          path: vepReport.html
        name: vepReport.html
      - localCopy:
          disk: local-disk
          path: variant_nozzle_table.dat
        name: variant_nozzle_table.dat
      - localCopy:
          disk: local-disk
          path: variant_effect_output.txt
        name: variant_effect_output.txt
      - localCopy:
          disk: local-disk
          path: dstat.log.txt
        name: dstat.log.txt
      - localCopy:
          disk: local-disk
          path: nozzle.RData
        name: nozzle.RData
      - localCopy:
          disk: local-disk
          path: nozzle.html
        name: nozzle.html
      - localCopy:
          disk: local-disk
          path: variants.vcf
        name: variants.vcf
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - mountPoint: /cromwell_root
          name: local-disk
          sizeGb: 78
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 7
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
    pipelineArgs:
      clientId: ''
      inputs:
        __extra_config_gcs_path: gs://cromwell-auth-broad-firecloud-benchmark/00f5bfe7-c698-4460-aaf9-78e3b26159fc_auth.json
        exec: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/exec.sh
        mutect_nozzle_task.fcCallStats-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-MutectFC_Task/MuTectFC.call_stats.txt
        mutect_nozzle_task.vcfFile-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-GatherAndOncotate_Task/MuTect1.call_stats.vcf.filtered.vcf.annotated.vcf
        mutect_nozzle_task.vepReportHTML-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt_summary.html
        mutect_nozzle_task.vepVCFFile-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt
      labels: {}
      logging:
        gcsPath: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/mutect_nozzle_task.log
      outputs:
        df.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/df.log.txt
        dstat.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/dstat.log.txt
        force_called.call_stats.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/force_called.call_stats.txt
        mutect_nozzle_task-rc.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/mutect_nozzle_task-rc.txt
        nozzle.RData: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.RData
        nozzle.html: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.html
        variant_effect_output.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_effect_output.txt
        variant_nozzle_table.dat: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_nozzle_table.dat
        variants.vcf: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variants.vcf
        vepReport.html: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/vepReport.html
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: ''
          name: local-disk
          sizeGb: 78
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 7
        noAddress: false
        preemptible: false
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
      serviceAccount:
        email: default
        scopes:
        - https://www.googleapis.com/auth/genomics
        - https://www.googleapis.com/auth/compute
  runtimeMetadata:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
    computeEngine:
      diskNames:
      - local-disk-204270014222369576
      instanceName: ggp-204270014222369576
      machineType: us-central1-b/n1-standard-2
      zone: us-central1-b
  startTime: '2017-02-25T11:32:37Z'
name: operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
[email protected]:~/benchmarking$ 
[email protected]:~/benchmarking$ 

As can be seen in it, it took about 2 minutes to run. In contrast, it is still "Running" in the UI

CallingGroup_Workflow.mutect_nozzle_taskHide
Call #1:
ID:operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
Status:
Running
Started:February 25, 2017, 6:32 AM (10 hours ago)
Ended:Pending...
Inputs:Show
Outputs:None
stdout:mutect_nozzle_task-stdout.log
stderr:mutect_nozzle_task-stderr.log
JES log:mutect_nozzle_task.log

This is the last task of the last workflow in the submission that is causing the entire submission to be marked "Submitted" rather than "Done"

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    This prepare step was started Friday, but the worklfow finally marked it as done today

    CallingGroup_Workflow.CallSomaticMutations_131_Prepare_TaskHide
    Call #1:
    ID:operations/EIj7-IqnKxjW56_Bhqv2zP8BIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
    Status:Done
    Started:February 24, 2017, 1:24 PM (3 days ago)
    Ended:February 27, 2017, 7:35 AM (4 hours ago)
    Inputs:Show
    Outputs:Show
    stdout:CallSomaticMutations_131_Prepare_Task-stdout.log
    stderr:CallSomaticMutations_131_Prepare_Task-stderr.log
    JES log:CallSomaticMutations_131_Prepare_Task.log
    

    Over the weekend I did some visual inspection of the quota for this project (broa-dfirecloud-benchmark)
    and there was plenty of quota available. See attached screenshot that I took Sunday morning (2/26)

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    This task (mentioned above) took about 18 hours to get marked done.
    I note that it is the last task in the workflow.
    I note (as mentioned above) that it finished in about two minutes

    CallingGroup_Workflow.mutect_nozzle_taskHide
    Call #1:
    ID:operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
    Status:Done
    Started:February 25, 2017, 6:32 AM (3 days ago)
    Ended:February 26, 2017, 12:28 AM (2 days ago)
    Inputs:Show
    Outputs:Show
    stdout:mutect_nozzle_task-stdout.log
    stderr:mutect_nozzle_task-stderr.log
    JES log:mutect_nozzle_task.log
    
    
Sign In or Register to comment.