We've moved!
You can find our new documentation site and support forum for posting questions here.

task is done but Cromwell marks it as "Running" prevents entire submission from being marked Done

esalinasesalinas BroadMember, Broadie ✭✭✭

I have a task that is the last task in the workflow. Its gcloud alpha genomics describe output is:

[email protected]:~/benchmarking$ gcloud alpha genomics operations describe operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
done: true
metadata:
  '@type': type.googleapis.com/google.genomics.v1.OperationMetadata
  clientId: ''
  createTime: '2017-02-25T11:32:33Z'
  endTime: '2017-02-25T11:34:10Z'
  events:
  - description: start
    startTime: '2017-02-25T11:33:08.421318432Z'
  - description: pulling-image
    startTime: '2017-02-25T11:33:10.946862092Z'
  - description: localizing-files
    startTime: '2017-02-25T11:33:45.985306414Z'
  - description: running-docker
    startTime: '2017-02-25T11:33:53.274120769Z'
  - description: delocalizing-files
    startTime: '2017-02-25T11:33:54.498750004Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.html"
    startTime: '2017-02-25T11:33:55.901959069Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.RData"
    startTime: '2017-02-25T11:33:57.060829757Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variants.vcf"
    startTime: '2017-02-25T11:33:58.718975202Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_effect_output.txt"
    startTime: '2017-02-25T11:34:00.694531228Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/vepReport.html"
    startTime: '2017-02-25T11:34:02.030846479Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/force_called.call_stats.txt"
    startTime: '2017-02-25T11:34:03.722244615Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/df.log.txt"
    startTime: '2017-02-25T11:34:05.004386006Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/dstat.log.txt"
    startTime: '2017-02-25T11:34:06.319586373Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/mutect_nozzle_task-rc.txt"
    startTime: '2017-02-25T11:34:07.393543466Z'
  - description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_nozzle_table.dat"
    startTime: '2017-02-25T11:34:08.572725025Z'
  - description: ok
    startTime: '2017-02-25T11:34:09.729690266Z'
  labels: {}
  projectId: broad-firecloud-benchmark
  request:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
    ephemeralPipeline:
      docker:
        cmd: /bin/bash /cromwell_root/exec.sh
        imageName: broadinstitute/broadmutationcalling_beta:benchmark_nozzle_1
      inputParameters:
      - name: __extra_config_gcs_path
      - localCopy:
          disk: local-disk
          path: exec.sh
        name: exec
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt
        name: mutect_nozzle_task.vepVCFFile-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-MutectFC_Task/MuTectFC.call_stats.txt
        name: mutect_nozzle_task.fcCallStats-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-GatherAndOncotate_Task/MuTect1.call_stats.vcf.filtered.vcf.annotated.vcf
        name: mutect_nozzle_task.vcfFile-0
      - localCopy:
          disk: local-disk
          path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt_summary.html
        name: mutect_nozzle_task.vepReportHTML-0
      name: CallingGroup_Workflow
      outputParameters:
      - localCopy:
          disk: local-disk
          path: mutect_nozzle_task-rc.txt
        name: mutect_nozzle_task-rc.txt
      - localCopy:
          disk: local-disk
          path: df.log.txt
        name: df.log.txt
      - localCopy:
          disk: local-disk
          path: force_called.call_stats.txt
        name: force_called.call_stats.txt
      - localCopy:
          disk: local-disk
          path: vepReport.html
        name: vepReport.html
      - localCopy:
          disk: local-disk
          path: variant_nozzle_table.dat
        name: variant_nozzle_table.dat
      - localCopy:
          disk: local-disk
          path: variant_effect_output.txt
        name: variant_effect_output.txt
      - localCopy:
          disk: local-disk
          path: dstat.log.txt
        name: dstat.log.txt
      - localCopy:
          disk: local-disk
          path: nozzle.RData
        name: nozzle.RData
      - localCopy:
          disk: local-disk
          path: nozzle.html
        name: nozzle.html
      - localCopy:
          disk: local-disk
          path: variants.vcf
        name: variants.vcf
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - mountPoint: /cromwell_root
          name: local-disk
          sizeGb: 78
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 7
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
    pipelineArgs:
      clientId: ''
      inputs:
        __extra_config_gcs_path: gs://cromwell-auth-broad-firecloud-benchmark/00f5bfe7-c698-4460-aaf9-78e3b26159fc_auth.json
        exec: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/exec.sh
        mutect_nozzle_task.fcCallStats-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-MutectFC_Task/MuTectFC.call_stats.txt
        mutect_nozzle_task.vcfFile-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-GatherAndOncotate_Task/MuTect1.call_stats.vcf.filtered.vcf.annotated.vcf
        mutect_nozzle_task.vepReportHTML-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt_summary.html
        mutect_nozzle_task.vepVCFFile-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-VEP_Task/variant_effect_output.txt
      labels: {}
      logging:
        gcsPath: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/mutect_nozzle_task.log
      outputs:
        df.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/df.log.txt
        dstat.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/dstat.log.txt
        force_called.call_stats.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/force_called.call_stats.txt
        mutect_nozzle_task-rc.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/mutect_nozzle_task-rc.txt
        nozzle.RData: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.RData
        nozzle.html: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/nozzle.html
        variant_effect_output.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_effect_output.txt
        variant_nozzle_table.dat: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variant_nozzle_table.dat
        variants.vcf: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/variants.vcf
        vepReport.html: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/89cdfe00-4a00-4780-a0df-fc2a4371ab12/CallingGroup_Workflow/00f5bfe7-c698-4460-aaf9-78e3b26159fc/call-mutect_nozzle_task/vepReport.html
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: ''
          name: local-disk
          sizeGb: 78
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 7
        noAddress: false
        preemptible: false
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
      serviceAccount:
        email: default
        scopes:
        - https://www.googleapis.com/auth/genomics
        - https://www.googleapis.com/auth/compute
  runtimeMetadata:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
    computeEngine:
      diskNames:
      - local-disk-204270014222369576
      instanceName: ggp-204270014222369576
      machineType: us-central1-b/n1-standard-2
      zone: us-central1-b
  startTime: '2017-02-25T11:32:37Z'
name: operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
[email protected]:~/benchmarking$ 
[email protected]:~/benchmarking$ 

As can be seen in it, it took about 2 minutes to run. In contrast, it is still "Running" in the UI

CallingGroup_Workflow.mutect_nozzle_taskHide
Call #1:
ID:operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
Status:
Running
Started:February 25, 2017, 6:32 AM (10 hours ago)
Ended:Pending...
Inputs:Show
Outputs:None
stdout:mutect_nozzle_task-stdout.log
stderr:mutect_nozzle_task-stderr.log
JES log:mutect_nozzle_task.log

This is the last task of the last workflow in the submission that is causing the entire submission to be marked "Submitted" rather than "Done"

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    This prepare step was started Friday, but the worklfow finally marked it as done today

    CallingGroup_Workflow.CallSomaticMutations_131_Prepare_TaskHide
    Call #1:
    ID:operations/EIj7-IqnKxjW56_Bhqv2zP8BIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
    Status:Done
    Started:February 24, 2017, 1:24 PM (3 days ago)
    Ended:February 27, 2017, 7:35 AM (4 hours ago)
    Inputs:Show
    Outputs:Show
    stdout:CallSomaticMutations_131_Prepare_Task-stdout.log
    stderr:CallSomaticMutations_131_Prepare_Task-stderr.log
    JES log:CallSomaticMutations_131_Prepare_Task.log
    

    Over the weekend I did some visual inspection of the quota for this project (broa-dfirecloud-benchmark)
    and there was plenty of quota available. See attached screenshot that I took Sunday morning (2/26)

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    This task (mentioned above) took about 18 hours to get marked done.
    I note that it is the last task in the workflow.
    I note (as mentioned above) that it finished in about two minutes

    CallingGroup_Workflow.mutect_nozzle_taskHide
    Call #1:
    ID:operations/EJzrrKinKxiohvSf7s_t6gIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
    Status:Done
    Started:February 25, 2017, 6:32 AM (3 days ago)
    Ended:February 26, 2017, 12:28 AM (2 days ago)
    Inputs:Show
    Outputs:Show
    stdout:mutect_nozzle_task-stdout.log
    stderr:mutect_nozzle_task-stderr.log
    JES log:mutect_nozzle_task.log
    
    
Sign In or Register to comment.