Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.

additional observation of Unusually long submission delay

esalinasesalinas BroadMember, Broadie ✭✭✭

This post is related to : http://gatkforums.broadinstitute.org/firecloud/discussion/9675/unusually-long-submission-delay#latest

Today, I seem to be observing delays (like those from the previously-linked post) . For example the task below took about an hour but when no delays are observed takes 15 minutes.

CallingGroup_Workflow.Mutect1_TaskHide
Call #1:
Operation:operations/EM_OupnIKxiM_Ja1z8KslRIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
Status:Done
Cache Result:Miss
Started:June 7, 2017, 10:58 AM (2 hours ago)
Ended:June 7, 2017, 12:05 PM (1 hour ago)
Inputs:Show
Outputs:Show
stdout:Mutect1_Task-0-stdout.log
stderr:Mutect1_Task-0-stderr.log
JES log:Mutect1_Task-0.log

Here is an example of a task without delay:

CallingGroup_Workflow.Mutect1_TaskHide
Call #1:
Operation:operations/EISuoPrHKxjTyYeOsfrlrJMBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
Status:Done
Cache Result:Miss
Started:June 6, 2017, 5:20 PM (20 hours ago)
Ended:June 6, 2017, 5:38 PM (19 hours ago)
Inputs:Show
Outputs:Show
stdout:Mutect1_Task-0-stdout.log
stderr:Mutect1_Task-0-stderr.log
JES log:Mutect1_Task-0.log


The operations descriptions for each are shown here. Once each is "created" (see "createTime") then the endTime is about 17-18 minutes later. For the first operation however, cromwell shows a time of closer to 1 hour (10:58-12:05).

wm8b1-75c:~ esalinas$ gcloud alpha genomics operations describe  operations/EM_OupnIKxiM_Ja1z8KslRIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
done: true
metadata:
  '@type': type.googleapis.com/google.genomics.v1.OperationMetadata
  clientId: ''
  createTime: '2017-06-07T15:31:31Z'
  endTime: '2017-06-07T15:47:40Z'
  events:
  - description: start
    startTime: '2017-06-07T15:32:30.575789627Z'
  - description: pulling-image
    startTime: '2017-06-07T15:32:32.802412488Z'
  - description: localizing-files
    startTime: '2017-06-07T15:33:46.826171474Z'
  - description: running-docker
    startTime: '2017-06-07T15:36:11.041332740Z'
  - description: delocalizing-files
    startTime: '2017-06-07T15:47:33.418218436Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/Mutect1_Task-0-rc.txt"
    startTime: '2017-06-07T15:47:35.055168003Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.call_stats.txt"
    startTime: '2017-06-07T15:47:36.535007878Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.power.wig.txt"
    startTime: '2017-06-07T15:47:37.895861890Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.coverage.wig.txt"
    startTime: '2017-06-07T15:47:39.271279483Z'
  - description: ok
    startTime: '2017-06-07T15:47:40.594967031Z'
  labels:
    cromwell-workflow-id: cromwell-9b7dc36e-e577-419b-9ace-93abaf3e8b52
    cromwell-workflow-name: callinggroup-workflow
    wdl-task-name: mutect1-task
  projectId: broad-firecloud-benchmark
  request:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
    ephemeralPipeline:
      description: ''
      docker:
        cmd: /bin/bash /cromwell_root/exec.sh
        imageName: broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb
      inputParameters:
      - defaultValue: ''
        description: ''
        name: __extra_config_gcs_path
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bam
        name: Mutect1_Task.normalBam-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
        name: Mutect1_Task.refFastaDict-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bai
        name: Mutect1_Task.normalBamIdx-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
        name: Mutect1_Task.dbSNPVCF-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
        name: Mutect1_Task.refFasta-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bai
        name: Mutect1_Task.tumorBamIdx-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bam
        name: Mutect1_Task.tumorBam-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-CallSomaticMutations_131_Prepare_Task/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.01.interval_list
        name: Mutect1_Task.mutectIntervals-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
        name: Mutect1_Task.readGroupBlackList-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: exec.sh
        name: exec
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
        name: Mutect1_Task.refFastaIdx-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
        name: Mutect1_Task.normalPanel-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
        name: Mutect1_Task.cosmicVCF-0
      name: CallingGroup_Workflow
      outputParameters:
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: Mutect1_Task-0-rc.txt
        name: Mutect1_Task-0-rc.txt
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: HCC1954_100_gene_pair.MuTect1.call_stats.txt
        name: HCC1954_100_gene_pair.MuTect1.call_stats.txt
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: HCC1954_100_gene_pair.MuTect1.power.wig.txt
        name: HCC1954_100_gene_pair.MuTect1.power.wig.txt
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: HCC1954_100_gene_pair.MuTect1.coverage.wig.txt
        name: HCC1954_100_gene_pair.MuTect1.coverage.wig.txt
      pipelineId: ''
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: /cromwell_root
          name: local-disk
          readOnly: false
          sizeGb: 19
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 3
        noAddress: false
        preemptible: true
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
    pipelineArgs:
      clientId: ''
      inputs:
        Mutect1_Task.cosmicVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
        Mutect1_Task.dbSNPVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
        Mutect1_Task.mutectIntervals-0: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-CallSomaticMutations_131_Prepare_Task/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.01.interval_list
        Mutect1_Task.normalBam-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bam
        Mutect1_Task.normalBamIdx-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bai
        Mutect1_Task.normalPanel-0: gs://firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
        Mutect1_Task.readGroupBlackList-0: gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
        Mutect1_Task.refFasta-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
        Mutect1_Task.refFastaDict-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
        Mutect1_Task.refFastaIdx-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
        Mutect1_Task.tumorBam-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bam
        Mutect1_Task.tumorBamIdx-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bai
        __extra_config_gcs_path: gs://cromwell-auth-broad-firecloud-benchmark/9b7dc36e-e577-419b-9ace-93abaf3e8b52_auth.json
        exec: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/exec.sh
      labels:
        cromwell-workflow-id: cromwell-9b7dc36e-e577-419b-9ace-93abaf3e8b52
        cromwell-workflow-name: callinggroup-workflow
        wdl-task-name: mutect1-task
      logging:
        gcsPath: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/Mutect1_Task-0.log
      outputs:
        HCC1954_100_gene_pair.MuTect1.call_stats.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.call_stats.txt
        HCC1954_100_gene_pair.MuTect1.coverage.wig.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.coverage.wig.txt
        HCC1954_100_gene_pair.MuTect1.power.wig.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.power.wig.txt
        Mutect1_Task-0-rc.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/3791f6f9-5fba-495c-908a-9ea035c9ad26/CallingGroup_Workflow/9b7dc36e-e577-419b-9ace-93abaf3e8b52/call-Mutect1_Task/shard-0/Mutect1_Task-0-rc.txt
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: ''
          name: local-disk
          readOnly: false
          sizeGb: 19
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 3
        noAddress: false
        preemptible: true
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
      serviceAccount:
        email: default
        scopes:
        - https://www.googleapis.com/auth/genomics
        - https://www.googleapis.com/auth/compute
  runtimeMetadata:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
    computeEngine:
      diskNames:
      - local-disk-1309054444811697676
      instanceName: ggp-1309054444811697676
      machineType: us-central1-c/n1-standard-1
      zone: us-central1-c
  startTime: '2017-06-07T15:31:38Z'
name: operations/EM_OupnIKxiM_Ja1z8KslRIg3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
wm8b1-75c:~ esalinas$ gcloud alpha genomics operations describe operations/EISuoPrHKxjTyYeOsfrlrJMBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
done: true
metadata:
  '@type': type.googleapis.com/google.genomics.v1.OperationMetadata
  clientId: ''
  createTime: '2017-06-06T21:20:49Z'
  endTime: '2017-06-06T21:38:16Z'
  events:
  - description: start
    startTime: '2017-06-06T21:22:09.204113273Z'
  - description: pulling-image
    startTime: '2017-06-06T21:22:11.618630382Z'
  - description: localizing-files
    startTime: '2017-06-06T21:23:32.952560276Z'
  - description: running-docker
    startTime: '2017-06-06T21:25:52.819738503Z'
  - description: delocalizing-files
    startTime: '2017-06-06T21:38:07.995202628Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.coverage.wig.txt"
    startTime: '2017-06-06T21:38:09.698515389Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/Mutect1_Task-0-rc.txt"
    startTime: '2017-06-06T21:38:11.524526146Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.call_stats.txt"
    startTime: '2017-06-06T21:38:12.964677052Z'
  - description: copied 1 file(s) to "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.power.wig.txt"
    startTime: '2017-06-06T21:38:14.384774842Z'
  - description: ok
    startTime: '2017-06-06T21:38:15.834811124Z'
  labels:
    cromwell-workflow-id: cromwell-418a4c92-0821-4695-acc8-291472136fb9
    cromwell-workflow-name: callinggroup-workflow
    wdl-task-name: mutect1-task
  projectId: broad-firecloud-benchmark
  request:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
    ephemeralPipeline:
      description: ''
      docker:
        cmd: /bin/bash /cromwell_root/exec.sh
        imageName: broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb
      inputParameters:
      - defaultValue: ''
        description: ''
        name: __extra_config_gcs_path
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bam
        name: Mutect1_Task.normalBam-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
        name: Mutect1_Task.refFastaDict-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bai
        name: Mutect1_Task.normalBamIdx-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
        name: Mutect1_Task.dbSNPVCF-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-CallSomaticMutations_131_Prepare_Task/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.01.interval_list
        name: Mutect1_Task.mutectIntervals-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
        name: Mutect1_Task.refFasta-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bai
        name: Mutect1_Task.tumorBamIdx-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bam
        name: Mutect1_Task.tumorBam-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
        name: Mutect1_Task.readGroupBlackList-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: exec.sh
        name: exec
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
        name: Mutect1_Task.refFastaIdx-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
        name: Mutect1_Task.normalPanel-0
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
        name: Mutect1_Task.cosmicVCF-0
      name: CallingGroup_Workflow
      outputParameters:
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: Mutect1_Task-0-rc.txt
        name: Mutect1_Task-0-rc.txt
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: HCC1954_100_gene_pair.MuTect1.call_stats.txt
        name: HCC1954_100_gene_pair.MuTect1.call_stats.txt
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: HCC1954_100_gene_pair.MuTect1.power.wig.txt
        name: HCC1954_100_gene_pair.MuTect1.power.wig.txt
      - defaultValue: ''
        description: ''
        localCopy:
          disk: local-disk
          path: HCC1954_100_gene_pair.MuTect1.coverage.wig.txt
        name: HCC1954_100_gene_pair.MuTect1.coverage.wig.txt
      pipelineId: ''
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: /cromwell_root
          name: local-disk
          readOnly: false
          sizeGb: 19
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 3
        noAddress: false
        preemptible: true
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
    pipelineArgs:
      clientId: ''
      inputs:
        Mutect1_Task.cosmicVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
        Mutect1_Task.dbSNPVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
        Mutect1_Task.mutectIntervals-0: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-CallSomaticMutations_131_Prepare_Task/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.01.interval_list
        Mutect1_Task.normalBam-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bam
        Mutect1_Task.normalBamIdx-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954_BL.100_gene_250bp_pad.bai
        Mutect1_Task.normalPanel-0: gs://firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
        Mutect1_Task.readGroupBlackList-0: gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
        Mutect1_Task.refFasta-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
        Mutect1_Task.refFastaDict-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
        Mutect1_Task.refFastaIdx-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
        Mutect1_Task.tumorBam-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bam
        Mutect1_Task.tumorBamIdx-0: gs://firecloud-tcga-open-access/tutorial/bams/HCC1954.100_gene_250bp_pad.bai
        __extra_config_gcs_path: gs://cromwell-auth-broad-firecloud-benchmark/418a4c92-0821-4695-acc8-291472136fb9_auth.json
        exec: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/exec.sh
      labels:
        cromwell-workflow-id: cromwell-418a4c92-0821-4695-acc8-291472136fb9
        cromwell-workflow-name: callinggroup-workflow
        wdl-task-name: mutect1-task
      logging:
        gcsPath: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/Mutect1_Task-0.log
      outputs:
        HCC1954_100_gene_pair.MuTect1.call_stats.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.call_stats.txt
        HCC1954_100_gene_pair.MuTect1.coverage.wig.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.coverage.wig.txt
        HCC1954_100_gene_pair.MuTect1.power.wig.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/HCC1954_100_gene_pair.MuTect1.power.wig.txt
        Mutect1_Task-0-rc.txt: gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/72eea5be-ec36-4632-a962-d73c3723b2a4/CallingGroup_Workflow/418a4c92-0821-4695-acc8-291472136fb9/call-Mutect1_Task/shard-0/Mutect1_Task-0-rc.txt
      projectId: broad-firecloud-benchmark
      resources:
        bootDiskSizeGb: 10
        disks:
        - autoDelete: false
          mountPoint: ''
          name: local-disk
          readOnly: false
          sizeGb: 19
          source: ''
          type: PERSISTENT_HDD
        minimumCpuCores: 1
        minimumRamGb: 3
        noAddress: false
        preemptible: true
        zones:
        - us-central1-b
        - us-central1-c
        - us-central1-f
      serviceAccount:
        email: default
        scopes:
        - https://www.googleapis.com/auth/genomics
        - https://www.googleapis.com/auth/compute
  runtimeMetadata:
    '@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
    computeEngine:
      diskNames:
      - local-disk-10617684529294468307
      instanceName: ggp-10617684529294468307
      machineType: us-central1-f/n1-standard-1
      zone: us-central1-f
  startTime: '2017-06-06T21:20:57Z'
name: operations/EISuoPrHKxjTyYeOsfrlrJMBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
wm8b1-75c:~ esalinas$ 

Best Answer

Answers

  • dlivitzdlivitz Member, Broadie

    I am also getting long submission delays for simple tasks.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I am looking into this again, thank you for letting us know, @esalinas.

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @KateN here is a recent observation of the issue :

    Call #2:
    Operation:operations/ELuIlsXIKxjxp5mBkYXm320g3orbxIMcKg9wcm9kdWN0aW9uUXVldWU
    Status:Done
    Started:June 8, 2017, 12:25 PM (1 hour ago)
    Ended:June 8, 2017, 1:31 PM (53 minutes ago)
    Inputs:Show
    Outputs:Show
    stdout:ContEST_Task-stdout.log
    stderr:ContEST_Task-stderr.log
    JES log:ContEST_Task.log
    

    From the UI text (copied/pasted) above the job looks to have taken nearly 1 hour to run ,but inspection of the gcloud alpha genomics operations describe shows that it took closer to 20 minutes

    {
      "done": true,
      "metadata": {
        "startTime": "2017-06-08T16:59:32Z",
        "request": {
          "@type": "type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest",
          "ephemeralPipeline": {
            "name": "QC_Workflow",
            "description": "",
            "projectId": "broad-firecloud-benchmark",
            "docker": {
              "imageName": "broadinstitute/[email protected]:b2caf8864681b54b9c9825822be47f221ca577f84faa0b3240f7010712b9dfd3",
              "cmd": "/bin/bash /cromwell_root/exec.sh"
            },
            "inputParameters": [{
              "name": "__extra_config_gcs_path",
              "description": "",
              "defaultValue": ""
            }, {
              "name": "ContEST_Task.normalBam-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/bams/C835.HCC1143_BL.4.bam",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.exomeIntervals-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/reference/whole_exome_agilent_1.1_refseq_plus_3_boosters_plus_10bp_padding_minus_mito.Homo_sapiens_assembly19.targets.interval_list",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.HapMapVCF-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/reference/hg19_population_stratified_af_hapmap_3.3.fixed.vcf",
                "disk": "local-disk"
              }
            }, {
              "name": "exec",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "exec.sh",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.tumorBamIdx-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/bams/C835.HCC1143.2.bai",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.refFastaDict-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.normalBamIdx-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/bams/C835.HCC1143_BL.4.bai",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.refFastaIdx-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.refFasta-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.SNP6Bed-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/reference/SNP6.hg19.interval_list",
                "disk": "local-disk"
              }
            }, {
              "name": "ContEST_Task.tumorBam-0",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "firecloud-tcga-open-access/tutorial/bams/C835.HCC1143.2.bam",
                "disk": "local-disk"
              }
            }],
            "resources": {
              "preemptible": false,
              "bootDiskSizeGb": 10,
              "disks": [{
                "name": "local-disk",
                "source": "",
                "readOnly": false,
                "autoDelete": false,
                "sizeGb": 26,
                "mountPoint": "/cromwell_root",
                "type": "PERSISTENT_HDD"
              }],
              "minimumCpuCores": 1,
              "minimumRamGb": 3,
              "noAddress": false,
              "zones": ["us-central1-b", "us-central1-c", "us-central1-f"]
            },
            "outputParameters": [{
              "name": "ContEST_Task-rc.txt",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "ContEST_Task-rc.txt",
                "disk": "local-disk"
              }
            }, {
              "name": "contamination_validation.array_free.txt",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "contamination_validation.array_free.txt",
                "disk": "local-disk"
              }
            }, {
              "name": "contest_validation.output.tsv",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "contest_validation.output.tsv",
                "disk": "local-disk"
              }
            }, {
              "name": "fraction_contamination.txt",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "fraction_contamination.txt",
                "disk": "local-disk"
              }
            }, {
              "name": "contamination.base_report.txt",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "contamination.base_report.txt",
                "disk": "local-disk"
              }
            }, {
              "name": "contamination.af.txt",
              "description": "",
              "defaultValue": "",
              "localCopy": {
                "path": "contamination.af.txt",
                "disk": "local-disk"
              }
            }],
            "pipelineId": ""
          },
          "pipelineArgs": {
            "serviceAccount": {
              "email": "default",
              "scopes": ["https://www.googleapis.com/auth/genomics", "https://www.googleapis.com/auth/compute"]
            },
            "logging": {
              "gcsPath": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/ContEST_Task.log"
            },
            "clientId": "",
            "outputs": {
              "contamination.base_report.txt": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contamination.base_report.txt",
              "fraction_contamination.txt": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/fraction_contamination.txt",
              "ContEST_Task-rc.txt": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/ContEST_Task-rc.txt",
              "contamination_validation.array_free.txt": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contamination_validation.array_free.txt",
              "contest_validation.output.tsv": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contest_validation.output.tsv",
              "contamination.af.txt": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contamination.af.txt"
            },
            "projectId": "broad-firecloud-benchmark",
            "inputs": {
              "ContEST_Task.tumorBamIdx-0": "gs://firecloud-tcga-open-access/tutorial/bams/C835.HCC1143.2.bai",
              "ContEST_Task.HapMapVCF-0": "gs://firecloud-tcga-open-access/tutorial/reference/hg19_population_stratified_af_hapmap_3.3.fixed.vcf",
              "ContEST_Task.refFasta-0": "gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta",
              "ContEST_Task.SNP6Bed-0": "gs://firecloud-tcga-open-access/tutorial/reference/SNP6.hg19.interval_list",
              "ContEST_Task.refFastaDict-0": "gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict",
              "ContEST_Task.tumorBam-0": "gs://firecloud-tcga-open-access/tutorial/bams/C835.HCC1143.2.bam",
              "__extra_config_gcs_path": "gs://cromwell-auth-broad-firecloud-benchmark/ebec769f-f099-4ba9-919f-ffbc8bc10df5_auth.json",
              "ContEST_Task.normalBam-0": "gs://firecloud-tcga-open-access/tutorial/bams/C835.HCC1143_BL.4.bam",
              "ContEST_Task.normalBamIdx-0": "gs://firecloud-tcga-open-access/tutorial/bams/C835.HCC1143_BL.4.bai",
              "ContEST_Task.refFastaIdx-0": "gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai",
              "exec": "gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/exec.sh",
              "ContEST_Task.exomeIntervals-0": "gs://firecloud-tcga-open-access/tutorial/reference/whole_exome_agilent_1.1_refseq_plus_3_boosters_plus_10bp_padding_minus_mito.Homo_sapiens_assembly19.targets.interval_list"
            },
            "labels": {
              "cromwell-workflow-id": "cromwell-ebec769f-f099-4ba9-919f-ffbc8bc10df5",
              "cromwell-workflow-name": "qc-workflow",
              "wdl-task-name": "contest-task"
            },
            "resources": {
              "preemptible": false,
              "bootDiskSizeGb": 10,
              "disks": [{
                "name": "local-disk",
                "source": "",
                "readOnly": false,
                "autoDelete": false,
                "sizeGb": 26,
                "mountPoint": "",
                "type": "PERSISTENT_HDD"
              }],
              "minimumCpuCores": 1,
              "minimumRamGb": 3,
              "noAddress": false,
              "zones": ["us-central1-b", "us-central1-c", "us-central1-f"]
            }
          }
        },
        "clientId": "",
        "createTime": "2017-06-08T16:59:27Z",
        "projectId": "broad-firecloud-benchmark",
        "endTime": "2017-06-08T17:18:43Z",
        "labels": {
          "wdl-task-name": "contest-task",
          "cromwell-workflow-id": "cromwell-ebec769f-f099-4ba9-919f-ffbc8bc10df5",
          "cromwell-workflow-name": "qc-workflow"
        },
        "@type": "type.googleapis.com/google.genomics.v1.OperationMetadata",
        "events": [{
          "description": "start",
          "startTime": "2017-06-08T17:00:36.147071681Z"
        }, {
          "description": "pulling-image",
          "startTime": "2017-06-08T17:00:39.390045659Z"
        }, {
          "description": "localizing-files",
          "startTime": "2017-06-08T17:02:05.313744421Z"
        }, {
          "description": "running-docker",
          "startTime": "2017-06-08T17:08:28.762219010Z"
        }, {
          "description": "delocalizing-files",
          "startTime": "2017-06-08T17:18:28.542487629Z"
        }, {
          "description": "copied 1 file(s) to \"gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/ContEST_Task-rc.txt\"",
          "startTime": "2017-06-08T17:18:30.439532217Z"
        }, {
          "description": "copied 1 file(s) to \"gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contamination_validation.array_free.txt\"",
          "startTime": "2017-06-08T17:18:32.884161650Z"
        }, {
          "description": "copied 1 file(s) to \"gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contest_validation.output.tsv\"",
          "startTime": "2017-06-08T17:18:34.595334258Z"
        }, {
          "description": "copied 1 file(s) to \"gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/fraction_contamination.txt\"",
          "startTime": "2017-06-08T17:18:36.415836697Z"
        }, {
          "description": "copied 1 file(s) to \"gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contamination.base_report.txt\"",
          "startTime": "2017-06-08T17:18:38.492440380Z"
        }, {
          "description": "copied 1 file(s) to \"gs://fc-bd7b9b59-7254-432a-860d-a4779571cebb/4007dde2-afb3-47cd-a0a8-f62f2343a835/QC_Workflow/ebec769f-f099-4ba9-919f-ffbc8bc10df5/call-ContEST_Task/attempt-2/contamination.af.txt\"",
          "startTime": "2017-06-08T17:18:40.668134233Z"
        }, {
          "description": "ok",
          "startTime": "2017-06-08T17:18:42.362379784Z"
        }],
        "runtimeMetadata": {
          "@type": "type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata",
          "computeEngine": {
            "instanceName": "ggp-7908206772817908721",
            "zone": "us-central1-f",
            "machineType": "us-central1-f/n1-standard-1",
            "diskNames": ["local-disk-7908206772817908721"]
          }
        }
      },
      "name": "operations/ELuIlsXIKxjxp5mBkYXm320g3orbxIMcKg9wcm9kdWN0aW9uUXVldWU"
    }
    
    
  • esalinasesalinas BroadMember, Broadie ✭✭✭

    okay @KateN thanks for the update

Sign In or Register to comment.