Timing off : some values reset

esalinasesalinas BroadMember, Broadie ✭✭✭

I have a submission and a workflow that started earlier today.

The first thing in the WDL is a "scatter" step.

Some tasks successfully "hit" in call-caching when the submission was started. The "hits" occur
soon after the submission.

In workflow here, the cache hit and call seems to have its timing values reset to sooner (closer to 3:18) rather than
left at their original "hit" time (closer to 1:02). The timing data is not correct and it is also confusing

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I seem to recall a previous thread of yours where the timing was off as well. I'll put in a ticket to investigate why this appears to be happening, as I agree that changing times is not a good thing to have.

  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited September 2017

    @KateN I saw this submission/workflow be "born" and I saw that task in particular get a cache "hit" maybe 5-10 min or so after the submission (closer to 1:02).

    what's also strange is in the same workflow a cache "miss" has the same value for start/end times, but the task actually takes on the order of minutes or hours to run. Moreover the value "3:18" appears again here.

    Call #166:
    Operation:operations/EPP4xcvsKxiU8uuFlLOXpeMBIPCeybKCFioPcHJvZHVjdGlvblF1ZXVl
    Status:Done
    Cache Result:Miss
    Started:September 28, 2017, 3:18 PM (20 hours ago)
    Ended:September 28, 2017, 3:18 PM (20 hours ago)
    Inputs:Show
    Outputs:Show
    stdout:token_make-165-stdout.log
    stderr:token_make-165-stderr.log
    JES log:token_make-165.log
    
    

    The operations describe verifies this (that it takes on the order of minutes and hours to run) as seen by looking in the "events" array and the timings in it:

    wm8b1-75c:MSMuTect_v1.0 esalinas$ gcloud alpha genomics operations describe  operations/EPP4xcvsKxiU8uuFlLOXpeMBIPCeybKCFioPcHJvZHVjdGlvblF1ZXVl --format json
    {
      "done": true,
      "metadata": {
        "@type": "type.googleapis.com/google.genomics.v1.OperationMetadata",
        "clientId": "",
        "createTime": "2017-09-28T17:03:31Z",
        "endTime": "2017-09-28T18:16:06Z",
        "events": [
          {
            "description": "start",
            "startTime": "2017-09-28T17:04:45.455225345Z"
          },
          {
            "description": "pulling-image",
            "startTime": "2017-09-28T17:04:47.114674671Z"
          },
          {
            "description": "localizing-files",
            "startTime": "2017-09-28T17:06:05.579603310Z"
          },
          {
            "description": "running-docker",
            "startTime": "2017-09-28T17:16:52.214515153Z"
          },
          {
            "description": "delocalizing-files",
            "startTime": "2017-09-28T18:15:06.573500364Z"
          },
          {
            "description": "copied 1 file(s) to \"gs://fc-6c731e3e-3c48-4077-85bc-e041579064e1/9adb666c-3151-4ed7-9421-21ffcdc12e1f/token_wf/58e415b0-8f55-4294-8a76-ffcde5e29ea7/call-token_make/shard-165/token_make-165-rc.txt\"",
            "startTime": "2017-09-28T18:15:07.595988856Z"
          },
          {
            "description": "copied 1 file(s) to \"gs://fc-6c731e3e-3c48-4077-85bc-e041579064e1/9adb666c-3151-4ed7-9421-21ffcdc12e1f/token_wf/58e415b0-8f55-4294-8a76-ffcde5e29ea7/call-token_make/shard-165/C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam.token.dat\"",
            "startTime": "2017-09-28T18:16:02.536230681Z"
          },
          {
            "description": "ok",
            "startTime": "2017-09-28T18:16:06.369478448Z"
          }
        ],
        "labels": {
          "cromwell-workflow-id": "cromwell-58e415b0-8f55-4294-8a76-ffcde5e29ea7",
          "wdl-task-name": "token-make"
        },
        "projectId": "broad-firecloud-testing",
        "request": {
          "@type": "type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest",
          "ephemeralPipeline": {
            "description": "",
            "docker": {
              "cmd": "/bin/bash /cromwell_root/exec.sh",
              "imageName": "broadinstitute/[email protected]:d83be8bcdc8e13ef943d44d6da11d46556363d35e507f44b00735f01e5abed3d"
            },
            "inputParameters": [
              {
                "defaultValue": "",
                "description": "",
                "name": "__extra_config_gcs_path"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "7008814a-277f-4fd4-aa61-a513f9d8108b/NCI-GDC/current/TCGA/TCGA-PAAD/WXS/Aligned_Reads/771d0391-d61d-40ed-9a0c-923fbf49dc57/C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bai"
                },
                "name": "token_make.bamIdx-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/annotation_db_hg38_token_pon.tar"
                },
                "name": "token_make.refTar-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "7008814a-277f-4fd4-aa61-a513f9d8108b/NCI-GDC/current/TCGA/TCGA-PAAD/WXS/Aligned_Reads/771d0391-d61d-40ed-9a0c-923fbf49dc57/C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam"
                },
                "name": "token_make.bam-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "exec.sh"
                },
                "name": "exec"
              }
            ],
            "name": "token_wf",
            "outputParameters": [
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "token_make-165-rc.txt"
                },
                "name": "token_make-165-rc.txt"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam.token.dat"
                },
                "name": "C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam.token.dat"
              }
            ],
            "pipelineId": "",
            "projectId": "broad-firecloud-testing",
            "resources": {
              "bootDiskSizeGb": 10,
              "disks": [
                {
                  "autoDelete": false,
                  "mountPoint": "/cromwell_root",
                  "name": "local-disk",
                  "readOnly": false,
                  "sizeGb": 67,
                  "source": "",
                  "type": "PERSISTENT_HDD"
                }
              ],
              "minimumCpuCores": 1,
              "minimumRamGb": 2,
              "noAddress": false,
              "preemptible": true,
              "zones": [
                "us-central1-b",
                "us-central1-c",
                "us-central1-f"
              ]
            }
          },
          "pipelineArgs": {
            "clientId": "",
            "inputs": {
              "__extra_config_gcs_path": "gs://cromwell-auth-broad-firecloud-testing/58e415b0-8f55-4294-8a76-ffcde5e29ea7_auth.json",
              "exec": "gs://fc-6c731e3e-3c48-4077-85bc-e041579064e1/9adb666c-3151-4ed7-9421-21ffcdc12e1f/token_wf/58e415b0-8f55-4294-8a76-ffcde5e29ea7/call-token_make/shard-165/exec.sh",
              "token_make.bam-0": "gs://7008814a-277f-4fd4-aa61-a513f9d8108b/NCI-GDC/current/TCGA/TCGA-PAAD/WXS/Aligned_Reads/771d0391-d61d-40ed-9a0c-923fbf49dc57/C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam",
              "token_make.bamIdx-0": "gs://7008814a-277f-4fd4-aa61-a513f9d8108b/NCI-GDC/current/TCGA/TCGA-PAAD/WXS/Aligned_Reads/771d0391-d61d-40ed-9a0c-923fbf49dc57/C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bai",
              "token_make.refTar-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/annotation_db_hg38_token_pon.tar"
            },
            "labels": {
              "cromwell-workflow-id": "cromwell-58e415b0-8f55-4294-8a76-ffcde5e29ea7",
              "wdl-task-name": "token-make"
            },
            "logging": {
              "gcsPath": "gs://fc-6c731e3e-3c48-4077-85bc-e041579064e1/9adb666c-3151-4ed7-9421-21ffcdc12e1f/token_wf/58e415b0-8f55-4294-8a76-ffcde5e29ea7/call-token_make/shard-165/token_make-165.log"
            },
            "outputs": {
              "C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam.token.dat": "gs://fc-6c731e3e-3c48-4077-85bc-e041579064e1/9adb666c-3151-4ed7-9421-21ffcdc12e1f/token_wf/58e415b0-8f55-4294-8a76-ffcde5e29ea7/call-token_make/shard-165/C546.TCGA-HZ-7918-10A-01D-2154-08.1_gdc_realn.bam.token.dat",
              "token_make-165-rc.txt": "gs://fc-6c731e3e-3c48-4077-85bc-e041579064e1/9adb666c-3151-4ed7-9421-21ffcdc12e1f/token_wf/58e415b0-8f55-4294-8a76-ffcde5e29ea7/call-token_make/shard-165/token_make-165-rc.txt"
            },
            "projectId": "broad-firecloud-testing",
            "resources": {
              "bootDiskSizeGb": 10,
              "disks": [
                {
                  "autoDelete": false,
                  "mountPoint": "",
                  "name": "local-disk",
                  "readOnly": false,
                  "sizeGb": 67,
                  "source": "",
                  "type": "PERSISTENT_HDD"
                }
              ],
              "minimumCpuCores": 1,
              "minimumRamGb": 2,
              "noAddress": false,
              "preemptible": true,
              "zones": [
                "us-central1-b",
                "us-central1-c",
                "us-central1-f"
              ]
            },
            "serviceAccount": {
              "email": "default",
              "scopes": [
                "https://www.googleapis.com/auth/genomics",
                "https://www.googleapis.com/auth/compute"
              ]
            }
          }
        },
        "runtimeMetadata": {
          "@type": "type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata",
          "computeEngine": {
            "diskNames": [
              "local-disk-16378005907683604756"
            ],
            "instanceName": "ggp-16378005907683604756",
            "machineType": "us-central1-f/n1-standard-1",
            "zone": "us-central1-f"
          }
        },
        "startTime": "2017-09-28T17:03:33Z"
      },
      "name": "operations/EPP4xcvsKxiU8uuFlLOXpeMBIPCeybKCFioPcHJvZHVjdGlvblF1ZXVl"
    }
    wm8b1-75c:MSMuTect_v1.0 esalinas$ 
    
    
Sign In or Register to comment.