Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

A resource limit has delayed the operation

I am running a WDL script using the wdl_runner tools ( github.com/broadinstitute/wdl-runner ) that wrap Cromwell. My script has executed successfully, but it seems that the program believes it cannot deallocate due to a rate limit? I checked the GCP Quotas and we're nowhere near a quota limit for API requests, so I am trying to understand whether this issue is Cromwell specific?


```

gcloud alpha genomics operations describe projects/ukbb-analyses/operations/11116485566567687044
done: false
metadata:
'@type': type.googleapis.com/google.genomics.v2alpha1.Metadata
createTime: '2019-04-27T17:23:00.563834Z'
events:
- description: 'A resource limit has delayed the operation: generic::resource_exhausted:
deallocating: deleting instance: Rate Limit Exceeded'
details:
'@type': type.googleapis.com/google.genomics.v2alpha1.DelayedEvent
cause: 'generic::resource_exhausted: deallocating: deleting instance: Rate Limit
Exceeded'
metrics:
- UNKNOWN
timestamp: '2019-04-27T17:52:23.882359Z'
- description: 'A resource limit has delayed the operation: generic::resource_exhausted:
deallocating: deleting instance: Rate Limit Exceeded'
details:
'@type': type.googleapis.com/google.genomics.v2alpha1.DelayedEvent
cause: 'generic::resource_exhausted: deallocating: deleting instance: Rate Limit
Exceeded'
metrics:
- UNKNOWN
timestamp: '2019-04-27T17:42:09.256947Z'
- description: Stopped running "/bin/sh -c gsutil -m -q cp /google/logs/output gs://ukbb_v2/projects/jamesp/wdl_runner/logging"
details:
'@type': type.googleapis.com/google.genomics.v2alpha1.ContainerStoppedEvent
actionId: 6
exitStatus: 0
stderr: ''
timestamp: '2019-04-27T17:42:08.268399Z'

```

Answers

  • jamespjamesp Member
    Also, it seems that the Markdown backticks have no effect on formatting despite the sidebar's suggestion?
  • jamespjamesp Member
    Note, after an additional ~13 minutes of idle time, it completed. Any sense of whether this is a Cromwell issue, or something I should raise to Google?

    ```
    gcloud alpha genomics operations describe projects/ukbb-analyses/operations/11116485566567687044
    done: true
    metadata:
    '@type': type.googleapis.com/google.genomics.v2alpha1.Metadata
    createTime: '2019-04-27T17:23:00.563834Z'
    endTime: '2019-04-27T18:29:37.091816Z'
    events:
    - description: Worker released
    details:
    '@type': type.googleapis.com/google.genomics.v2alpha1.WorkerReleasedEvent
    instance: google-pipelines-worker-3bdc692486800de8b5617dc872a3fd77
    zone: us-central1-a
    timestamp: '2019-04-27T18:29:37.091816Z'
    - description: 'A resource limit has delayed the operation: generic::resource_exhausted:
    deallocating: deleting instance: Rate Limit Exceeded'
    details:
    '@type': type.googleapis.com/google.genomics.v2alpha1.DelayedEvent
    cause: 'generic::resource_exhausted: deallocating: deleting instance: Rate Limit
    Exceeded'
    metrics:
    - UNKNOWN
    timestamp: '2019-04-27T18:16:40.326193Z'
    ```
Sign In or Register to comment.