Is it possible to set a timeout for WDL tasks that run on Google cloud? Some backends (e.g. BCS) seem to support a timeout runtime attribute, as indicated in cromwell.examples.conf. Is there such a setting for JES backend?
It's possible to apply such a timeout on the Pipelines API v2 backend. For the use case you have in mind, would it be sufficient to have a workflow option with such a timeout?
We’ve observed the same issue and, afaik the Cromwell team is looking into it. Thanks for reporting!
If you could provide operation IDs, that’d be hugely helpful though!
In Pipelines API v2, there's a jobId parameter for each Cromwell task call. It's in the form projects/your-project/operations/1234567890. This is what we need to pass to Cromwell developers so they could troubleshoot it. You can get the value of jobId for each task call from Cromwell API.
Btw, we're also experiencing these as random issues. I also think the problem is on Google side, rather than Cromwell.
I've made https://github.com/broadinstitute/cromwell/issues/4946 to track this request.
Note for @Ruchi - I think it could be a relatively minimal change to wire through the existing timeout runtime attribute (currently BCS only) to PAPIv2. The code line to change is https://github.com/broadinstitute/cromwell/blob/develop/supportedBackends/google/pipelines/v2alpha1/src/main/scala/cromwell/backend/google/pipelines/v2alpha1/GenomicsFactory.scala#L135
Yep, you can see jobId for each task call inside metadata.json.