Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.
Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

stopping a pipeline using cromwell API

afriedenafrieden Cambridge, MAMember ✭
in Ask the Cromwell + WDL Team

I am able to query for a pipeline I am running. However, is there a way to kill a workflow if I have that workflow's id?

For instance, I will have the following from a query on the GET /workflows/{version}/query endpoint:

{
  "results": [
    {
      "id": "74444045-0c99-4ef8-acdb-7a57a4486f30",
      "name": "SimpleVariantSelection",
      "status": "Running",
      "start": "2017-07-10T15:20:02-04:00"
    }
  ]
}

I want to cancel the job with id 74444045-0c99-4ef8-acdb-7a57a4486f30. How do I do this using the API? Thanks!

Tagged:
· Share on Twitter Share on Google+

Best Answers

  • afriedenafrieden Cambridge, MA ✭
    Accepted Answer

    Interesting, when I submitted via the API on the POST /workflows/{version} the job was submitted and then the abort endpoint worked on this new submitted job. Submitting via the command line does NOT allow the abort endpoint to work.

    · Share on Twitter Share on Google+

Answers

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @afrieden, you should be able to use the workflow abort endpoint. POST /api/workflows/:version/:id/abort More docs here.

    · Share on Twitter Share on Google+
  • afriedenafrieden Cambridge, MAMember ✭

    Does not seem to work. I ran the get query:
    GET http://localhost:8000/api/workflows/v1/query?status=Running

    { "results": [ { "id": "51e73872-0b2d-431c-8989-0eb699c38f3e", "name": "SimpleVariantSelection", "status": "Running", "start": "2017-07-11T10:12:21-04:00" } ] }

    then I ran the abort post:
    POST http://localhost:8000/api/workflows/v1/51e73872-0b2d-431c-8989-0eb699c38f3e/abort

    { "status": "error", "message": "Couldn't abort 51e73872-0b2d-431c-8989-0eb699c38f3e because no workflow with that ID is in progress" }

    Am I missing something?

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    @afrieden It's quite likely something is wrong with the Abort endpoint as there are certain known issues. I can link this post to an existing issue in our backlog. If you query metadata for this particular workflow, have any of the jobs transitioned into the Aborted state?

    · Share on Twitter Share on Google+
  • afriedenafrieden Cambridge, MAMember ✭

    No I have tried numerous times and none have made it to the abort state. As far as I can tell it doesn't work yet.

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    It seems like your workflow might have already stopped running (which is likely if nothing is being logged by the Cromwell server anymore) and that's why /abort endpoint is returning with a message ".. workflow with that ID is in progress." Would you mind running this query on your workflow so I can get an idea of the current status of the jobs inside your workflow?
    http://localhost:8000/api/workflows/v2/51e73872-0b2d-431c-8989-0eb699c38f3e/metadata?includeKey=executionStatus&expandSubWorkflows=false

    You have a few options on what to try next depending on what your goal is. If you'd like to make sure that the actual tasks stop running, then depending on the backend you're utilizing, there are ways to manually kill all jobs.

    If all jobs have reached a terminal state already and you'd like to push the workflow to Aborted or some terminal state, it might help to restart the Cromwell server.

    Meanwhile, I've logged the issue you're seeing into an existing ticket for fixing aborts.

    · Share on Twitter Share on Google+
  • afriedenafrieden Cambridge, MAMember ✭

    Still didn't work. Ran the metadata GET request right before and after sending abort. Metadata GET sent back:

    { "calls": { "SimpleVariantSelection.haplotypeCaller": [ { "executionStatus": "Running", "attempt": 1, "shardIndex": -1 } ] }, "id": "718a1620-62bf-4263-bde4-6161af7fa04b" }

    Abort POST returned back error:
    { "status": "error", "message": "Couldn't abort 718a1620-62bf-4263-bde4-6161af7fa04b because no workflow with that ID is in progress" }

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    I'm sorry, I should have been more clear, the GET /metadata request was only to get a count of the number of jobs running inside this workflow, not an attempt at aborting the workflow.

    Things you've tried but didn't work:
    1. The /abort endpoint

    Things you could try:
    1. Restart the Cromwell Server
    2. Get the job ID for SimpleVariantSelection.haplotypeCaller from workflow metadata and kill the job manually (which would mean doing it using something other than the Cromwell API).

    · Share on Twitter Share on Google+
  • afriedenafrieden Cambridge, MAMember ✭
    Accepted Answer

    Interesting, when I submitted via the API on the POST /workflows/{version} the job was submitted and then the abort endpoint worked on this new submitted job. Submitting via the command line does NOT allow the abort endpoint to work.

    · Share on Twitter Share on Google+
  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev admin

    @afrieden when you use the command line, you're starting a whole new cli Cromwell which is disjoint from any server Cromwell that you happen to also be running. That could be why you weren't able to abort the command line version?

    · Share on Twitter Share on Google+
  • afriedenafrieden Cambridge, MAMember ✭

    @ChrisL thats my guess why. Surprised since I can query it with API when I run on command line, just can't abort.

    · Share on Twitter Share on Google+
Sign In or Register to comment.