stopping a pipeline using cromwell API

afriedenafrieden Cambridge, MAMember

I am able to query for a pipeline I am running. However, is there a way to kill a workflow if I have that workflow's id?

For instance, I will have the following from a query on the GET /workflows/{version}/query endpoint:

{
  "results": [
    {
      "id": "74444045-0c99-4ef8-acdb-7a57a4486f30",
      "name": "SimpleVariantSelection",
      "status": "Running",
      "start": "2017-07-10T15:20:02-04:00"
    }
  ]
}

I want to cancel the job with id 74444045-0c99-4ef8-acdb-7a57a4486f30. How do I do this using the API? Thanks!

Tagged:

Best Answers

  • afriedenafrieden Cambridge, MAMember
    Accepted Answer

    Interesting, when I submitted via the API on the POST /workflows/{version} the job was submitted and then the abort endpoint worked on this new submitted job. Submitting via the command line does NOT allow the abort endpoint to work.

Answers

  • RuchiRuchi Member, Broadie, Moderator, Dev

    Hey @afrieden, you should be able to use the workflow abort endpoint. POST /api/workflows/:version/:id/abort More docs here.

  • afriedenafrieden Cambridge, MAMember

    Does not seem to work. I ran the get query:
    GET http://localhost:8000/api/workflows/v1/query?status=Running

    { "results": [ { "id": "51e73872-0b2d-431c-8989-0eb699c38f3e", "name": "SimpleVariantSelection", "status": "Running", "start": "2017-07-11T10:12:21-04:00" } ] }

    then I ran the abort post:
    POST http://localhost:8000/api/workflows/v1/51e73872-0b2d-431c-8989-0eb699c38f3e/abort

    { "status": "error", "message": "Couldn't abort 51e73872-0b2d-431c-8989-0eb699c38f3e because no workflow with that ID is in progress" }

    Am I missing something?

  • RuchiRuchi Member, Broadie, Moderator, Dev

    @afrieden It's quite likely something is wrong with the Abort endpoint as there are certain known issues. I can link this post to an existing issue in our backlog. If you query metadata for this particular workflow, have any of the jobs transitioned into the Aborted state?

  • afriedenafrieden Cambridge, MAMember

    No I have tried numerous times and none have made it to the abort state. As far as I can tell it doesn't work yet.

  • RuchiRuchi Member, Broadie, Moderator, Dev

    It seems like your workflow might have already stopped running (which is likely if nothing is being logged by the Cromwell server anymore) and that's why /abort endpoint is returning with a message ".. workflow with that ID is in progress." Would you mind running this query on your workflow so I can get an idea of the current status of the jobs inside your workflow?
    http://localhost:8000/api/workflows/v2/51e73872-0b2d-431c-8989-0eb699c38f3e/metadata?includeKey=executionStatus&expandSubWorkflows=false

    You have a few options on what to try next depending on what your goal is. If you'd like to make sure that the actual tasks stop running, then depending on the backend you're utilizing, there are ways to manually kill all jobs.

    If all jobs have reached a terminal state already and you'd like to push the workflow to Aborted or some terminal state, it might help to restart the Cromwell server.

    Meanwhile, I've logged the issue you're seeing into an existing ticket for fixing aborts.

  • afriedenafrieden Cambridge, MAMember

    Still didn't work. Ran the metadata GET request right before and after sending abort. Metadata GET sent back:

    { "calls": { "SimpleVariantSelection.haplotypeCaller": [ { "executionStatus": "Running", "attempt": 1, "shardIndex": -1 } ] }, "id": "718a1620-62bf-4263-bde4-6161af7fa04b" }

    Abort POST returned back error:
    { "status": "error", "message": "Couldn't abort 718a1620-62bf-4263-bde4-6161af7fa04b because no workflow with that ID is in progress" }

  • RuchiRuchi Member, Broadie, Moderator, Dev

    I'm sorry, I should have been more clear, the GET /metadata request was only to get a count of the number of jobs running inside this workflow, not an attempt at aborting the workflow.

    Things you've tried but didn't work:
    1. The /abort endpoint

    Things you could try:
    1. Restart the Cromwell Server
    2. Get the job ID for SimpleVariantSelection.haplotypeCaller from workflow metadata and kill the job manually (which would mean doing it using something other than the Cromwell API).

  • afriedenafrieden Cambridge, MAMember
    Accepted Answer

    Interesting, when I submitted via the API on the POST /workflows/{version} the job was submitted and then the abort endpoint worked on this new submitted job. Submitting via the command line does NOT allow the abort endpoint to work.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    @afrieden when you use the command line, you're starting a whole new cli Cromwell which is disjoint from any server Cromwell that you happen to also be running. That could be why you weren't able to abort the command line version?

  • afriedenafrieden Cambridge, MAMember

    @ChrisL thats my guess why. Surprised since I can query it with API when I run on command line, just can't abort.

Sign In or Register to comment.