Forum Login Issue:
Currently the "Log in with Google" button redirects you to a "Page not found." This is an issue that our forum vendors are working on fixing. In the meantime, while on the "Page not found" you can edit the URL to delete the second gatk, firecloud, or wdl (depending on what subforum you are acessing).
ex: https://gatkforums.broadinstitute.org/gatk/gatk/entry/...

Cromwell's call caching is stupid at times

oskarvoskarv BergenMember

I had a workflow crash because I had set the wrong file path in the json file, when I fixed it the workflow started from the first step even though I had finished that step successfully. The file path was for the step that crashed, obviously, so it should restart at the latest successful step, but it doesn't. The call caching function should be smart enough to recognize that a change in the json file affects a specific step instead of assuming that any change in the json file means that everything needs to be run again. This isn't an edge case and should be built in already.

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    @oskarv this is indeed built in already! :smile:.

    Assuming you've got call caching configured and set up correctly, what you're probably seeing is just Cromwell replaying jobs in logs as it call-caches past them.

    Cromwell still logs that it's going to "run task A" even if that "run" ends up being a call cache "copy file A to directory B" run.

  • oskarvoskarv BergenMember

    @ChrisL said:
    @oskarv this is indeed built in already! :smile:.

    Assuming you've got call caching configured and set up correctly, what you're probably seeing is just Cromwell replaying jobs in logs as it call-caches past them.

    Cromwell still logs that it's going to "run task A" even if that "run" ends up being a call cache "copy file A to directory B" run.

    That's odd, every other call caching function works as it should, but in this specific case it literally started from the beginning instead of replaying the jobs as it should have.
    Perhaps you could take a look at my configuration files to see if there's anything missing?
    Here's application.conf: https://github.com/oskarvid/wdl_deepvariant/blob/master/cromwell-mysql/cromwell/app-config/application.conf
    and here's workflowoptions.json: https://github.com/oskarvid/wdl_deepvariant/blob/master/cromwell-mysql/cromwell/workflow-options/workflowoptions.json
    I'm using cromwell 30-2.
    Can you see if there's anything wrong with it? It would be nice to find the cause of the problem.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    The use of "filepath" might be causing problems - but it sounds like only the path of the final task will have changed in your inputs?

    If you're running in server mode, you can use the call cache diff endpoint to debug this - if you navigate to the swagger page (eg if Cromwell is running on localhost: http://localhost:8000/swagger/index.html) you can scroll down to the call caching diff endpoint, see the fields it expects and try it out directly in the swagger UI (you'll need to enter the workflow ID of the original task, and the task which should have call cached from it - it'll tell you why it was a miss)

Sign In or Register to comment.