On Monday and Tuesday, November 12-13, the communications team will be out of the office for a U.S. federal holiday and a team event. We will be back in action on November 14th and apologize for any inconvenience this may cause. Thank you for using the forum.

How to delete execution data form old runs?

If we have a workflow of this form:

Inputs -> Temporary Files 1 -> Important Files 1 -> Temp files 2 -> Temp files 3 -> Important Files 2 -> Temp files 4 ...

In a setting where we're using cromwell in server mode backed with mysql for call-caching, eventually we'll need to delete old temporary files to avoid running out of space (assuming we already have some mechanism for moving the important files out somewhere else).

Deleting intermediate files has been discussed here, however, deleting the workflow execution directory right after running the workflow would invalidate call-caching (would it? I guess so, because there would be no output file to create a link to) .

I'm looking for a way to delete old executions, for example, a daily cron job deleting all executions older than 1 year. If I just go and delete the executions directories, would the mysql database get corrupted? What would be involved in such clean up process? Does cromwell provide any features to help with this? E.g. I imagine a DELETE workflow API endpoint would be useful here.

Thank you!

Best Answer

Answers

  • Thank you that's very useful!
    Regarding the mysql database, will invalid rows get deleted, or just get marked as invalid? I worry that the database may grow indefinitely, especially if we run thousands of tasks per day.

Sign In or Register to comment.