I can't find documentation on how Cromwell Server copes if it loses database connections mid transaction. What would happen if a)there is a minor loss of connectivity or b) the database goes completely?
When it loses database connectivity, Cromwell will no longer be able to do anything involving a database connection, including (but not exhaustively):
If it loses its database, you might as well just shut down Cromwell because it won't reconnect (even if the database comes back online) without a service restart.
Luckily, Cromwell will also not do anything destructive either - if it has already submitted jobs to a cloud backend for execution, they won't be lost, for example. When the database comes back and Cromwell is restarted, it will restart its workflows from where they left off, and reconnect to any existing jobs still running on cloud backends.
EDIT: If Cromwell is brought down at the same time that the database connection is lost, it can run through a well-tested "reconnection" process when it is restarted against a reconnected database, which should be able to recover any running jobs at the point where the disconnection occurred.
EDIT: You can probably disregard the lower half of my earlier response. I've just tried this again and the result was a little more mixed:
So I think the more correct answer here is: