Preemptible instances: historical data of Google Compute Engine VM running time before preemption?
Google makes zero guarantee on when your preemptible instance could become preempted.
Preemptible instances typically cost about 1/4 of regular instances, but it is difficult to make an informed decision on whether to request preemptible instances without any data to build a predictive model of time to preemption...
Question 1. Now that people have been running FireCloud or Google Cloud for a while... Is there any historical data available?
By default, Cromwell request a non-preemptible instance. Has anyone else been setting the runtime attribute
preemptible to a non-zero number and collected data like below?
Question 2. Is there a programmatic way of extracting preemption data from your own FireCloud account?
A sample of the data that I am hoping to collect:
Note: Only the running time of preemptible instances were measured (non-preemptible instances were omitted). If a task completes/fails/aborts before preemption, the observation is censored (indicated by +).
Remark: As seen above, one of our instances was taken away after 5 minutes... No guarantee, sure. But come on, 5 minutes?! And that's after two previous preemptions!
With this type of data, we could build a predictive model to calculate the expectation of time to preemption and make an informed decision using decision theory to minimize expected compute cost.