Putting WDL/Cromwell into production in Google Cloud
All of the help has been amazing and we've been able to port a number of our workflows into WDL in less than a week - amazing! I think it really speaks to the power of the platform that complex pipelines can be ported that fast.
My questions now are pretty in the weeds:
There is a section in the cromwell docs on monitoring, where you can point cromwell to a monitoring script stored in a Google bucket, and it will run that. Do you have an example of what such a script would look like? It would be great to know what the utilization and time spent is on each step so we can figure out what the requirements are more precisely.
More broadly, I am trying to figure out how we can monitor success/failure of the analysis of different samples. In an ideal world, there would be a way to actively write to a log file when each new step of a run of a given pipeline has started, and subsequently succeeded or failed. Do you have any recommendations for how we can get this up and going?