Cromwell 27: Better, faster, stronger
Last week we released Cromwell 27, the latest and greatest Cromwell to date! Much of our work this quarter has been directed towards improving the general speed and scalability of Cromwell.
But before you do, have a look at the new feature highlights below!
Breaking Changes: update to Slick 3.2
You will need to make a change to the database stanza, see the Change log for full details.
Migration for cached jobs
As a part of our call caching improvements you will need to migrate your database in order to preserve previously cached jobs. See the Change log for full details on how to execute the migration.
Call Caching improvements
Prior to Cromwell 27, if you had a floating tag (such as
docker: "ubuntu:latest") in your WDL, Cromwell would not call cache for that job. Now in version 27, Cromwell will resolve the floating tag into a hash and Cromwell can call cache those tasks. If the same floating tag appears later within the same workflow then Cromwell will consistently use the same hash to run the task.
To increase the likelihood that your workflow tasks have a call cache-hit, we recommend that you specify the Docker image rather than use
ubuntu:latest. Sometimes Cromwell is not certain that it is using the same hash as a previous task, in which case it will run the task from scratch (rather than risk call caching erroneously). For more information, see the Change log.
Stay tuned to the WDL Blog for an upcoming post about call caching.
We added a new WDL function called
basename. This allows WDL authors to get just the file name from a file by removing the directory path, for example.
We also added the ability to coerce
Map objects into
Pairs, in order to directly scatter over
You can now specify how to look up a Docker hash. Within the Docker section of the configuration you can choose
local to look for the image on the machine where Cromwell is running, or
remote to look up the image on Docker Hub or GCR.
We also added hash lookup support for Quay.io.
Google Service Account
We added support for JSON file format for Google Service Account credentials. PEM credentials are deprecated and support may be removed in the future.
read_X standard library functions limit accepted file sizes, which differ by type. For more information see the reference.conf.