Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Can runtime parameter values be generated dynamically?

Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

Say you have a task where the runtime requirements (like disk size and memory) can vary a lot depending on e.g. the size of the input files for a given run. To minimize costs, you want to be able to allocate those runtime parameters depending on what is needed for each job; and to minimize effort you want to do that dynamically, without having to change the WDL script or manually changing input values for each job.

This can be done by using the size() function to check the size of the files, then tacking on a bit of arithmetic to adjust as needed (eg if you want the task to have a bit of extra headroom). In principle you could use size() directly in the runtime block, since it will be evaluated before localization. However, size() returns a float (i.e. a type of decimal number) which is not valid for any property you want to set that requires an integer.

Obviously if you can convert the float to int, then it will work fine. The problem is we don't currently have a nice civilized conversion function or equivalent numerical type coercion mechanism. So as a workaround you have to use an ugly hack, as originally reported here:

  Float f = size(infile)
  String s = f 
  String string_before_decimal = sub(s, "\\..*", "") 
  Int final_int = string_before_decimal

  runtime {
    memory = final_int
  }

This will hopefully be improved in future evolutions of the WDL standard library, so that we will no longer need the hack. But in the meantime at least this will work.

You should be able to add whatever headroom arithmetic you like (e.g. multiplying or adding to either the number produced by size() or the one that is produced by the hack) to calculate the final allocation value.

Tagged:
Sign In or Register to comment.