What does the WDL development cycle look like?

Ok, this is a "soft" question but one that I could use some help with. One thing I've found is that, even when using a test data set, developing WDL code is slow, mostly because the programs running it are slow. And, right now, what will often happen is that a command that works file in the container it was developed in will error out in WDL because of typos, syntax, etc. So, my question is:

Is there any way to launch a WDL script from a specified step in the workflow (assuming that you have pointers to all of the files that would go in at that step)? It would be great to be able to develop each step in my pipeline iteratively on top of the results that came before.

More broadly, thoughts on how to have quick iteration cycle with WDL would be helpful.

Thanks!
-d

Tagged:

Best Answer

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev ✭✭

    Hey @dannykwells it sounds like you might benefit from the Call Caching feature - it remembers previous calls based on the set of inputs, command line, docker image, etc and can short-circuit re-running them in future workflow instances if nothing's changed.

    If you enable this you should be able to change one task and have all the preceding calls be read from cache instead of re-evaluated.

    Does that sound like the sort of thing you were after? To enable this you'll need to set up a mysql database

  • dannykwellsdannykwells San FranciscoMember ✭✭

    That would be helpful, yes. Do you have a document on how to set this up with a Google Cloud backend (using, ideally, the SQL tables in google?)

    Also, any details on how to handle caching and if there are any caveats to be aware of would be great. Thanks!

  • dannykwellsdannykwells San FranciscoMember ✭✭

    Ok, great, got this working on GCP. Had to do some hacking on the .conf file to get all the connections to the SQL table to work, it could be useful to have some more docs here.

    Thanks again!

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev ✭✭

    @dannykwells that's great news that you got it working!

    Documentation is always a struggle... for what it's worth I think the best way to improve is to ask for feedback so feel free to let us know where specifically you had trouble and what you think would improve the situation!

    (I almost hesitate to mention it, but if you wanted to submit a documentation improvement PR to our github project that would be very well received too!)

  • dannykwellsdannykwells San FranciscoMember ✭✭

    Hi @ChrisL : sounds good. I will talk to some folks and be back in touch.

Sign In or Register to comment.