This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.
Feedback on initial version of bcbio WDL converted from CWL
I've been working to run bcbio (https://github.com/chapmanb/bcbio-nextgen) with Cromwell using conversion of bcbio generated CWL (http://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html) to WDL. I have an in-progress CWL to WDL converter (https://github.com/chapmanb/bcbio-nextgen/blob/master/scripts/utils/cwltool2wdl.py) that generates reasonable looking WDL for most of the functionality of CWL we're using:
The idea is to practically get bcbio running, and also define an interoperable subset of WDL and CWL. This implementation currently supports workflows, tasks and nested workflows. It converts CWL records to WDL objects and also has scatter functionality.
I have most of the parts in place but have a couple of questions about the best way to integrate some last steps prior to starting to actually try runs with Cromwell. I'm completely open to changing bcbio outputs to make it more compatible with WDL but not sure about the best way to do it. The two areas I'm not sure about are:
Can you scatter over multiple variables simultaneously? In CWL this is a dotproduct scatter. I have generated speculative WDL output, but am not sure if this is actually supported in some way:
Do I need to put everything into an Object first and then scatter over that? I do that in some places using CWL Records mapped to Objects:
Is there a way to import output information from a big JSON object similar to CWL's cwl.output.json?
or is this reasonable to add to the standard library? I mocked up what I imagined it looking like as
I am open to outputting something else that is easier for ingesting into WDL. It looks like
read_jsonmight do some of this if I generate a per-output json file but I'm not sure how to encode File references.
Thanks for any suggestions and thoughts. I'm excited to open a conversation on this and provide some interoperability between WDL and CWL so we can make use of bcbio across multiple platforms.