To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Optional task inputs for imported workflows

I recently moved some shared code into a workflow that I can import, and I noticed that there is no possibility to change some of the default arguments for tasks within the import workflow.
When I import a task, wdltool inputs puts in (optional) config keys for all the possible inputs for the imported task. But for workflows, this is not the case.

I was wondering if this is a bug or by design? I could put the optional arguments into the workflow itself as a workaround, but that would lead to a lot of duplicated argument definitions, and it would be easy to miss some. Especially if you have imported workflows that import other workflows themselves.

I have tried to manually add the arguments I wanted to the arguments.json file, but they seem to get ignored. (No errors though)

Issue · Github
by Geraldine_VdAuwera

Issue Number
2083
State
open
Last Updated
Assignee
Array

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie
    Hi Redmar, I think this is probably a bug. Can you give a small code example so we can make sure we understand your use case and observations?
  • For the file main.wdl

    import "share.wdl" as share
    
    workflow wf {
        String user
    
        call share.hello as hello {
            input:  user = user
        }
        call share.howdie as howdie {
            input: user  = user
        }
    }
    

    For the imported tasks and workflows share.wdl

    task hello {
        String user
        String? greeting
    
        command {
            echo "${default="Hello" greeting} ${user}"
        }
        output {
            String text= read_string(stdout())
        }
    }
    
    workflow howdie {
        String user
    
        call hello {
            input: user = user
        }
        output {
            String hi = hello.text
        }
    }
    

    As you can see below, the optional input for the task hello is specified, but not for the workflow howdie that includes the task. I expected to also see an optional input for wf.howdie.hello.greeting. I don't think that would lead to conflicts, since all names are fully qualified.

    $ wdltool inputs main.wdl 
    {
      "wf.user": "String",
      "wf.hello.greeting": "(optional) String?"
    }
    
  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev
    edited May 2017

    I also suspect this is a bug, but I'd also like to throw in my personal style preference.

    My personal preference would be to express this with a passthrough variable, e.g. adding to the main WDL:

    workflow wf {
      String user
      String? greeting
      call share.hello as hello { input:
        user = user
        greeting = greeting
      }
    

    I like this because it means all inputs to wf are declared right there, at the top of wf. Besides being easy to find, the interface to wf can stay the same regardless of its underlying imports changing or being replaced.

  • @ChrisL

    I use that as a workaround now for setting that I really need to modify, but its a lot of duplicate effort, in my opinion. Because every time I change something to the imported workflow (like adding a new parameter), I also have to modify all workflows that use it to add the passthrough variable.

  • @Geraldine_VdAuwera @ChrisL

    I just ran into this issue again, any news on implementing (optional) inputs for sub-workflows in wdl?

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie

    I think I heard some chatter around that recently. @ChrisL , any thoughts?

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev

    So we looked into this and the reason for this is that workflow calls encapsulate any inner calls (rather than letting inputs from the JSON trickle all the way down). That has a bunch of benefits including:

    • Workflow calls looking identical to task calls from inputs json.
    • Making it easy to find out what inputs are required for each call, whether workflow or task.
    • Letting far-nested sub-workflows alter their implementation without external callers having to change their inputs JSON, so long as the sub-workflow interface doesn't change.

    The downside (which I realize is annoying) is that you have to have a pass throughs variable per sub-workflow input, I'm hopeful that the structs in WDL draft 3 are going to make that much easier (since they'll let you bundle all the variables together and ingest everything as a single input declaration)

    FWIW the existing bug is that wdltool (or now womtool) should be rejecting the validation rather than creating nested inputs in the file which Cromwell ignores or fails with. I believe @Ruchi is looking into this actually?

Sign In or Register to comment.