Latest Release: 03/12/19
Release Notes can be found here.

workflow function "sub" is too impatient. Needs to wait until inputs are ready....

yfarjounyfarjoun Broad InstituteDev ✭✭✭

Here's an example workflow that I think should work:

workflow testSub {

    call MakeString{}

    String no_and=sub(MakeString.out,"_and","")

}

task MakeString{
    command{
        echo hello_and_goodbye 
    }

     runtime {
      docker: "python:2.7"
      memory: "1 GB"
    }

    output {
        String out = read_string(stdout())
    }
}

Running this however give the following error:

Workflow has invalid declarations: Could not evaluate workflow declarations: testSub.no_and: Invalid parameters for engine function sub: (Failure(wdl4s.exception.OutputVariableLookupException: Could not find outputs for call testSub.MakeString at index None),Success(WdlString(_and)),Success(WdlString())).

Which indicates (to me) that the function sub isn't waiting until the task MakeString runs and has an output, and then it errors out because it can't find one of its inputs...This is not the case for the function "size" for example, which will patiently wait until its inputs are ready.

This causes people to write workarounds that involve putting the sub command inside a subsequent task, piggybacking on the fact that the surrounding task will wait for the output...see for example (private repo) https://github.com/broadinstitute/dsde-pipelines/pull/323/files

Best Answer

Answers

  • yfarjounyfarjoun Broad InstituteDev ✭✭✭

    The workaround is to treat sub as a non-engine command...which kind-of defeats the purpose:

    workflow testSub {
    
        call MakeString{}
    
        call SubTask{
            input:
                string=MakeString.out,
                pattern= "_and",
                replacement= ""}
    }
    
    task MakeString {
        command{
            echo hello_and_goodbye 
        }
    
         runtime {
          docker: "python:2.7"
          memory: "1 GB"
        }
    
        output {
            String out = read_string(stdout())
        }
    }
    
     task SubTask{
    
        String string
        String pattern
        String replacement
    
        command{}
    
        output {
            String out=sub(string, pattern, replacement)
        }
    
        runtime {
          docker: "python:2.7"
          memory: "1 GB"
        }
     }
    
    
    

    since once I've spun-up a VM I don't need to use sub...I can use sed...or anything for that matter!

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    This sounds like a change to the functionality of Cromwell/WDL itself. I agree it would be useful to have the sub function wait to hear back from MakeString before deciding the output does not exist, and I will speak with the Cromwell team about getting this developed.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin
    edited October 2017

    hey @yfarjoun what Cromwell are you using? This sounds like something that got fixed in the last few versions. If not then it's 100% a bug that we should fix

  • yfarjounyfarjoun Broad InstituteDev ✭✭✭

    that was on firecloud...cromwell 29.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    Oh, I thought FC is still on 28?

    I'll double check this against 29 just in case but I'm fairly confident this has been fixed

  • yfarjounyfarjoun Broad InstituteDev ✭✭✭

    you are correct. version 28. sorry..

  • yfarjounyfarjoun Broad InstituteDev ✭✭✭

    OK, I just checked in FC again. Still broken.

  • yfarjounyfarjoun Broad InstituteDev ✭✭✭

    Thanks for looking for a workaround. That will not be needed at this point (I'm not blocked by this) but it's an annoying bug. (I posted my workaround here and I'm not going to swap one clunky workaround for another...)

    Hopefully Firecloud will move to C30 soon and this problem will be solved.

Sign In or Register to comment.