New to the forum? Ask your questions here!

Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

We are facing a high volume of spam that we are working to prevent. Due to this, new users are only able to comment on current question threads, not create new ones. Please post your WDL/Cromwell question here if this applies to you. We apologize for this inconvenience and should have this resolved in the next few days.

Answers

  • natecholsnatechols Member

    Cross-posting from the GATK form: When I tried to use the SGE backend I quickly ran into problems with executables not being on path. Is there a way to propagate $PATH for a workflow? @ChrisL responded that "When you submit jobs to SGE without docker, I believe you're relying on whatever environment the cluster has set up for you. If that sounds OK because the environment should be ok, you might just need to use a fully qualified path to the tool you want to use." Unfortunately this is not an option, since the executable path will vary. I need to be able to modify the environment seen by the SGE jobs. (Or another cluster system - a colleague showed me how to propagate PATH in SGE, but I'm hoping for a more general solution.)

  • natecholsnatechols Member
    edited November 29

    Second question: when a qsub job itself exits with non-zero status, Cromwell isn't picking this up, and it hangs forever waiting for the task to finish. qacct -j $JOB shows the exit status as 1, but this isn't picked up by Cromwell. The only files in the task execution directory are:
    script script.submit stderr stderr.submit stdout stdout.submit
    The stderr has only "Illegal variable name."; stderr.submit is empty. No other clue what failed or why, and Cromwell is still running.

    This isn't always reproducible either - I just saw a task get retried 3 times, initially with the expected error (my unrelated bug crashing the task), correctly captured in rc and with qsub exiting 0, but the last time qsub failed entirely and Cromwell hung. It's clearly a Bash syntax problem but I don't know why it's inconsistent.

    Post edited by natechols on
  • natecholsnatechols Member

    I figured out a workaround for my first question - not elegant, but good enough for now.

  • DorjeeDorjee Member

    I have 3 tasks and the 3rd (last) one should kick-in only when the 1st and 2nd tasks are completed. How do I WDL this type of dependency? Please help.

    I guess, I could take multiple inputs and then check if both inputs are valid? Is there any other way to accomplish this?

    Eg:

    workflow TestWorkflow {
        call taskA
        call taskB
        call taskC {
            input: 
                inA = taskA.out, 
                inB = taskB.out
              } 
        }
    

    Thanks in advance,

  • RuchiRuchi Member, Broadie, Moderator, Dev admin
    edited December 3

    Hey @natechols,

    Setting a variable

    There might be two possible ways of handling the case of setting an env variable accessible a job:
    1. Setting the path variable directly inside the command block of a task, which is the most common approach.
    2. One could set a variable directly inside the SGE config stanza (example taken from here):

    backend.providers.SGE.config {
      runtime-attributes = """
      Int cpu = 1
      Float? memory_gb
      String? sge_queue
      String? sge_project
      """
    }
    

    Here one can see that cpu is set to 1 by default for all jobs run on this Cromwell SGE configuration. Similarly, you could add a line for a path variable that all jobs in this Cromwell config can access:

      String gatk = "/path/to/gatk/jar"
    

    It could be referenced inside of any task as:

    task taskName {
       command {
        java -jar ${gatk} ...
      }
    }
    

    Hanging job

    One suggestion is to add a set -e to the start of every command block, so that it forces the job to exit with a non-zero return code if there's an error. The logs you listed that were present doesn't include an rc file, and that is need to ensure that Cromwell will mark the job failed.

    Secondarily, if you notice that the set -e doesn't help, a secondary thing to try is to set a key in your SGE config check-alive that uses a given command to monitor if the SGE job is alive:
    https://cromwell.readthedocs.io/en/develop/tutorials/HPCIntro/#how-cromwell-checks-if-an-hpc-job-is-alive
    You'd want to pair that configuration with what interval should Cromwell poll qsub to check if jobs are still alive:
    https://cromwell.readthedocs.io/en/develop/backends/HPC/#exit-code-timeout

    Hope this helps.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    @Dorjee

    There is an upcoming feature to address this more elegantly:
    https://github.com/openwdl/wdl/pull/162

    For now, the best way to handle is exactly as you suggested. If both taskA and taskB produce some kind of output, then taskC will automatically only run if both taskA and taskB succeeded. If I can help review any specific workflow, please let me know.

    Thanks!

  • natecholsnatechols Member

    New question: it appears that conditionals can't be used to evaluate a boolean parameter that have a default value. If I do this:

    workflow my_workflow {
      Boolean run_clustering
    
      if (run_clustering) {
        ...
      }
    

    it works - as long as I pass an explicit value for foo.run_clustering in my inputs - but if I change it to Boolean run_clustering = true, I get this error message: "Unable to build WOM node for If '$if_0': An if block must be given a boolean expression but instead got 'run_clustering' (a Boolean?)". Is this a bug, or am I doing something wrong?

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @natechols,

    So the before is what you posted above and the after is:

    workflow my_workflow {
      Boolean run_clustering = true
    
      if (run_clustering) {
        ...
      }
    

    and this doesn't work?

  • natecholsnatechols Member

    @Ruchi sorry, I left out a "?":

    workflow my_workflow {
      Boolean? run_clustering = true
    
      if (run_clustering) {
        ...
      }
    

    Am I using the question mark incorrectly?

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @natechols

    You're totally using the question mark correctly. The problem is that WDL can't coerce an optional to a non-optional type. The if condition requires a non-optional Boolean, so another way to approach this is:

    workflow my_workflow {
    
      input {
        Boolean run_clustering = true
      }
    
      if (run_clustering) {
        ...
      }
    
    

    With this input {} block, you can override run_clustering to be false from our inputs json if you'd like.

    https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#declared-inputs-defaults-and-overrides

  • natecholsnatechols Member

    That makes sense, but I haven't been using the input {} block in workflows so far - what is the functional difference between this versus declaring inputs in the outer "scope" of the workflow block? (That's what the giant GOTC_PairedEndSingleSampleWf.wdl example does, so I assumed it was proper form...)

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @natechols

    The input {} block is a recent addition so you may not see it on a lot of workflows yet. It's a new recommended practice to distinguish between inputs that have a declared default value , and the input block allows them to be overridable but the outer scope prevents override.

  • natecholsnatechols Member

    @Ruchi how recent? I'm using Cromwell 36 and it gave me a syntax error.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    @natechols

    I forgot, but you'll also need to add a version to the top of your file, as the input {} syntax is supported by the newest version of wdl (1.0), which needs to be specified. By default Cromwell assumes draft-2 (old version) as thats what most people use out of the box.

    version 1.0
    
    workflow w {...}
    
Sign In or Register to comment.