To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Is there a way to tell Cromwell to ignore the "runtime section" of the tasks in a WDL script?

Is there a way to tell Cromwell to ignore the "runtime section" of the tasks in a WDL script? Our Linux servers don't let us use Docker, so it would be nice to be able to write a single WDL script that will run in three different situations: (1) Firecloud, (2) on my local Mac using a Docker image, and (3) on one of our Linux servers.

At the moment, I can run the same WDL script in all three situations, but for #3, at the moment I have to manually edit the WDL script to comment out the runtime sections. Hopefully, there's a better way, but at the moment, this is the only way I know how to accomplish this goal. (Short of writing a little program to strip out the runtime sections programmatically.)

I've tried defining a Boolean input parameter for the script and then doing something along the lines of

runtime {
    docker: if useDocker then "nessus/mite-seq-process" else ""
}

But unfortunately, that doesn't work. If useDocker is false, Cromwell will try to use a Docker image with the empty string as its name. That fails, of course.

Tagged:

Best Answer

Answers

  • RuchiRuchi Member, Broadie, Dev

    Hey @nessus42,

    There is no way to selectively ignore certain runtime attributes, but you can configure your own Local backend that ignores docker by removing the submit-docker stanza in the default Local backend.
    A sample workflow options file:

    Local {
          actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
          config {
    
            # Limits the number of concurrent jobs
            #concurrent-job-limit = 5
    
            run-in-background = true
            # `script-epilogue` configures a shell command to run after the execution of every command block.
            #
            # If this value is not set explicitly, the default value is `sync`, equivalent to:
            # script-epilogue = "sync"
            #
            # To turn off the default `sync` behavior set this value to an empty string:
            # script-epilogue = ""
    
            runtime-attributes = """
            String? docker
            String? docker_user
            """
            submit = "/bin/bash ${script}"
            #submit-docker = """
            #docker run \
            #  --rm -i \
            #  ${"--user " + docker_user} \
            #  --entrypoint /bin/bash \
            #  -v ${cwd}:${docker_cwd} \
            #  ${docker} ${script}
            #"""
    
    
    
            # Root directory where Cromwell writes job results.  This directory must be
            # visible and writeable by the Cromwell process as well as the jobs that Cromwell
            # launches.
            root = "cromwell-executions"
    
            filesystems {
              local {
                localization: [
                  "hard-link", "soft-link", "copy"
                ]
    
                caching {
                  # When copying a cached result, what type of file duplication should occur. Attempted in the order listed below:
                  duplication-strategy: [
                    "hard-link", "soft-link", "copy"
                  ]
    
                  # Possible values: file, path
                  # "file" will compute an md5 hash of the file content.
                  # "path" will compute an md5 hash of the file path. This strategy will only be effective if the duplication-strategy (above) is set to "soft-link",
                  # in order to allow for the original file path to be hashed.
                  hashing-strategy: "file"
    
                  # When true, will check if a sibling file with the same name and the .md5 extension exists, and if it does, use the content of this file as a hash.
                  # If false or the md5 does not exist, will proceed with the above-defined hashing strategy.
                  check-sibling-md5: false
                }
              }
            }
    
            default-runtime-attributes {
              failOnStderr: false
              continueOnReturnCode: 0
            }
          }
    
  • Thanks Ruchi!

    Unfortunately, Cromwell still tries to use Docker even if I use the config file that you provided. (I know that it's actually using the config file b/c if I introduce a syntax error into it, then Cromwell dies quickly, complaining about the syntax error.)

    Even while using the provided Cromwell config file, I see this in the terminal output:

    executing: docker run \
      --rm -i \
       \
      --entrypoint /bin/bash \
      -v /Volumes/Ext-SSD/dalan-ext/s.big/local/wdl/mite-seq/cromwell-executions/fetchAndProcessFiles/51def993-4991-42b6-9a7c-cb54f4f945e2/call-expandPreMadeDataDir:/cromwell-executions/fetchAndProcessFiles/51def993-4991-42b6-9a7c-cb54f4f945e2/call-expandPreMadeDataDir \
      nessus/mite-seq-process@sha256:6a3ced61ebfd7056f6177c941e7d42adbfcf724c53796f67b2d8f5568f9bc562 /cromwell-executions/fetchAndProcessFiles/51def993-4991-42b6-9a7c-cb54f4f945e2/call-expandPreMadeDataDir/execution/script
    

    Thanks for your help!
    |>oug (dalan@broadinstitute.org)

  • That seemed to work. Thanks!!!

  • P.S. How do I get this forum to say that I'm a Broadie? My email address is set to "dalan@broadinstitute.org".

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev

    I'm going to ping the existing ticket that requested optional docker values with this thread.

Sign In or Register to comment.