runnig wdl workflow with cromwell on google cloud

I am trying to run a wdl workflow using cromwell on google cloud.
I got the cromwell github repo, modified the configuration file with my google specific parameters, but have trouble to launch a workflow.
I am failing to find a detailed tutorial on installation, launch of cromwell with backend google cloud.

Any help would be greatly appreciated.

Idan

Answers

  • RuchiRuchi Member, Broadie, Moderator, Dev

    Hey @idan_g, the docs around setting up a google backend with step-by-step instructions are being drafted right now! Until those are published, there are some technical docs in the Cromwell repo that could help: https://github.com/broadinstitute/cromwell#google-jes-backend. If you can paste the error you're seeing, we can help determine what's missing from your configuration, or if you'd like to share the configuration and hide any secrets, we can point out any missing keys. Thanks!

  • hey @Ruchi thank you for your reply.
    I am not sure I am running cromwell properly. I have installed it with homebrew, so running :

    cromwell -version

    returns:

    cromwell: 28-5fd2237-SNAP

    How do I make sure the execution uses my configuration file? Where the wdl script should be placed, locally? in the bucket? if I have input files, should they be placed in the bucket or locally? If I can get a simple example of running cromwell with some simple input on google cloud - that would be fantastic.

    Thanks

  • RuchiRuchi Member, Broadie, Moderator, Dev

    Hey @idan_g,

    I'm not sure how to pass in a configuration file when running Cromwell through the homebrew install --and hopefully someone more experienced then me at it can chime in on how to do that? However, an easy option is to download the latest jar (https://github.com/broadinstitute/cromwell/releases/tag/28.2) from from the cromwell release page. As for your wdl, I would recommend saving that in a local dir so you can access/modify it easily. Your input files should be in a google directory and their paths should be specified in a workflow inputs json (https://github.com/broadinstitute/wdl/blob/develop/SPEC.md#specifying-workflow-inputs-in-json), and the inputs json can be stored locally. Then your command line would look like:
    java -Dconfig.file=<local_path_to_config_file> -jar <local_path_to_cromwell_jar> run <local_path_to_wdl> <local_path_to_inputs>

  • kshakirkshakir Broadie, Dev

    Hi @idan_g,

    The README probably has the best explanations that exists, but the simplest example I can think of is the integration tests that cromwell uses.

    How do I make sure the execution uses my configuration file?

    -Dconfig.file=myConfig.conf

    NOTE The first line of your config must be #include "application" or #include "application.conf". There are important defaults in the original config file that must be included.

    Where the wdl script should be placed, locally? in the bucket?

    The wdl lives locally. You pass a single wdl via java -jar cromwell.jar run my.wdl, or start a server with java -jar cromwell.jar server and then pass the wdl over HTTP.

    The server mode runs an endpoint that listens on http://localhost:8000 by default. You can connect in a browser and a Swagger web UI will load. From there you can discover and interact with the actual REST endpoints.

    if I have input files, should they be placed in the bucket or locally?

    All inputs for Google JES must be in a bucket. Find and look at the inputs json used in the example below, and see how it references paths at say gs://bucket/path/to/file.

    If I can get a simple example of running cromwell with some simple input on google cloud - that would be fantastic.

    Here is a snapshot of the commands used to run our current nightly tests. Everything to launch this command, including a template for the Google JES test config, is committed in git under src/bin/travis/resources.

  • hey @Ruchi thank you very much, I will try and report back the errors (if there would be any)

Sign In or Register to comment.