How do I enable call-caching for local cromwell runs?

shleeshlee CambridgeMember, Broadie, Moderator
edited January 17 in Ask the WDL team

Hi,

I am running cromwell-30.1 locally on a cloud VM.

Is it sufficient to add -Dcall-caching.enabled=true -Dlookup-docker-hash=false to my run commands (as outlined in https://gatkforums.broadinstitute.org/wdl/discussion/7572/how-do-i-enable-call-caching-in-cromwell) to enable call-caching? Or do I need to add some custom configurations as well? I am reading https://cromwell.readthedocs.io/en/develop/Configuring/#database, but it's not clear to me how to create, modify and place a custom config file.

Thanks for your help.

Post edited by shlee on

Answers

  • danbdanb Member, Broadie

    cache results will not persist between each time you run via the run command. I think you want to start via server and perhaps configure a mysql server to persist the results of each run.

  • danbdanb Member, Broadie

    Also see the beginning of that configuration section in the docs for steps on how to setup the conf file in the first place

    https://cromwell.readthedocs.io/en/develop/Configuring/#overview

  • shleeshlee CambridgeMember, Broadie, Moderator

    Thanks @danb. I'll try to follow the directions in the link you sent.

  • shleeshlee CambridgeMember, Broadie, Moderator
    edited January 19

    Just following up on what I've done so far, in case others are looking for the solution.

    In the my.conf file, here are the lines I've changed:

    # Optional call-caching configuration.
    call-caching {
      # Allows re-use of existing results for jobs you've already run
      # (default: false)
      #enabled = true
    

    and (with help from @bshifaw):

    database {
      # This specifies which database to use#
    
     profile = "slick.jdbc.MySQLProfile$"
     db {
          url = "jdbc:mysql://localhost/db_shlee_test?useSSL=false&rewriteBatchedStatements=true"
           user = "root"
           password = ""
           driver = "com.mysql.jdbc.Driver"
           numThreads = 100
           connectionTimeout = 5000 
      }
    
      insert-batch-size = 2000
    
       migration {
         read-batch-size = 100000
         write-batch-size = 100000
       }
    }
    

    Here, db_shlee_test refers to my local system's mysql database. This I create by

    1. downloading and installing mysql (I used brew install mysql on my Mac),
    2. configuring with mysql_secure_installation,
    3. starting the server with mysql.server start,
    4. logging into the server with mysql -u root -p (use password from step 2),
    5. and creating the database with CREATE DATABASE db_shlee_test;

    I haven't gotten as far as testing call-caching on my local system. But I believe all I have to do is refer to the my.conf file when I run a cromwell job, e.g. with

    java -Dconfig.file=/path/to/my.conf -jar cromwell.jar...
    
    Post edited by shlee on
  • shleeshlee CambridgeMember, Broadie, Moderator

    For instructions on how to install brew for linux, see http://linuxbrew.sh/.

  • dannykwellsdannykwells San FranciscoMember

    Hi @shlee to be clear, when you run
    java -Dconfig.file=/path/to/my.conf -jar cromwell.jar...
    you are running the server command, or "run"?

    For doing local development (on a cloud VM), what is the best way to do that will call caching enabled?

Sign In or Register to comment.