How do I enable call-caching for local cromwell runs?

shleeshlee CambridgeMember, Broadie ✭✭✭✭✭
edited January 2018 in Ask the Cromwell + WDL Team

Hi,

I am running cromwell-30.1 locally on a cloud VM.

Is it sufficient to add -Dcall-caching.enabled=true -Dlookup-docker-hash=false to my run commands (as outlined in https://gatkforums.broadinstitute.org/wdl/discussion/7572/how-do-i-enable-call-caching-in-cromwell) to enable call-caching? Or do I need to add some custom configurations as well? I am reading https://cromwell.readthedocs.io/en/develop/Configuring/#database, but it's not clear to me how to create, modify and place a custom config file.

Thanks for your help.

Post edited by shlee on

Answers

  • danbdanb Member, Broadie ✭✭✭

    cache results will not persist between each time you run via the run command. I think you want to start via server and perhaps configure a mysql server to persist the results of each run.

  • danbdanb Member, Broadie ✭✭✭

    Also see the beginning of that configuration section in the docs for steps on how to setup the conf file in the first place

    https://cromwell.readthedocs.io/en/develop/Configuring/#overview

  • shleeshlee CambridgeMember, Broadie ✭✭✭✭✭

    Thanks @danb. I'll try to follow the directions in the link you sent.

  • shleeshlee CambridgeMember, Broadie ✭✭✭✭✭
    edited January 2018

    Just following up on what I've done so far, in case others are looking for the solution.

    In the my.conf file, here are the lines I've changed:

    # Optional call-caching configuration.
    call-caching {
      # Allows re-use of existing results for jobs you've already run
      # (default: false)
      #enabled = true
    

    and (with help from @bshifaw):

    database {
      # This specifies which database to use#
    
     profile = "slick.jdbc.MySQLProfile$"
     db {
          url = "jdbc:mysql://localhost/db_shlee_test?useSSL=false&rewriteBatchedStatements=true"
           user = "root"
           password = ""
           driver = "com.mysql.jdbc.Driver"
           numThreads = 100
           connectionTimeout = 5000 
      }
    
      insert-batch-size = 2000
    
       migration {
         read-batch-size = 100000
         write-batch-size = 100000
       }
    }
    

    Here, db_shlee_test refers to my local system's mysql database. This I create by

    1. downloading and installing mysql (I used brew install mysql on my Mac),
    2. configuring with mysql_secure_installation,
    3. starting the server with mysql.server start,
    4. logging into the server with mysql -u root -p (use password from step 2),
    5. and creating the database with CREATE DATABASE db_shlee_test;

    I haven't gotten as far as testing call-caching on my local system. But I believe all I have to do is refer to the my.conf file when I run a cromwell job, e.g. with

    java -Dconfig.file=/path/to/my.conf -jar cromwell.jar...
    
    Post edited by shlee on
  • shleeshlee CambridgeMember, Broadie ✭✭✭✭✭

    For instructions on how to install brew for linux, see http://linuxbrew.sh/.

  • dannykwellsdannykwells San FranciscoMember ✭✭

    Hi @shlee to be clear, when you run
    java -Dconfig.file=/path/to/my.conf -jar cromwell.jar...
    you are running the server command, or "run"?

    For doing local development (on a cloud VM), what is the best way to do that will call caching enabled?

Sign In or Register to comment.