How to enable call caching using database.

In the document, for call caching, In order for call caching to be used on any previously run jobs, it is best to configure Cromwell to point to a MySQL database instead of the default in-memory database.

How to configure to use database? As long as database information provided and call caching flag is enabled in configuration file, will call caching automatically point to the defined database?

Thanks
Jing

Best Answer

  • lij41lij41
    Accepted Answer

    Thank you so much @ChrisL, @EADG, @danb.

    I successfully configured and ran the call caching after seeing your posts.

    But since Cromwell release AWS Batch support, I tested call caching in this release, it seems to me the call caching was broken when I have aws batch enabled. When I tried to run hello world example (print hello message to a file), every time it will submit the job to aws batch queue and create a new output file.

    I am running cromwell-35.jar.

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    That's correct - if your config file specifies a database (like in https://cromwell.readthedocs.io/en/stable/Configuring/#database) then it will automatically be used by Cromwell.

    To enable call caching you need to opt-in via a few extra config options: https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching

  • EADGEADG KielMember ✭✭✭

    Hi @lij41,

    A while ago I wrote are a short manual for my colleagues, maybe it is helpful for you too:
    A short guide to run MySQL-DB in a docker-container and use it for cromwell-server.
    Prerequisite:
    => cromewell >= 29 (https://github.com/broadinstitute/cromwell/releases)
    => docker already installed (https://docs.docker.com/engine/installation/)

    1) Pull the MySQL docker image from dockerhub:
    docker pull mysql:5.5 (You can choose another version if you want)

    2) Start the MySQL docker container with the following line:
    docker run -p 3306:3306 --name NameOfTheContainer -e MYSQL_ROOT_PASSWORD=YourPassword-e MYSQL_DATABASE=DatabaseName -e MYSQL_USER=ChooseAName -e MYSQL_PASSWORD=YourOtherPassword -d mysql/mysql-server:latest
    The -p option maps the standard port 3306 from the container to the 3306 on your machine running the container, this is necessary for cromwell to communicate with the database inside the docker container. You can choose names and passwords freely but keep them in mind for the cromwell configuration file.

    3) Update your application.conf file.
    database { profile = "slick.jdbc.MySQLProfile$" db{ driver = "com.mysql.jdbc.Driver" url = "jdbc:mysql://localhost/DatabaseName?useSSL=false" user = "ChooseAName" password = "YourOtherPassword" connectionTimeout = 5000 } }

    Add the line above, below the all other lines in your application.conf. Replace "DatabaseName","ChooseAName" and "YourOtherPassword" with the values you choose in step 2, preserving the double qoutes.

    Test it, by running your server with the updated application.conf.
    java -Dconfig.file=/path/to/application.conf/ -jar ...

    The guide was tested on Debian 8.
    ```

    Greetings EADG

  • danbdanb Member, Broadie ✭✭✭

    I attempted a minimal DB config w/ call caching here

  • lij41lij41 Member
    Accepted Answer

    Thank you so much @ChrisL, @EADG, @danb.

    I successfully configured and ran the call caching after seeing your posts.

    But since Cromwell release AWS Batch support, I tested call caching in this release, it seems to me the call caching was broken when I have aws batch enabled. When I tried to run hello world example (print hello message to a file), every time it will submit the job to aws batch queue and create a new output file.

    I am running cromwell-35.jar.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @lij41 ,

    Are you running Cromwell in server mode? Is there any chance you could share the metadata from a hello world run? It contains information about the cause of why call caching failed -- so it would make it a lot easier to debug.

    Thanks!

  • lij41lij41 Member

    No. I did not run in server mode. But call caching shall work both run and server mode, right? How would you like me to share metadata? dump database to a file and upload? What are the tables you would like to see?

    Thanks

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @lij41,

    Yes, Call caching should work in both modes. If you run Cromwell as a server, then you can call the /metadata endpoint which returns a JSON object, which can be attached here.

    Step 1: Setup a server
    https://cromwell.readthedocs.io/en/stable/tutorials/ServerMode/

    Step 2: Run a workflow! (uuid-1)

    Step 3: Run workflow again! (uuid-2)

    Step 4: Get workflow metadata
    https://cromwell.readthedocs.io/en/stable/api/RESTAPI/#get-workflow-and-call-level-metadata-for-a-specified-workflow

    If you could share the response, that would be great.

    Thanks!

  • lij41lij41 Member

    Here is the metadata for the second round. The metadata for the first round is also similar.
    {
    "workflowName": "test",
    "actualWorkflowLanguageVersion": "draft-2",
    "submittedFiles": {
    "workflow": "task hello {\n String name\n command {\n echo 'Hello ${name}!' > \"hello${name}.txt\"\n }\n output {\n File response = \"hello${name}.txt\"\n }\n runtime {\n docker: \"ubuntu:latest\"\n }\n}\n\nworkflow test {\n call hello\n}\n",
    "root": "",
    "options": "{\n\n}",
    "inputs": "{\"test.hello.name\":\"World\"}",
    "workflowUrl": "",
    "labels": "{}"
    },
    "calls": {
    "test.hello": [
    {
    "executionStatus": "Done",
    "stdout": "s3://s4-somaticgenomicsrd-valinor/cromwell-execution/test/ffb214ab-8b28-4389-9622-c59beb0c13b1/call-hello/hello-stdout.log",
    "backendStatus": "Succeeded",
    "commandLine": "echo 'Hello World!' > \"helloWorld.txt\"",
    "shardIndex": -1,
    "outputs": {
    "response": "s3://s4-somaticgenomicsrd-valinor/cromwell-execution/test/ffb214ab-8b28-4389-9622-c59beb0c13b1/call-hello/helloWorld.txt"
    },
    "runtimeAttributes": {
    "failOnStderr": "false",
    "queueArn": "arn:aws:batch:us-east-1:267795504649:job-queue/GenomicsHighPriorityQue-ae4256f76f07d96",
    "disks": "local-disk /cromwell_root",
    "continueOnReturnCode": "0",
    "docker": "ubuntu:latest",
    "maxRetries": "0",
    "cpu": "1",
    "cpuMin": "1",
    "noAddress": "false",
    "zones": "us-east-1a",
    "memoryMin": "2 GB",
    "memory": "2 GB"
    },
    "callCaching": {
    "allowResultReuse": true,
    "hit": false,
    "result": "Cache Miss",
    "hashes": {
    "output count": "C4CA4238A0B923820DCC509A6F75849B",
    "runtime attribute": {
    "docker": "66E19F14150E71B0E42CA8557A69C5F9",
    "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA",
    "failOnStderr": "68934A3E9455FA72420237EB05902327"
    },
    "output expression": {
    "File response": "D60C182DE4045F1C60D7B55507F75606"
    },
    "input count": "C4CA4238A0B923820DCC509A6F75849B",
    "backend name": "2267EF43AEF6BB551F414FEC2390F68A",
    "command template": "450143B1E23E152B5A10E6799E402200",
    "input": {
    "String name": "362DC0726AA6A2A62E9F7773FD901DF1"
    }
    },
    "effectiveCallCachingMode": "ReadAndWriteCache",
    "hitFailures": [
    {
    "2f58eee9-1b0f-4436-a4ad-48eb305655e9:test.hello:-1": [
    {
    "causedBy": [
    {
    "causedBy": [],
    "message": "The specified key does not exist. (Service: S3Client; Status Code: 404; Request ID: C669EBD8405F1085)"
    }
    ],
    "message": "[Attempted 1 time(s)] - NoSuchKeyException: The specified key does not exist. (Service: S3Client; Status Code: 404; Request ID: C669EBD8405F1085)"
    }
    ]
    },
    {
    "02306258-436a-4372-ab54-2dcd83c42b47:test.hello:-1": [
    {
    "causedBy": [
    {
    "causedBy": [],
    "message": "The specified key does not exist. (Service: S3Client; Status Code: 404; Request ID: CC2E88C00D5E3176)"
    }
    ],
    "message": "[Attempted 1 time(s)] - NoSuchKeyException: The specified key does not exist. (Service: S3Client; Status Code: 404; Request ID: CC2E88C00D5E3176)"
    }
    ]
    },
    {
    "caa2b9fd-aa68-465b-abea-ac1fcfa17bdd:test.hello:-1": [
    {
    "causedBy": [
    {
    "causedBy": [],
    "message": "The specified key does not exist. (Service: S3Client; Status Code: 404; Request ID: 706FA41111E8041F)"
    }
    ],
    "message": "[Attempted 1 time(s)] - NoSuchKeyException: The specified key does not exist. (Service: S3Client; Status Code: 404; Request ID: 706FA41111E8041F)"
    }
    ]
    }
    ]
    },
    "inputs": {
    "name": "World"
    },
    "returnCode": 0,
    "jobId": "e363c645-6cd2-486a-9138-f66f27e25e35",
    "backend": "AWSBATCH",
    "end": "2018-12-03T18:58:51.652Z",
    "dockerImageUsed": "[email protected]:6d0e0c26489e33f5a6f0020edface2727db9489744ecc9b4f50c7fa671f23c49",
    "stderr": "s3://s4-somaticgenomicsrd-valinor/cromwell-execution/test/ffb214ab-8b28-4389-9622-c59beb0c13b1/call-hello/hello-stderr.log",
    "callRoot": "s3://s4-somaticgenomicsrd-valinor/cromwell-execution/test/ffb214ab-8b28-4389-9622-c59beb0c13b1/call-hello",
    "attempt": 1,
    "executionEvents": [
    {
    "startTime": "2018-12-03T18:58:31.121Z",
    "description": "FetchingCachedOutputsFromDatabase",
    "endTime": "2018-12-03T18:58:31.142Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.703Z",
    "description": "FetchingCachedOutputsFromDatabase",
    "endTime": "2018-12-03T18:58:30.724Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.638Z",
    "description": "PreparingJob",
    "endTime": "2018-12-03T18:58:30.671Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.724Z",
    "description": "BackendIsCopyingCachedOutputs",
    "endTime": "2018-12-03T18:58:31.099Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.099Z",
    "description": "CheckingCallCache",
    "endTime": "2018-12-03T18:58:31.121Z"
    },
    {
    "startTime": "2018-12-03T18:58:50.332Z",
    "description": "UpdatingCallCache",
    "endTime": "2018-12-03T18:58:50.757Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.368Z",
    "description": "BackendIsCopyingCachedOutputs",
    "endTime": "2018-12-03T18:58:31.524Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.539Z",
    "description": "Pending",
    "endTime": "2018-12-03T18:58:30.540Z"
    },
    {
    "startTime": "2018-12-03T18:58:50.757Z",
    "description": "UpdatingJobStore",
    "endTime": "2018-12-03T18:58:51.652Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.142Z",
    "description": "BackendIsCopyingCachedOutputs",
    "endTime": "2018-12-03T18:58:31.333Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.671Z",
    "description": "CheckingCallCache",
    "endTime": "2018-12-03T18:58:30.703Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.333Z",
    "description": "CheckingCallCache",
    "endTime": "2018-12-03T18:58:31.352Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.540Z",
    "description": "RequestingExecutionToken",
    "endTime": "2018-12-03T18:58:30.637Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.352Z",
    "description": "FetchingCachedOutputsFromDatabase",
    "endTime": "2018-12-03T18:58:31.368Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.524Z",
    "description": "CheckingCallCache",
    "endTime": "2018-12-03T18:58:31.543Z"
    },
    {
    "startTime": "2018-12-03T18:58:30.637Z",
    "description": "WaitingForValueStore",
    "endTime": "2018-12-03T18:58:30.638Z"
    },
    {
    "startTime": "2018-12-03T18:58:31.543Z",
    "description": "RunningJob",
    "endTime": "2018-12-03T18:58:50.332Z"
    }
    ],
    "backendLogs": {
    "log": "s3://s4-somaticgenomicsrd-valinor/cromwell-execution/test/ffb214ab-8b28-4389-9622-c59beb0c13b1/call-hello/hello.log"
    },
    "start": "2018-12-03T18:58:30.539Z"
    }
    ]
    },
    "outputs": {
    "test.hello.response": "s3://s4-somaticgenomicsrd-valinor/cromwell-execution/test/ffb214ab-8b28-4389-9622-c59beb0c13b1/call-hello/helloWorld.txt"
    },
    "actualWorkflowLanguage": "WDL",
    "id": "ffb214ab-8b28-4389-9622-c59beb0c13b1",
    "inputs": {
    "test.hello.name": "World"
    },
    "labels": {
    "cromwell-workflow-id": "cromwell-ffb214ab-8b28-4389-9622-c59beb0c13b1"
    },
    "submission": "2018-12-03T18:58:27.593Z",
    "status": "Succeeded",
    "end": "2018-12-03T18:58:52.970Z",
    "start": "2018-12-03T18:58:29.479Z"
    }

Sign In or Register to comment.