How to add plumbing between tasks in docker containers
Disclaimer first: I'm a WDL/Cromwell/Docker newb.
I'm trying to (basically) run a 2-task linear workflow that will eventually be run in FireCloud. In this workflow, the output of task1 is used as input for task2.
I'm following the steps in "(howto) Use my own tools in FireCloud". I put my tool in a Docker image in Dockerhub. In my WDL workflow, for each task, I've referenced the Docker image in DockerHub.
Here's where I run into problems. When cromwell gets to task2, it automatically mounts a task2-specific directory. However, the input file for task2 (i.e., the output from task1) is in the task1-specific directory which is invisible to task2's running Docker container. As such, I get a file not found error when task2 tries to use the file.
I don't think it's a WDL syntax issue because if I comment out the runtime docker statements in the .wdl file, the workflow completes successfully. Therefore, I think something must be awry with my understanding of how cromwell and docker communicate. When using multiple Docker containers, how do I tell them where the last step's files are? One solution is to combine the tasks so only 1 Docker container is created. However, this solution would make the code less modular...
This question is somewhat similar to "Accessing local files in host machine while running WDL/CROMWELL script / Mount volume automatically" but is slightly different because I need help localizing intermediate files, not workflow input files.
My .wdl, cromwell stdout and stderr, and task stderr files are on Google Drive here (Sorry, my internet browser had trouble attaching the files in this post.)
OS: Ubuntu 16.04.1