Cannot find output when running docker container with cromwell

lwestlak

Hi there, I currently have a method that I now run with Docker and was hoping to put on Firecloud. The way the docker is currently structured, I run a wrapper script within the container and that writes output files to a directory within the container, and I then run a docker cp to get the output files on my local system. In trying to write a WDL, I am able to run the process (stdout log shows output from the script running) but not extract the output (get a file not found error) when I specify the output file using the path within the docker container. I am not sure how to either mount a volume so that the Cromwell can read the outputs or modify my docker so things are written somewhere else.

lwestlak

I should note that I am running this locally with cromwell 39

AdelaideR

Hi @lwestlak

It would be helpful to take a look at your WDL if you do not mind.

When the docker is being run by cromwell, it is similar to being inside an interactive docker. It is useful to make an output directory that the code points to.

command {

        set -eo pipefail 
        mkdir output_dir
        python mypythoncode.py -all -the -parameters

    }

output {
        File python_command_output = "output_dir/results.tar.gz"
    }

Then, it is important to set up the json to recognize your outputs.

But please post your WDL and the errors that are generated when you run it.

lwestlak

Hi @AdelaideR , thanks for getting back to me. I've attached below the WDL, workflow log, and standard out. In the last few lines of the stdout I inspected the directory where I expect my output files to be and they appear to be there (although I'm not sure if this might be a symlink issue?) Thanks again!

AdelaideR

Okay @lwestlak, I will take a look and get back to you soon.

AdelaideR

@lwestlak

I have not figured out about the outputs yet, but I did see that this error message popped up early in the standard out:

Unauthorized to get docker hash lwestlak/tangent-repo:latest

I tried to pull that docker, I don't have permissions. Is there a way to make the repo available?

It looks like the cromwell execution did not have access to that docker, for starters.

lwestlak

Apologies I edited the permissions it should work now

AdelaideR

I apologize if I was not clear, @lwestlak - that error appeared in your standard output, indicating that the docker may not have been pulled during the workflow by the cromwell engine.

Please take a look at this discussion.

It has links to relevant discussions about this permission issue.