Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.

Cannot find output when running docker container with cromwell

Hi there, I currently have a method that I now run with Docker and was hoping to put on Firecloud. The way the docker is currently structured, I run a wrapper script within the container and that writes output files to a directory within the container, and I then run a docker cp to get the output files on my local system. In trying to write a WDL, I am able to run the process (stdout log shows output from the script running) but not extract the output (get a file not found error) when I specify the output file using the path within the docker container. I am not sure how to either mount a volume so that the Cromwell can read the outputs or modify my docker so things are written somewhere else.
Tagged:

Answers

  • lwestlaklwestlak Member
    I should note that I am running this locally with cromwell 39
  • AdelaideRAdelaideR Member admin

    Hi @lwestlak

    It would be helpful to take a look at your WDL if you do not mind.

    When the docker is being run by cromwell, it is similar to being inside an interactive docker. It is useful to make an output directory that the code points to.

    command {
    
            set -eo pipefail 
            mkdir output_dir
            python mypythoncode.py -all -the -parameters
    
        }
    
    output {
            File python_command_output = "output_dir/results.tar.gz"
        }
    
    

    Then, it is important to set up the json to recognize your outputs.

    But please post your WDL and the errors that are generated when you run it.

  • lwestlaklwestlak Member
    Hi @AdelaideR , thanks for getting back to me. I've attached below the WDL, workflow log, and standard out. In the last few lines of the stdout I inspected the directory where I expect my output files to be and they appear to be there (although I'm not sure if this might be a symlink issue?) Thanks again!
  • AdelaideRAdelaideR Member admin

    Okay @lwestlak, I will take a look and get back to you soon.

  • AdelaideRAdelaideR Member admin

    @lwestlak

    I have not figured out about the outputs yet, but I did see that this error message popped up early in the standard out:

    Unauthorized to get docker hash lwestlak/tangent-repo:latest
    

    I tried to pull that docker, I don't have permissions. Is there a way to make the repo available?

    It looks like the cromwell execution did not have access to that docker, for starters.

  • lwestlaklwestlak Member
    Apologies I edited the permissions it should work now
  • AdelaideRAdelaideR Member admin

    I apologize if I was not clear, @lwestlak - that error appeared in your standard output, indicating that the docker may not have been pulled during the workflow by the cromwell engine.

    Please take a look at this discussion.

    It has links to relevant discussions about this permission issue.

Sign In or Register to comment.