Docker file mapping error?

JuliaJoungJuliaJoung Member, Broadie

Hi,

I'm using a Wdl that I've used for cellranger back in June successfully, but now I am getting a strange docker file mapping error. In the logs file, it used to say "Docker file /cromwell_root/exec.sh maps to host location /mnt/local-disk/exec.sh." but now it says "Docker file /cromwell_root/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
maps to host location /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz" or "Docker file /cromwell_root/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180726_samplesheet.csv maps to host location /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180726_samplesheet.csv" if I change the Docker. The files that it is randomly choosing to map are in the input.

Do you have any suggestions on what might be happening?

Best,
Julia

Best Answer

Answers

  • bshifawbshifaw admin Member, Broadie, Moderator admin

    Hi @JuliaJoung ,

    What does the error message say?
    What docker container are you using?
    Does this error only arise when using a different docker? Perhaps the latest docker image being used contain changes not compatible with commands in the task. Check by running the WDL with the original docker image.

    /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    fc-733ca028-212e-4aa6-9448-c1691fc9132f is the workspace you're working on.
    I'm uncertain about 180813_NB501148_0418_AHJLV2BGX7.tgz and 180726_samplesheet.csv.

  • JuliaJoungJuliaJoung Member, Broadie
    edited August 2018

    Hi,

    This is the full header of the error message:

    2018/08/19 19:34:46 I: Switching to status: pulling-image
    2018/08/19 19:34:46 I: Calling SetOperationStatus(pulling-image)
    2018/08/19 19:34:46 I: SetOperationStatus(pulling-image) succeeded
    2018/08/19 19:34:46 I: Writing new Docker configuration file
    2018/08/19 19:34:46 I: Pulling image "marcusczi/[email protected]:a1070845aa01633bfad810434fd93daf9d59c28a18d4ce28aca894232fca841a"
    2018/08/19 19:35:43 I: Pulled image "marcusczi/[email protected]:a1070845aa01633bfad810434fd93daf9d59c28a18d4ce28aca894232fca841a" successfully.
    2018/08/19 19:35:43 I: Switching to status: localizing-files
    2018/08/19 19:35:43 I: Calling SetOperationStatus(localizing-files)
    2018/08/19 19:35:43 I: SetOperationStatus(localizing-files) succeeded
    2018/08/19 19:35:43 I: Docker file /cromwell_root/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
     maps to host location /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    .
    2018/08/19 19:35:43 I: Running command: sudo gsutil -q -m cp gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
     /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    2018/08/19 19:35:44 E: command failed: CommandException: No URLs matched: gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    CommandException: 1 file/object could not be transferred.
     (exit status 1)
    2018/08/19 19:35:44 W: cp failed: gsutil -q -m cp gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
     /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    , command failed: CommandException: No URLs matched: gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    CommandException: 1 file/object could not be transferred.
    

    Previously, the log file looks like this:

    2018/02/20 21:40:31 I: Switching to status: pulling-image
    2018/02/20 21:40:31 I: Calling SetOperationStatus(pulling-image)
    2018/02/20 21:40:31 I: SetOperationStatus(pulling-image) succeeded
    2018/02/20 21:40:31 I: Writing new Docker configuration file
    2018/02/20 21:40:31 I: Pulling image "marcusczi/[email protected]:a1070845aa01633bfad810434fd93daf9d59c28a18d4ce28aca894232fca841a"
    2018/02/20 21:41:32 I: Pulled image "marcusczi/[email protected]:a1070845aa01633bfad810434fd93daf9d59c28a18d4ce28aca894232fca841a" successfully.
    2018/02/20 21:41:32 I: Switching to status: localizing-files
    2018/02/20 21:41:32 I: Calling SetOperationStatus(localizing-files)
    2018/02/20 21:41:32 I: SetOperationStatus(localizing-files) succeeded
    2018/02/20 21:41:32 I: Docker file /cromwell_root/exec.sh maps to host location /mnt/local-disk/exec.sh.
    2018/02/20 21:41:32 I: Running command: sudo gsutil -q -m cp gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/4c23c252-b0bd-42a1-8970-e590b700289c/cellranger/805a84b5-5dce-43d5-85d7-099ce46eae4b/call-cellranger_mkfastq/exec.sh /mnt/local-disk/exec.sh
    2018/02/20 21:41:34 I: Docker file /cromwell_root/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180210_NB501148_0300_AH7WCGBGX5.tgz maps to host location /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180210_NB501148_0300_AH7WCGBGX5.tgz.
    2018/02/20 21:41:34 I: Running command: sudo gsutil -q -m cp gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180210_NB501148_0300_AH7WCGBGX5.tgz /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180210_NB501148_0300_AH7WCGBGX5.tgz
    2018/02/20 21:46:29 I: Docker file /cromwell_root/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180211_NB501148_0301_AH7WK3BGX5.tgz maps to host location /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180211_NB501148_0301_AH7WK3BGX5.tgz.
    2018/02/20 21:46:29 I: Running command: sudo gsutil -q -m cp gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180211_NB501148_0301_AH7WK3BGX5.tgz /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180211_NB501148_0301_AH7WK3BGX5.tgz
    2018/02/20 21:48:59 I: Docker file /cromwell_root/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180201_samplesheet.csv maps to host location /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180201_samplesheet.csv.
    2018/02/20 21:48:59 I: Running command: sudo gsutil -q -m cp gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180201_samplesheet.csv /mnt/local-disk/fc-733ca028-212e-4aa6-9448-c1691fc9132f/180201_samplesheet.csv
    2018/02/20 21:49:00 I: Done copying files.
    

    It seems that the main difference is that it no longer finds the exec.sh file. I'm currently using "marcusczi/cellranger_clean" for the docker, which did not seem to have been updated within the last year. 180813_NB501148_0418_AHJLV2BGX7.tgz and 180726_samplesheet.csv are files for my input. When the original docker didn't work, I tried switching dockers, but that only changed the error slightly.

    Does this help explain the error? Let me know if anything else is unclear.

    Best,
    Julia

    Post edited by bshifaw on
  • bshifawbshifaw admin Member, Broadie, Moderator admin

    Hi @JuliaJoung

    Looks like FC is having a hard time finding you input file:

    CommandException: No URLs matched: gs://fc-733ca028-212e-4aa6-9448-c1691fc9132f/180813_NB501148_0418_AHJLV2BGX7.tgz
    

    Perhaps the path to the file is incorrect or the file in not present in the workspace bucket. Please confirm your input file 180813_NB501148_0418_AHJLV2BGX7.tgz is located in your workspace bucket.
    If it is indeed present please share your workspace with [email protected] and tell us the name of the workspace as well as any submission and workflow ids involved.

  • JuliaJoungJuliaJoung Member, Broadie

    Hi @bshifaw,

    Yes I checked that the path to the file is indeed correct and located in my workspace bucket. I just shared my workspace with [email protected] The name of the workspace is juliajoung-zhanglab/scRNAseq and the most recent one I ran was 2a2b12fe-6d56-472d-9978-0c0eaab4b5cb (different docker) and 14cf68a9-6458-4352-aa50-4b2ffcffb26b (original docker that worked previously).

    Best,
    Julia

  • KateNKateN admin Cambridge, MAMember, Broadie, Moderator admin

    Thank you for sharing your workspace @JuliaJoung. Taking a look now.

  • JuliaJoungJuliaJoung Member, Broadie

    Hi @bshifaw,

    Thanks that fixed it! It was a strange formatting error.

    Julia

Sign In or Register to comment.