Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Localization via hard link has failed

EADGEADG KielMember ✭✭✭

Hello,

i got a general problem with the localization of the workflow-files, cromwell allways copy the files instead of creating a hardlink, which consumes at least a lot of diskspace. During the workflow i got the following warn-message:
2013-10-27 19:32:14,238 WARN - Localization via hard link has failed: /data/cromwell_runs/2017/April/cromwell-executions/PairedEndSingleSampleWorkflow/d82e4595-d587-4e70-81e8-2d019685b179/call-BaseRecalibratorNormal/shard-212/inputs/data/cromwell_runs/2017/April/cromwell-executions/PairedEndSingleSampleWorkflow/d82e4595-d587-4e70-81e8-2d019685b179/call-SortAndFixTagsNormal/execution/SampleXYZ_S2.lane1.sorted.bai -> /data/cromwell_runs/2017/April/cromwell-executions/PairedEndSingleSampleWorkflow/d82e4595-d587-4e70-81e8-2d019685b179/call-SortAndFixTagsNormal/execution/SampleXYZ_S2.lane1.sorted.bai: Die Operation ist nicht erlaubt [Operation not permitted]

Do you now how to fix this ?

I am running a local pipeline on a Debian 8 Server and observe this issue with cromewell 21/24/25.

Thank you in advance!

Greetings EADG

Tagged:

Best Answer

  • EADGEADG Kiel ✭✭✭
    Accepted Answer

    Hi @ChrisL,

    i solved the Problem with the cromwell-26 and by adding the ngs-user to the docker-container and add

    runtime-attributes = """
            String? docker
            String? docker_user
            """
            submit = "/bin/bash ${script}"
            submit-docker = """docker run \ --rm -i \ --user docker_user \ -v ${cwd}:${docker_cwd} \ ${docker} \ /bin/bash ${script} """
    

    to my application.conf and the additional runtime-attribute:

    docker_user: "ngs"
    

    to my wdl.file.

    Greetings EADG

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev admin

    Hi, @EADG!

    This looks like a permissions issue between the user/system that the Cromwell service is running on, and the file system that it's using. In particular I'd double check whether Cromwell has to access this directory as a remote or mounted FS that doesn't allow hard links.

    You could also try to create a similar link yourself, e.g. log into the Cromwell server and run ln src dest with the src being the original file and the dest being somewhere similar to what Cromwell is trying to write. This link might also be able to help debug any permissions issues: https://unix.stackexchange.com/questions/233275/hard-link-creation-permissions

    Let me know if any of that helps!

  • EADGEADG KielMember ✭✭✭

    Hi @ChrisL thank for your tipp,

    i tryed to create a hardlink from anywhere in the cromwell-execution folder but fail. I take a look at the rights for the outputfiles and see that the rc-file and the desired outputfile are owned by root. While the execution folder and all other files are owned by the user. The user/other only has read permissions. I think this is the reason why the location via hardlink fails...All Folders/files are created by the cromwell-engine. Have i done something wrong, which cause that behavior ?

    I know that all programms inside the docker-container are running with root-rights, maybe that the reseaon why the output-files are owned by root..

  • EADGEADG KielMember ✭✭✭

    Hi @ChrisL,

    I found this in the reference.conf :

            runtime-attributes = """
            String? docker
            String? docker_user
            """
            submit = "/bin/bash ${script}"
            submit-docker = """
            docker run \
              --rm -i \
              ${"--user " + docker_user} \
              -v ${cwd}:${docker_cwd} \
              ${docker} \
              /bin/bash ${script}
              """
    

    I think this snippet will force cromwell to run the docker Container as the specific user and not as root. Unfortunately cromwell (24/25) ignore that command (i use java -Dconfig.file=application.conf) do you know why ?

    Here my complete application.conf:

    akka {
      loggers = ["akka.event.slf4j.Slf4jLogger"]
      logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
    }
    system {
      max-concurrent-workflows = 2
      new-workflow-poll-rate = 2
      max-workflow-launch-count = 1
      abort-jobs-on-terminate=true
    }
    spray.can {
      server {
        request-timeout = 40s
      }
      client {
        request-timeout = 40s
        connecting-timeout = 40s
      }
    backend {
      default = "Local"
      providers {
        Local {
          config {
            concurrent-job-limit = 40
            run-in-background = true
            runtime-attributes = """
            String? docker
            String? docker_user
            """
            submit = "/bin/bash ${script}"
            submit-docker = """
            docker run \
              --rm -i \
              --user ngs \
              -v ${cwd}:${docker_cwd} \
              ${docker} \
              /bin/bash ${script}
            """
            filesystems = {
              local {
                localization: [
                 "soft-link","hard-link","copy"
                ]
              }
            }
          }
        }
      }
    }
    }
    

    Maybe you can help...

    Thanks in advance!
    EADG

  • ChrisLChrisL Cambridge, MAMember, Broadie, Dev admin

    Cromwell should log the "start run" command that it's using, so you should be able to double check that it's respecting your config file.

    Bear in mind also that the user you're specifying there is the docker user, not the host user. Unless the user-IDs happen to match in both, the user "xyz" in your docker run won't be the same as the user "xyz" in your host system.

  • EADGEADG KielMember ✭✭✭

    Hi @ChrisL,

    sorry for my late response, I was on a business trip last week. I do a double check, first i change some arguments in my application.conf file to non-valid values, like concurrent-job-limit = $ and started. Cromwell throw a wrong value error-message and terminated. So I know that cromwell respecting my config file. Then I changed the value back to valid one and start cromwell again. The warn message persist...As I checked the script.submit, I see that Cromwell use the standard command to run the docker-container.

    Is there an additional command what I should use in the workflow.wdl to invoke docker with a non-root user ? Is there a way to change submit-docker = "docker run --rm -v ${cwd}:${docker_cwd} -i ${docker} /bin/bash < ${script}" to
    submit-docker = "docker run --rm --user docker_user -v ${cwd}:${docker_cwd} -i ${docker} /bin/bash < ${script}" ?

    Maybe I am thinking about this the wrong way...?

    Greetings EADG

  • EADGEADG KielMember ✭✭✭
    Accepted Answer

    Hi @ChrisL,

    i solved the Problem with the cromwell-26 and by adding the ngs-user to the docker-container and add

    runtime-attributes = """
            String? docker
            String? docker_user
            """
            submit = "/bin/bash ${script}"
            submit-docker = """docker run \ --rm -i \ --user docker_user \ -v ${cwd}:${docker_cwd} \ ${docker} \ /bin/bash ${script} """
    

    to my application.conf and the additional runtime-attribute:

    docker_user: "ngs"
    

    to my wdl.file.

    Greetings EADG

Sign In or Register to comment.