[warn] Localization via hard link has failed Invalid cross-device link

ppanteliadppanteliad BostonMember, Broadie

Ηello, I'm trying to run GATK best practices and I've run in to a wall when the SamToFastqAndBwaMem is executed.

First things first: in the .inputs.json I specified my .fa.alt via absolute path and I get this warning: [warn] Localization via hard link has failed:

More specifically the SamToFastqAndBwaMem task, always exits with 1, meaning that I get this WorkflowManagerActor Workflow 8992b121-6c74-4ab8-aba3-07ecd748ef04 failed (during ExecutingWorkflowState): Job PairedEndSingleSampleWorkflow.SamToFastqAndBwaMem:0:1 exited with return code 1 which has not been declared as a valid return code. I suspect this error comes from the else-statement

# else ref_alt is empty or could not be found else exit 1;
Is this true? Has anyone else encountered this error?
My version of cromwel is 0.25


  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie

    Hi there, can you clarify what you mean by "GATK best practices"? What WDL/method are you running?

  • ppanteliadppanteliad BostonMember, Broadie

    @Geraldine_VdAuwera said:
    Hi there, can you clarify what you mean by "GATK best practices"? What WDL/method are you running?

    PublicPairedSingleSampleWf_160927.wdl and it's accompanying files from broad. GATK is for variant discovery analysis. And broad generated these files so we can run it. If you scroll down the .wdl (line) to line 87 you will see the line, I think is causing this error

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie
    Ah, I'm quite familiar with that script, my team publishes it :)

    Are you running it locally then? Could you post the JSON inputs file? It looks like the program isn't finding the inputs.
  • ppanteliadppanteliad BostonMember, Broadie

    @Geraldine_VdAuwera Sorry for the late response.
    But isn't it weird?

      "PairedEndSingleSampleWorkflow.sample_name": "NA12878",
      "PairedEndSingleSampleWorkflow.flowcell_unmapped_bams": [
      "PairedEndSingleSampleWorkflow.final_gvcf_name": "NA12878.g.vcf.gz",
      "PairedEndSingleSampleWorkflow.unmapped_bam_suffix": ".bam",
      "##_COMMENT2": "INTERVALS",
       "PairedEndSingleSampleWorkflow.scattered_calling_intervals" : [],
      "PairedEndSingleSampleWorkflow.wgs_calling_interval_list": "/path/to/.interval_list",
      "PairedEndSingleSampleWorkflow.HaplotypeCaller.contamination": 0,
      "PairedEndSingleSampleWorkflow.ref_dict": "/path/to/.dict",
      "PairedEndSingleSampleWorkflow.ref_fasta": "/path/to/.fa",
      "PairedEndSingleSampleWorkflow.ref_fasta_index": "/path/to/.fa.fai",
      "PairedEndSingleSampleWorkflow.ref_alt": "/path/to/.fa.alt",
      "PairedEndSingleSampleWorkflow.ref_sa": "/path/to/.fa.sa",
      "PairedEndSingleSampleWorkflow.ref_amb": "/path/to/.fa",
      "PairedEndSingleSampleWorkflow.ref_bwt": "/path/to/.fa.bwt",
      "PairedEndSingleSampleWorkflow.ref_ann": "/path/to/.fa.ann",
      "PairedEndSingleSampleWorkflow.ref_pac": "/path/to/.fa",
      "PairedEndSingleSampleWorkflow.dbSNP_vcf": "/path/to/.vcf",
      "PairedEndSingleSampleWorkflow.dbSNP_vcf_index": "/path/to/.vcf.idx",
      "PairedEndSingleSampleWorkflow.known_indels_sites_VCFs": [
      "PairedEndSingleSampleWorkflow.known_indels_sites_indices": [
      "PairedEndSingleSampleWorkflow.agg_small_disk": 200,
      "PairedEndSingleSampleWorkflow.agg_medium_disk": 300,
      "PairedEndSingleSampleWorkflow.agg_large_disk": 400,
      "PairedEndSingleSampleWorkflow.agg_preemptible_tries": 3,
      "PairedEndSingleSampleWorkflow.flowcell_small_disk": 200,
      "PairedEndSingleSampleWorkflow.flowcell_medium_disk": 300,
      "PairedEndSingleSampleWorkflow.preemptible_tries": 3

    I can't post the exact files we are using.
    But this is the format in which we are referencing the needed files.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie
    The if-else statement is just a check to make sure we're working with the right reference. Technically it shouldn't be needed if the inputs are set up correctly; I'm not sure why the pipelines team (who authors these) felt it necessary. But in any case i don't think you're even reaching that command; if the program is failing to localize files that usually means it can't access them in the first place.

    I understand you can't provide the exact files, but can you tell me what platform you're running on and what is the type of location your files are in?
  • ppanteliadppanteliad BostonMember, Broadie

    the platform is lsf and it's clustered connected NAS storage

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie
    Ah, ok. I don't have direct experience using Cromwell in that environment. Have you tested an analogous toy script to make sure that file localization works correctly in your setup?
  • ppanteliadppanteliad BostonMember, Broadie

    I've used simple bash scripts. And they worked fine with absolute paths on that specific system.

  • esalinasesalinas BroadMember, Broadie

    @Geraldine_VdAuwera I wonder if when docker is run by cromwell that it might mount (using "-v") and that the filesystem mounting/mountpoints (double mount? or mount not on a boot disk, but on some already mounted file system) could be causing an issue? what do you think?

  • jsotojsoto Broad InstituteMember, Broadie, Dev

    @ppanteliad Have you been able to run any jobs through cromwell using the lsf backend and did those jobs do any kind of file localization? If there is a base case that works then we can try to piece together what your next steps should be.

    We also have office hours every week on Wednesday(Today) from 2-3 in 75A-04-Yosemite-4001 where we could answer your questions.

  • Was this question ever answered as I am having similar localization problems on a LSF server while using Cromwell

  • Does Cromwell 27 have some sort of softlink option, running locally on single node?

  • KateNKateN Cambridge, MAMember, Broadie, Moderator

    I'm not familiar with LSF, as we specialize here on FireCloud. I will move your question to the WDL forum where I believe they will be able to answer your question.

  • mcovarrmcovarr Cambridge, MAMember, Broadie, Dev

    Hi @tphillip, for a non-Docker config Cromwell defaults to trying these localization strategies in this order:

            filesystems {
              local {
                localization: [
                  "hard-link", "soft-link", "copy"

    The failure of "hard-link" is to be expected if input files reside on a different filesystem than the Cromwell execution directory. This is not a concern (it's a warning and not an error), Cromwell should fall back to using "soft-link" instead.

Sign In or Register to comment.