Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Failure during localization

amr@broadinstitute.orge[email protected] Member, Broadie

I've got a cromwell server up and running and I tried out my first WDL. I get the error below. I've pasted my WDL and JSON below the error:

2016-06-27 08:01:07,766 cromwell-system-akka.actor.default-dispatcher-9 ERROR - Failures during localization
java.lang.UnsupportedOperationException: Could not localize /cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/scripts/version0.2/SampleData/Mouse-A2-single_S2_L001_R1_001.fastq  -> /cil/shed/apps/internal/cromwell/cromwell-executions/smartseq_amr/21612af1-2c5f-400a-a53f-6ac66ec47674/call-RunSTARAlignment/cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/scripts/version0.2/SampleData/Mouse-A2-single_S2_L001_R1_001.fastq 
        at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$$anonfun$localize$1$3.apply(SharedFileSystem.scala:243) ~[cromwell-0.19.jar:0.19]
        at scala.Option.getOrElse(Option.scala:121) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$class.localize$1(SharedFileSystem.scala:242) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$class.adjustFile$1(SharedFileSystem.scala:264) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$class.localizeWdlValue(SharedFileSystem.scala:271) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.sge.SgeBackend.localizeWdlValue(SgeBackend.scala:60) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$$anonfun$16.apply(SharedFileSystem.scala:223) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:225) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$$anonfun$17.apply(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19]
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[cromwell-0.19.jar:0.19]
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[cromwell-0.19.jar:0.19]
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) ~[cromwell-0.19.jar:0.19]
        at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.local.SharedFileSystem$class.adjustSharedInputPaths(SharedFileSystem.scala:224) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.sge.SgeBackend.adjustSharedInputPaths(SgeBackend.scala:60) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.sge.SgeBackend.adjustInputPaths(SgeBackend.scala:67) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.sge.SgeBackend.instantiateCommand(SgeBackend.scala:260) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand$lzycompute(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.BackendCallJobDescriptor.instantiateCommand(JobDescriptor.scala:52) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.sge.SgeBackend$$anonfun$execute$1.apply(SgeBackend.scala:84) ~[cromwell-0.19.jar:0.19]
        at cromwell.engine.backend.sge.SgeBackend$$anonfun$execute$1.apply(SgeBackend.scala:82) ~[cromwell-0.19.jar:0.19]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[cromwell-0.19.jar:0.19]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[cromwell-0.19.jar:0.19]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) ~[cromwell-0.19.jar:0.19]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [cromwell-0.19.jar:0.19]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [cromwell-0.19.jar:0.19]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [cromwell-0.19.jar:0.19]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [cromwell-0.19.jar:0.19]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [cromwell-0.19.jar:0.19]

WDL:

task RunSTARAlignment {
  File fastq1
  File fastq2
  String sample_name
  String genome_dir
  String output_dir
  String pipeline_path
  command {
    sh ${pipeline_path}/RunSTARAlignment.sh ${fastq1} ${fastq2} ${sample_name} ${genome_dir} ${output_dir}
  }
  output {
    File genome_alignment = "${output_dir}/alignments/${sample_name}.Aligned.sortedByCoord.out.bam"
    File trans_alignment =  "${output_dir}/alignments/${sample_name}.Aligned.Aligned.toTranscriptome.out.bam"
  }
  runtime {
    docker: "cowmoo/cil-dev:latest"
    memory: "4G"
    cpu: "3"
    zones: "us-central1-c us-central1-a"
    disks: "/mnt/mnt1 3 SSD, /mnt/mnt2 500 HDD"
  }
}

workflow smartseq_amr {
  call RunSTARAlignment
}

JSON:

{
        "smartseq_amr.RunSTARAlignment.fastq1": "/cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/scripts/version0.2/SampleData/Mouse-A2-single_S2_L001_R1_001.fastq ",
        "smartseq_amr.RunSTARAlignment.fastq2": "/cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/scripts/version0.2/SampleData/Mouse-A2-single_S2_L001_R2_001.fastq",
        "smartseq_amr.RunSTARAlignment.sample_name": "Mouse-A2-single",
        "smartseq_amr.RunSTARAlignment.genome_dir": "/cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/Resources/Mouse10",
        "smartseq_amr.RunSTARAlignment.output_dir": "/cil/shed/sandbox/amr/smartseq/test",
        "smartseq_amr.RunSTARAlignment.pipeline_path": "/cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/scripts/version0.2/"
}

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Thank you for reposting your question here; there was a bit of confusion about where to ask, as I understand.

    My first question is: are your file paths in your JSON absolute paths? I'm not familiar with your file structure, but on my computer using the local backend, file paths begin with:

    /Users/knoblett/Documents/...
    
  • amr@broadinstitute.orge[email protected] Member, Broadie

    Hello Kate, thank you for responding. Yes, my file paths are absolute paths.

  • dshihdshih BostonMember, Broadie
    edited July 2016

    What flags did you start cromwell with? I also encountered localization failure when I only allowed cromwell to soft link or hard link. This is a bug in release 0.19, persisted in 0.19_hotfix, and was fixed in the developmental branch (currently a.k.a 0.20).

    I would also double-check that the path in question does in fact point to a file:

    ls -l /cil/shed/apps/internal/RNA_utilities/SmartSeq_StarBasedPipeline/scripts/version0.2/SampleData/Mouse-A2-single_S2_L001_R1_001.fastq
    
  • amr@broadinstitute.orge[email protected] Member, Broadie

    I ran Cromwell as follows:

    java -Dconfig.file=engine/src/main/resources/application.conf -jar target/scala-2.11/cromwell-0.19.jar server

    I don't see a way to list the flags with -h or --help.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    From the logs, it looks like you are using the SGE backend. Your WDL specifies a few runtime options which look to be JES-specific. Docker, for example, is a parameter that is ignored by SGE. Are you expecting the input file to become available from the docker image? If so, that would be the reason for your file localization error; the docker never becomes available.

Sign In or Register to comment.