CommandLineFunction with List[File] as input

Hi,

I've been using Queue extensively recently, and it's great. I'm now trying to run a Qfunction that takes a list of input files, and uses them all as dependencies just as they would be if they were single inputs. Here's an example of what I want:

  case class reportWriter(inFiles:List[File], file:File, report:DataReport) extends CommandLineFunction {
    @Input(doc="files to rely on") val files:List[File] = inFiles 
    @Output(doc="report file to write") val reportFile:File = file
    def commandLine = "reportWriter " + inFiles + ">" + reportFile
    this.isIntermediate = false
    this.jobName = "writeReport"
    this.analysisName = this.jobName
  }

I the call populate a List[File] parameters in my pipeline, and pass it as an argument like so:

  add( GeneralUtils.writeReport(fileList, reportFile, report) )

However, the files in my List[File] are not treated as dependencies and the job is run the first thing that happens in the pipeline.

Is there a way to use a List[File] as input to a CommandLineFunction, and have it use all elements of the List as dependencies?

Best Answer

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie

    Hmm, I don't think I've seen this usage in any of our code. I believe the system treats the list file itself as the dependency, not its contents, so if you provide the list file up front, the dependency is satisfied from the get-go. One hacky solution would be to have an extra command-line function that creates a copy of the list file once the previous job is done, and use the copy's name as input. Then the dependency is only satisfied once the previous jobs have run and the new list file is created.

  • Sorry, I mean that List[File] is an array of Files, scala style. I dont have a file with a list of files, but rather an array of input files in the qscript itself.

    Something like "this is my final job, that should create a report of all patients in the pipeline".

    Daniel

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie

    Oh, got it, thanks for clarifying. I think that should be feasible but I'll have to ask @kshakir‌ to confirm.

  • Thanks, I'm now using a List[File] which works great.

Sign In or Register to comment.