On Monday and Tuesday, November 12-13, the communications team will be out of the office for a U.S. federal holiday and a team event. We will be back in action on November 14th and apologize for any inconvenience this may cause. Thank you for using the forum.

Cromwell workflow options: Output Copying

I'm trying to get cromwell to copy output files to another directory for further automatic processing.

I've set final_workflow_outputs_dir, but I'm confused about the behavior I see. I was expecting cromwell to copy the files to the specified folder, but instead it just creates a new file structure there, which again includes the name of the workflow, the workflow UUID, and the call-* folders. This isn't really an improvement over the normal cromwell-executions folder.

Is there another way to just copy the output files from the workflow into a certain folder, without recreating the cromwell folder structure? I'm only interested in seeing the latest results in the output folder, and I don't want to search all UUID's to find the latest output files.

Also, I noticed that the output from sub-workflows is not copied over at all, even though it has been specified as part of the outer workflow's output, and the files do show up in the final json output. Is this a bug?

Best Answer

Answers

  • dayzcooldayzcool Member

    Hi,
    I need your help on relevant issue. I used cromwell 33.1 and cromwell is running in server mode.
    I have trouble in using final_workflow_outputs_dir.
    I have confirmed that your test setup works for me, but the error below occurs during FinalizingWorkflowState if used for GATK4's germline CNV wdl.

    Workflows run successfully. However, in the final stage, it is looking for a directory that does not exist and fails. /cromwell-working-dir in the log below is the working directory of cromwell process.

    Thank you!

    2018-07-19 11:11:42,170 cromwell-system-akka.dispatchers.engine-dispatcher-105 ERROR - WorkflowManagerActor Workflow e9c09014-09bb-4061-8c72-858113835cd8 failed (during FinalizingWorkflowState): cromwell.core.CromwellFatalException: java.nio.file.NoSuchFileException: /cromwell-working-dir/sample-515
            at cromwell.core.CromwellFatalException$.apply(core.scala:18)
            at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:38)
            at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
            at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
            at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
            at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
            at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
            at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
            at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
            at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
            at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
            at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
            at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
            at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
            at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
            at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
            at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
    Caused by: java.nio.file.NoSuchFileException: /cromwell-working-dir/sample-515
            at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
            at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
            at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
            at java.nio.file.Files.copy(Files.java:1274)
            at better.files.File.copyTo(File.scala:663)
            at cromwell.core.path.BetterFileMethods.copyTo(BetterFileMethods.scala:425)
            at cromwell.core.path.BetterFileMethods.copyTo$(BetterFileMethods.scala:424)
            at cromwell.core.path.DefaultPath.copyTo(DefaultPathBuilder.scala:55)
            at cromwell.engine.io.nio.NioFlow.$anonfun$copy$1(NioFlow.scala:65)
            at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
            at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)
            at scala.util.Success.$anonfun$map$1(Try.scala:251)
            at scala.util.Success.map(Try.scala:209)
            at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
            at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
            at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
            ... 12 more
    
    2018-07-19 11:11:42,172 cromwell-system-akka.dispatchers.engine-dispatcher-122 INFO  - WorkflowManagerActor WorkflowActor-e9c09014-09bb-4061-8c72-858113835cd8 is in a terminal state: WorkflowFailedState
    
Sign In or Register to comment.