Issues running test workflow locally using docker

Hello,

I am new to cromwell and trying to learn how to run a simple test workflow locally before I try it in Google cloud. However, not having much luck running it locally when using docker. I have docker installed and the service running on a local vm. Here is what my wdl file looks like:

--
workflow jes_workflow {
call jes_task
}
task jes_task {
command {
echo "Hello JES!" > out.txt
}
runtime {
docker: "ubuntu:latest"
continueOnReturnCode: true
}
output {
File out = "out.txt"
}

}

Here is the error I am getting:

--

return exit code

exit $rc
[2018-08-09 18:44:36,85] [info] BackgroundConfigAsyncJobExecutionActor [bdd46509jes_workflow.jes_task:NA:1]: job id: 21344
[2018-08-09 18:44:36,85] [info] BackgroundConfigAsyncJobExecutionActor [bdd46509jes_workflow.jes_task:NA:1]: Status change from - to WaitingForReturnCodeFile
[2018-08-09 18:44:38,49] [info] BackgroundConfigAsyncJobExecutionActor [bdd46509jes_workflow.jes_task:NA:1]: Status change from WaitingForReturnCodeFile to Done
[2018-08-09 18:44:39,05] [error] WorkflowManagerActor Workflow bdd46509-3a46-4f7e-abe7-ecd53f745cd1 failed (during ExecutingWorkflowState): java.io.FileNotFoundException: Could not process output, file not found: /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/out.txt
at cromwell.backend.sfs.SharedFileSystem.mapJobWomFile(SharedFileSystem.scala:149)
at cromwell.backend.sfs.SharedFileSystem.mapJobWomFile$(SharedFileSystem.scala:145)
at cromwell.backend.sfs.SharedFileSystemJobCachingActorHelper$$anon$1.mapJobWomFile(SharedFileSystemJobCachingActorHelper.scala:13)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.mapOutputWomFile(SharedFileSystemAsyncJobExecutionActor.scala:237)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.mapOutputWomFile$(SharedFileSystemAsyncJobExecutionActor.scala:236)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.mapOutputWomFile(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$outputValueMapper$1(StandardAsyncExecutionActor.scala:759)
at wom.WomFileMapper$.$anonfun$mapWomFiles$1(WomFileMapper.scala:25)
at scala.util.Try$.apply(Try.scala:209)
at wom.WomFileMapper$.mapWomFiles(WomFileMapper.scala:25)
at cromwell.backend.standard.StandardAsyncExecutionActor.outputValueMapper(StandardAsyncExecutionActor.scala:759)
at cromwell.backend.standard.StandardAsyncExecutionActor.outputValueMapper$(StandardAsyncExecutionActor.scala:758)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.outputValueMapper(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$evaluateOutputs$1(StandardAsyncExecutionActor.scala:781)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$5(OutputEvaluator.scala:66)
at cats.data.EitherT.$anonfun$flatMap$1(EitherT.scala:80)
at scala.util.Success.flatMap(Try.scala:247)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:35)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:14)
at cats.data.EitherT.flatMap(EitherT.scala:78)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$4(OutputEvaluator.scala:65)
at cats.data.EitherT.$anonfun$flatMap$1(EitherT.scala:80)
at scala.util.Success.flatMap(Try.scala:247)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:35)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:14)
at cats.data.EitherT.flatMap(EitherT.scala:78)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$1(OutputEvaluator.scala:64)
at scala.util.Success.flatMap(Try.scala:247)
at cromwell.backend.OutputEvaluator$.foldFunction$1(OutputEvaluator.scala:35)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$13(OutputEvaluator.scala:79)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:122)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:118)
at scala.collection.immutable.List.foldLeft(List.scala:86)
at cromwell.backend.OutputEvaluator$.fromOutputPorts$1(OutputEvaluator.scala:79)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$17(OutputEvaluator.scala:115)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2018-08-09 18:44:39,06] [info] WorkflowManagerActor WorkflowActor-bdd46509-3a46-4f7e-abe7-ecd53f745cd1 is in a terminal state: WorkflowFailedState

[2018-08-09 18:44:43,18] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.

Any guidance would be appreciated! Thanks in advance.

Best Answer

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    Hi @jagga13 !

    This looks like a pretty standard workflow to me - maybe to help debug it, could you let us know -

    • What Cromwell version you're using
    • What your backend config looks like (I think there shouldn't be here, but always double check to remove any sensitive info, if there is any)

    Thanks!

  • jagga13jagga13 Member

    Thanks Chris for the response. I am using cromwell-34.jar and I did not explicitly define a backend since I thought it would use the localhost by default. That might be the missing piece? Just did this:

    --
    $ cat jes_task.wdl
    workflow jes_workflow {
    call jes_task
    }
    task jes_task {
    command {
    echo "Hello JES!" > out.txt
    }
    runtime {
    docker: "ubuntu:latest"
    continueOnReturnCode: true
    }
    }
    output {
    File out = "out.txt"
    }

    $ java -jar cromwell-34.jar run jes_task.wdl

    -J

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @jagga13,

    I can recreate your error if my task simply never produces "out.txt". Something is preventing the command from running properly. A few questions...

    • Are you running Cromwell from inside a docker container?
    • From where you ran Cromwell, can you try running: ls -R /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/
    • Can you also examine the contents of your stdout/stderr files?
      cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout
      and
      cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr

    Thanks!

  • jagga13jagga13 Member
    edited August 10

    Thanks Ruchi for your response. I am running cromwell inside a openstack instance that has docker installed. My assumption is that a docker container would be spun up to run the command that I have provided via the wdl file. Let me know if that is not correct.

    Here is what I see:

    $ ls -R /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/
    /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/:
    execution

    /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution:
    docker_cid rc script script.background script.submit stderr.background stdout.background

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout
    cat: /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout: No such file or directory

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout.background
    21344
    b495329d3cec736b73ff6a83badb3c31d9c9427943b13a6e2d7bd542e50bfe39

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr
    cat: /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr: No such file or directory

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr.background
    /bin/bash: /cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/script: Permission denied

    Thanks again for your assistance with this! Its much appreciated.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @jagga13,

    I believe this /bin/bash: /cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/script: Permission denied means the command script never ran. Cromwell requires root access, is it possible to grant it that?

  • I am running this workflow as root though from within the openstack instance. I was initially doing it as a normal user but switched to root when I saw that message. However, getting that error even with root.

  • jagga13jagga13 Member
    edited August 11

    Ok, so I tried to just do this on google cloud and enabled the Google Genomics API in my project. Based on the documentation here is what I did but getting a network related error message:

    $ cat hello.wdl
    task hello {
    String addressee
    command {
    echo "Hello ${addressee}! Welcome to Cromwell . . . on Google Cloud!"
    }
    output {
    String message = read_string(stdout())
    }
    runtime {
    docker: "ubuntu:latest"
    }
    }

    workflow wf_hello {
    call hello

    output {
    hello.message
    }
    }

    $ cat hello.inputs
    {
    "wf_hello.hello.addressee": "World"
    }

    created google.conf with the appropriate information for my project/gcs_bucket and ran the workflow with:

    $ java -Dconfig.file=google.conf -jar cromwell-34.jar run hello.wdl -i hello.inputs

    However, getting this error:

    [2018-08-11 01:45:13,57] [info] PipelinesApiAsyncBackendJobExecutionActor [8f55bf4dwf_hello.hello:NA:1]: Status change from - to Failed
    [2018-08-11 01:45:14,51] [error] WorkflowManagerActor Workflow 8f55bf4d-9389-40e6-a469-dbd434394dd8 failed (during ExecutingWorkflowState): java.lang.Exception: Task wf_hello.hello:NA:1 failed. The job was stopped before the command finished. PAPI error code 3. Invalid value for field 'resource.networkInterfaces[0].network': 'https://www.googleapis.com/compute/v1/projects/gred-sb-001-7304802c/global/networks/default'. The referenced network resource cannot be found.

    I followed the documentation on http://cromwell.readthedocs.io/en/stable/search.html?q=docker. Is there a step I am missing here?

    Thanks,
    -J

  • I don't see a config in the google backend where I can set a network/subnet name. Do I have to use a vpc with the exact name default? Is this not configurable?

    Thanks,
    -J

  • jagga13jagga13 Member
    Accepted Answer

    Moving discussion to another thread since the initial post might not be doable and now testing with GCP instead.

Sign In or Register to comment.