Attention:
The frontline support team will be unavailable to answer questions until May27th 2019. We will be back soon after. Thank you for your patience and we apologize for any inconvenience!

Issues running test workflow locally using docker

Hello,

I am new to cromwell and trying to learn how to run a simple test workflow locally before I try it in Google cloud. However, not having much luck running it locally when using docker. I have docker installed and the service running on a local vm. Here is what my wdl file looks like:

--
workflow jes_workflow {
call jes_task
}
task jes_task {
command {
echo "Hello JES!" > out.txt
}
runtime {
docker: "ubuntu:latest"
continueOnReturnCode: true
}
output {
File out = "out.txt"
}

}

Here is the error I am getting:

--

return exit code

exit $rc
[2018-08-09 18:44:36,85] [info] BackgroundConfigAsyncJobExecutionActor [bdd46509jes_workflow.jes_task:NA:1]: job id: 21344
[2018-08-09 18:44:36,85] [info] BackgroundConfigAsyncJobExecutionActor [bdd46509jes_workflow.jes_task:NA:1]: Status change from - to WaitingForReturnCodeFile
[2018-08-09 18:44:38,49] [info] BackgroundConfigAsyncJobExecutionActor [bdd46509jes_workflow.jes_task:NA:1]: Status change from WaitingForReturnCodeFile to Done
[2018-08-09 18:44:39,05] [error] WorkflowManagerActor Workflow bdd46509-3a46-4f7e-abe7-ecd53f745cd1 failed (during ExecutingWorkflowState): java.io.FileNotFoundException: Could not process output, file not found: /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/out.txt
at cromwell.backend.sfs.SharedFileSystem.mapJobWomFile(SharedFileSystem.scala:149)
at cromwell.backend.sfs.SharedFileSystem.mapJobWomFile$(SharedFileSystem.scala:145)
at cromwell.backend.sfs.SharedFileSystemJobCachingActorHelper$$anon$1.mapJobWomFile(SharedFileSystemJobCachingActorHelper.scala:13)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.mapOutputWomFile(SharedFileSystemAsyncJobExecutionActor.scala:237)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.mapOutputWomFile$(SharedFileSystemAsyncJobExecutionActor.scala:236)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.mapOutputWomFile(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$outputValueMapper$1(StandardAsyncExecutionActor.scala:759)
at wom.WomFileMapper$.$anonfun$mapWomFiles$1(WomFileMapper.scala:25)
at scala.util.Try$.apply(Try.scala:209)
at wom.WomFileMapper$.mapWomFiles(WomFileMapper.scala:25)
at cromwell.backend.standard.StandardAsyncExecutionActor.outputValueMapper(StandardAsyncExecutionActor.scala:759)
at cromwell.backend.standard.StandardAsyncExecutionActor.outputValueMapper$(StandardAsyncExecutionActor.scala:758)
at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.outputValueMapper(ConfigAsyncJobExecutionActor.scala:193)
at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$evaluateOutputs$1(StandardAsyncExecutionActor.scala:781)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$5(OutputEvaluator.scala:66)
at cats.data.EitherT.$anonfun$flatMap$1(EitherT.scala:80)
at scala.util.Success.flatMap(Try.scala:247)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:35)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:14)
at cats.data.EitherT.flatMap(EitherT.scala:78)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$4(OutputEvaluator.scala:65)
at cats.data.EitherT.$anonfun$flatMap$1(EitherT.scala:80)
at scala.util.Success.flatMap(Try.scala:247)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:35)
at cats.instances.TryInstances$$anon$1.flatMap(try.scala:14)
at cats.data.EitherT.flatMap(EitherT.scala:78)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$1(OutputEvaluator.scala:64)
at scala.util.Success.flatMap(Try.scala:247)
at cromwell.backend.OutputEvaluator$.foldFunction$1(OutputEvaluator.scala:35)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$13(OutputEvaluator.scala:79)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:122)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:118)
at scala.collection.immutable.List.foldLeft(List.scala:86)
at cromwell.backend.OutputEvaluator$.fromOutputPorts$1(OutputEvaluator.scala:79)
at cromwell.backend.OutputEvaluator$.$anonfun$evaluateOutputs$17(OutputEvaluator.scala:115)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2018-08-09 18:44:39,06] [info] WorkflowManagerActor WorkflowActor-bdd46509-3a46-4f7e-abe7-ecd53f745cd1 is in a terminal state: WorkflowFailedState

[2018-08-09 18:44:43,18] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.

Any guidance would be appreciated! Thanks in advance.

Best Answer

Answers

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    Hi @jagga13 !

    This looks like a pretty standard workflow to me - maybe to help debug it, could you let us know -

    • What Cromwell version you're using
    • What your backend config looks like (I think there shouldn't be here, but always double check to remove any sensitive info, if there is any)

    Thanks!

  • Thanks Chris for the response. I am using cromwell-34.jar and I did not explicitly define a backend since I thought it would use the localhost by default. That might be the missing piece? Just did this:

    --
    $ cat jes_task.wdl
    workflow jes_workflow {
    call jes_task
    }
    task jes_task {
    command {
    echo "Hello JES!" > out.txt
    }
    runtime {
    docker: "ubuntu:latest"
    continueOnReturnCode: true
    }
    }
    output {
    File out = "out.txt"
    }

    $ java -jar cromwell-34.jar run jes_task.wdl

    -J

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @jagga13,

    I can recreate your error if my task simply never produces "out.txt". Something is preventing the command from running properly. A few questions...

    • Are you running Cromwell from inside a docker container?
    • From where you ran Cromwell, can you try running: ls -R /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/
    • Can you also examine the contents of your stdout/stderr files?
      cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout
      and
      cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr

    Thanks!

  • jagga13jagga13 Member
    edited August 2018

    Thanks Ruchi for your response. I am running cromwell inside a openstack instance that has docker installed. My assumption is that a docker container would be spun up to run the command that I have provided via the wdl file. Let me know if that is not correct.

    Here is what I see:

    $ ls -R /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/
    /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/:
    execution

    /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution:
    docker_cid rc script script.background script.submit stderr.background stdout.background

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout
    cat: /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout: No such file or directory

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stdout.background
    21344
    b495329d3cec736b73ff6a83badb3c31d9c9427943b13a6e2d7bd542e50bfe39

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr
    cat: /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr: No such file or directory

    $ cat /root/cromwell/cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/stderr.background
    /bin/bash: /cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/script: Permission denied

    Thanks again for your assistance with this! Its much appreciated.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @jagga13,

    I believe this /bin/bash: /cromwell-executions/jes_workflow/bdd46509-3a46-4f7e-abe7-ecd53f745cd1/call-jes_task/execution/script: Permission denied means the command script never ran. Cromwell requires root access, is it possible to grant it that?

  • I am running this workflow as root though from within the openstack instance. I was initially doing it as a normal user but switched to root when I saw that message. However, getting that error even with root.

  • jagga13jagga13 Member
    edited August 2018

    Ok, so I tried to just do this on google cloud and enabled the Google Genomics API in my project. Based on the documentation here is what I did but getting a network related error message:

    $ cat hello.wdl
    task hello {
    String addressee
    command {
    echo "Hello ${addressee}! Welcome to Cromwell . . . on Google Cloud!"
    }
    output {
    String message = read_string(stdout())
    }
    runtime {
    docker: "ubuntu:latest"
    }
    }

    workflow wf_hello {
    call hello

    output {
    hello.message
    }
    }

    $ cat hello.inputs
    {
    "wf_hello.hello.addressee": "World"
    }

    created google.conf with the appropriate information for my project/gcs_bucket and ran the workflow with:

    $ java -Dconfig.file=google.conf -jar cromwell-34.jar run hello.wdl -i hello.inputs

    However, getting this error:

    [2018-08-11 01:45:13,57] [info] PipelinesApiAsyncBackendJobExecutionActor [8f55bf4dwf_hello.hello:NA:1]: Status change from - to Failed
    [2018-08-11 01:45:14,51] [error] WorkflowManagerActor Workflow 8f55bf4d-9389-40e6-a469-dbd434394dd8 failed (during ExecutingWorkflowState): java.lang.Exception: Task wf_hello.hello:NA:1 failed. The job was stopped before the command finished. PAPI error code 3. Invalid value for field 'resource.networkInterfaces[0].network': 'https://www.googleapis.com/compute/v1/projects/gred-sb-001-7304802c/global/networks/default'. The referenced network resource cannot be found.

    I followed the documentation on http://cromwell.readthedocs.io/en/stable/search.html?q=docker. Is there a step I am missing here?

    Thanks,
    -J

  • I don't see a config in the google backend where I can set a network/subnet name. Do I have to use a vpc with the exact name default? Is this not configurable?

    Thanks,
    -J

  • jagga13jagga13 Member
    Accepted Answer

    Moving discussion to another thread since the initial post might not be doable and now testing with GCP instead.

Sign In or Register to comment.