Spark Backend With Cromwell 0.29

@jgentry with current cromwell version. If following wdl and input are used with Spark backend i see an error stating runtime key not found.
`

task sparkjob_with_yarn_cluster {
File input_jar
String output_base
String entry_point
Int cores
String memory

    command {
            ${input_jar} ${output_base}
    }

    runtime {
            appMainClass: "${entry_point}"
            executorMemory: "${memory}"
            executorCores: "${cores}"
    }
    output {
            File out = "${output_base}"
      }

}

workflow sparkBETest {
call sparkjob_with_yarn_cluster
}

and Input

<br />{
"sparkBETest.sparkjob_with_yarn_cluster.memory": "4G",
"sparkBETest.sparkjob_with_yarn_cluster.output_base":"/home/himanshu/Documents/test/sparkOutput",
"sparkBETest.sparkjob_with_yarn_cluster.entry_point": "SimpleApp",
"sparkBETest.sparkjob_with_yarn_cluster.deployMode": "client",
"sparkBETest.sparkjob_with_yarn_cluster.cores": "12",
"sparkBETest.sparkjob_with_yarn_cluster.input_jar": "/home/himanshu/Documents/test/sparktest_2.11-1.0.jar"
}

Error: 

Runtime attributes evaluation:
key not found: entry_point
key not found: memory
key not found: cores
akka.actor.ActorInitializationException: akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-d7f7f13a-5690-4b69-8f14-3acd94f5ad82/WorkflowExecutionActor-d7f7f13a-5690-4b69-8f14-3acd94f5ad82/d7f7f13a-5690-4b69-8f14-3acd94f5ad82-EngineJobExecutionActor-sparkBETest.sparkjob_with_yarn_cluster:NA:1/d7f7f13a-5690-4b69-8f14-3acd94f5ad82-BackendJobExecutionActor-d7f7f13a:sparkBETest.sparkjob_with_yarn_cluster:-1:1: exception during creation

Best Answer

Answers

Sign In or Register to comment.