Problem with running gatk haplotypecaller in cromwell with spark backend

Hi, trying to run gatk in cromwell with spark backend on amazon emr
Backend is configured as in documentation: cromwell.readthedocs.io/en/develop/backends/Spark/
Sample wdl is from here: https://software.broadinstitute.org/wdl/documentation/article?id=7159
Tested versions of cromwell: 30, 30.1
Tested versions of gatk: 3.8, 4.beta1, 4.beta5(for version 4, key -T removed, -o replaced with -O)
When I try to run it with default config(with local backend) it running successfully, but when I try to override backend with spark(as in documentation) it fails with error:
[error] WorkflowManagerActor Workflow 2438bdd9-ee47-4511-bf73-a2a34b275685 failed (during ExecutingWorkflowState): Execution process failed. Spark returned non zero status code: 2 java.lang.IllegalStateException: Execution process failed. Spark returned non zero status code: 2 at cromwell.backend.impl.spark.SparkJobExecutionActor.resolveExecutionResult(SparkJobExecutionActor.scala:110) at cromwell.backend.impl.spark.SparkJobExecutionActor.executeTask(SparkJobExecutionActor.scala:92) at cromwell.backend.impl.spark.SparkJobExecutionActor.$anonfun$taskLauncher$1(SparkJobExecutionActor.scala:210) at scala.util.Try$.apply(Try.scala:209) at cromwell.backend.impl.spark.SparkJobExecutionActor.taskLauncher(SparkJobExecutionActor.scala:208) at cromwell.backend.impl.spark.SparkJobExecutionActor.execute(SparkJobExecutionActor.scala:84) at cromwell.backend.BackendJobExecutionActor$$anonfun$receive$1.$anonfun$applyOrElse$1(BackendJobExecutionActor.scala:57) at cromwell.backend.BackendLifecycleActor.performActionThenRespond(BackendLifecycleActor.scala:44) at cromwell.backend.BackendLifecycleActor.performActionThenRespond$(BackendLifecycleActor.scala:40) at cromwell.backend.impl.spark.SparkJobExecutionActor.performActionThenRespond(SparkJobExecutionActor.scala:30) at cromwell.backend.BackendJobExecutionActor$$anonfun$receive$1.applyOrElse(BackendJobExecutionActor.scala:57) at akka.actor.Actor.aroundReceive(Actor.scala:514) at akka.actor.Actor.aroundReceive$(Actor.scala:512) at cromwell.backend.impl.spark.SparkJobExecutionActor.aroundReceive(SparkJobExecutionActor.scala:30) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) at akka.actor.ActorCell.invoke(ActorCell.scala:496) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I checked script in ../execution
#!/bin/sh cd /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/execution spark-submit --master local --total-executor-cores 1 --deploy-mode client --class org.broadinstitute.hellbender.Main --executor-memory 1gb InstantiatedCommand(java -jar /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/inputs/sample/inputs/gatk-package-4.beta.5-local.jar \ HaplotypeCaller \ -R /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/inputs/sample/inputs/human_g1k_b37_20.fasta \ -I /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/inputs/sample/inputs/NA12878_wgs_20.bam \ -O NA12878.raw.indels.snps.vcf,List()) echo $? > rc
and seems InstantiatedCommand(,List()) should not be in bash script.
When I remove it from script(and java -jar) and run it manually, it runs successfully on spark.
Same error if I remove java -jar in wdl.

Answers

  • henderj8henderj8 Cincinnati Children's Hospital Member

    Hi Sergejs,

    We are running into the same issue while attempting to set up Cromwell with a Spark backend running through our on premise spark cluster. Were you able to figure out a solution to the "InstantiatedCommand" problem? We can not seem to figure out where it is coming from/why it is present in the script. Any information on this would be helpful.

    Thank you in advance!

Sign In or Register to comment.