To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

Problem with running gatk haplotypecaller in cromwell with spark backend

Hi, trying to run gatk in cromwell with spark backend on amazon emr
Backend is configured as in documentation: cromwell.readthedocs.io/en/develop/backends/Spark/
Sample wdl is from here: https://software.broadinstitute.org/wdl/documentation/article?id=7159
Tested versions of cromwell: 30, 30.1
Tested versions of gatk: 3.8, 4.beta1, 4.beta5(for version 4, key -T removed, -o replaced with -O)
When I try to run it with default config(with local backend) it running successfully, but when I try to override backend with spark(as in documentation) it fails with error:
[error] WorkflowManagerActor Workflow 2438bdd9-ee47-4511-bf73-a2a34b275685 failed (during ExecutingWorkflowState): Execution process failed. Spark returned non zero status code: 2 java.lang.IllegalStateException: Execution process failed. Spark returned non zero status code: 2 at cromwell.backend.impl.spark.SparkJobExecutionActor.resolveExecutionResult(SparkJobExecutionActor.scala:110) at cromwell.backend.impl.spark.SparkJobExecutionActor.executeTask(SparkJobExecutionActor.scala:92) at cromwell.backend.impl.spark.SparkJobExecutionActor.$anonfun$taskLauncher$1(SparkJobExecutionActor.scala:210) at scala.util.Try$.apply(Try.scala:209) at cromwell.backend.impl.spark.SparkJobExecutionActor.taskLauncher(SparkJobExecutionActor.scala:208) at cromwell.backend.impl.spark.SparkJobExecutionActor.execute(SparkJobExecutionActor.scala:84) at cromwell.backend.BackendJobExecutionActor$$anonfun$receive$1.$anonfun$applyOrElse$1(BackendJobExecutionActor.scala:57) at cromwell.backend.BackendLifecycleActor.performActionThenRespond(BackendLifecycleActor.scala:44) at cromwell.backend.BackendLifecycleActor.performActionThenRespond$(BackendLifecycleActor.scala:40) at cromwell.backend.impl.spark.SparkJobExecutionActor.performActionThenRespond(SparkJobExecutionActor.scala:30) at cromwell.backend.BackendJobExecutionActor$$anonfun$receive$1.applyOrElse(BackendJobExecutionActor.scala:57) at akka.actor.Actor.aroundReceive(Actor.scala:514) at akka.actor.Actor.aroundReceive$(Actor.scala:512) at cromwell.backend.impl.spark.SparkJobExecutionActor.aroundReceive(SparkJobExecutionActor.scala:30) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) at akka.actor.ActorCell.invoke(ActorCell.scala:496) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I checked script in ../execution
#!/bin/sh cd /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/execution spark-submit --master local --total-executor-cores 1 --deploy-mode client --class org.broadinstitute.hellbender.Main --executor-memory 1gb InstantiatedCommand(java -jar /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/inputs/sample/inputs/gatk-package-4.beta.5-local.jar \ HaplotypeCaller \ -R /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/inputs/sample/inputs/human_g1k_b37_20.fasta \ -I /sample/cromwell-executions/helloHaplotypeCaller/74df28a0-8781-42f5-ab99-2c6a3f646cba/call-haplotypeCaller/inputs/sample/inputs/NA12878_wgs_20.bam \ -O NA12878.raw.indels.snps.vcf,List()) echo $? > rc
and seems InstantiatedCommand(,List()) should not be in bash script.
When I remove it from script(and java -jar) and run it manually, it runs successfully on spark.
Same error if I remove java -jar in wdl.

Answers

  • henderj8henderj8 Cincinnati Children's Hospital Member

    Hi Sergejs,

    We are running into the same issue while attempting to set up Cromwell with a Spark backend running through our on premise spark cluster. Were you able to figure out a solution to the "InstantiatedCommand" problem? We can not seem to figure out where it is coming from/why it is present in the script. Any information on this would be helpful.

    Thank you in advance!

Sign In or Register to comment.