Spark

edited June 26 in Dictionary

In a nutshell, Spark is a piece of software that GATK4 uses to do multithreading, which is a form of parallelization that allows a computer (or cluster of computers) to finish executing a task sooner. You can read more about multithreading and parallelism in GATK here. The Spark software library is open-source and maintained by the Apache Software Foundation. It is very widely used in the computing industry and is one of the most promising technologies for accelerating execution of analysis pipelines.


Not all GATK tools use Spark

Tools that can use Spark generally have a note to that effect in their respective Tool Doc.

- Some GATK tools exist in distinct Spark-capable and non-Spark-capable versions

The "sparkified" versions have the suffix "Spark" at the end of their names. Many of these are still experimental; down the road we plan to consolidate them so that there will be only one version per tool.

- Some GATK tools only exist in a Spark-capable version

Those tools don't have the "Spark" suffix.


You don't need a Spark cluster to run Spark-enabled GATK tools!

If you're working on a "normal" machine (even just a laptop) with multiple CPU cores, the GATK engine can still use Spark to create a virtual standalone cluster in place, and set it to take advantage of however many cores are available on the machine -- or however many you choose to allocate. See the example parameters below and the local-Spark tutorial for more information on how to control this. And if your machine only has a single core, these tools can always be run in single-core mode -- it'll just take longer for them to finish.

To be clear, even the Spark-only tools can be run on regular machines, though in practice a few of them may be prohibitively slow (SV tools and PathSeq). See the Tool Docs for tool-specific recommendations.

If you do have access to a Spark cluster, the Spark-enabled tools are going to be extra happy but you may need to provide some additional parameters to use them effectively. See the cluster-Spark tutorial for more information.

Example command-line parameters

Here are some example arguments you would give to a Spark-enabled GATK tool:

  • --spark-master local[*] -> "Run on the local machine using all cores"
  • --spark-master local[2] -> "Run on the local machine using two cores"
  • --spark-master spark://23.195.26.187:7077 -> "Run on the cluster at 23.195.26.187, port 7077"
  • --spark-runner GCS --cluster my_cluster -> "Run on my_cluster in Google Dataproc"

You don't need to install any additional software to use Spark in GATK

All the necessary software for using Spark, whether it's on a local machine or a Spark cluster, is bundled within the GATK itself. Just make sure to invoke GATK using the gatk wrapper script rather than calling the jar directly, because the wrapper will select the appropriate jar file (there are two!) and will set some parameters for you.

Post edited by Geraldine_VdAuwera on

Comments

  • SkyWarriorSkyWarrior TurkeyMember

    Hi,
    --sparkMaster needs to be replaced with --spark-master for GATK 4.0.0.0

  • how to run PathSeqPipelineSpark on the local machine using 8 cores?

  • DBPZDBPZ Member

    I'm not sure if something has changed after GATK-4.0.0.0 :

    1) spark is no longer included in the GATK package -- it required the "spark-submit" command in $PATH.

    2) the argument names have changed : now it is "--spark-runner SPARK --spark-master $SPARK_URL ".

    3) the "spark-submit" command failed because the string (the "-D" options to java) for the "extraJavaOptions" argument was not quoted. I wrote a wrapper to add quotes around these "-D" options.

    4) the "--spark-master" argument was given to the command (for example, SortSam) submitted to spark, causing an error. I removed it in the wrapper.

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @DBPZ
    Hi,

    Are you saying these changes happened after the first official release of GATK4 or after the beta releases? The team is actively developing the Spark tools and they are in beta, so some changes are to be expected.

    Thanks,
    Sheila

  • manolismanolis Member
    edited June 8

    Hi @Sheila,

    do you have any news about this tutorial (Spark - How To)?

    All the best

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @manolis
    Hi,

    I don't have any news. I think the team is holding off on producing documentation for Spark until the Spark tools are out of beta. Perhaps this Dictionary entry will help a bit.

    -Sheila

Sign In or Register to comment.