In a nutshell, Spark is a piece of software that GATK4 uses to do multithreading, which is a form of parallelization that allows a computer (or cluster of computers) to finish executing a task sooner. You can read more about multithreading and parallelism in GATK here. The Spark software library is open-source and maintained by the Apache Software Foundation. It is very widely used in the computing industry and is one of the most promising technologies for accelerating execution of analysis pipelines.
Not all GATK tools use Spark
Tools that can use Spark generally have a note to that effect in their respective Tool Doc.
- Some GATK tools exist in distinct Spark-capable and non-Spark-capable versions
The "sparkified" versions have the suffix "Spark" at the end of their names. Many of these are still experimental; down the road we plan to consolidate them so that there will be only one version per tool.
- Some GATK tools only exist in a Spark-capable version
Those tools don't have the "Spark" suffix.
You don't need a Spark cluster to run Spark-enabled GATK tools!
If you're working on a "normal" machine (even just a laptop) with multiple CPU cores, the GATK engine can still use Spark to create a virtual standalone cluster in place, and set it to take advantage of however many cores are available on the machine -- or however many you choose to allocate. See the example parameters below and the local-Spark tutorial for more information on how to control this. And if your machine only has a single core, these tools can always be run in single-core mode -- it'll just take longer for them to finish.
To be clear, even the Spark-only tools can be run on regular machines, though in practice a few of them may be prohibitively slow (SV tools and PathSeq). See the Tool Docs for tool-specific recommendations.
If you do have access to a Spark cluster, the Spark-enabled tools are going to be extra happy but you may need to provide some additional parameters to use them effectively. See the cluster-Spark tutorial for more information.
Example command-line parameters
Here are some example arguments you would give to a Spark-enabled GATK tool:
--sparkMaster local[*]-> "Run on the local machine using all cores"
--sparkMaster local-> "Run on the local machine using two cores"
--sparkMaster spark://18.104.22.168:7077-> "Run on the cluster at 22.214.171.124, port 7077"
--sparkRunner GCS --cluster my_cluster-> "Run on my_cluster in Google Dataproc"
You don't need to install any additional software to use Spark in GATK
All the necessary software for using Spark, whether it's on a local machine or a Spark cluster, is bundled within the GATK itself. Just make sure to invoke GATK using the
gatk wrapper script rather than calling the jar directly, because the wrapper will select the appropriate jar file (there are two!) and will set some parameters for you.