There has been work done with HaplotypeCallerSpark so that it now matches the output of regular HaplotypeCaller. You can use the parameter --strict to invoke this behavior but please note that there are still modifications being made to improve performance as listed in this ticket.
Hello @wlai - Sorry for the delayed response. It seems that this error has been seen due to insufficient memory allocated for the process. Would you be able to re-try by increasing memory? Here is a forum discussion with the same error and the user's solution listed as well for reference?
The GATK 4 release promotes the use of Sparktools as the primary method of threading. Each tool that is supported by Spark contains the suffix "Spark". HaplotypeCallerSpark for example, is the Spark version of HaplotypeCaller in GATK 4.0. You can use this tool if you wish to perform threading with HaplotypeCaller.