How to resolve log4j version clash on SPARK

Hi,
by default SPARK 2 has no support for log4j2. It uses Log4j 1.2.17. When I use GATK 4.0.0 build "as-is" (depends on Log4j2) all jobs on SPARK fail due to version clash. How this is supposed to be solved?

As a workaround I was able:

  • relocate Log4j to different package
  • make sure it is included in the {{-spark}} jar (it is excluded by default)

But I believe there must be better and cleaner way of running current GATK 4.0.0 on SPARK.

Thanks!

Issue · Github
by Sheila

Issue Number
2887
State
open
Last Updated

Answers

  • Hi,

    and thanks for the comment. Since I am interested in running GATK-4.0.0.0 with Spark too,
    could you please give us a few more details on the workaround you've done?

    I would like to run the GATK pipeline in combination with Spark-2.1.0.
    For the current version of GATK, do we need Log4j 1.2.17? what do you mean by relocating to different package?

    Thanks in advance for you answer!

    Best,
    Giuseppe

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @zykajan @SergioBenvenuti
    Hi,

    I asked someone from the team to get back to you.

    -Sheila

Sign In or Register to comment.