Hi GATK Users,

Happy Thanksgiving!
Our staff will be observing the holiday and will be unavailable from 22nd to 25th November. This will cause a delay in reaching out to you and answering your questions immediately. Rest assured we will get back to it on Monday November 26th. We are grateful for your support and patience.
Have a great holiday everyone!!!

Regards
GATK Staff

How to resolve log4j version clash on SPARK

Hi,
by default SPARK 2 has no support for log4j2. It uses Log4j 1.2.17. When I use GATK 4.0.0 build "as-is" (depends on Log4j2) all jobs on SPARK fail due to version clash. How this is supposed to be solved?

As a workaround I was able:

  • relocate Log4j to different package
  • make sure it is included in the {{-spark}} jar (it is excluded by default)

But I believe there must be better and cleaner way of running current GATK 4.0.0 on SPARK.

Thanks!

Issue · Github
by Sheila

Issue Number
2887
State
open
Last Updated

Answers

  • Hi,

    and thanks for the comment. Since I am interested in running GATK-4.0.0.0 with Spark too,
    could you please give us a few more details on the workaround you've done?

    I would like to run the GATK pipeline in combination with Spark-2.1.0.
    For the current version of GATK, do we need Log4j 1.2.17? what do you mean by relocating to different package?

    Thanks in advance for you answer!

    Best,
    Giuseppe

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @zykajan @SergioBenvenuti
    Hi,

    I asked someone from the team to get back to you.

    -Sheila

Sign In or Register to comment.