To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

How to resolve log4j version clash on SPARK

Hi,
by default SPARK 2 has no support for log4j2. It uses Log4j 1.2.17. When I use GATK 4.0.0 build "as-is" (depends on Log4j2) all jobs on SPARK fail due to version clash. How this is supposed to be solved?

As a workaround I was able:

  • relocate Log4j to different package
  • make sure it is included in the {{-spark}} jar (it is excluded by default)

But I believe there must be better and cleaner way of running current GATK 4.0.0 on SPARK.

Thanks!

Issue · Github
by Sheila

Issue Number
2887
State
open
Last Updated

Answers

  • Hi,

    and thanks for the comment. Since I am interested in running GATK-4.0.0.0 with Spark too,
    could you please give us a few more details on the workaround you've done?

    I would like to run the GATK pipeline in combination with Spark-2.1.0.
    For the current version of GATK, do we need Log4j 1.2.17? what do you mean by relocating to different package?

    Thanks in advance for you answer!

    Best,
    Giuseppe

  • SheilaSheila Broad InstituteMember, Broadie, Moderator

    @zykajan @SergioBenvenuti
    Hi,

    I asked someone from the team to get back to you.

    -Sheila

Sign In or Register to comment.