LOG4J has been very popular logging library in Java world for years. LOG4J2 is even better. In Aug 2015 Log4j development team officially announced end of life for Log4j version 1.* and urged to start using Log4j2.
While upgrading from log4j to log4j2 is easy and can be done without changing existing code, real-life application have a lot dependencies which still come pre-configured to use old log4j and switching to new library is not always easy. Here we discuss how to start using log4j2 on Spark. Spark 2.1 is still using old slf4j library and though switching is possible, it is not documented and require manual steps on installed Spark system.
The following steps are required to switch Spark to use LOG4J2 logger:
1.Download log4j2 jars and unpack to locations accessible by Spark on each Spark node
wget http://apache.mirrors.lucidnetworks.net/logging/log4j/2.8.2/apache-log4j-2.8.2-bin.tar.gz
tar xzf apache-log4j-2.8.2-bin.tar.gz
cd apache-log4j-2.8.2-bin
mkdir -p /usr/local/spark/extra_jars
for f in log4j-1.2-api-2.8.2.jar log4j-api-2.8.2.jar log4j-api-scala_2.11-2.8.2.jar log4j-core-2.8.2.jar log4j-slf4j-impl-2.8.2.jar
do
cp $f /usr/local/spark/extra_jars/
done
2. Add new jars to SPARK_CLASSPATH variable using Spark configuration script
vi /usr/local/spark/conf/spark-env.sh
-----
SPARK_CLASSPATH=/usr/local/spark/extra_jars/log4j-1.2-api-2.8.2.jar
SPARK_CLASSPATH=/usr/local/spark/extra_jars/log4j-api-2.8.2.jar:$SPARK_CLASSPATH
SPARK_CLASSPATH=/usr/local/spark/extra_jars/log4j-api-scala_2.11-2.8.2.jar:$SPARK_CLASSPATH
SPARK_CLASSPATH=/usr/local/spark/extra_jars/log4j-core-2.8.2.jar:$SPARK_CLASSPATH
SPARK_CLASSPATH=/usr/local/spark/extra_jars/log4j-slf4j-impl-2.8.2.jar:$SPARK_CLASSPATH 3. Remove old log4j jars from standard Spark classpath
cd /usr/local/spark/jars
mv log4j-1.2.17.jar log4j-1.2.17.jar.bak
mv slf4j-log4j12-1.7.16.jar slf4j-log4j12-1.7.16.jar.bak
mv slf4j-log4j12-1.7.16.jar slf4j-log4j12-1.7.16.jar.bak
4. Restart Spark services.
That’s it! Now Spark uses logs using log4j2 library.
Leave a Reply