Spark Application with Kafka Consumer and Hbase
1. Maven Project pom.xml. Import all required dependencies, and package all dependencies, resources into a single jar. This is the best option else you will […]
Java
Kafka, Spark, Hbase, Hadoop, Java
Step 1: Install and Setup Hadoop Cluster Follow the link here Step 2: Install and Setup Hbase Follow the link here Step 3: Install and […]
Log4j2 and Spark 2.3.1
LOG4J has been very popular logging library in Java world for years. LOG4J2 is even better. In Aug 2015 Log4j development team officially announced end of life for […]
Log4J 2 XML Configuration and Usage
To include Log4j2 in your project, include below dependency in your project pom.xml: <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <version>2.6.1</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>2.6.1</version> </dependency> Log4j2.xml for […]
Kafka Producer and Consumer Java Json Examples
Currently, there are 2 ways to write and read from kafka, via producer and consumer or kafka stream. Data are write once to kafka via […]
Install Spark 2.3.x on YARN with Hadoop 3.x
This is the preferred method for Spark Installation, as Yarn is started as ResourceManager in hadoop, this reduces the transaction time required when installing Spark […]