Spark Application with Kafka Consumer and Hbase
1. Maven Project pom.xml. Import all required dependencies, and package all dependencies, resources into a single jar. This is the best option else you will […]
Month: August 2018
Kafka, Spark, Hbase, Hadoop, Java
Step 1: Install and Setup Hadoop Cluster Follow the link here Step 2: Install and Setup Hbase Follow the link here Step 3: Install and […]
Log4j2 and Spark 2.3.1
LOG4J has been very popular logging library in Java world for years. LOG4J2 is even better. In Aug 2015 Log4j development team officially announced end of life for […]
Log4J 2 XML Configuration and Usage
To include Log4j2 in your project, include below dependency in your project pom.xml: <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <version>2.6.1</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>2.6.1</version> </dependency> Log4j2.xml for […]
Hbase and Zookeeper Znode
The HBase root znode path is configurable using hbase-site.xml, and by default the location is “/hbase”. All the znodes referenced below will be prefixed using […]
Kafka Producer and Consumer Java Json Examples
Currently, there are 2 ways to write and read from kafka, via producer and consumer or kafka stream. Data are write once to kafka via […]
Eclipse and Maven Packaging Jar
1: add build instructions to pom.xml: <build> <plugins> <plugin> <!– Build an executable JAR –> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <mainClass>com.kafka.client.Run</mainClass> </manifest> </archive> </configuration> </plugin> […]
Kafka 2.12-2 installation with Hbase Zookeeper
1: Download the code and extract Download the 2.0.0 release and un-tar it. cd /usr/local tar -xzf kafka_2.12-2.0.0.tgz ln -s kafka_2.12-2.0.0 kafka cd kafka mkdir […]
Hbase Commands
1. Connect to HBase. $ hbase shell hbase(main):001:0> Display HBase Shell Help Text. Type help and press Enter, to display some basic usage information for […]
Install Spark 2.3.x on YARN with Hadoop 3.x
This is the preferred method for Spark Installation, as Yarn is started as ResourceManager in hadoop, this reduces the transaction time required when installing Spark […]