Kafka 2.12-2 installation with Hbase Zookeeper

1: Download the code and extract

Download the 2.0.0 release and un-tar it.
cd /usr/local
tar -xzf kafka_2.12-2.0.0.tgz
ln -s kafka_2.12-2.0.0 kafka
cd kafka
mkdir logs
chown -R hadoop:hadoop /usr/local/kafka_2.12-2.0.0

2. setup environment path

vi ~./bashrc
export PATH=$PATH:/usr/local/kafka/bin:/usr/local/hadoop/bin:$SPARK_HOME/bin:$HBASE_HOME/bin:/etc/alternatives/java_sdk_1.8.0/bin

3. Edit configuration file:

vi config/server.properties
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://yourdomain:9092 (make sure the domain name can be resolved)
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/log/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
group.initial.rebalance.delay.ms=0

zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

if you would like to create more broker, copy server.properties to server-1.properties, change the below:
broker.id=1
listeners=PLAINTEXT://:9093
advertised.listeners=PLAINTEXT://yourdomain:9093 
until server-Nth.properties

4. The below must be pointing to the zookeeper setup while configuring hbase:

zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

ensure Hbase is startup before startup kafka

if your kafka installation is not on the same server as Hbase, you will have to run an instance of zookeeper:

zookeeper-server-start.sh config/zookeeper.properties

5. edit kafka server stop script:

vi bin/kafka-server-stop.sh
Change PIDS line to below:
PIDS=$(ps ax | grep -i 'kafkaServer' | grep java | grep -v grep | awk '{print $1}')

6. Start/Stop kafka server

kafka-server-start.sh -daemon config/server.properties
kafka-server-stop.sh

for multiple broker:
kafka-server-start.sh -daemon config/server-1.properties
kafka-server-start.sh -daemon config/server-2.properties

7. Test Kafka

run the command below:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test"

bin/kafka-topics.sh --list --zookeeper localhost:2181
test

8. Send some messages
Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default, each line will be sent as a separate message.

Run the producer and then type a few messages into the console to send to the server.

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message 

Kafka also has a command line consumer that will dump out messages to standard output.

> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message

Now create a new topic with a replication factor of three:

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic 

Run the "describe topics" command:

> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic   PartitionCount:1    ReplicationFactor:3 Configs:
    Topic: my-replicated-topic  Partition: 0    Leader: 1   Replicas: 1,2,0 Isr: 1,2,0

9. Test High Availability
Let's kill Broker 1:

> ps aux | grep server-1.properties
7564 ttys002    0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java...
> kill -9 7564

Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:

> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic   PartitionCount:1    ReplicationFactor:3 Configs:
    Topic: my-replicated-topic  Partition: 0    Leader: 2   Replicas: 1,2,0 Isr: 2,0
But the messages are still available for consumption even though the leader that took the writes originally is down:
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
...