1: Download the code and extract
Download the 2.0.0 release and un-tar it. cd /usr/local tar -xzf kafka_2.12-2.0.0.tgz ln -s kafka_2.12-2.0.0 kafka cd kafka mkdir logs chown -R hadoop:hadoop /usr/local/kafka_2.12-2.0.0
2. setup environment path
vi ~./bashrc export PATH=$PATH:/usr/local/kafka/bin:/usr/local/hadoop/bin:$SPARK_HOME/bin:$HBASE_HOME/bin:/etc/alternatives/java_sdk_1.8.0/bin
3. Edit configuration file:
vi config/server.properties broker.id=0 listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://yourdomain:9092 (make sure the domain name can be resolved) num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/usr/log/kafka/logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 group.initial.rebalance.delay.ms=0 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000
if you would like to create more broker, copy server.properties to server-1.properties, change the below: broker.id=1 listeners=PLAINTEXT://:9093 advertised.listeners=PLAINTEXT://yourdomain:9093 until server-Nth.properties
4. The below must be pointing to the zookeeper setup while configuring hbase:
zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 ensure Hbase is startup before startup kafka
if your kafka installation is not on the same server as Hbase, you will have to run an instance of zookeeper:
zookeeper-server-start.sh config/zookeeper.properties
5. edit kafka server stop script:
vi bin/kafka-server-stop.sh Change PIDS line to below: PIDS=$(ps ax | grep -i 'kafkaServer' | grep java | grep -v grep | awk '{print $1}')
6. Start/Stop kafka server
kafka-server-start.sh -daemon config/server.properties kafka-server-stop.sh for multiple broker: kafka-server-start.sh -daemon config/server-1.properties kafka-server-start.sh -daemon config/server-2.properties
7. Test Kafka
run the command below: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Created topic "test" bin/kafka-topics.sh --list --zookeeper localhost:2181 test
8. Send some messages
Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default, each line will be sent as a separate message.
Run the producer and then type a few messages into the console to send to the server.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test This is a message This is another message
Kafka also has a command line consumer that will dump out messages to standard output.
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning This is a message This is another message
Now create a new topic with a replication factor of three:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
Run the "describe topics" command:
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs: Topic: my-replicated-topic Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
9. Test High Availability
Let's kill Broker 1:
> ps aux | grep server-1.properties 7564 ttys002 0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java... > kill -9 7564
Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs: Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 1,2,0 Isr: 2,0 But the messages are still available for consumption even though the leader that took the writes originally is down: > bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic ...
Leave a Reply