How to set Kafka and Zookeeper 2

After you had configured the zookeeper, now you could set Kafka in the cluster machine

1 download Kafka

go to http://kafka.apache.org/downloads.html and download relevant version. Be aware, download the binary version!

wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz

2 unzip and install in a folder

tar -zxvf kafka_2.11-0.10.0.1.tgz
cd kafka_2.11-0.10.0.1

3 introduce the folder in Kafka

xx/bin this is for Kafka start / stop / status ...etc... xx/config this is where the Kafka server configuration file stored xx/libs this is the Kafka class/method library

4 configure the server.property file

create Kafka-logs folder first

broker.id=0 # must be unique listeners=PLAINTEXT://192.168.1.70:9092 # broker ip + port port=9092 log.dirs=kafka-logs zookeeper.connect=server70:2181,server71:2181,server72:2181 # where to listen the zookeeper zookeeper.connection.timeout.ms=1000000 num.network.threads=2
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/"kafka_folder"/kafka-logs
num.partitions=2 # how many partitions for one topic log.retention.hours=168 #how many hours data will be stored log.segment.bytes=536870912 # the size that each segments could be in a topic, if larger, then create new segment log.retention.check.interval.ms=60000
log.cleaner.enable=false
default.replication.factor = 1 # how many replications system will create for each partition

5 start Kafka

first start zookeeper and run in back end (don't stop the zookeeper when running the kafka)

bin/zookeeper-server-start.sh config/zookeeper.properties &

then start the Kafka

bin/kafka-server-start.sh config/server.properties &

6 stop Kafka

stop Kafka first and then stop zookeeper

bin/kafka-server-stop.sh

bin/zookeeper-server-stop.sh