How to set Kafka and Zookeeper 1

Now you know how Kafka works and let's see how to set the Kafka with multiple brokers

1 Setting the Zookeepr

1 download and unzip zookeeper in any folder:

[root@ZookeeperA server]# tar –zxvf zookeeper-3.4.9.tar.gz

2 setup JDK

download Oracle JDK java 8 (not recommended JDK 9, it will have some conflict later) npm install the JDK
go to /etc/profile to set the JDK path environment

#Java Env

export JAVA_HOME=/usr/java/jdk-9.0.4

export PATH=$PATH:$JAVA_HOME/bin

If you could not find Java home path, use which java to find the path remember to source the profile to update the change

3 setup zookeeper environment in /etc/profile

export ZOOKEEPER_HOME=/usr/server/zookeeper-3.4.9

export PATH=$PATH:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf

Same, if you could not find the Zookeeper path, just change the path to where your zookeeper had been unzipped

[root@ZookeeperA /]# source /etc/profile

source the profile again to make sure the change updated

4 create data and log folder for zookeeper

[root@ZookeeperA zookeeper-3.4.9]# mkdir data

[root@ZookeeperA zookeeper-3.4.9]# mkdir log

5 change the hostname for each node machine within the system

Assume you had 3 machine in the system and their ips are:192.168.1.70/192.168.1.71/192.168.1.72 then change the /etc/hosts file on every machine

127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4

::1        localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.70 ZookeeperA

192.168.1.71 ZookeeperB

192.168.1.72 ZookeeperC

6 create the zoo.cfg file in each machine

In zookeeper folder zookeeper-2.4.9/conf/, you will find a file called zoo_sample.cfg and then copy it as zoo.cfg (don't use other name)

[root@ZookeeperA conf]# cp zoo_sample.cfg zoo.cfg

7 update the zoo.cgf file

change

dataDir=/usr/server/zookeeper-3.4.9/data

and

dataLogDir=/usr/server/zookeeper-3.4.9/log

and add below code

server.70=ZookeeperA:2888:3888

server.71=ZookeeperB:2888:3888

server.72=ZookeeperC:2888:3888

format of the above server is server.id=host:port:port

8 create myid file in each node cluster

[root@ZookeeperA data]# vi myid

go to zookeeper data fold which you created in step 4 and crate a myid file in the fold then add a number you had used in step 7, which is the server id in this example, it is 70, 71, 72

So go to 192.168.1.70 and type 70 into myid file 192.168.1.71 is 71 in myid 192.168.1.72 is 72 in myid

9 firewall setting

make sure nothing block for the port you had used in zookeeper

[root@ZookeeperA zookeeper]# firewall-cmd --permanent --zone=public --add-port=2888/tcp

[root@ZookeeperA zookeeper]# firewall-cmd --permanent --zone=public --add-port=3888/tcp

[root@ZookeeperA zookeeper]# firewall-cmd --permanent --zone=public --add-port=2181/tcp

10 start zookeeper in each cluster

[root@ZookeeperA bin]# zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /usr/server/zookeeper-3.4.9/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

if you could get below info by type jps

[root@ZookeeperA bin]# jps

2983 QuorumPeerMain

3128 Jps

that means all good for the cluster

you could check the status by using command

[root@ZookeeperB bin]# zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /usr/server/zookeeper-3.4.9/bin/../conf/zoo.cfg

Mode: follower

So the cluster you are playing is a follower now

Also if you had any errors in the starting, you could use below command to check:

[root@ZookeeperA bin]# zkServer.sh start-foreground

Next page is the kafka setting