Getting Started on Kafka (Windows)

1. Start up zookeeper server

Open a command prompt in C:\kafka and enter this command:

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Zookeeper is running on Port 2181

2. Start the Kafka server

Open a new command prompt and start the Kafka server:

.\bin\windows\kafka-server-start.bat .\config\server.properties

3. Create a topic

Open a new command prompt in C:\kafka\bin\windows:

kafka-topics.bat --create --bootstrap-server localhost:9092 --topic mytopic

4. Create a Producer Console

kafka-console-producer.sh --broker-list <BROKER_LIST> --topic <TOPIC_NAME>

Open another new command prompt in C:\kafka\bin\windows:

kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic

You should see a > prompt when the producer is ready.

5. Create a Consumer Console

kafka-console-consumer.sh --bootstrap-server <BROKER_LIST> --topic <TOPIC_NAME>

Open another new command prompt in C:\kafka\bin\windows:

kafka-console-consumer.bat --topic mytopic --bootstrap-server localhost:9092 --from-beginning

6. Paste Data to Producer and Observe Consumer

Sample data for use:

{"Name":"Kenneth","Age":"35","Gender":"Male"}
{"Name":"Gemma","Age":"25","Gender":"Female"}
{"Name":"Amy","Age":"15","Gender":"Male"}

Referencing the image below, once you paste a row of data into the producer terminal (left), the data appears on the consumer terminal (right) almost instantaneously.

In Git Bash/ Linux

Similar commands collated from kafka quickstart

# Download and extract
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0

# Start the ZooKeeper service
bin/zookeeper-server-start.sh config/zookeeper.properties
# Start and hide logs on terminal
bin/zookeeper-server-start.sh config/zookeeper.properties > /dev/null 2>&1 &
# verify that Zookeeper is indeed running
ps -aef

# Start the Kafka broker service in another terminal
bin/kafka-server-start.sh config/server.properties

# Create a topic in another terminal
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

# Show details of a topic
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic: quickstart-events        TopicId: z_zqwwkiT6ueXVJxi3EAdA PartitionCount: 1       ReplicationFactor: 1     Configs:
        Topic: quickstart-events        Partition: 0    Leader: 0       Replicas: 0     Isr: 0

# Run Producer Console in another terminal
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092

# Run Consumer Console in another terminal
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

# Write event to topic at Producer console
This is my first message
This is my second message

Demonstrating 3 Brokers Replica

# Create 2 other brokers
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
ll config/serv*

Update these 3 areas in server-1 and server-2 file:

# SERVER-1
# id of the broker
broker.id=1
# broker port
listeners=PLAINTEXT://:9093
# log files directories 
log.dirs=/tmp/kafka-logs-1

# SERVER-2
# id of the broker
broker.id=2
# broker port
listeners=PLAINTEXT://:9094
# log files directories 
log.dirs=/tmp/kafka-logs-2

Now start all 3 brokers

bin/kafka-server-start.sh config/server.properties
bin/kafka-server-start.sh config/server-1.properties
bin/kafka-server-start.sh config/server-2.properties

Now Create a topic with 2 partitions and 3 replication factor across 3 brokers

bin/kafka-topics.sh --create --topic TestTopicXYZ --partitions 2 --replication-factor 3 --bootstrap-server localhost:9092, localhost:9093, localhost:9094

Lastly describe the topic to examine:

bin/kafka-topics.sh --describe --topic TestTopicXYZ --bootstrap-server localhost:9092,localhost:9093, localhost:9094

Topic: TestTopicXYZ     TopicId: 65thxHOeTIGSJJ5L_jl-XQ PartitionCount: 2       ReplicationFactor: 3    Configs:
        Topic: TestTopicXYZ     Partition: 0    Leader: 0       Replicas: 0,2,1 Isr: 0,2,1
        Topic: TestTopicXYZ     Partition: 1    Leader: 2       Replicas: 2,1,0 Isr: 2,1,0

What is Kafka?

Publish/Subscribe systems

With Publish/Subscribe systems, each event can be processed by multiple consumers who are listening to the topic. So multiple consumers can all get the same messages from the producer.

Pub/Sub systems introduce the concept of a topic. We can have multiple topics (to categorize your messages) and multiple producers writing to each topic and multiple consumers reading from a topic.

Distributed
Kafka is like a messaging system. Kafka Cluster is made up of more than 1 Kafka servers. Each Kafka server is referred to as a Broker. A Kafka cluster will have multiple brokers, therefore it is distributed application.
Fault-Tolerant
In Kafka cluster, messages are replicated in multiple brokers. A message published in Broker 1 will also be published in Broker 3. Therefore Kafka is fault-tolerant as message is replicated and not lost even if Broker 1 is down.
Language Agnostic
Data transferred (called Messages) are byte arrays so you can use JSON or a data format like Avro. Messages are immutable and have a timestamp, a value and optional key/headers

Kafka Architecture

Every Kafka ecosystem can have multiple producers/publishers.
A Kafka system consists of several components:

Producer - Any service that pushes messages to Kafka is a producer.
Consumer - Any service that consumes (pulls) messages from Kafka is a consumer. There can be multiple Consumer groups in the Kafka ecosystem. Consumers must be associated in a Consumer groups.
Kafka Brokers - the kafka cluster is composed of a network of nodes called brokers. Each machine/instance/container running a Kafka process is called a broker.

In a Kafka cluster, the individual servers are referred to as brokers. Brokers will handle:

Receiving Messages - accepting messages from producers
Storing Messages - storing them on disk organized by topic. Each message has a unique offset identifying it.
Serving Messages - sending messages to consumer services when the consumer requests it

Topics - A Kafka server has multiple topics. Every topics can have multiple partitions. Messages in Kafka are stored in topics, which are the core structure for organizing and storing your messages. You can think of a topic as a log file that stores all the messages and maintains their order. New message are appended to the end of the topic.