Currently when we need to connect large information systems with different formats and volumes we will depend on tools that are in charge of being in the middle of our architecture, organizing and distributing all that information operating in high availability, this is where the integration options we have could fit very well Apache Kafka would fit our project very well.

What is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
https://kafka.apache.org

Get Apache Kafka

At the time of writing this post it is in version 2.6

https://kafka.apache.org/downloads

Kafka Architecture

• Brokers: In charge of containing the topics of the cluster
• Publishers: They are in charge of putting the messages on the topics
• Consumers: They extract the messages from the topics to process them
• Cluster: A set of Kafka nodes
• Topics: Flow of data that have a same context
• Partitions: The corresponding divisions of a topic
• Zookeeper: Orchest all nodes in the cluster

Zookeeper

To use Kafka we need to first start Zookeeper, in the installation folder there is the shell command in the folder bin. Which is:

$ bin/zookeeper-server-start.sh config/zookeeper.properties

Kafka

Now we start Kafka, we need to locate ourselves in the bin directory where the shell is located.

$ bin/kafka-server-start.sh config/server.properties

Creating the topic

To generate a topic we must execute:

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic my-topic 

List all topics

$ bin/kafka-topics.sh --list --zookeeper localhost:2181

Get information about the topic

$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic 

Delete a topic

$ bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic my-topic 

Creating a consumer

$ bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092

Creating a producer

$ bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092
> Mensaje 1
> Mensaje 2