How would you perform a Kafka operations from your local machine?
CodeWithVed
Posted on November 22, 2024
Let's say you have a cloud Kafka in your application (e.g., AWS MSK). To form the connection from your local machine without code, follow these instructions:
Download Apache Kafka from here.
Extract the downloaded zip file and navigate to the "bin" folder.
In the "bin" folder, you will find all the .sh files, e.g., kafka-topics.sh, kafka-console-consumer.sh.
Once you've completed the setup, there's one more important step:
- You need to create a client.properties file for AWS MSK. The configuration would mostly be like this:
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
Now we are done with the complete setup and ready to execute the first command:
### To create a Topic:
./kafka-topics.sh --create --bootstrap-server <bootstrap-server/s> --topic <topic-name> --command-config client.properties
Now you have created the topic, simply lets check the way to describe our created topic:
Describe a Topic:
./kafka-topics.sh --describe --bootstrap-server <bootstrap-server/s> --topic <topic> --command-config client.properties
see, its that easy to describe by specifying the --describe flag with the kafka-topics.sh command.
Let's move forward, by producing some message in existing topic:
Produce messages in Kafka topic:
./kafka-console-producer.sh --bootstrap-server <bootstrap-server/s> --producer.config client.properties --topic <topic-name>
Similarly lets check for consumer,
Consume messages from Kafka topic:
./kafka-console-consumer.sh --bootstrap-server <bootstrap-server/s> --consumer.config client.properties --topic <topic-name>
What if you wanted to list down all the existing topics then,
List All Topics in Kafka:
./kafka-topics.sh --bootstrap-server <bootstrap-server/s> --list --command-config client.properties
If you wanted to list down all the consumer groups then,
List All Consumer Groups in Kafka:
./kafka-consumer-groups.sh --bootstrap-server <bootstrap-server/s> --list --command-config client.properties
If you don't know what is consumer group and how it will help you then read this blog it will give you detail idea, and how you can use it in your application:
https://dev.to/codewithved/the-kafka-conundrum-choosing-between-consumer-groups-and-partitions-for-efficient-message-consumption-b8n
Posted on November 22, 2024
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.