TheLazy.Dev
Portfolio GitHub

Blog

springboot |kafka |java |stream

Connecting Kafka to Spring Boot: A Step-by-Step Guide Using KafkaTemplate

amailath

Anbumani

Posted on January 4, 2024

Connecting Kafka to Spring Boot: A Step-by-Step Guide Using KafkaTemplate

Hello, Tech enthusiasts! Today, we're going to learn how to connect to one of the most widely used distributed streaming platforms, Kafka, from Spring Boot. This will be a brief write-up as we'll be utilizing KafkaTemplate, which makes our lives easier. Let's dive in.

If you like to learn in action, below is the video version that goes over the implementation.

Overview

We will create a Rest endpoint that will be responsible for posting the message into the Kafka topic and the consumer who is also part of the same project listening to the topic will print the received message on the console.

Project Set-up

Let's create a spring boot application with the following dependencies.
Add "spring-boot-starter-web" for the rest endpoint creation.



<dependency>             
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
</dependency>
Enter fullscreen mode Exit fullscreen mode

Here comes the Kafka dependency will do the magic of producing & consuming from Kafka via KafkaTemplate.



<dependency>
        <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
Enter fullscreen mode Exit fullscreen mode

As the next step will add below configuration to establish the connection with Kafka broker. Spring boot will auto-configure the KafkaTemplate with these properties.

Make sure to update the server and group-id below.

spring:
kafka:
consumer:
bootstrap-servers:
- localhost:9092
group-id: ytube-group

auto-offset-reset: earliest

key-deserializer:

value-deserializer: 

producer:
  bootstrap-servers:
    - localhost:9092
Enter fullscreen mode Exit fullscreen mode 

### Consumer Configuration:

- bootstrap-servers: Specifies the Kafka server(s) that the consumer will connect to. In this case, it's set to a single server at localhost:9092.

- group-id: Identifies the consumer group to which this consumer belongs. It helps Kafka keep track of the progress of different consumer groups.

- auto-offset-reset: (Commented out) This property determines what happens when there is no initial offset or the current offset does not exist. The earliest option means it will start reading from the beginning of the topic.

- key-deserializer and value-deserializer: (Commented out) These properties define the deserializer classes for the key and value of the messages. Depending on your use case, you might need to specify appropriate deserializer classes.

### Producer Configuration:

- bootstrap-servers: Similar to the consumer, this property specifies the Kafka server(s) that the producer will connect to. In this case, it's also set to a single server at localhost:9092.

## KafkaMessageProducer
With KafkaTemplate, it's as simple as writing a letter. Create a KafkaProducer service, autowire KafkaTemplate, and start sending messages.
Enter fullscreen mode Exit fullscreen mode

@Component
public class KafkaMessageProducer {

@Autowired
KafkaTemplate<String, String> kafkaTemplate;

public void sendMessage(String topic, String message){
    kafkaTemplate.send(topic, message);
}
Enter fullscreen mode Exit fullscreen mode

}


## KafkaMessageConsumer
Of course, every good story has two sides. Create a KafkaConsumer service to catch the messages on the other end. KafkaTemplate sends, and KafkaConsumer receives – it's the yin and yang of our messaging world. 

> We configure the consumer with the help of KafkaListener.
Enter fullscreen mode Exit fullscreen mode

@Component
public class MessageConsumer {

@KafkaListener(topics = "ytube-topic1", groupId = "ytube-group")
public void consumerMessage(String message){
    System.out.println("Received Message "+ message);
}
Enter fullscreen mode Exit fullscreen mode

}


Let's make the rest endpoint to post the message into the Kafka topic. 

> We have autowired the kafka message producer and passed the topic name and message.
Enter fullscreen mode Exit fullscreen mode 
@Autowired
KafkaMessageProducer messageProducer;

@PostMapping("/sendMessage")
public String sendMessage(@RequestParam("message") String message){
    messageProducer.sendMessage("ytube-topic1", message);
    return "Message has been sent successfully "+message;
}
Enter fullscreen mode Exit fullscreen mode 

Let's execute the application and see the message posted to Kafka topic and received by the consumer. In real world, the consumer can be another microservice that consumes the message to process it further. 

Kudos, you made it to the end! Thanks for hanging out with us. If you have more questions or just enjoyed the ride, come back soon. Happy exploring!

Checkout the completed code here 

  

    

      GitHub logo
      
        amaialth
       / 
        kafkademo
      
    

    

      Demo/Starter project to connect Spring boot with Kafka
    

  

  

    



Simple Kafka Demo





The application demonstrates the use of KafkaTemplate. We will use Kafka template to produce and consume the message.










  
View on GitHub
Enter fullscreen mode Exit fullscreen mode
💖 💪 🙅 🚩
amailath
Anbumani

Posted on January 4, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Connecting Kafka to Spring Boot: A Step-by-Step Guide Using KafkaTemplate
springboot Connecting Kafka to Spring Boot: A Step-by-Step Guide Using KafkaTemplate

January 4, 2024