Sometimes during an interview, the interviewer might notice on your resume that you have experience using a message queue like Kafka, and then ask you why you chose to use a message queue. Before you answer this question, you should understand that what the interviewer really wants to know is your company's business scenario, the technological challenges you faced, how you addressed those challenges, and why you chose the solutions you did. After you provide the project background, you can then delve into the technology. There are three key points for using a message queue:

• Decoupling
• Asynchronous processing
• Peak shaving

Decoupling

Consider this scenario: System A sends a message to Systems B, C, and D using an API interface. What happens if System E also needs this message? Conversely, what if System C no longer requires the message? In this situation, the developer of System A may become overwhelmed.

In this scenario, System A is tightly coupled with the other systems. However, if we use a message queue (MQ), System A can produce a message and send it to the MQ, completing its task. Regardless of which system needs the message, they can retrieve it from the MQ. If a system no longer requires the message, it can simply unsubscribe from it. None of these actions require System A's direct involvement.

Asynchronous Processing

In this scenario, when a user initiates a request from the frontend page, it takes 100ms in System A, 200ms in System B, 250ms in System C, and 300ms in System D. Consequently, the total processing time for this request is nearly 900ms, resulting in a poor user experience.

Ideally, every operation initiated by the user should be handled within 200ms to ensure a seamless experience.

By using a message queue (MQ), System A can receive the user request, process it in 100ms, and then send the result to the MQ, allowing Systems B, C, and D to handle the message asynchronously (within the bounds of business requirements). This approach reduces the overall processing time to just 100ms, transforming the user experience from awful to awesome.

Peak Shaving

Every day from 12 AM to 12 PM, System A operates peacefully, with a query-per-second (QPS) rate of 50. However, at 12 PM, the QPS suddenly spikes to 5000 or more. This surge in requests places significant stress on the database queries.

When using MySQL, a single MySQL architecture can handle 2000+ queries per second, which represents its upper limit. A QPS of 5000+ would cause the system to break down. However, after the peak, the QPS returns to a lower rate, around 50+ QPS.

In this scenario, if we use a message queue (MQ), during the peak time of 5000 requests per second, the system can handle 2000 requests per second, leaving 3000 requests to be stored in the MQ. Over the course of an hour, the MQ would store millions of messages, preventing the system from breaking down during peak times. After this period, System A would process the remaining messages until all have been handled.

Conclusion

While message queue (MQ) offers several advantages as discussed above, it introduces complexities and potential new issues. Apart from the mentioned benefits, its implementation can reduce system availability. Once integrated into your system, MQ becomes a core component, necessitating measures to ensure its stability and prevent breakdowns.

Another challenge that may arise is the issue of consistency. Due to the decoupling and asynchronous nature of systems, it's possible for some components, such as Systems A and C, to process successfully while others, like Systems B and D, encounter failures. In such cases, a compensation mechanism may be necessary to address this situation, although the specific approach will depend on the nature of your project.

Ultimately, as with any technology, there are both positive and negative aspects to consider when implementing a message queue.