What Are Microservices?

Microservices are independently deployable services modeled around a business domain. They communicate with each other via networks, and as an architecture choice offer many options for solving the problems you may face. They are a type of service-oriented architecture, albeit one that is opinionated about how service boundaries should be drawn and that independent deployability is key. From a technology viewpoint, microservices expose the business capabilities that they encapsulate via one or more network endpoints. They also encapsulate data storage and retrieval, exposing data, via well-defined interfaces.

Communication between microservices must be efficient and robust. With lots of small services interacting to complete a single business activity, this can be a challenge. Speaking about difficulty, I would like to write about microservice patterns that I'm gonna write publisher subscriptions. These days I find have time to develop technically. While watching a training video on Linkedin, I remembered the work I had done at the company. I said why didn't you write a short article? Today's content is a pattern that we know publisher-subscriber. The reason is we need multiple subscribers. You will find them in this article.

• Details of pattern
• Definition of Entities
• Cross-Account Integration with Amazon SNS

"Pub/Sub" stands for "Publish/Subscribe," and it's a messaging pattern used in distributed systems and event-driven architectures. In a Pub/Sub system, the communication between different components (or services) is facilitated through the concept of channels or topics. The publisher-subscriber design architecture is below.

What are the use cases:

• Multiple tasks
• Multiple responders

1. Publishers: Entities that generate and send messages without knowing who or what might be listening. Publishers "publish" messages to specific channels or topics.

2. Subscribers: Entities that express interest in receiving messages on specific channels or topics. Subscribers "subscribe" to these channels and receive any messages published to those channels.

3. Broker (Message Queue or Middleware): This is an intermediary component that manages the distribution of messages. It takes care of routing messages from publishers to subscribers based on the channels or topics to which they are subscribed.

The key idea is that publishers and subscribers are decoupled. Publishers don't need to know anything about the subscribers, and vice versa. This decoupling allows for a more flexible and scalable system, where components can be added or removed without affecting the others.

I needed to pub&sub implementation in the last 2 companies that I worked for. Specifically, it was Tiko. Before the business changed at Tiko, my team was responsible for web scraping and web crawling. The main area is collecting, storing, and sharing data with cross domains. The challenge is one crawl operation collects an average of 5 million data and needs to be shared as fast and without losing data. Each team represents a different AWS account. The problem is how to send a message to an SQS in a different account. In the following section, I will describe how to develop steps and explain the subscription access policy rule. In 2023 AWS released the Document db streaming which means every CRUD operation can be a trigger event. I used the streaming process because it is fully managed, with no downtime, and is consistent, and reliable.

Steps:

• Set up the document db streaming settings.
• Create a new Serverless application
• Create an SQS with the name of the router queue
• Add trigger Serverless to SQS
• Create 3 SNS Topics for listing separation by country code (use filter policy).

The topics have already subscribed to the router-service lambda application. That architecture works as event-driven when listing data and writing the database that throws the event.

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:GetTopicAttributes",
        "SNS:SetTopicAttributes",
        "SNS:AddPermission",
        "SNS:RemovePermission",
        "SNS:DeleteTopic",
        "SNS:Subscribe",
        "SNS:ListSubscriptionsByTopic",
        "SNS:Publish"
      ],
      "Resource": "arn:aws:sns:eu-central-1:owner-account-id:distributer-topic",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "owner-account-id"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::team-1:root",
          "arn:aws:iam::team-2:root"
        ]
      },
      "Action": [
        "SNS:Subscribe",
        "SNS:Publish"
      ],
      "Resource": "arn:aws:sns:eu-central-1:owner-account-id:distributer-topic"
    }
  ]
}

Advantages:

1. Loose Coupling: Publishers and subscribers are independent of each other, promoting a more modular and maintainable architecture.
2. Scalability: New subscribers or publishers can be added easily without affecting the existing components.
3. Asynchronous Communication: Pub/Sub systems often enable asynchronous communication, allowing components to operate independently and potentially improving system responsiveness.

Last words

• Every team manages own applications which means no dependency
• Almost fully managed and event-driven system that was created.
• Manage the data send speed with lambda concurrency.
• Any team subscribes to any Topic. That provides more flexibility
• Every team works own SQS and SQS DLQ.

Reference:

• https://docs.aws.amazon.com/sns/latest/dg/sns-send-message-to-sqs-cross-account.html
• https://docs.aws.amazon.com/lambda/latest/dg/with-documentdb.html
• https://learn.microsoft.com/en-us/azure/architecture/microservices/design/interservice-communication
• Monolith to Microservices(Sam Newman)