Here is the scenario:
We are running a microservices style architecture on AWS using API Gateway and EC2 instances. Let’s say there are five services: Photo Service, Topics Service, User Posts Service, User Messages Service and User Profile Service. There is also an SQS Consumer service, the “Message Worker”. Each microservice has its own database. Here is the sequence of events when a user makes a request to the User Posts Service (for example):
API Gateway routes the request to the User Posts Service.
The User Posts Service updates the necessary items in its database.
The User Posts Service publishes a message to the Message Worker (SQS).
The Message Worker duplicates the message for as many services necessary, i.e. if the User Comments Service, User Profile Service and User Messages Service all need to be updated, it duplicates the message 3 times.
The Message Worker sends the data to each service via HTTP, and each service responds via HTTP indicating that they have received the message or that it failed to update.
Is this an efficient way to handle messaging between microservices? One concern is that the Message Worker is a single point of failure, but what is the alternative? We are considering creating a queue for each microservice and publishing the messages to the queues via SNS. Any other suggestions to make this more efficient?
We are restricted in what services we can use due to compliance, so using a "smart queue" with message broker capabilities like ZeroMQ / RabbitMQ isn't possible.
Best Answer
"We are considering creating a queue for each microservice and publishing the messages to the queues via SNS."
Yes, then your queues can subscribe to the notification and you no longer need the message service. Essentially its fan out routing. Your microservices become queue workers
However, lots of queues and notifications can become complicated. SQS + SNS is a bit clunky in some respects. I would be tempted to limit my use of it to cases where I definitely have long running async tasks.
If you can fit 4 or 5 micro-services on one ec2 instance because they are all small simple operations that aren't stressed, you've simply separated them for architectural concerns. Then it might be worth just having them call each other when required.
It's very easy to take queues to the nth degree and have millions of nano services communicating via sns messages. It's not that this doesn't work, but you do have to monitor these queues and worry about dropped and double sent messages, full queues, error queues etc.