Microservices – Architecture for Queuing Based Technology in Multi-Tenant Environment

cmessage-queuemicroservicesrabbitmqweb-api

Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance issues and dictated by third party software.

The platform is essentially a naïve Micro-Service pattern followed by being accessed in a Multi-Tenant fashion.

We have a core API that inserts core data, but the core data needs to be complimented by other data, in this example: Assets, C1 and C2.

Now the idea is to insert core data, then publish to a queue, each module needs to subscribe, compliment the data and then publish back into the queue, which then is subscribed via the other modules sequentially. By that point all data is complimented giving a full set of data, which then is finally subscribed via the last module, which then does some super-uber magic and returned.

We expect volume of thousands of requests and possibly millions in due course. Performance/speed is absolutely vital but more performance. Limiting timeouts and hang-time, is also vital and can be achieved via parallel/background tasks. It's the throughput volume that is the highest priority.

Diagram

Questions:

  1. Is such architecture feasible?
  2. Can queuing based technology have/contain the ability to
    publish, subscribe the same data sequentially until that data is deemed complete? each time we receive the data
    compliment the data and push back i.e. publish

Best Answer

Distributed Queue-based solutions have a number of advantages for systems that are asynchronous especially when they are throughput focused. I would not recommend using distributed queues as part of synchronous transactions.

Pros

  • Volume can be leveled out. Processing will continue at it's maximum rate and overflow will be managed on queues.
  • If designed properly, scaling is easy. Just add new processing nodes to pull from the queues.
  • Downtime on processing nodes does not (generally) prevent new requests from being accepted.

Cons

  • Queues require memory and/or disk. If you need to retain messages when queuing is down, you must write each message to disk. IO overhead can be significant.
  • Depending on the technology, you need to make a lot of decisions up front like how much depth you will support on a queue and how large the messages can be.
  • The queuing systems I have worked with do not handle large messages efficiently. There are workarounds but they can be complicated.
  • It can be difficult to find people with experience designing and developing these solutions and also people who know how to support the infrastructure.
  • Unlike a read on a database, get on a queue is destructive. It's easy for developers to commit messages to early resulting in data loss.

I can't overstate the last con: there are so many ways messages can get lost in the shuffle and it can be very difficult to detect that it is happening. For this reason, I would highly recommend you keep a transaction ledger that at the very least records every request that comes in and it's final state. The other thing to consider is that some popular queueing systems have some deep flaws. For example, one system that I have used checks every message on a queue each time you attempt to read the next message. That means if you are falling behind on processing messages, each read gets slower and slower resulting in a downwards spiral of worsening performance. Caveat emptor. I've also worked with weird configurations that make a queue work like a stack which created really unwelcome results under heavy load.

Your Design

This seems like a pretty typical queue-based approach. You have messages coming in the front door and pass them along an assembly-line solution.

What's not clear from your diagram is whether you will have mutliple queues or you literally plan to 'push back' to the same queue each step reads from. If you are planning to use a single queue, don't! I strongly advise that you use a separate queue for each stage of the process like in the diagram below:

enter image description here

You can make it work with one queue but don't get fancy. You'll just create a bunch of problems that don't need to exist. On top of that, it makes it much harder to support. With separate queues, you can see in an instant where you are falling behind: the queue with non-zero depth is being written to faster than it's being read from. And that's one last point that I've seen people struggle with. Over the long run, your depths should all be going to zero. If you have queues that are growing over time, you are heading for trouble.

One other think I advise that isn't strictly about queuing. Avoid updates to records in the DB in your processes. Use inserts only. Updating records will create contention across your processing nodes and make the solution scaling less effective.

Related Topic