Database – Aggregation of data from two Microservices

algorithmsbusiness-logicdatabasedatabase-designmicroservices

I have two Microservices A and B.

  • B Microservice has a large set of an entity called User.
  • A Microservice stores the User entity in its own DB if User is configured by an agent. There is no flag available in Microservice B's DB to find if the User is configured.

I want to find all the list of all unconfigured user by page (Set(B)-Set(A)).

How should I go about querying if User data is too large in B Microservice?

Best Answer

There is no good way to do this. Because your data is split across multiple microservices, you need to join this data manually. So there needs to be some piece of code like:

def find_unconfigured_users():
  for user in B.all_users():
    if A.has_configured_user(user):
      yield user

This is going to be tedious and slow, but because you are using microservices you don't have any reasonable choice. (The unreasonable choice would be to access B's data directly, without going through the microservice interface.)

Where this functionality should live depends … possibly, this should be part of the Microservice A itself, i.e. part of the service that deals with configured users.

When data is split like this over multiple microservices this might be a necessary evil, or could be an indication of a deeper problem. Domain-Driven Design has the concept of a bounded context – a self-contained part of the problem domain your software system is trying to solve. One bounded context should not be split across multiple microservices!

You might want to check which bounded context which aspects of a User belongs to. Why is there a need to deal with configured and unconfigured users, and why is that distinct from users in general? It is legitimate if different bounded contexts have different but related concepts of a “User”, but you have to be clear how they relate. Here, it seems like one User entity is effectively shared across different contexts, and therefore causing problems. Possibly, the solution could be to include the configuredness of a user into the main User context (B).

“The schema is fixed” is not necessarily a good reason to avoid doing this, but perhaps cannot be influenced by you. No technological solution will be able to solve organizational dysfunction – especially not microservices. You'll then have to fall back to tediously checking every user, as in the code snippet above.

Related Topic