In an event-driven microservices architecture, services typically need to update their domain state and publish an integration event to a service bus at the same time (Either both operations complete, or none).
Simple example:
The ordering microservice receives a REST API call to create an order. The microservice needs to:
1) Create the order entity and persist it in the database
2) Publish an OrderCreated integration event through an event bus
When using a relational database this is typically achieved using the outbox pattern: a single entry is saved in the database that indicates that event X needs to be published. This entry is saved as part of the same transaction that contains the domain-state changes. A background process then polls these entries and publishes the events. This means that the event will eventually be published, making the system eventually consistent.
However, NoSQL databases do not favor the idea of updating multiple documents in a single transaction, and many of them do not support it without ugly workarounds. Below is a list of potential solutions (Some more ugly than others):
1. Outbox pattern variation:
Outbox pattern, but instead of having a separate collection of documents for the pending events, they will be saved as part of the domain entity. Each domain entity will encapsulate a collection of events that remain to be published and a background process will poll such entities and publish the events.
Cons:
If the background process publishes the event but fails to remove itThis shouldn't really be a problem if updates are idempotent or if the event handler is able to identify duplicate events.
from the domain entity, it will re-publish it.- Domain entities are
corrupted with integration events.
2. Event sourcing:
Event sourcing makes this problem go away but is very complex to implemented and a big overhead for small microservices.
Cons:
- Complex, might need complete re-design of the way services work with data.
3. Listening to own events:
The service will only publish an event that is also subscribed to (It will not update its state as part of the same operation). When the service bus sends the event back for handling, the service will update its domain entity.
Cons:
- Other microservices may handle the event before the origin microservice. This may cause problems if they assume that the event already happened when in fact it hasn't.
Are there any other solutions to this problem? Which is the best one?
Best Answer
You can't. You are writing to two different databases (NoSQL + Service Bus) - you cannot guarantee consistency between the two without introducing distributed transactions which are bound to cause headaches, slowdowns and ultimately failures.
Event sourcing is the way to go. Event sourcing is not as much of an overhead as it is believed to be, in fact, it is easy to implement (No need for frameworks, easy to build something production-ready), reduces overall complexity and increases performance for large scale applications.
Some comments about the other two approaches:
Approach #1: This seems to be a downgraded variation of ES. It is temporarily persisting the events and using the latest snapshot as the source of truth. I think this is a clear downgrade from event sourcing because:
Approach #3: This is essentially an attempt to use the service bus as an event store - it is not meant to be an event store, don't do it. I've seen some systems that use something like Kafka as an event store, but this is because Kafka leverages events as its core principle and provides some event-sourcing features out of the box.
I recommend going through Greg Young's talks & papers about event sourcing, he's known to be one of the event sourcing gurus.