Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B creates an entity of its own based on that event, and C makes a synchronous call to D to perform an action. Finally D receives both A and B's entity and performs the action requested by C which requires both A and B's entities.
Note: A,B, and D are essentially CRUD services which also have REST APIs, while C is business logic and has no state. A, B, and D are intended to be product-agnostic services in a cloud environment.
This flow works fine when dealing with updating existing entities as service D stores a partial copy of the data it receives from A and B, but it creates a sort of race condition when A creates a new entity. Note that even if A and B have created their entities before C performs the call, there's no guarantee that D has read both events.
What are common ways of dealing with this? I've come up with several, but none of them seem particularly great.
- Retry Pattern
- Which entity performs the retry? Since this action needs to eventually be performed, I assume having it in service D is a bad idea as it then can't distinguish from bad input through the REST API, or requires duplicated functionality to differentiate between an event and a REST call.
- Even if that was acceptable, it pushes business logic into a CRUD service.
- Webhook between D and B
- Again, puts the logic into a CRUD service (kinda).
- Thread sleep on C
- These anywhere in a running service generally seems like a horrible idea…
- Doesn't help with the condition where B goes down while the rest stay up.
Edit in response to answers:
- Have C wait for the two Events before it invokes D
- This unfortunately doesn't guarantee that D has read those same events, even though it is more likely.
- C can't send the full contents of the event from A with the request, just the ID of the object without duplicating the REST interface methods on D; other services to come won't have anything except the ID when they request D's service(s). In all other cases, all the necessary information will reside in D to perform its service, the awkward case seems to be only for new entities.
- Have D wait for both create events from A and B and then do the logic without C having to tell it to.
- This seems to break the single-responsibility principle.
- A, B, and D are intended to be 'product-agnostic' services, see edit to the note above. The implication is that this would push logic into a service that is used by multiple products, even though the logic is specific to only one product.
Best Answer
Think of your situation as data and processes that depend on it. What results is this:
A
creates entitya
B
creates entityb
D
performs a processd
which depends ona
andb
d
should happen is made by serviceC
D
has to do its duty whenC
tells it to, butC
does not provide all dependencies for that process.Thats the problem how i see it. Now on to the solution: If you want to keep
D
as simple as possible,C
has to provide all necessary information toD
. There is no way around that.If it is acceptable to put some logic into
D
(depending on the language and frameworks in use, that logic might happen to be 2 LoC), you can wait for all data inD
:a
is created with ID11
b
is created with ID22
and referes toa#11
C
listens for the create-event ofb
and then tellsD
the following: "As soon as you havea#11
andb#22
, dod
with them.D
waits for both events and then does its thing.If you are using tools like Promises and ReactiveX, this becomes a really simple thing:
This gets a lot harder when you have to persists the waits for the entities. In that case, a scheduled job may be the better choice.