One of the major issues that I have seen occur in a system with microservices is the way transactions work when they span over different services. Within our own architecture, we have been using distributed transactions to resolve this, but they come with their own issues. Especially deadlocks have been a pain so far.
Another option seems to be some kind of custom-made transaction manager, which knows the flows within your system, and will take care of the rollbacks for you as a background process spanning over your entire system (so it will tell the other services to rollback and if they're down, notify them later on).
Is there another, accepted option? Both of these seem to have their disadvantages. The first one could cause deadlocks and a bunch of other issues, the second one could result in data inconsistency. Are there better options?
Best Answer
The usual approach is to isolate those microservices as much as possible - treat them as single units. Then transactions can be developed in context of the service as a whole (ie not part of usual DB transactions, though you can still have DB transactions internal to the service).
Think how transactions occur and what kind make sense for your services then, you can implement a rollback mechanism that un-does the original operation, or a 2-phase commit system that reserves the original operation until told to commit for real. Of course both these systems mean you're implementing your own, but then you're already implementing your microservices.
Financial services do this kind of thing all the time - if I want to move money from my bank to your bank, there is no single transaction like you'd have in a DB. You don't know what systems either bank is running, so must effectively treat each like your microservices. In this case, my bank would move my money from my account to a holding account and then tell your bank they have some money, if that send fails, my bank will refund my account with the money they tried to send.