Microservices – Ordering Integration Events

From what you're describing, it strikes me that you could have a problem with the threading model for your event processing loop (ie. the RabbitMQ message handler) and how ASP.NET's concurrency works.

There's two ways to tackle this: in-process and out-of-process.

In-process:

Use a dedicated background thread for your event loop, one that is isolated from the threads that are processing HTTP requests.
Do this in Application_Start. That will give you the concurrency you're looking for: handling incoming HTTP requests & listening for messages at the same time.

Unfortunately, you're doing this in an application framework that's not been designed for this style of work processing. Phil Haack explores why a long running background thread, and an event processing loop is exactly that, in ASP.NET is problematic So, depending on your context, you may be happy to ignore the problems surface in that post. The main one to consider is that application pools, by default, get torn down after 20 minutes. That's fundamentally in-compatible with the always-on process model an event pulling loop needs.

Out-of-process:

Stop handling RabbitMQ messages in your application and expose each type of message processing as an HTTP triggered RESTy API call instead. Introduce a RabbitMQ-to-HTTP bridge process (a windows service, or even a console application) that will listen for Rabbit messages and trigger the appropriate RESTy method.

The downside of this approach is increased deployment complexity and you have to consider distribution as part of your failure scenarios. The nice bit is that your main application runs off HTTP only and your RabbitMQ message processor itself is very straightforward, with no threading worries. All this should facilitate development and testing, despite the increased level of distribution.

Variation: instead of going HTTP-centric for your main application, go Rabbit-centric, and build an HTTP -> RabbitMQ bridge instead and convert your existing REST calls into message triggered equivalents. I don't recommend that, because you're always dependent on a very particular message broker to do work, while HTTP is a ubiquitous integration technology.

In my experience, most questions about transactionability and microservices are caused by the following two reasons:

1. The transactional data is placed in different microservices: This is wrong by definition. Data that should be modified transactionaly belongs to the same service. Grouping the right data in the right service is very tricky and you need to put a lot of thought into it. If you just create microservices based on entities (Orders, Users, Inventory, Products, etc), then you will find the same problem over and over, plus you'll end up with N services depending on N other services to perform their purpose. Therefore: put all data that changes transactionally together.

2. The operation is not transactional in the first place: You are trying to solve with technology something that should be solved by business rules. This is your scenario in my opinion. Orders and Inventory are not transactional. You obviously don't want people buying products out of stock, but you have to accept that there will be always a slight chance that this happens (what if your inventory numbers are correct, but someone at the warehouse drops a box and breaks the last item?). Businesses have been dealing with these sort of problems far longer than transactions existed. Therefore, talk to business and ask what you should do when an order comes in and you are out of stock. If proper stock management is in place, you'll probably know already when new items will be available, so you can email the customer with the new delivery date and offer a refund if she doesn't accept (as an example).

As a side note, your sentence "an order microservice which then emits events to deduct inventory from the inventory microservice" sounds wrong. It might be just the sentence wording, but just in case, I want to point out that the wording suggests that what you do is emitting a Command, not an Event. A command is imperative, you tell someone to do something (DeductInventoryCommand). An Event is something that has already happened (OrderPlacedEvent).

Ideally, you don't want one microservice telling another one what to do, as this effectively breaks the autonomy of the service that receives the command. Instead, develop your services to be able to perform their business capability based on the information they "hear" from other microservices, but autonomously. A Logistics service can listen to the placed orders (and cancelled orders) and take care of all the logistics process without being told what to do. Billing should be able to do its job without being told what to do either, and so on.