What can be the design of a distributed high available trade exchange

I am working on design of a financial xchange system and especially the order-matching part.

I couldn't find a clear/complete design article especially for scalable/high available order-maching system. There lots of implementations which gives an idea about the system but none of the solutions are scalable and high available.

I want to give details about the problem and a possible solution. I want to ask whether such a solution work, applicable and you have any suggestions.

Let me start with the algoritm definition.

Order-matching algorithm

Although the algorithm has variations and will be possibly more complex in real world, it can be simply summarized as below. You have buyers and sellers in the system. Buyers creates buying orders while sellers creates selling orders in realtime. Order-matching algorithms duty is to match suitable
selling orders with buying orders ( and vice versa ) again in real-time.

Assume that some traders enter the orders shown below. (Below 2 tables and explanation is taken from https://www.apress.com/gp/book/9781590595640 which is a great book. )

| Time  | Trader   | Buy/Sell | Quantity | Price                      |
|-------|----------|----------|----------|----------------------------|
| 10:01 | Anthony  | Buy      | 300      | $20                        |
| 10:05 | Anu      | Sell     | 300      | $20.10                     |
| 10:08 | Nicola   | Buy      | 200      | $20                        |
| 10:09 | Jason    | Sell     | 500      | $19.80                     |
| 10:10 | Jeff     | Sell     | 400      | $20.20                     |
| 10:15 | Nicholas | Buy      | 500      | Market Price Order ( MPO ) |
| 10:18 | Kumar    | Buy      | 300      | $20.10                     |
| 10:20 | Doe      | Sell     | 600      | $20                        |
| 10:29 | Sally    | Buyy     | 700      | $19.80                     |

The exchange will send an order acknowledgment to the traders’ trading terminals and fill the order book as below

|---------------------------------------------|---|-------------------------------------------|
|                 Buyer                       |   |                  Seller                   |
|---------------------------------------------|---|-------------------------------------------|
| Timestamp | Name     | Quantity | Buy Price |   | Sell Price | Quantity | Name  | Timestamp |
|-----------|----------|----------|-----------|---|------------|----------|-------|-----------|
| 10:15     | Nicholas | 500      | Market    |   | $19.80     | 500      | Jason | 10:09     |
| 10:18     | Kumar    | 300      | $20.10    |   | $20        | 600      | Doe   | 10:20     |
| 10:01     | Anthony  | 300      | $20       |   | $20.10     | 300      | Anu   | 10:05     |
| 10:08     | Nicola   | 200      | $20       |   | $20.20     | 400      | Jeff  | 10:10     |
| 10:29     | Sally    | 700      | $19.80    |   |            |          |       |           |
|-----------|----------|----------|-----------|---|------------|----------|-------|-----------|

If the traders in the previous table submit their orders, the market will match the orders as follows:

• Nicholas’s buy order at market price will match Jason’s sellorder.This will result in the first trade, and both the orders will be removed from the order book.
• Kumar’s order of 300 buy will get matched to Doe’s order of 600 sell. Interestingly, this order will get matched at $20.10 even though Doe wanted to sell at $20. Exchange systems are designed to protect the interests of both buyers and sellers. Since there was a passive order from Kumar willing to buy at $20.10 and Doe’s order comes in later asking only for $20, she will still get $20.10. Since Kumar’s order is completely filled, it will be removed completely from the order book. However, Doe’s order of 600 is only half-filled. So, 300 shares of Doe will remain in the order book.
• In the next step, Anthony’s buy order of 300 shares will get fully filled by Doe’s balance of 300 at $20, and both orders will be removed from the order book.
• Now Nicola wants to buy 200 at $20, but Jeff will sell only at $20.20. There is no agreement in price; hence, there will be no further matching, and the matching system will wait either for one of the parties to adjust the price or for a new order at a price where either a buy or a sell can be matched.

Questions, questions and questions

Trading platforms are living systems which means you have all the time selling orders and buying orders coming into system and prices are changes rapidly. Matching-algorithm should have very little latency because any latency may result in money loss for users.

( As I understood but not sure ) it is very hard ( if not impossible ) to implement matching algorithm in distributed manner. Where you have a google/pubsub and consumers listening on orders and doing their matching magic. Because (again) latency is a very hard requirement and putting extra queues may break it. Also sequentiality of orders is very important because all the algorithm based on this. Putting a distributed or concurrent matching logic possibly will break the sequentiality.

• If a single-threaded solution is implemented how can these systems scale?
• If a single-threaded and not distributed solution is implemented how can we ensure high availability?
• Because latency is one of the most critical requirements when we should persist orders, or trades to db? All these operations are async?
• Seems lmax-disruptor is a solution for solving latency problem within a single machine pushing limits of performance. If yes, how we can make it high available?

Possible Architecture

I am trying to put a solution at least working for some assumptions. Please comment on architecture and flow also.

 Seller/Buyer

          +
          |
          | 1 - place order
          |
          |
          |
    2     |
  +-------v-------+
  |               |      3      +--------------------+           6
  | Order Service +------------->   orders queue     +----------------------+
  |               |             +---------+----------+                      |
  +---------------+                       |                                 |
          |4                              |5                                |
          |                               |                                 |
     +----v-----+                         |                                 |
     |          |               +---------v----------+            +---------v----------+
     | Database |          7    |                    |   8        |                    |
     |          <---------------+  Trading Service   +------------>  Trading Service   |
     +----------+               |  (order+matching)  |            |  (order+matching)  |
                                |                    |            |   cold instance    |
                                |                    |            |                    |
                                +--------------------+            +--------------------+

• Seller or buyer places order
• Order service gets this order ( REST )
• Order is put into queue for further processing
• Order is saved into DB async
• Order is fetched by trading-service. Trading service fills its own in-memory structure for order-matching. Order matching is continuosly works on data set.
• Cold instance also fetches the same data from queue.
• When trading service finds some matcing it send this matching to database Async.
• In order to maintain cold instance up to date, matched orders are notified to cold instance also in async.

More and more questions

• Is there any way to eliminate the latency originating from order-queue
• Maintaning hot and cold Trading Service is a very hard job. A distributed cache like hazelcast can be used for distributed memory management but I think it would be slower than lmax disruptor. Is it right?
• Is there any way to distribute lmax-disruptors on distrubuted machines?