Rest – Microservices and data storage

api-designArchitecturemicroservicesrestsharding

I'm considering moving a monolithic REST API to a microservice architecture, and I'm getting a bit confused about data storage.
As I see it, some of the benefits of microservices would be:

  • Horizontally scalable – I can run multiple redundant copies of a microservice to cope with load and/or a server going down.
  • Loosely coupled – I can change internal implementations of microservices without having to change the others, and I can independently deploy and change them etc…

My problem is with data storage. As I see it there are several options:

  1. A single Database service shared by all microservices – this would seem to completely eliminate any benefit of loose coupling.
  2. A locally installed database instance on each microservice – I can't see a way of horizontally scaling this, so I don't think it would be an option.
  3. Each microservice has it's own database service – this seems the most promising, as it preserves the benefits of loose coupling and horizontal scaling (using redundant database copies and/or sharding across several)

To me, the third option seems to be the only option, but it seems incredibly heavyweight to me, and a very overengineered solution. If I'm understanding it right, then for a simple application with 4-5 microservices I'd have to run 16-20 servers – two actual microservice instances per microservice (in case of server failure, and for deploying without downtime), and two database service instances per microservice (in case of server failure etc…).

This, quite frankly, seems slightly ridiculous. 16-20 servers to run a simple API, bearing in mind that a realistic project will probably have more than 4-5 services? Is there some fundamental concept that I'm missing that will explain this?

Some things that may help while answering:

  • I'm the sole developer on this project, and will be for the foreseeable future.
  • I'm using Node.js and MongoDB, but I'd be interested in language-agnostic answers – an answer might even be that I'm just using the wrong technologies!

Best Answer

Of your three options, the first (a single, shared database) and the third (a "database service") are the most common.

The first is called an integration database. This is generally not seen as a good solution in a microservice architecture. It does add coupling to your services. It also makes it very easy for one service to simply bypass the other services and query into a database directly. You could lose any kind of data integrity or validation provided by the application level not enforced at the database level.

Your third idea is called an application database. And you're right - it allows you to enforce the loose coupling at the API level between services and allows you to more easily scale services at the database level. It also makes it easier to replace the underlying database technology to something appropriate with each service, just as you can change technology or other implementation details of each service. Very flexible.

However, I'd propose an intermediate solution.

Instead of standing up a database service for every microservice, stand up a schema for every service. If you are using multiple database technologies, you may need to split slightly differently, but the idea would be to minimize the number of database servers that you are running, but make it very easy to split out a service into its own database server if and when it becomes necessary. As long as you only allow a database to access its own schema, you have the advantages of an application database but without the overhead of database servers existing for every application or service.

However, as a solo developer, I would challenge the entire notion of microservices at this point in time - Martin Fowler writes about the Monolith First and the Microservice Premium, Simon Brown talks about modular monoliths, and DHH talks about the Majestic Monolith. I'm not sure how well your monolith is organized, but refactor and organize it. Identify components and make clean separations between them for extracting pieces into a service easily. The same goes for your database structure. Focus on good, clean, component-based architecture that can support refactoring into services. Microservices add a lot of overhead for a single developer to build and support in operations. However, once you actually have a need to scale part of the system, use your monitoring and reporting systems to identify the bottlenecks, extract to a service, and scale as necessary.

Related Topic