I'm beginning a new project at work, and will likely be nearly the sole developer on the project, though one or two other developers will need to integrate existing applications or simple scripts into the main project. The project needs to handle small-scale bulk and streaming data ingest/processing, and both event-driven and on-demand code execution. Some parts of the framework will be heavily CPU bound, and some parts could be heavily I/O bound; most of the data must live on a single machine, but we're able to create a cluster and connect VM's to increase available compute power. There will probably be one or more small web applications that depend on services provided this core framework. The main language will be Python for just about everything.
My question is whether or not I should take a microservices approach to an effort like this or stick with a monolithic application, given that I'll be doing most of the development by myself. My thought is that microservices (using Nameko) provide a natural separation between elements of the framework that have different execution models (data pipelines, event-launched, on-demand, web applications, etc.) and a clear way to distribute the workload and communication across multiple processes. My concern is that I'd probably end up with a Kubernetes cluster to manage (I'm familiar with Docker, but still fairly new to Kubernetes), multiple services (rabbitmq, redis, etc.) required just to facilitate running the system, and potentially a lot of small chunks of code to actually implement all the necessary capabilities that we'll need in the framework.
For a project with little more than a single developer, do microservices still simplify developing and maintaining a complicated system like this? Are there methods/systems/frameworks I should consider using instead, or to reduce the overhead involved in designing the system this way?
Best Answer
Microservices are generally undesirable because they turn your software into a distributed system – and distributed systems make everything a lot more difficult. But a service-oriented architecture has some important benefits:
As you will be the sole developer, you do not need the flexibility to develop the services independently.
But you note that some parts may be CPU-bound. So it might be desirable to scale those independently from the rest of the application. If that is the case, that doesn't mean that you have to turn the whole project into a microservice-architecture. You only need to move that CPU-intensive part into its own service, and can keep the rest in a convenient monolith. Along which lines the system should be split up is difficult to tell, but in general the DDD idea of “bounded contexts” is a good guideline.
Note that monoliths are not bad. Monoliths do not equal a huge messy unmaintainable project. Where you could split the system into different microservices, you can also split the system into different components within a monolith. The separation between these components is just more visible and more clearly enforced in a service-oriented architecture. This also means that for a well-designed system, it should be fairly easy to turn a component into a service at some later point. So you don't have to decide right now, you can move to microservices if and when a monolith has proven itself unsuitable.
Consider also Martin Fowler's concept of the Microservice Premium (2015): microservices introduce substantial complexity of their own, in addition to the base complexity of your system. You have to pay this “premium” in terms of reduced productivity. This means that for simple projects, microservices make you less productive. This changes for more complex projects: whereas a monolithic solution might become increasingly difficult to work with, a microservice architecture scales much better and requires roughly constant effort. You have to know whether the extra initial effort of microservices is worth it given your software system. Since you are asking this question, the answer is probably “no”. Fowler continues: