We are designing a system based on independent microservices (connected via a RabbitMq bus). The code will (for the first components at least) be written in python (both python2 and python3). We have already a monolith application implementing some of the business logic, which we want to refactor as microservices, and extend. One question that worries me is:
What is the best way to share code between the different microservices. We have common helper functions (data processing, logging, configuration parsing, etc), which must be used by several microservices.
The microservices themselves are going to be developed as separate projects (git repositories). The common libraries can be developed as a self-contained project too. How do I share these libraries between the microservices?
I see several approaches:
- copy around the version of the library that is needed for each microservice, and update as needed
- release the common libraries to an internal PyPi, and list those libraries as dependencies in the microservice's requirements
- include the library repository as a git submodule
I would like to read a bit more about suggested approaches, best practices, past experiences before deciding on how to proceed. Do you have any suggestion or link?
Best Answer
Your second option is definitely the way to go. Break out common libraries and install them onto your local PyPi server.
Option 1 is horrendous because improvements to the libraries will be difficult to propagate to others who could use it.
Option 3 is similar to option 1.
The common pattern is to setup Jenkins so that when you push to master of a library repo, it does a python build and uploads it automatically to the PyPi repo. Once you write this build script, you'll never have to worry about packaging libraries and uploading them manually to PyPi. And with this option, all library updates will be instantly available to be possibly upgraded into other microservices.
Setting up your own PyPi server is very easy. I like this guide