We've got a Jenkins CI server that fetches our code from Git, builds it, makes a Docker image, and then ships it off to some production servers.
Our project is primarily written in Python, so "building" involves running
pip install -r requirements.txt
That works fine, except it's kind of slow. It has to fetch packages over the network, plus it has to build C libraries for a few of them (and 'lxml' is not small!).
In development, I've had success using pip-accel
to speed up this process. It has the same interface as pip
but it caches both the Python downloads and the built C code, so
pip-accel install -r requirements.txt
is fast.
I want to do this for our production builds, but I'm running into some obstacles.
Obviously, pip-accel
needs a directory in which to store the cache. Since our CI server is what runs the builds, that's the logical place to put it. But the pip install
command runs inside a fresh Docker container, so it can't just access a common directory on that server.
Docker "volumes" seem like they're designed for sharing directories with containers, but our build happens (surprise surprise) inside docker build
, and only docker run
lets you attach volumes. You can't attach volumes with docker build
.
Is there something I'm missing? How can I run a docker build
and share a cache folder with my host, outside of the container I'm in?
Best Answer
I have few reputations to comment on your question, so my answer has some questions as well.
I tried creating the same setup as yours but minified (based on your explanation above) and it seems to have improvements with the docker's own caching mechanisms
My example Dockerfile looks like this:
And requirements.txt looked like this:
After building this for first time (look a longer time), I added a new library "chardet==1.0.1" and now my requirements.txt looked like:
After running the docker build, it did use all the docker's own cache which had also the old pip libraries
Because, docker build by default has '--force-rm=false', '--no-cache=false'.
It might be a different story if your Jenkins CI runs this build as different users or in different hosts. Otherwise, its a matter of ordering the commands in Dockerfile.
If you still have questions, can you share your sample Dockerfile and also inform here how much/often your requirements.txt changes every jenkins build.