Docker – How to run docker build at lower priority

dockernice

I often have to re-build Docker images, and I do that inside the host I run my containers in. Occasionally I see that this puts considerable pressure on the CPU, so I thought I could run docker build under nice -n19, but this seems to make no difference in terms of how much docker yields when other processes need to run.

I'd rather not use Docker Hub repos, because this is private stuff and I'm trying to save every penny I can right now. I also know that I could set up another machine as a build/repo – I could use a machine at the office, for example – but I'm not sure how.

So, question: why does nice -n19 docker build ... seem to not be very helpful?

(bonus points for pointing me at docs on how to set up my own private build/repo machine)

Best Answer

why does nice -n19 docker build ... seem to not be very helpful?

The docker commands is a client of the docker daemon, which spawns the processes that runs in the containers. When you give a lower priority to the docker command, the priority of the docker daemon itself is unaffected, thus your build and exec runs at their default priority. It's like giving lower priority to your web browser does not mean that the web server is going to service your requests at lower priority.

how to set up my own private build/repo machine

For a basic separate build machine, you can just do docker export, scp, docker import. But for a more serious build system though, you may want to run a private docker registry. Some more useful documentations:

If you run your own private registry, you can do a build from your local workstation in office, then use docker push and docker pull to upload the docker image to your private registry and fetch it to wherever it needs to be.

First let's recall a few definitions

Docker images

Docker images are essentially a union filesystem + metadata. You can inspect the content of docker image union filesystem with the docker export command, and you can inspect a docker image metadata with the docker inspect command.

Data volumes

from the Docker user guide:

A data volume is a specially-designated directory within one or more containers that bypasses the Union File System to provide several useful features for persistent or shared data.

It is important to note here that a given volume (as the directory or file that contains data) is reusable only if it exists at least one docker container using it. Docker images don't have volumes, they only have metadata which eventually tells where volumes would be mounted on the union filesystem. Data volumes aren't either part of docker containers union filesystem, so where are they? under /var/lib/docker/volumes on the docker host (while containers are stored under /var/lib/docker/containers).

Data volume containers

That special type of container has nothing special. They are just stopped containers using a data volume with the sole and unique goal of having at least one container using that data volume. Remember, as soon as the last container (running or stopped) using a given data volume is deleted, that volume will become unreachable through the docker run --volumes-from option.

Working with data volume containers

How to create a data volume container

The image used to create a data volume container has no importance as such a container can remain stopped and still fill its purpose. So to create a data container named datatest_data for a volume in /datafolder you only need to run:

docker run --name datatest_data --volume /datafolder busybox true

Here base is the image name (a conveniently small one) and true is a command we provide just to avoid seeing the docker daemon complain about a missing command. Anyway after you have a stopped container named datatest_data with the sole purpose of allowing you to reach that volume with the --volumes-from option of the docker run command.

How to read from a data volume container

I know two ways of reading a data volume: the first one is through a container. If you cannot have a shell into an existing container to access that data volume, you can run a new container with the --volumes-from option for the sole purpose of reading that data.

For instance:

docker run --rm --volumes-from datatest_data busybox cat /datafolder/data.txt

The other way is to copy the volume from the /var/lib/docker/volumes folder. You can discover the name of the volume in that folder by inspecting the metadata of one of the container using the volume. See this answer for details.

Working with volumes (since Docker 1.9.0)

How to create a volume (since Docker 1.9.0)

Docker 1.9.0 introduced a new command docker volume which allows to create volumes :

docker volume create --name hello

How to read from a volume (since Docker 1.9.0)

Let say you created a volume named hello with docker volume create --name hello, you can mount it in a container with the -v option :

docker run -v hello:/data busybox ls /data

About committing & pushing containers

It should now be clear that since data volumes aren't part of a container (the union filesystem), committing a container to produce a new docker image won't persist any data that would be in a data volume.

Making backups of data volumes

The docker user guide has a nice article about making backups of data volumes.

Good article reagarding volumes: http://container42.com/2014/11/03/docker-indepth-volumes/

docker – Fixing DNS Resolution Issues on Ubuntu 14.04 Desktop Host

Woo, I found a post on github that solved my problem.

After Steve K. pointed out that it wasn't actually a DNS issue and was a connectivity issue, I was able to find a post on github that described how to fix this problem.

Apparently the docker0 network bridge was hung up. Installing bridge-utils and running the following got my Docker in working order:

apt-get install bridge-utils
pkill docker
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
service docker restart