Docker engine won’t let me remove old containers

docker

I have a docker host with a lot of exited containers. Through my inattention, the host server ran out of disk space due in large part to these containers. I tried to delete them with docker rm, but got an error about not having enough disk space to do the removal. I didn't realize the potential significance of this at the time, however, and no longer have the specific message.

I then freed up some space on the device and tried docker rm again, and now I get this message:

[me@machine ~]$ docker rm amazing_mahavira
Error response from daemon: Driver devicemapper failed to remove root filesystem 92c18d5ebe6cgc328225353064f30b1cb78e1ea668aad05c3d3d6e7095e20524:
devicemapper: Error running DeleteDevice dm_task_run failed

Trying to docker save my images also fails.

Here is the docker info storage output:

Storage Driver: devicemapper
 Pool Name: docker-253:0-273125642-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 92.09 GB
 Data Space Total: 107.4 GB
 Data Space Available: 14.24 MB
 Metadata Space Used: 587.1 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 14.24 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data

I see this link with a similar problem and no suggested solution other than that it looks like a bug. I searched the docker github for similar issues and saw none.

What is the best way for me to clean all this up so that I can have a working docker host again? There are no containers or images that I absolutely must retain, though I really would like to save some images. Can I resurrect this installation, or am I doomed?

The server is RHEL 7.2, FWIW.

Best Answer

I've never been able to fix docker devmapper loop-file corruption.

If rebooting the whole machine can't fix it and you can't save your images with docker save, I'd mark this docker setup a total loss.

A fast and simple way to get it back to pristine, working condition, is this:

sudo service docker stop
sudo rm /var/lib/docker/*
sudo service start

This will delete all your current docker data, so images, containers, volumes, networks, anything. Docker will recreate all needed files and it'll be like new.

As near future steps it would probably be good to not only put monitoring on the disk space, but also swap to one of the other storage drivers.

See the docker docs about this subject. Easiest would probably be either the overlay (or overlay2 if you have a 4.0+ kernel) if your kernel supports it, or aufs (probably not that easy on RHEL, since not officially supported).

Another, slightly more challenging (still not really hard) solution is switching devicemapper to direct-lvm instead of loop-lvm (what it is now). RHEL Atomic docs mention docker-storage-setup which should help/automate setting up direct-lvm, but I have limited experience there

Related Topic