I have a docker host with a lot of exited containers. Through my inattention, the host server ran out of disk space due in large part to these containers. I tried to delete them with docker rm
, but got an error about not having enough disk space to do the removal. I didn't realize the potential significance of this at the time, however, and no longer have the specific message.
I then freed up some space on the device and tried docker rm
again, and now I get this message:
[me@machine ~]$ docker rm amazing_mahavira
Error response from daemon: Driver devicemapper failed to remove root filesystem 92c18d5ebe6cgc328225353064f30b1cb78e1ea668aad05c3d3d6e7095e20524:
devicemapper: Error running DeleteDevice dm_task_run failed
Trying to docker save
my images also fails.
Here is the docker info
storage output:
Storage Driver: devicemapper
Pool Name: docker-253:0-273125642-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 92.09 GB
Data Space Total: 107.4 GB
Data Space Available: 14.24 MB
Metadata Space Used: 587.1 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 14.24 MB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
I see this link with a similar problem and no suggested solution other than that it looks like a bug. I searched the docker github for similar issues and saw none.
What is the best way for me to clean all this up so that I can have a working docker host again? There are no containers or images that I absolutely must retain, though I really would like to save some images. Can I resurrect this installation, or am I doomed?
The server is RHEL 7.2, FWIW.
Best Answer
I've never been able to fix docker devmapper loop-file corruption.
If rebooting the whole machine can't fix it and you can't save your images with
docker save
, I'd mark this docker setup a total loss.A fast and simple way to get it back to pristine, working condition, is this:
This will delete all your current docker data, so images, containers, volumes, networks, anything. Docker will recreate all needed files and it'll be like new.
As near future steps it would probably be good to not only put monitoring on the disk space, but also swap to one of the other storage drivers.
See the docker docs about this subject. Easiest would probably be either the
overlay
(oroverlay2
if you have a 4.0+ kernel) if your kernel supports it, oraufs
(probably not that easy on RHEL, since not officially supported).Another, slightly more challenging (still not really hard) solution is switching devicemapper to direct-lvm instead of loop-lvm (what it is now). RHEL Atomic docs mention
docker-storage-setup
which should help/automate setting up direct-lvm, but I have limited experience there