Debian – Docker hello world gives ‘no space left on device’

cgroupdebiandocker

The command docker run hello-world gives me this error:
System error: write /dev/cgroup/docker/<hash>/cgroup.procs: no space left on device

It's a Debian Wheezy, with a 3.16 kernel from backports. (previously upgraded from squeeze)

/dev is mounted as devtmpfs of 10MB udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=1019797,mode=755)

Does docker have a config option I need to change?

Full output:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world 
535020c3e8ad: Pull complete 
af340544ed62: Already exists 
library/hello-world:latest: The image you are pulling has been verified.
Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:d5fbd996e6562438f7ea5389d7da867fe58e04d581810e230df4cc073271ea52
Status: Downloaded newer image for hello-world:latest
WARNING: Your kernel does not support memory swappiness capabilities, memory swappiness discarded.
Error response from daemon: Cannot start container 0ecfcae485a362fffcbf40e26db61356be62a8093f8bcea5dc37147f1a5a47e0: [8]     System error: write /dev/cgroup/docker/0ecfcae485a362fffcbf40e26db61356be62a8093f8bcea5dc37147f1a5a47e0/cgroup.procs: no space left on

Docker info:

docker info
Containers: 7
Images: 5
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 19
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 8
Total Memory: 7.796 GiB
Name: ...
ID: WZ7K:E5CI:YYXJ:GXL6:Q2NL:5S27:NL4T:6KMM:Z4PF:4N56:PZI3:NPMY
WARNING: No memory limit support
WARNING: No swap limit support

df:

Filesystem              Type      Size  Used Avail Use% Mounted on
rootfs                  rootfs    107G   41G   64G  39% /
udev                    devtmpfs   10M     0   10M   0% /dev
tmpfs                   tmpfs     799M  244K  799M   1% /run
/dev/disk/by-label/\x2f ext3      107G   41G   64G  39% /
tmpfs                   tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                   tmpfs     2.4G     0  2.4G   0% /run/shm
/dev/sda1               ext3       88M   51M   33M  61% /boot
cgroup                  tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/disk/by-label/\x2f ext3      107G   41G   64G  39% /var/lib/docker/aufs

Best Answer

Add cgroup_enable=memory swapaccount=1 to your grub and reboot the OS.

$ sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

$ sudo update-grub

$ sudo reboot

Reference

Related Solutions

Debian – Cgroup – no space left on device

I have been able to reproduce and solve the issue doing :

/bin/echo 0 >/cgroups/test/cpuset.mems
/bin/echo 0 >/cgroups/test/cpuset.cpus

Seems that CPUs and Memory Nodes' list for the cpuset were created with empty values when the test folder was created.

Hope it will work for you also.

Debian – No space left on device error, but df reports as more space available

It could be that the /tmp/ directory itself is filled with stale PHP sessions that are not getting cleaned up; meaning the source of the issues might be isolated to the /tmp/ directory itself. If that is the case I would just remove all /tmp/sess_* files. First, list all of the sess_* files like this:

ls -la /tmp/sess_*

Or you can get count with wc like this:

ls -la /tmp/sess_* | wc -l

Now once you get some confirmation there is some insane number of files there, go ahead and run this command to delete the /tmp/sess_* files:

sudo rm -rf /tmp/sess_*

And the ephemeral session files will be blown away.

But another brute force—but relatively safe—way to deal wit this is to blow away the /tmp directory itself, recreate the /tmp directory and reboot the server.

Since the /tmp directory is basically a coding holding pen for cached material, there is nothing valid that should be in there. So my best advice is to run the following command to remove & rebuidl the /tmp directory.

rm -rf /tmp && mkdir /tmp/ && chown root:root /tmp && chmod 1777 /tmp

Now that one liner is basically a list of shell commands connected by && that will first delete /tmp, recreate /tmp, change the ownership of /tmp back to root:root and then set proper permissions to the /tmp directory. If you wish you can run each command one by one if you feel safer doing it that way.

sudo rm -rf /tmp 
sudo mkdir /tmp
sudo chown root:root /tmp
sudo chmod 1777 /tmp

Once that is done, I would recommend rebooting the server. Things should be calm cleared up again.

Best Answer

Related Solutions

Debian – Cgroup – no space left on device

Debian – No space left on device error, but df reports as more space available

Related Topic