Mongodb suddenly fails with message “No space left on device”

centos6mongodb

Mongodb fails with message "No space left on device". Mongo log:

$ sudo tail -n 5 ./mongodb/mongod.log
2016-11-10T07:19:46.821-0700 E STORAGE  [initandlisten] WiredTiger (28) [1478787586:821178][9468:0x7f54e7014d40], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongo/WiredTiger.turtle.set: handle-open: open: No space left on device
2016-11-10T07:19:46.821-0700 E STORAGE  [initandlisten] WiredTiger (28) [1478787586:821251][9468:0x7f54e7014d40], txn-recover: Recovery failed: No space left on device
2016-11-10T07:19:46.830-0700 I -        [initandlisten] Assertion: 28595:28: No space left on device
2016-11-10T07:19:46.830-0700 I STORAGE  [initandlisten] exception in initAndListen: 28595 28: No space left on device, terminating
2016-11-10T07:19:46.830-0700 I CONTROL  [initandlisten] dbexit:  rc: 100

but I have enough space and inodes:

$ df
Filesystem         1K-blocks       Used Available Use% Mounted on
/dev/ploop35670p1 2113654704 1530326948 497220988  76% /
none                 8388608          4   8388604   1% /dev
none                 8388608        160   8388448   1% /dev/shm
$ df -i
Filesystem           Inodes   IUsed     IFree IUse% Mounted on
/dev/ploop35670p1 134217728 1575620 132642108    2% /
none                2097152     155   2096997    1% /dev
none                2097152      51   2097101    1% /dev/shm

OS version:

$rpm -qa | grep centos-release
centos-release-6-8.el6.centos.12.3.x86_64

Mongo version:

$ mongod --version
db version v3.2.10
git version: 79d9b3ab5ce20f51c272b4411202710a082d0317
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
    distmod: rhel62
    distarch: x86_64
    target_arch: x86_64

How can I find source of the problem?

Best Answer

Your "server" is actually a container on an OpenVZ host. It is not your VPS which is out of disk space, but the host itself. Contact your hosting provider to report the problem. And stop using OpenVZ VPSes.