Linux – MongoDB not starting: Unable to create/open lock file No space left on device Is a mongod instance already running

linuxmongodb

MongoDB was already running for a couple of months then stopped suddenly.

I ran:

service mongod restart 
Restarting mongod (via systemctl):  Job for mongod.service failed because
the control process exited with error code. See "systemctl status mongod.service" 
and "journalctl -xe" for details

then:

systemctl status mongod.service
mongod.service - SYSV: Mongo is a scalable, document-oriented database.
   Loaded: loaded (/etc/rc.d/init.d/mongod; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-12-21 14:57:30 EST; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 22675 ExecStop=/etc/rc.d/init.d/mongod stop (code=exited, status=0/SUCCESS)
  Process: 28515 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....
mongod[28515]: Error starting mongod. /var/run/mongodb/mongod.pid exists.
systemd[1]: mongod.service: control process exited, code=exited status=1
systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
systemd[1]: Unit mongod.service entered failed state.
systemd[1]: mongod.service failed.

i checked the mongodb.log and got these:

I CONTROL  ***** SERVER RESTARTED *****
I CONTROL  [initandlisten] MongoDB starting : pid=24312 port=27017 dbpath=/var/lib/mongo 64-bit host=server-11
I CONTROL  [initandlisten] db version v3.0.15
I CONTROL  [initandlisten] git version: b8ff507269c382bc100fc52f75f48d54cd42ec3b
I CONTROL  [initandlisten] allocator: tcmalloc
I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
I STORAGE  [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: /var/lib/mongo/mongod.lock errno:28 No space left on device Is a mongod instance already running?, terminating
I CONTROL  [initandlisten] dbexit:  rc: 100

which is weird because running df -h shows I still have a lot of space:

[root@server-11 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G  4.4G   15G  24% /
devtmpfs        485M     0  485M   0% /dev
tmpfs           494M     0  494M   0% /dev/shm
tmpfs           494M   57M  438M  12% /run
tmpfs           494M     0  494M   0% /sys/fs/cgroup
tmpfs            99M     0   99M   0% /run/user/0

[root@server-11 /]# df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
/dev/xvda1     1310720 68253 1242467    6% /
devtmpfs        123983   303  123680    1% /dev
tmpfs           126358     1  126357    1% /dev/shm
tmpfs           126358   495  125863    1% /run
tmpfs           126358    16  126342    1% /sys/fs/cgroup
tmpfs           126358     1  126357    1% /run/user/0

i also ran ls -la /var/lib/mongo

total 376860
drwxr-xr-x.  3 mongod mongod      4096 Dec 21 16:18 .
drwxr-xr-x. 38 root   root        4096 Aug 17 13:25 ..
-rw-------.  1 mongod mongod  67108864 Aug 17 14:23 admin.0
-rw-------.  1 mongod mongod  16777216 Aug 17 14:23 admin.ns
drwxr-xr-x.  2 mongod mongod      4096 Dec 14 14:28 journal
-rw-------.  1 mongod mongod  67108864 Dec 20 16:59 learning_locker.0
-rw-------.  1 mongod mongod 134217728 Dec 20 16:59 learning_locker.1
-rw-------.  1 mongod mongod  16777216 Dec 20 16:59 learning_locker.ns
-rw-------.  1 mongod mongod  67108864 Dec 14 14:26 local.0
-rw-------.  1 mongod mongod  16777216 Dec 14 14:26 local.ns
-rw-r--r--.  1 mongod mongod        69 Aug 17 13:29 storage.bson

I installed mongodb with the root user and have been restarting mongo with the same user. Im not sure why it would change all of a sudden.

Best Answer

More often than not, this error message is misleading. As you saw, the disk space is not the problem.

If mongod failed and died before clearing the lockfile (/var/lib/mongo/mongod.lock), the lockfile is still there, the new instance will not be able to create a new lockfile and will die with this error message.

To restart mongod, first make sure no mongod instance is running, delete the lockfile, and start it.

Other cause for this message is when you started mongod with one user (e.g. mongod) and tried to start it again under another user (www-data or nobody). In this case, the user www-data will not have permission to write on the lock directory. If you run ls -la /var/lib/mongo you can see which user is owner of the directory and if you are starting mongod with the correct user.

Related Topic