Why Does MongoDB Give an Out Of Disk Space Error When Space Is Available

fedoramongodb

I am running RocketChat on a Fedora 27 (64bit) server and it uses MongoDB. Details are currently:

Version of Rocket.Chat Server: 0.64.0-rc.0
Operating System: Fedora 27 64 bit
Deployment Method(snap/docker/tar/etc): tar
Number of Running Instances: 1
Node Version: 8.9.4
mongoDB Version: 3.4.11

Roughly every 24 hours the instance crashes and I have to restart the mongod service to get it up and running again.

I can see the following in the log:

2018-04-18T00:07:47.635+0000 E STORAGE [thread2] WiredTiger error (28) [1524010067:625767][8003:0x7fae865e9700], file:index-545-5179165362493312193.wt, WT_SESSION.checkpoint: /var/lib/mongodb/index-545-5179165362493312193.wt: handle-write: pwrite: failed to write 4096 bytes at offset 28672: No space left on device
2018-04-18T00:07:47.637+0000 E STORAGE [thread2] WiredTiger error (28) [1524010067:637297][8003:0x7fae865e9700], file:index-545-5179165362493312193.wt, WT_SESSION.checkpoint: index-545-5179165362493312193.wt: fatal checkpoint failure: No space left on device

There is over 12GB of free space on the partition that hosts the mongod database, so I am unclear what the cause of this is. The server uses KVM and not OpenVZ.

Anyone have a clue what might be causing this?

Best Answer

In the end I discovered that this problem was caused by other processes in the system that were exhausting the storage temporarily. case closed.

Related Solutions

Ubuntu – MongoDB upgrade on Ubuntu causing “Low Disk Space” error message

Are you running a 64-bit system? Journaling is automatically enabled on 64-bit systems as of Mongo 1.9.2, and this can take up quite a bit of space. You can safely disable journaling and clear out the old journal files (which can be quite big).

To disable from config file, you will need to COMMENT OUT or DELETE the line that says journal = true (if it exists) and add a line that says nojournal = true. Or, if you're running Mongo via command line, specify --nojournal. Regardless, make sure you restart Mongo before going to the next step.

To delete the journal files, go to your {dbpath}/journal and remove the entire journal subdirectory. This will clear out the journal files and should clean up quite a bit of space.

Hope this helps!

When does the replication from primary to secondary happen in mongodb

Mongo syncs instantly, so there is something wrong with your Replica set.

MongoDB replica sets are something that you need to get right from when they are first set up. If they aren't set up correctly, they can be difficult to fix.

Configuration of the Replica Set should (normally) be done from the master only. If your Set isn't live yet, the best option may be to re-create it.

Also, not sure what robomongo is, but you're probably better off using the native mongo client to find out what is going on.

The rs.status() command should give you output like this

rs0:SECONDARY> rs.status()
{
"set" : "rs0",
"date" : ISODate("2014-03-10T10:42:27Z"),
"myState" : 2,
"syncingTo" : "mongo-master:27017",
"members" : [
    {
        "_id" : 0,
        "name" : "mongo-master:27017",
        "health" : 1,
        "state" : 1,
        "stateStr" : "PRIMARY",
        "uptime" : 3008469,
        "optime" : Timestamp(1394448146, 1),
        "optimeDate" : ISODate("2014-03-10T10:42:26Z"),
        "lastHeartbeat" : ISODate("2014-03-10T10:42:26Z"),
        "lastHeartbeatRecv" : ISODate("2014-03-10T10:42:26Z"),
        "pingMs" : 1
    },
    {
        "_id" : 3,
        "name" : "mongo-slave3:27017",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "uptime" : 3012206,
        "optime" : Timestamp(1394448146, 1),
        "optimeDate" : ISODate("2014-03-10T10:42:26Z"),
        "self" : true
    },
    {
        "_id" : 4,
        "name" : "mongo-slave4:27017",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "uptime" : 890533,
        "optime" : Timestamp(1394448146, 1),
        "optimeDate" : ISODate("2014-03-10T10:42:26Z"),
        "lastHeartbeat" : ISODate("2014-03-10T10:42:26Z"),
        "lastHeartbeatRecv" : ISODate("2014-03-10T10:42:26Z"),
        "pingMs" : 0,
        "syncingTo" : "mongo-master:27017"
    }
],
"ok" : 1
}

Related Topic