Non-mapped virtual memory & total number of connections

mongodbnosql

We have two MongoDB data nodes (replica set) – Primary & Secondary. I noticed that the non-mapped virtual memory is relatively high and wondering if they are hurting our MongoDB performance (The server usually peaked at around 6-7K queries per sec).

In MMS, it was stated: "The most common case of usage of a high amount of memory for non-mapped is that there are very many connections to the database."

So we checked the memory usage with db.serverStatus().mem in our Secondary:

{
    "bits" : 64,
    "resident" : 6846,
    "virtual" : 416797,
    "supported" : true,
    "mapped" : 205549,
    "mappedWithJournal" : 411098,
    "note" : "virtual minus mapped is large. could indicate a memory leak"
}

Note: We are using 2.0.4 and now the default stack size should be 1MB per connection.

The current number of connections is around 1.1K, but the non-mapped virtual memory (virtual-mappedWithJournal) is around 5699 MB.
The trend is quite stable so I can't say there is a leak here, but where is the memory gone?

Any idea?

Best Answer

First, let me say that if it is relatively stable you don't have a memory leak, and that note in serverStatus() has been removed in 2.2 because of the frequency of false positives.

Non-mapped virtual memory is MongoDB's internal data structures and threads' stacks, essentially anything not backed by files on disk. This is generally driven, as you mention, by the connection stack at 1MB per connection. In your case it sounds like that should be around the 1.1-1.2 GB mark (unless you had a connection spike and the memory had not yet been reclaimed).

If you are doing a lot of inline map/reduce, in-memory sorts etc. they can also push up the usage of non-mapped memory. If you install MMS (which is free), non-mapped memory is one of the stats tracked over time and you can then correlate increases in the non-mapped stat with the activity on your database easily. Though, if you have not been running it you may need to restart the process to track the usage from scratch again.

This type of usage analysis is often easier than using pmap (or similar) to try to tie back the memory to specific pieces inside a process

Finally, if non-mapped usage is out of control, it can occasionally be caused by issues with libc malloc in 2.0 - this, among other reasons, was why the 2.2 version shipped with TCMalloc instead. So, 2.2 may well be worth checking out if this remains high without an obvious reason.

Related Topic