High CPU usage on mongodb server when idle

This is an older question, but some important things to provide are:

• What version of MongoDB are you running?

• Can each server in the cluster communicate to every other server in the cluster on the assigned port (27018)? This includes the database shards and the mongos (balancer).

• The shards have to talk amongst themselves to replicate the data, and since the Sending and Receiving nodes are in different zones, are they in the same security group? Are there local firewalls preventing communication?

• Consider inspecting the amount of open file descriptors for the mongod process on each server. There is a hard max of 20k enforced in the server code, but typically there is another limit in place. Here's some commands to help:

  lsof -p <pid of mongod> | wc -l
  
  su - mongod # or whatever user mongod is running as
  ulimit -n # => some systems default to 1024
  

MongoDB uses memory mapped files - which can be much larger then the actual physical (or swap) available on your machine. The total virtual memory used will usually be the size of all the data on disk at the time as a result.

The resident memory will be the stat that represents the actual working set used, though depending on your resources on the box and the usage pattern this will tend to grow over time and approach the total physical RAM on the host. Basically, unless the OS sees a need to page out old data (memory pressure) it will stay in RAM.

When you restart MongoDB this basically starts again from scratch. So if you have a known time frame for your actual working data set (say you only ever use the last 7 days), then the set size after that time period (assuming regular, predictable use) will represent your approximate working set size in terms of RAM.

More info to be found in the Docs here:

http://www.mongodb.org/display/DOCS/Caching

http://www.mongodb.org/display/DOCS/Checking+Server+Memory+Usage

Finally, MMS (which is free) does a good job of mapping this out for you over time - Mapped, Virtual, Resident and non-Mapped as well as plenty of other stats (look for "i" over each graph for an explanation).

http://mms.10gen.com