We have some 64 bits Linux boxes (RHEL6) running on Microsoft Windows 2012 Server (Hypervisor) with the same problem. Those Linux guest servers are running in MS Windows Cloud (Hyper-V) with 16 hosts servers with 256 GB of RAM each one.
They start with the following memory usage:
# free -m
total used free shared buffers cached
Mem: 48259 653 47606 0 19 106
-/+ buffers/cache: 527 47732
Swap: 13999 0 13999
# cat /proc/meminfo
MemTotal: 49418204 kB
MemFree: 48749868 kB
Buffers: 20080 kB
Cached: 108564 kB
SwapCached: 0 kB
Active: 149652 kB
Inactive: 98856 kB
Active(anon): 120124 kB
Inactive(anon): 1884 kB
Active(file): 29528 kB
Inactive(file): 96972 kB
Unevictable: 4 kB
Mlocked: 4 kB
SwapTotal: 14335992 kB
SwapFree: 14335992 kB
Dirty: 788 kB
Writeback: 0 kB
AnonPages: 122196 kB
Mapped: 39844 kB
Shmem: 2132 kB
Slab: 51832 kB
SReclaimable: 14696 kB
SUnreclaim: 37136 kB
KernelStack: 5656 kB
PageTables: 15840 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 39045092 kB
Committed_AS: 490856 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 131964 kB
VmallocChunk: 34359602252 kB
HardwareCorrupted: 0 kB
AnonHugePages: 28672 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8128 kB
DirectMap2M: 50323456 kB
(Ordered by RES)
# top
top - 11:26:52 up 1 min, 1 user, load average: 1.71, 0.73, 0.27
Tasks: 609 total, 1 running, 608 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.1%hi, 0.0%si, 0.0%st
Mem: 49418204k total, 674472k used, 48743732k free, 20472k buffers
Swap: 14335992k total, 0k used, 14335992k free, 111720k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1859 root 20 0 103m 28m 5784 S 0.0 0.1 0:00.85 Xvnc
2037 root 20 0 508m 25m 8876 S 0.7 0.1 0:01.18 gnome-settings-
2064 root 20 0 373m 17m 12m S 0.0 0.0 0:01.11 nautilus
2141 root 20 0 464m 16m 12m S 0.0 0.0 0:00.14 clock-applet
2063 root 20 0 319m 14m 10m S 0.0 0.0 0:00.28 gnome-panel
2082 root 20 0 307m 12m 9100 S 0.0 0.0 0:00.11 nm-applet
2139 root 20 0 381m 12m 9748 S 0.0 0.0 0:00.08 gdm-user-switch
2093 root 20 0 442m 11m 9104 S 0.0 0.0 0:00.13 gnome-volume-co
2116 root 20 0 299m 11m 9476 S 0.0 0.0 0:00.10 wnck-applet
2118 root 20 0 307m 11m 8768 S 0.0 0.0 0:00.06 trashapplet
...
And 6 minutes later the memory is consumed by an unknown process or by the kernel:
uptime ; free -m
11:31:52 up 6 min, 1 user, load average: 1.05, 0.93, 0.47
total used free shared buffers cached
Mem: 48259 25296 22963 0 21 160
-/+ buffers/cache: 25115 23144
Swap: 13999 0 13999
]# cat /proc/meminfo
MemTotal: 49418204 kB
MemFree: 23514240 kB
Buffers: 21600 kB
Cached: 164428 kB
SwapCached: 0 kB
Active: 210768 kB
Inactive: 108108 kB
Active(anon): 133036 kB
Inactive(anon): 2332 kB
Active(file): 77732 kB
Inactive(file): 105776 kB
Unevictable: 8 kB
Mlocked: 8 kB
SwapTotal: 14335992 kB
SwapFree: 14335992 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 132892 kB
Mapped: 41976 kB
Shmem: 2516 kB
Slab: 52624 kB
SReclaimable: 17628 kB
SUnreclaim: 34996 kB
KernelStack: 5752 kB
PageTables: 15756 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 39045092 kB
Committed_AS: 654848 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 131964 kB
VmallocChunk: 34359602252 kB
HardwareCorrupted: 0 kB
AnonHugePages: 45056 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8128 kB
DirectMap2M: 50323456 kB
(Ordered by RES)
# top
top - 11:32:45 up 7 min, 1 user, load average: 1.02, 0.94, 0.50
Tasks: 607 total, 1 running, 606 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49418204k total, 25904096k used, 23514108k free, 21600k buffers
Swap: 14335992k total, 0k used, 14335992k free, 164428k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1859 root 20 0 103m 28m 6176 S 0.0 0.1 0:00.87 Xvnc
2037 root 20 0 508m 25m 8916 S 0.0 0.1 0:01.19 gnome-settings-
2064 root 20 0 373m 17m 12m S 0.0 0.0 0:01.61 nautilus
2141 root 20 0 464m 16m 12m S 0.0 0.0 0:00.15 clock-applet
2063 root 20 0 319m 14m 10m S 0.0 0.0 0:00.29 gnome-panel
1788 root 20 0 779m 13m 5808 S 0.0 0.0 0:00.18 scxcimserver
2082 root 20 0 307m 12m 9100 S 0.0 0.0 0:00.11 nm-applet
2139 root 20 0 381m 12m 9748 S 0.0 0.0 0:00.08 gdm-user-switch
2093 root 20 0 442m 11m 9104 S 0.0 0.0 0:00.13 gnome-volume-co
2116 root 20 0 299m 11m 9476 S 0.0 0.0 0:00.10 wnck-applet
...
The sum of RSS memory reported by ps in MB is about 388:
# ps aux | awk '{sum+=$6} END {print sum / 1024}'
387.898
What else can I check to find out what is consuming the memory?
Best Answer
It sounds like Hyper-V is stealing the memory because your host is over-provisioned, a process called memory ballooning (or Dynamic Memory in the Hyper-V world).