Solaris prstat RSS does not add up

solaris

So, on a Solaris 10 10/09 zone, I am seeing all resident memory being used up. In attempting to located the process that is allocating all our physical memory, I notice that the RSS totals "prstat" gives do not add up to the entire total RSS used in the zone.

There are no shared memory segments, and pmap -x reports the same usage numbers as are reported for the individual processes in prstat.

Here is some command output. As you can see, adding the RSS for the top processes is nowhere close to the overall RSS reported (5063M).

# prstat -n 20 -s rss -Z
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
18481 nobody     14G 1457M cpu18   30    0   0:20:51 4.3% java/272
18970 nobody    435M  399M sleep   59    0   0:02:44 1.2% java/30
19083 nobody    371M  363M sleep   59    0   0:02:08 0.4% java/47
18803 nobody    330M  257M sleep   59    0   0:09:51  13% java/366
22260 nobody    410M  132M sleep   59    0   3:52:07 0.3% java/23
12430 daemon     81M   35M sleep   59    0   0:00:23 0.0% httpd/28
12429 daemon     87M   33M sleep   59    0   0:00:22 0.0% httpd/28
...
very low usage processes
....
ZONEID    NPROC  SWAP   RSS     MEMORY  TIME      CPU  ZONE
17        93    3197M   5063M   99%     24:57:19  27%  cygna

Any idea where all the physical memory has gone?

Best Answer

resident and shared overlap, or: shared memory pages are not exclusive. e.g. if multiple processes have libc mapped into their memory space then these will be shared (until COW makes them not shared). The output of pmap may help.

Also see this answer for some insight, though bear in mind that Solaris and Linux memory management differ in a few details (the most important being the approach to overcommit): https://stackoverflow.com/questions/1612939/why-does-the-sun-jvm-continue-to-consume-ever-more-rss-memory-even-when-the-heap

Related Topic