Used memory on Solaris 10

memorymemory usagesolarissolaris-10

One more question about memory on Solaris 10.

A top shows me that I have 672 MB free memory :

130 processes: 126 sleeping, 2 zombie, 2 on cpu
CPU states: 95.1% idle,  3.9% user,  1.0% kernel,  0.0% iowait,  0.0% swap
Memory: 16G phys mem, 672M free mem, 2048M total swap, 2023M free swap

A vmstat shows me the same :

kthr      memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr rm s0 s1 s2   in   sy   cs us sy id
0 0 0 564744 687896  3  13  0  0  0  0  0  0  0  0  0  354  667  752  1  1 98

But when I do a prstat -a -s size I get this :

NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
   45 orbixadm 1449M 1592M   9.7%   4:46:53 0.4%
   48 root      146M  160M   1.0%   8:09:49 1.2%
    3 user1      46M  204M   1.2%   0:00:45 0.0%
    9 webservd   46M   14M   0.1%   0:00:00 0.0%
    5 ctxsrvr    28M   32M   0.2%   4:54:51 0.0%
   11 user2      23M   34M   0.2%   0:00:37 0.2%
    4 user3    4840K   11M   0.1%   0:00:01 0.0%
    1 smmsp    1456K 4552K   0.0%   0:00:24 0.0%
    2 daemon   2128K 6224K   0.0%   0:06:32 0.0%
    1 user4    1232K 3608K   0.0%   0:00:00 0.0%
    1 nagios    376K 2472K   0.0%   0:15:18 0.0%

and as you can see, the sum of RSS values does not reach 15GB of memory, and even if I add SWAP values to it.

So my question is : which command do I believe ?

If top and vmstat give me the good result, where are my 15GB used memory ?
If not, why do they show me that ?

Edit:
the result for the command : % echo ::memstat | mdb -k

Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    1687138             13180   82%
Anon                       137110              1071    7%
Exec and libs               47107               368    2%
Page cache                  95277               744    5%
Free (cachelist)            22248               173    1%
Free (freelist)             69592               543    3%

Total                     2058472             16081
Physical                  2055442             16058

Edit 2:

Ok, now i can see the memory used by ARC cache.
But with some new tests, now I have :

2066 MB used (prstat -Z and echo ::memstat | mdb -k result)
1193 MB free (top result)
8859 MB ARC cache (kstat zfs::arcstats:size result)

Which give us more or less 12 GB of memory, while my system has 16 GB.
Maybe I missed something else, but where are the other 4 GB ?

Best Answer

ZFS is likely using most of your memory as ARC cache. Should you want to know how your RAM is used, run this command as root:

# echo ::memstat | mdb -k

On Solaris 10 10/09 and newer, this displays something like this:

Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      60569               236   16%
ZFS File Data               53270               208   14%
Anon                        41305               161   11%
Exec and libs                5891                23    2%
Page cache                   1190                 4    0%
Free (cachelist)             7006                27    2%
Free (freelist)            212607               830   56%

Total                      381838              1491

As you see, there is a line stating how much of the RAM is used to cache ZFS file data. Unfortunately, you are running an older Solaris 10 release so memstat doesn't show this ZFS statistic separately. It is included with the Kernel used memory which is confusing. A kernel shouldn't use 13 GB of RAM under normal circumstances.

Anyway, there is still a way to display the full ARC size on your server.

Just run this command:

# kstat zfs::arcstats:size
module: zfs                             instance: 0
name:   arcstats                        class:    misc
        size                            273469024

It shows that on my machine, 273 MB of RAM are currently used to handle the ZFS ARC cache. memstat shows that from these 273 MB, 208 MB are used as file cache. Up to these 208 MB of RAM could be released automatically on demand should applications need it.

Now lets look at processes memory usage. If you use the -Z option with prstat, it shows a summary per zone under the per process statistics. Here the global (and only) zone is using 185 MB of RAM. This should (roughly) match the sum of all processes rss column.

# prstat -Z
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
   741 noaccess  129M  113M sleep   59    0   0:00:35 1,4% java/18
   973 root     5148K  832K run     29    0   0:00:00 0,4% script/1
   972 root     5072K  900K sleep   59    0   0:00:00 0,2% script/1
   998 root     7148K 2812K cpu0    49    0   0:00:00 0,1% prstat/1
   974 root     3456K  968K sleep   49    0   0:00:00 0,1% ksh/1
     5 root        0K    0K sleep   99  -20   0:00:01 0,1% zpool-rpool/37
   241 root     5400K 1608K sleep   59    0   0:00:00 0,0% VBoxService/5
    77 root     7620K 2356K sleep   59    0   0:00:00 0,0% devfsadm/7
   969 root     3372K  936K sleep   59    0   0:00:00 0,0% script/1
   126 root     9664K 2844K sleep   59    0   0:00:00 0,0% nscd/31
   480 root     9420K 2036K sleep   59    0   0:00:00 0,0% sendmail/1
    11 root     9164K 7860K sleep   59    0   0:00:29 0,0% svc.configd/17
     1 root     2504K 1432K sleep   59    0   0:00:00 0,0% init/1
   413 root       15M 9644K sleep   59    0   0:00:00 0,0% fmd/19
   377 root     6536K 2848K sleep   59    0   0:00:02 0,0% inetd/4
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     0       48  177M  185M    12%   0:01:24 2,5% global

These 185 MB correspond to sum of two lines in memstat output: "Anon" which is RAM used by applications to store data and "Exec and libs" which is the applications and their libraries code.

Related Topic