Used memory on Solaris 10
Solution 1
ZFS is likely using most of your memory as ARC cache. Should you want to know how your RAM is used, run this command as root:
# echo ::memstat | mdb -k
On Solaris 10 10/09 and newer, this displays something like this:
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 60569 236 16%
ZFS File Data 53270 208 14%
Anon 41305 161 11%
Exec and libs 5891 23 2%
Page cache 1190 4 0%
Free (cachelist) 7006 27 2%
Free (freelist) 212607 830 56%
Total 381838 1491
As you see, there is a line stating how much of the RAM is used to cache ZFS file data. Unfortunately, you are running an older Solaris 10 release so memstat doesn't show this ZFS statistic separately. It is included with the Kernel used memory which is confusing. A kernel shouldn't use 13 GB of RAM under normal circumstances.
Anyway, there is still a way to display the full ARC size on your server.
Just run this command:
# kstat zfs::arcstats:size
module: zfs instance: 0
name: arcstats class: misc
size 273469024
It shows that on my machine, 273 MB of RAM are currently used to handle the ZFS ARC cache. memstat shows that from these 273 MB, 208 MB are used as file cache. Up to these 208 MB of RAM could be released automatically on demand should applications need it.
Now lets look at processes memory usage. If you use the -Z option with prstat, it shows a summary per zone under the per process statistics. Here the global (and only) zone is using 185 MB of RAM. This should (roughly) match the sum of all processes rss column.
# prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
741 noaccess 129M 113M sleep 59 0 0:00:35 1,4% java/18
973 root 5148K 832K run 29 0 0:00:00 0,4% script/1
972 root 5072K 900K sleep 59 0 0:00:00 0,2% script/1
998 root 7148K 2812K cpu0 49 0 0:00:00 0,1% prstat/1
974 root 3456K 968K sleep 49 0 0:00:00 0,1% ksh/1
5 root 0K 0K sleep 99 -20 0:00:01 0,1% zpool-rpool/37
241 root 5400K 1608K sleep 59 0 0:00:00 0,0% VBoxService/5
77 root 7620K 2356K sleep 59 0 0:00:00 0,0% devfsadm/7
969 root 3372K 936K sleep 59 0 0:00:00 0,0% script/1
126 root 9664K 2844K sleep 59 0 0:00:00 0,0% nscd/31
480 root 9420K 2036K sleep 59 0 0:00:00 0,0% sendmail/1
11 root 9164K 7860K sleep 59 0 0:00:29 0,0% svc.configd/17
1 root 2504K 1432K sleep 59 0 0:00:00 0,0% init/1
413 root 15M 9644K sleep 59 0 0:00:00 0,0% fmd/19
377 root 6536K 2848K sleep 59 0 0:00:02 0,0% inetd/4
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 48 177M 185M 12% 0:01:24 2,5% global
These 185 MB correspond to sum of two lines in memstat output: "Anon" which is RAM used by applications to store data and "Exec and libs" which is the applications and their libraries code.
Solution 2
The memory is filled with unmapped pages of data read from disk. It's kept in memory because those files may be read again and keeping the data in memory saves a disk read. Free memory is forever wasted, so the computer tries to keep as little of it as possible.
For example, say you run a program. The program terminates. The program is still in memory, but those pages of memory are not used by any process since the program isn't running. If the system isn't under memory pressure, the pages are kept in memory. If the program runs again, this will save the effort of making it free just to have to allocate more memory for the program and then read it in again. And if the pages are needed for something else, it's still a win for the system because it's easier to move a page of memory directly from use to another than make it free only to make it used again.
Memory is not a saveable resource. If you leave 1GB free for an hour, anything you could have done with that data is forever lost.
Jeremy C.
Updated on September 18, 2022Comments
-
Jeremy C. almost 2 years
One more question about memory on Solaris 10.
A top shows me that I have 672 MB free memory :
130 processes: 126 sleeping, 2 zombie, 2 on cpu CPU states: 95.1% idle, 3.9% user, 1.0% kernel, 0.0% iowait, 0.0% swap Memory: 16G phys mem, 672M free mem, 2048M total swap, 2023M free swap
A vmstat shows me the same :
kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr rm s0 s1 s2 in sy cs us sy id 0 0 0 564744 687896 3 13 0 0 0 0 0 0 0 0 0 354 667 752 1 1 98
But when I do a prstat -a -s size I get this :
NPROC USERNAME SWAP RSS MEMORY TIME CPU 45 orbixadm 1449M 1592M 9.7% 4:46:53 0.4% 48 root 146M 160M 1.0% 8:09:49 1.2% 3 user1 46M 204M 1.2% 0:00:45 0.0% 9 webservd 46M 14M 0.1% 0:00:00 0.0% 5 ctxsrvr 28M 32M 0.2% 4:54:51 0.0% 11 user2 23M 34M 0.2% 0:00:37 0.2% 4 user3 4840K 11M 0.1% 0:00:01 0.0% 1 smmsp 1456K 4552K 0.0% 0:00:24 0.0% 2 daemon 2128K 6224K 0.0% 0:06:32 0.0% 1 user4 1232K 3608K 0.0% 0:00:00 0.0% 1 nagios 376K 2472K 0.0% 0:15:18 0.0%
and as you can see, the sum of RSS values does not reach 15GB of memory, and even if I add SWAP values to it.
So my question is : which command do I believe ?
If top and vmstat give me the good result, where are my 15GB used memory ? If not, why do they show me that ?
Edit: the result for the command :
% echo ::memstat | mdb -k
Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 1687138 13180 82% Anon 137110 1071 7% Exec and libs 47107 368 2% Page cache 95277 744 5% Free (cachelist) 22248 173 1% Free (freelist) 69592 543 3% Total 2058472 16081 Physical 2055442 16058
Edit 2:
Ok, now i can see the memory used by ARC cache.
But with some new tests, now I have :2066 MB used
(prstat -Z and echo ::memstat | mdb -k result)
1193 MB free
(top result)
8859 MB ARC cache
(kstat zfs::arcstats:size result)Which give us more or less
12 GB
of memory, while my system has16 GB
.
Maybe I missed something else, but where are the other4 GB
? -
Jeremy C. over 12 yearsThank's for this well explained answer. I understand now why all my Solaris servers have more or less 90% RAM used.
-
Jeremy C. over 12 yearsThank's for your response too, the command's result is not really detailed, but there is the necessary to see what is using RAM.
-
jlliagre over 12 yearsCan you add its output by updating your question ? By the way, the answer you accepted is actually slighty incorrect as unmapped pages are reported as free RAM by Solaris, not as used which is the problem you are complaining of.
-
Jeremy C. over 12 yearsQuestion edited with the command result. You're right my questions are not totally answered. At least we can see that free memory is the same with top and vmstat than with ::memstat. But is there a meanning to detail what is ued by each process ?
-
jlliagre over 12 yearsWhat Solaris 10 update are you using (cat /etc/release) and are you using ZFS ?
-
Jeremy C. over 12 yearsit is Solaris 10 5/09, and yes I use ZFS