Linux – High Memory 64bit server, not using all ram

centoslinuxmemoryvirtual-memory

I have several high memory linux servers running centos 64bit, and after 10-20 days of uptime I have noticed on all of these servers they dont actually use all the ram available to then (they all have around 10gb free on a 48gb system ,and 20gb free on 64gb systems.

They are web servers, and have a working dataset (Eg active files ) of over the amount of ram in the server so I would have assumed that the page cache would grow to the point of using all ram then the pages from the page cache would be freed when / if needed.

for example :

top - 09:44:46 up 57 days,  9:32,  5 users,  load average: 6.44, 6.33, 6.27
Tasks: 680 total,   4 running, 676 sleeping,   0 stopped,   0 zombie
Cpu(s): 17.3%us,  3.3%sy,  0.0%ni, 79.0%id,  0.1%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  49313212k total, 39277524k used, 10035688k free,  1247064k buffers
Swap: 20643832k total,        0k used, 20643832k free, 20592968k cached

shows this server has been up for 57 days, however has 10GB of ram free that should be used in the page cache.

The following sysctl's are set from stock centos :

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians = 1
error: "kernel.maps_protect" is an unknown key
net.core.rmem_default = 8388608
net.core.wmem_default = 8388608
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 20000
vm.min_free_kbytes = 85536
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 6000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_fin_timeout = 40
net.ipv4.tcp_keepalive_time = 1000
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_intvl = 30
net.netfilter.nf_conntrack_tcp_timeout_established = 2000
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
kernel.panic = 40
fs.proc_can_see_other_uid = 0
net.ipv4.ipfrag_secret_interval = 6000
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_max_tw_buckets_ub = 720000
net.core.optmem_max = 25165824
fs.proc_can_see_other_uid = 0
fs.proc_super_gid = 32010
vm.swappiness = 2
fs.file-max = 400000
fs.suid_dumpable = 1
kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

iv tried altering now one to many sysctl's ! , and hope some kind soul may have seen this before, thanks in advance for any help.

Best Answer

Im coming back here to answer my own question!

The issue is caused when you have multi cpu's (not multi core) eg 2 cpus each with x cores

vm.zone_reclaim_mode=0

fixes this.

Related Topic