I've read a lot of this website to optimize my server but nothing has really helped me 🙁 Im having a memory trouble I think , I have just one website (Drupal) with 860K page views month but when the traffic increase, the load average goes up over +40, +70 etc with 100% of memory use causing the server going totally down
Actually when I just have restarted the server the total memory usage is about 80%
I don't know what to do ! I really cant believe this server cant handle this kind of traffic, please help me!
Specs
Processor #1 to #24
Intel Dual Xeon E5645 @ 2.40GHz
Cache 12288 KB
4GB Total RAM
Apache/2.2.19 -prefork- (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 PHP/5.2.17
500GB HD RAID 1
Drupal based website with Boost module and Cache Router (INNODB tables)
APC Installed
top (shift – m)
top - 23:05:37 up 19:42, 1 user, load average: 0.78, 0.74, 0.64
Tasks: 527 total, 1 running, 524 sleeping, 0 stopped, 2 zombie
Cpu(s): 1.7%us, 0.3%sy, 0.0%ni, 97.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4034276k total, 3774668k used, 259608k free, 279060k buffers
Swap: 6088624k total, 103616k used, 5985008k free, 1316080k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8768 mysql 15 0 1211m 880m 4784 S 12.0 22.4 20:43.44 mysqld
20475 nobody 16 0 429m 176m 27m S 0.0 4.5 0:00.44 httpd
20846 nobody 15 0 427m 176m 28m S 1.0 4.5 0:01.13 httpd
20775 nobody 15 0 422m 171m 29m S 0.0 4.4 0:01.22 httpd
20826 nobody 15 0 422m 171m 29m S 0.7 4.4 0:01.00 httpd
20827 nobody 15 0 423m 171m 28m S 0.7 4.4 0:00.61 httpd
20578 nobody 15 0 422m 171m 29m S 0.0 4.3 0:01.73 httpd
20833 nobody 15 0 422m 170m 28m S 0.0 4.3 0:00.84 httpd
20830 nobody 15 0 421m 170m 28m S 0.0 4.3 0:00.84 httpd
20681 nobody 15 0 422m 170m 28m S 1.0 4.3 0:00.93 httpd
20913 nobody 15 0 422m 170m 27m S 0.0 4.3 0:00.34 httpd
20914 nobody 15 0 422m 169m 27m S 0.0 4.3 0:00.60 httpd
20854 nobody 15 0 423m 167m 23m S 0.0 4.2 0:00.36 httpd
20911 nobody 16 0 418m 167m 28m S 0.3 4.2 0:00.70 httpd
httpd.conf
Timeout 300
TraceEnable On
ServerSignature Off
ServerTokens Full
FileETag All
StartServers 5
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>
ServerLimit 256
MaxClients 150
MaxRequestsPerChild 800
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100
my.cnf
[mysqld]
max_connections = 120
safe-show-database
skip-locking
key_buffer = 148M
max_allowed_packet = 14M
table_cache = 596
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 2M
myisam_sort_buffer_size = 64M
thread_cache_size = 24
query_cache_size= 128M
thread_concurrency = 48
wait_timeout = 45
innodb_file_per_table
innodb_log_file_size = 10485760
open_files_limit = 8192
tmp_table_size=200M
max_heap_table_size=200M
innodb_buffer_pool_size=596M
local-infile=1
log_slow_queries = /var/log/slow.log
long_query_time = 3
[mysqldump]
quick
max_allowed_packet = 16M
[mysqld_safe]
log-error=/var/log/mysqld.log
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 128M
sort_buffer_size = 64M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 128M
sort_buffer_size = 64M
read_buffer = 2M
write_buffer = 2M
Some graphs (this week)
- Load average / Filesystem usage (note swap disk) / MySQL stats
- http://i.imgur.com/si21Z.gif
Update:
- Apache Modules installed
- /etc/init.d/httpd -M and got these results dpaste.com/hold/615665
- /etc/init.d/httpd -l results dpaste.com/hold/615667
- PHP.ini http://dl.dropbox.com/u/33784/info.html
Top with server load over 200
top - 12:27:13 up 5 days, 9:04, 1 user, load average: 219.36, 189.93, 130.56
Tasks: 750 total, 1 running, 749 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.3%us, 1.0%sy, 0.1%ni, 49.7%id, 47.8%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 4034276k total, 4014052k used, 20224k free, 13404k buffers
Swap: 6088624k total, 3036872k used, 3051752k free, 71272k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15653 mysql 15 0 1444m 154m 3384 S 0.0 3.9 192:42.76 mysqld
23800 nobody 16 0 422m 96m 29m D 0.3 2.4 0:06.96 httpd
23527 nobody 16 0 422m 93m 29m D 0.0 2.4 0:05.11 httpd
23759 nobody 15 0 421m 91m 26m S 0.7 2.3 0:03.97 httpd
23842 nobody 16 0 422m 91m 26m D 0.3 2.3 0:04.88 httpd
23819 nobody 16 0 421m 91m 26m D 0.3 2.3 0:07.11 httpd
23739 nobody 16 0 421m 91m 26m D 0.0 2.3 0:10.27 httpd
23778 nobody 15 0 421m 91m 26m S 0.0 2.3 0:04.81 httpd
23790 nobody 15 0 421m 91m 26m S 0.0 2.3 0:03.86 httpd
23754 nobody 16 0 421m 91m 26m D 0.0 2.3 0:08.19 httpd
23700 nobody 16 0 421m 90m 26m D 0.3 2.3 0:05.45 httpd
23843 nobody 16 0 420m 90m 26m S 0.0 2.3 0:06.39 httpd
23510 nobody 16 0 426m 90m 24m D 0.0 2.3 0:04.98 httpd
23841 nobody 16 0 416m 89m 29m D 0.0 2.3 0:03.53 httpd
23836 nobody 15 0 414m 89m 30m S 0.0 2.3 0:05.82 httpd
23849 nobody 15 0 418m 88m 25m S 0.0 2.3 0:05.78 httpd
23833 nobody 16 0 429m 88m 24m D 0.0 2.3 0:05.59 httpd
23832 nobody 16 0 418m 88m 25m S 0.0 2.2 0:09.25 httpd
23746 nobody 16 0 428m 88m 25m D 0.0 2.2 0:04.13 httpd
23851 nobody 16 0 428m 88m 24m D 0.0 2.2 0:03.60 httpd
23816 nobody 15 0 418m 88m 25m S 0.0 2.2 0:07.00 httpd
23282 nobody 15 0 416m 87m 28m S 0.0 2.2 0:11.29 httpd
23742 nobody 15 0 416m 86m 26m S 0.0 2.2 0:09.37 httpd
23837 nobody 16 0 425m 86m 25m D 0.3 2.2 0:05.20 httpd
23093 nobody 16 0 430m 86m 24m D 0.0 2.2 0:04.19 httpd
23732 nobody 16 0 421m 86m 24m D 0.0 2.2 0:05.55 httpd
23772 nobody 15 0 415m 85m 29m S 1.0 2.2 0:14.55 httpd
Someone helped me to tweak Apache settings, but everything looks the same
I have enabled piped logging which should help with the memory issue.
I have also shortened the amount of requests an apache process will do
before it is cycled through memory.
I will really appreciate your help, I tried almost everything, I'm not really an sysadmin but we have not anyone right now to help us.
Thank you!
Best Answer
General rule of thumb -
Run this command when server is under load:
That will tell you approximately the average size of an Apache process.
This is not a dedicated web node so say 60% of the RAM is available for Apache.
4096 * .60 / AVERAGE_SIZE_HTTPD_PROCESS = approximately the # of MaxClients/requests you can service. Generally looks like your average is around 170MB so...
You can service 14 requests at a time. Reduce MaxClients to a more reasonable number like 20-25.
Cheers