I have two servers – one runs a PHP webservice, and the other is the database server. I have high load (~9.0), and I believe it is a configuration issue of some sort. The PHP processing is minimal, all the heavy lifting is done by the database server. I have approximately 6-8 requests per second.
System is CentOS/Plesk (not my choice).
top looks like this:
top - 19:54:38 up 4:38, 2 users, load average: 9.42, 9.10, 8.86
Tasks: 48 total, 12 running, 36 sleeping, 0 stopped, 0 zombie
Cpu(s): 67.4%us, 32.5%sy, 0.0%ni, 0.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2097152k total, 231120k used, 1866032k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11806 apache 16 0 50416 28m 4744 R 24.6 1.4 7:27.61 httpd
11770 apache 16 0 49140 27m 4764 R 23.6 1.4 7:32.80 httpd
11783 apache 16 0 50424 28m 4736 R 23.2 1.4 7:19.99 httpd
30217 apache 16 0 50412 28m 4732 R 21.6 1.4 3:33.00 httpd
26245 apache 16 0 50412 28m 4732 R 21.2 1.4 4:14.98 httpd
26256 apache 16 0 48880 27m 4732 R 21.2 1.3 4:15.62 httpd
13659 apache 15 0 52464 30m 4764 R 19.2 1.5 5:23.12 httpd
26255 apache 16 0 48880 27m 4748 R 17.3 1.3 4:16.62 httpd
30215 apache 16 0 48796 27m 4732 R 16.3 1.3 3:45.32 httpd
29703 apache 16 0 52208 30m 4764 R 12.3 1.5 3:46.21 httpd
Load on the database server is next to nothing – 0.05-0.07.
I'm not a server expert so I've been messing with this issue for a while now. I thought I fixed it a while back by running a yum update on the server – many packages updated, so I presumed it was a fixed bug in httpd or similar. The issue returned, I ran yum update again, the issue went away again. Now the issue is back and updating does not fix it – I find this whole thing quite curious.
netstat has many (~100) identical entries like this (IPs and hostnames obscured for NDA reasons):
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 123.123.123.123:12345 mydbserver.com:mysql TIME_WAIT
In addition it has another hundred or so entries for incoming connections.
I would appreciate any ideas or ways to diagnose this issue – I'm not a server guy primarily (I'm a programmer) and have been dropped in the deep end on this one. I'm reasonably certain the PHP webservice is not to blame – the fact that at one point the issue was resolved makes me feel that it is fine and it's a system config issue.
Best Answer