I'm experiencing a huge load on my server at the moment and I can't figure out why. When I use the 'top' command, there's hundreds of apache processes with the command "aux", but I can't find anything online that tells me what it means. The load is flapping between 50-150, which is a good 50-150 more than it usually is.
Netstat returns hundreds and hundreds of rows like this:
tcp 0 0 xxx.xxx.xxx.xxx:45216 61.155.202.205:80 CLOSE_WAIT 28863/aux
Almost all from 61.155.xxx.xxx (not sure if this is relevant information, but trying to give as much as possible).
The OS is CentOS: release 5.7 Final
We just run LAMP stack on it with about 30 websites that don't get much load (or so I thought). I've checked the logs for all of the vHosts but none seem to be getting many/any requests (not nearly enough to cause this trouble). I'm not sure if there are other logs I should be checking?
It started a couple of days ago; no changes made on the server as far as I'm aware.
Does anyone have any ideas for how I can track down what's causing the huge spike in load? Are there other commands/logs that I've missed that might be able to help me track down what the problem is?
Best Answer
That's not a connection from 61.155.xxx.xxx. That's a connection to a webserver on 61.155.202.205.
It looks very much like your webserver is making HTTP requests to other webservers on ADSL connections in China. Try a
tcpdump -n -A -s0 host 61.155.202.205to see what kind of data you are collecting. I suspect it's malicious.If it is malicious, refer to My server's been hacked! EMERGENCY.
The "many Apache processes" is most likely caused by the high load rather than causing the high load. Even at a load average of 50 I would expect to start seeing HTTP requests taking multiple seconds. At 150 it would be worse.