Linux – Kill processes if high load average

debiankernelkill-processlinuxwatchdog

Not so far ago LA on my server raised to 400 and I couldn't even login to server using ssh. Does exists any software, that can prevent this situations by automatically killing processes that making huge load on server?

PS. Debian 6.0.5

Best Answer

You can use a watchdog like Monit to watch over the processes you care about, and restart them if they consume excess resources.

Something like this would be used to monitor Apache:

 check process apache with pidfile /var/run/httpd.pid
       start program = "/etc/init.d/httpd start"
       stop program  = "/etc/init.d/httpd stop"
       if cpu > 40% for 2 cycles then alert
       if totalcpu > 60% for 2 cycles then alert
       if totalcpu > 80% for 5 cycles then restart
       if mem > 100 MB for 5 cycles then stop
       if loadavg(5min) greater than 10.0 for 8 cycles then stop

So, if the cpu% for the Apache process or any of its children are over 40%, send an alert. If it's above 80%, do a restart of Apache.

Monit will also start up Apache if it's not running for some reason, which is a reasonable way to keep critical services up (if you don't have something like Upstart available).

This assumes that you have a set of processes that you can target for this sort of monitoring. Presumably, you suspect a particular application may be a problem.

Related Topic