Linux – Question on SendMail daemon actions when load average is too high

I refer to this as a "Sorry Server". If you're using Apache 2.2 you can add another host to your LB pool as a hot spare, and when the actual app servers reach capacity your balancer will direct requests to the "Sorry Server" until the application servers become available again. Here's a rough idea:

<Proxy balancer://yourapp>
    BalancerMember http://10.0.0.1:8080 retry=5 max=50
    BalancerMember http://10.0.0.2:8080 retry=5 max=50
    BalancerMember http://10.0.0.3:8080 retry=5 max=50
    BalancerMember http://10.0.0.4:8080 retry=5 max=50
    # the hot standby on server2
    BalancerMember http://10.0.0.5:80 status=+H
</Proxy>
<Location /app>
    ProxyPass           balancer://yourapp
</Location>

Actually you could alternatively set up an extra vhost on your loadbalancer machine and have it serve the "Sorry Server" page itself. Hope that helps :)

You are probably having processes on UNINTERRUPTIBLE_SLEEP state, normally they are on that state because they are waiting on something from hardware like reading from a disk. Those processes are effectively sleeping (you have 244 sleeping processes) but they enter the Load Average calculation. Check your server IO with vmstat and see if you have many D statuses on top or ps to confirm.

EDIT: Seeing your vmstat output further confirms the IO problem. Your b column under procs show an average of 100 processes on uninterruptible sleep. Your bi column (blocks read from a block device) is very high, as well as the si column (memory read from swap/disk). Finally, under the cpu header, the wa column shows that your CPU spends more than 90% waiting for IO to complete.

You must check why you are having those problems with IO. It can be lack of server's capacity, processes running wild and some other reasons, but it's definitively IO.