Linux – Heavy load (100% ni) and CPU load average is 42.61, 42.24, 42.23

I would install atop from EPEL repository. Atop should help you show diagnose what is causing the %sys activity.

Atop also has a atop -r feature that will allow you to step through logs backward and fordward in time using t/T keys.

Also take a look at /proc/interrupts and through your /var/log/httpd/logs and sort those by ip to see if there is any suspect IP causing abnormal amounts of httpd traffic.

I would cron a cat /proc/interrupts to a log file. Look for high deltas in the interupts.

First off, you have run out of memory.

Your VPS has 2GB of RAM and 4GB of swap. The 2GB of RAM is all used up, and the swap is maybe 3/4 full. At this point, on any system I manage, there is an alarm that repeats every five minutes until someone acknowledges the problem, it is that serious.

So you need to figure out what's using all the memory. Unfortunately this may be difficult.

I can see that a single Apache process alone is using nearly 1GB of memory. This is obviously an outlier, since most of the others use less than 22MB each.

The reason this is difficult is that Apache lets you embed languages such as Perl or PHP directly into the process, and it is almost certainly such a script that has run away and eaten up all your RAM.

All your system can do at this point is thrash in and out of swap (which is exactly what iotop shows it doing) until someone shows it mercy with a well placed kill -9.

So your next step is to try to figure out what script went haywire and ate all your RAM.