Linux – Too many httpd processes running and CentOS server freeze

apache-2.2centoshttpdlinuxMySQL

I am running a high traffic website on a Dedicated-Virtual 16GB RAM CentOS server hosted by Media Temple.
Very often the server dies because of high CPU usage (%1500)
When i check the running processes using "top" command i see a too many httpd processes running which i think are causing the high usage of CPU.
I have tried to tune mysql, tune apache, my.cnf, httpd.cnf, removed Apache modules that i do not use but that does not solve the problem.
Please let me know what values do you need to know in my server config in order to help me diagnosis the problem.
Thanks in advance.

Best Answer

what does tail -f /var/log/messages say when you experience high system loads?

Are your processes very I/O or very CPU intensive?

One observation:

Looks like you have 18G physical memory, but zero swap space..., e.g. your top command shows "Swap: 0k total" ... that means that you do not have any Swap Space configured.

In general, you should always have a sufficient swap space on a UNIX system! Swap-size = 1...2-times RAM size is a good idea. Using a fast partition is a good idea. Really bad things happen if your UNIX system runs out of RAM and doesn't have Swap .. processes just die inexplicably.. that is a very bad thing! especially in production. Disk is cheap! add a generous swap partition! :-) or worst case you can also create a swap-file later.

On any UNIX system, you do need swap space -- because that is where processes live while they are not scheduled on one of the CPUs. If you don't have swap space, really bad things will happen, because your system will run out of places (RAM) where to create new processes -- it can not put them in swap, because there is none, so you will see extremely high loads, the system freezing up, and processes die seemingly inexplicably.

To check your memory and swap space in use, you can run free -k. You can also check the swap configuration by running swapon -s .. this will not show any output if no swap is configured.

As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap -- don't be stingy with the Swap, because "disk is dirt-cheap"!

Disclaimer: there are a few exceptions when you can live without swap, but on the other hand, configuring and not needing it does not hurt! :)

Two choices for creating swap space:

  • either create a swap partition, preferably on your fastest disk partition (best choice; or add a flash drive to your physical server)

  • or create a swap file (if you don't have disk space left to partition; this is also a good option if you later find out that you need more swap than you anticipated)

...then format that swap partition or swap file to make it usable for swapping with the mkswap command.

Once you enable the swap space with sudo swapon -a, you should see it listed in top and with swapon -s , and your system should behave much more nicely..

Articles about Swap Space:

https://www.linux.com/news/software/applications/8208-all-about-linux-swap-space

https://www.linux.com/learn/tutorials/442430-increase-your-available-swap-space-with-a-swap-file

http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

http://lissot.net/partition/partition-08.html

http://tldp.org/LDP/sag/html/swap-space.html

Some pointers:

man -k swap

man mkswap

man fstab , vim /etc/fstab


EDIT

as this question was migrated to ServerFault .. I'll add a comment here, because ppl are complaining that I quote the old rule: " 1..2-times RAM size == Swap size "

A 2TB SATA drive costs around $70 these days, that means 20GB costs you about 70 cents -- it's dirt cheap! I know 20GB sounds a lot, but think of it as a very inexpensive "worst case insurance"!

Somebody said: "if your system filled up 18GB and is swapping onto 20GB, then you have bigger problems"

Certainly true, but having generous Swap Space if unexpected things go wrong, will save your server and the processes on it from crashing completely! You might still be able to log in, examine the system while it's still running (slowly), and repair things (albeit slowly), and not have to reboot.. On a production server, there is really no reason not to use generous swap.

If I had a server with 18GB RAM and it's running MongoDB for example (which is very RAM intensive) .. I would certainly configure very generous Swap Space, perhaps even 40..60GB ... just in case something unexpected goes wrong..