I'm currently administrating a Gentoo server hosting a few dozen websites for friends and people with common interests. I may be moving to a new server soon, and with that comes the possibility to switch to another OS. I've enjoyed Gentoo's ability to let me tinker with USE flags and automatically patch upstream source code right before it's compiled and installed, but it does have some quirks (tail and cron having subtly broken behavior, deadlocks in kernel cgroups code, files being installed with randomly broken permissions), and the upkeep can be considerable (I can't seem to update perl without breaking half the system).
I conjecture that for a shared hosting system, two things are of most importance: security and stability. While I generally trust my users not to do something intentionally malicious, I can't trust them to never allow their accounts to be compromised (e.g. by installing and not updating a web app for which a vulnerability has been found). A user should not be able to access other users' data, interfere with the accessibility of other websites, or trash the server.
CentOS seems to be very popular on servers, although my personal limited experience with rpm/yum was rather discouraging (took me an hour to install one package). I heard nice things about Ubuntu Server as well. Or should I just stick to what I'm familiar with, and hope that my Gentoo problems are simply a result of misconfiguration or accumulated cruft that won't manifest in a clean install? (CentOS and Ubuntu Server come with SELinux/AppArmor, Gentoo has the Hardened Gentoo project.)
There's also the question of Apache configuration… running mod_php with a standard MPM is out of the question, as it'll allow scripts access to all users' data. CGI is a lot slower. I'm using the itk MPM right now, and it seems to work okay, but I wonder if there's a better solution – how do the big shared hosting providers (DreamHost etc.) do it?
Best Answer
CentOS is very popular because of the upstream support by a lot of enterprise vendors and the rock-solid nature of the system. A move to any system like this is going to make you feel like you are trapped in a cage. You're clearly someone who enjoys tweaking (which I can agree with!). That being said, I think for a shared hosting system CentOS or Ubuntu Server is definitely a better approach as it simplifies management a lot.
In either case you'll need to invest some time (far less than the time required to learn Gentoo, IMHO) in learning it's tools and quirks. Once you get used to using apt or yum they're both quite tolerable for simple software installation.
In either case if your focus is PHP/Apache webhosting you could simply build a system using CentOS or Ubuntu and custom build your own Apache/PHP to meet your exact requirements. If I were you I'd spend some time seeing if the system packages offer anything like the configuration you're looking for (I know they both offer various MPM modules, for example).
In RedHat EL5 at least the mod_php default installation runs all PHP scripts as the www-data user, which would not allow access to all users' files (there may be something I'm missing here).