We have an application that is running on a few (5 or so and will grow) boxes. The hardware is identical in all the machines, and ideally the software would be as well.
I have been managing them by hand up until now, and don't want to anymore (static ip addresses, disabling all necessary services, installing required packages…) . Can anyone balance the pros and cons of the following options, or suggest something more intelligent?
1: Individually install centos on all the boxes and manage the configs with chef/cfengine/puppet.
This would be good, as I have wanted an excuse to learn to use one of applications, but I don't know if this is actually the best solution.
2: Make one box perfect and image it. Serve the image over PXE and whenever I want to make modifications, I can just reboot the boxes from a new image.
How do cluster guys normally handle things like having mac addresses in the /etc/sysconfig/network-scripts/ifcfg* files? We use infiniband as well, and it also refuses to start if the hwaddr is wrong. Can these be correctly generated at boot?
I'm leaning towards the PXE solution, but I think monitoring with munin or nagios will be a little more complicated with this. Anyone have experience with this type of problem?
All the servers have SSDs in them and are fast and powerful.
Thanks,
matt.
Best Answer
Your cluster sounds more like an HPC cluster than an OLTP one like mine, but I think the setup I'm using would work for you too. I call it the "mpdehaan trifecta" because thats the ircnick of the guy who wrote or manages the three tools involved.
1.) Cobbler for base-build provisioning. Cobbler is a project that aims to be the intersection of your kickstart, pxe, yum-repo, dhcp, dns, etc systems. Its by far the easiest way to get a kickstart setup up and running, and you can grow into the other features as needed.
2.) Puppet for configuration management. Ideally your cobbler built hosts are very barebones configs that know just enough to phone home to your puppet server on startup. Puppet will then apply your configuration settings and keep them consistent across your environment in perpetuity.
3.) Func for ad-hoc commands to multiple machines in parallel. For instance "deploy a new svn checkout of the code and restart apache". Its pretty easy to just use func to call the same bash command on a group of servers much like cluster-ssh. If you really want to get into it you can write your own modules for it with some really simple python.
All three of these tools have good wiki's and active irc channels for help on freenode.