Long story short: Big old corporation, lots of UNIX/Linux servers.
I inherited responsibility for a bunch of scripts that that left a few years ago. One of them was a script that would be run every $X amount of months to globally update the root password across all of our servers.
The script is a mess of Shell Script and Expect, and it works on the SSH trust that is set up between all of our servers and a central command-and-control server.
The problem is, the script is a giant mess. The Expect commands are trying to account for every possible version of "passwd" that exists on any UNIX/Linux box out there – and they vary quite a bit.
As we're expanding and upgrading a lot of our infrastructure, the script is getting really unmanageable.
My question is: Is there a better way to do this? Assuming there's an already established SSH trust, what's the best way to change the root password on 3000+ servers at the same time?
Best Answer
Use Puppet.
Puppet is very flexible, easy to maintain and uses SSL. Might sound a bit overkill and you'll have to put some extra effort to build Puppet system up.
But. Most probably this is not the last mass-update you'll be doing to these machines. Puppet will and does save you a lot of time when actual whatever mass-update procedure begins and scripts are very readable/reusable.
At least this worked for me few years back and still I'm able to re-use some of those Puppet recipes (aka scripts). I have also used it in a bit smaller environments just make sure that every machine is actually having a known state.
I have proven it many times (at many companies) that all customised deployment scripts become pain in the butt after a while or when that next guy steps in. And as long as you carry a cellphone, old scripts will and do haunt you.
If you think that this actually sounds good, here's a great Puppet tutorial with virtual environment included to get you started.