Chef: how to run actions if git deploy has pulled new code

I work with multiple projects, so cjc's solution won't work for me. There's also an issue of common vs custom configuration (addresses etc are common to the company, there's also a bit of magic in the configs). The scheme I finally settled on is a bit of a hack, but it's a convenient-to-use one.

Instead of global ~/.chef, I use '.chef' subdirectory within chef-repo, which is not stored in git (it's added to .gitignore). I have also file config/knife.rb file which is checked into Git and contains shared configuration. It starts with this snippet:

root_dir = File.join(File.dirname(__FILE__), '..')
%w(knife-secrets.rb knife-local.rb).each do |conf_name|
  conf = File.join(root_dir, ".chef", conf_name)
  Kernel::load(conf) if File.exists? conf
end

This loads files .chef/knife-local.rb containing custom configuration (in basic version it's just OPSCODE_USER='username' constant that is used later on, but it can contain any knife config), and .chef/knife-secrets.rb which contains shared secrets (AWS keys etc).

Below that, there is regular knife config that uses constants defined in these files, e.g.:

client_key               "#{root_dir}/.chef/#{OPSCODE_USER}.pem"

This way, I achieve standarization of knife config across the company, which in turn means that any code snippet or knife invocation shared in a wiki will work for everyone. There is enough confusion and magic in knife itself - different configurations would only make it worse. Also, everyone gets benefit of small magic snippets, like this one to make knife ssh use login configured in user's ~/.ssh/config

There's also issue of shared secrets: chef server's validation key, AWS keys stored in knife-secrets.rb, EC2's SSH private key, encrypted data bag keys, and so on. We definitely don't want them to be stored in the repository - or, actually, anywhere where they aren't safely encrypted. So we distribute those files as a .tar.gz file, which is GPG-encrypted to everyone in the company, and shared over Dropbox.

Configuring all this is getting complicated, and I want people on team to actually use the thing, so there's the final element: rake init task which creates .chef directory, symlinks config/knife.rb there, decrypts and untars chef-secrets.tgz file, makes sure user's private Opscode Platform key is there and .chef/knife-local.rb is properly configured, symlinks knife plugins, and sets proper permissions on the directory and files inside. This task is set up so that it's safe to run it many times on already initialized repository (e.g. to update secrets or knife plugins).

There's also a helper task that repacks all the secrets, encrypts the tarball to everyone, and copies it into dropbox, to make it easier to add new employees or change secrets.

Regarding multiple environments: Chef has a feature called environments. I haven't used it yet, but it should do what you need. You can also strictly separate production environment (to avoid developers having any keys that relate in any way to production env) by having two separate Hosted Chef organizations or Chef servers. This knife.rb snippet show how to configure knife in a different way based on currently checked out branch - you can use this to set environment as well as chef server's url. There's also knife plugin called knife-flow , providing more complete, two-organization workflow.

You're doing:

export GIT_WORK_TREE=/tmp/nagiosworkdir
/usr/bin/git checkout -f $NEW_SHA1

in your hook. Although it's not touching your usual working-copy it is updating references in the git-dir (specifically the HEAD reference), as shown in your error:

...
remote: HEAD is now at da71aed... syntax error
...

Your hook is doing exit 1 to reject the update, but it's not (re)resetting the HEAD reference after failure.

I think you need to update the failure branch in your hook like so:

...
if [ "$NAGIOS_CHECK_STATUS" -ne 0 ]
   then
   echo "Your configs did not parse correctly, there was an error. Output follows."
   cat $GIT_WORK_TREE/check.out
   /usr/bin/git reset --hard $OLD_SHA1    # <-- Add This
   exit 1
else
   ...