GitLab post-receive hook not firing

gitgitlab

Apologies if this isn't the right stackexchange.

I have a GitLab install. It was installed over the top of a gitolite install that was only a few days old, and I assume this non-standard setup is at the root of my problem, but I cannot pin it down.

The problem is straightforward: post-receive hooks are not fired. This prevents 'project activity' appearing in GitLab. The problem looks like:

$ git push
#...
error: cannot run hooks/post-receive: No such file or directory

Hook Exists

The post-receive hook/symlink exists and is executable:

-rwxr-xr-x 1 git git 470 Oct  3  2012 .gitolite/hooks/common/post-receive
lrwxrwxrwx 1 git git  45 Oct  3  2012 repositories/project.git/hooks/post-receive -> /home/git/.gitolite/hooks/common/post-receive

It's Executable By GitLab

The gitlab user can execute the script (I have removed the /dev/null redirect and fed in blank input to get an 'OK' as output):

sudo su - gitlab -c /home/git/.gitolite/hooks/common/post-receive

OK

GitLab Can Find It

GitLab is looking for hooks in the correct location:

$ grep hooks /srv/gitlab/gitlab/config/gitlab.yml
  hooks_path: /home/git/.gitolite/hooks/

and

$ bundle exec rake gitlab:app:status RAILS_ENV=production
# ...
/home/git/.gitolite/hooks/common/post-receive exists? ............YES

GitLab Runs As Correct User

The GitLab software is being run by the gitlab user:

$ ps U gitlab
  PID TTY      STAT   TIME COMMAND
 3650 ?        Sl     0:59 unicorn_rails master -c /srv/gitlab/gitlab/config/unicorn.rb -E production -D
 3671 ?        Sl     0:43 unicorn_rails worker[0] -c /srv/gitlab/gitlab/config/unicorn.rb -E production -D
 3674 ?        Sl     0:42 unicorn_rails worker[1] -c /srv/gitlab/gitlab/config/unicorn.rb -E production -D
# ...

Environment

The env -i line in the hook is commonly cited as an issue. I think that would occur after this problem, but for completeness, redis-cli is found OK:

$ env -i redis-cli
redis>

I've run out of debugging ideas on this one. Does anybody have any suggestions?

Best Answer

OK, figured this one out. It's remotely possible this situation will apply to other people, so I've documented it.

Our infrastructure has all external SSH connections landing on a particular machine1. This isn't the machine running gitlab. This wasn't obvious because gitolite+gitlab works despite this. Our homedirs are NFS mounts and both the SSH machine and the gitlab machine see the gitolite files as 'local'. The one problem with this 'distributed' gitolite+gitlab is that the post-receive hook tries to call redis-cli, a binary which doesn't exist on the machine running the hook.

To solve the problem, I installed the redis-server package on the machine accepting SSH connections (unfortunately the redis-cli binary doesn't appear to be available separately). I then modified the redis line in /home/git/.gitolite/hooks/common/post-receive to the following:

redis-cli -h hostname rpush "resque:queue:post_receive" "{\"class\":\"PostReceive\",\"args\":[\"$reponame\",\"$oldrev\",\"$newrev\",\"$ref\",\"$GL_USER\"]}" > /dev/null 2>&1

The -h hostname switch is the only addition. This causes the redis rpush command to be sent to redis on the correct machine.

The missing binary error didn't surface due to the > /dev/null 2>&1 redirect in the hook script. Although I did remove it during debugging, I must have restored it before I attempted to trigger the hook via a git push, which otherwise would have echoed the error.

My assumption about my slightly nonstandard gitlab installation turned out to be incorrect. This problem would have occured regardless; it's due to a networking quirk. Anyhow, I hope this will be of use to somebody.

  1. In the case of git, we use external URLs even within the local network so that project URLs are consistent regardless of network environment.
Related Topic