Nginx – Mercurial not receiving push

mercurialnginx

I have a mercurial web-frontend (hgwebdir.cgi) installed on a server, and an installation of nginx was installed in front of it as a reverse proxy to the web-frontend as my friend suggested. However, whenever a large changeset is pushed (via a script), it would fail. I found an issue ticket @google-code that describe similar problem, and there is a solution that says (#39)

So the server side answer is: don't
send the 401 back early. Be as
slow/dumb as 'hg serve' and make the
hg client send the bundle twice.

How do I do that? My current nginx config

location /repo/testdomain.com {
    rewrite ^(.*) http://bpj.kkr.gov.my$1/hgwebdir.cgi;
}
location /repo/testdomain.com/ {
    rewrite ^(.*) http://bpj.kkr.gov.my$1hgwebdir.cgi;
}
location /repo/testdomain.com/hgwebdir.cgi {
    proxy_pass http://localhost:81/repo/testdomain.com/hgwebdir.cgi;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_buffering on;
    client_max_body_size 4096M;
    proxy_read_timeout 30000;
    proxy_send_timeout 30000;
}

From the access log we keep seeing 408 entries

incoming.ip.address - - [18/Nov/2009:08:29:31 +0800] "POST /repo/testdomain.com/hgwebdir.cgi/example_repository?cmd=unbundle&heads=73121b2b6159afc47cc3a028060902883d5b1e74 HTTP/1.1" 408 0 "-" "mercurial/proto-1.0"
incoming.ip.address - - [18/Nov/2009:08:37:14 +0800] "POST /repo/testdomain.com/hgwebdir.cgi/example_repository?cmd=unbundle&heads=73121b2b6159afc47cc3a028060902883d5b1e74 HTTP/1.1" 408 0 "-" "mercurial/proto-1.0"

Is there anything else I can do on the server because solving it on the server side is preferable :/

Further Findings

Bitbucket seems to have this solved ( Check liquidhg bitbucket project and the Diagnosis wiki page ) on the server side, can't find the config anywhere though :/

  1. What happens next varies depending
    on your server. Some servers refuse
    the BODY, simplying closing the pipe
    from the client and causing
    Mercurial to fail. Some, like Apache
    (at least the way I configure it,
    and that could be part of the
    problem) and nginx (they way
    BitBucket.org configures it), accept
    the BODY, though it may take a few
    retries. Bottom line: if Mercurial
    doesn't fail the push, it sends the
    changeset data at least once to a
    server that has already told it it
    lacks credentials (more on this at
    Blame).
  2. Assuming Mercurial is still running,
    it resends the "unbundle" request
    and data, this time with
    authentication.
  3. Finally, Apache accepts the data
    successfully. Nginx, OTOH, at least
    under BitBucket's configuration,
    seems to reassemble the previous
    body (the one that lacked
    authentication) and somehow keep
    Mercurial from re-sending the whole
    body.

Best Answer

I spent ages trying to get hgwebdir working with nginx and it is a bit of a hassle, because hgwebdir won't send the request to the browser for authentication. In the end I solved it like this:

server {
        listen 80;
        listen 10240;
        server_name code.zofrex.com;
        access_log /home/zofrex/websites/code.zofrex.com/logs/access.log;
        error_log /home/zofrex/websites/code.zofrex.com/logs/errors.log;

        location / {
                limit_except GET {
                        proxy_pass http://localhost:81;
                }
                fastcgi_pass 127.0.0.1:10001;
                include fastcgi_params;
        }

        location ~ ^/(HGBot|ZeroBotAHD|zoebot|FZeroZBot|RDPrototype|RSPS3000|zerobot|rmi)  {
                proxy_pass http://localhost:81;
        }

        include defaults;
}

server {
        listen 81;

        access_log /home/zofrex/websites/code.zofrex.com/logs/access_secure.log;
        error_log /home/zofrex/websites/code.zofrex.com/logs/errors_secure.log;

        location / {
                auth_basic           "Restricted";
                auth_basic_user_file /home/zofrex/passwords;
                fastcgi_pass 127.0.0.1:10001;
                include fastcgi_params;
        }

        include defaults;
}

This is the only way you can do it, really, as limit_except can't be combined with fastcgi_pass - in fact no conditionals can, but they be combined with proxy_pass. No idea why.

So how does this work? If a GET request comes in, it's handed off to hgwebdir straight away. If a POST (i.e. a push) comes in, the request upstreams to the proxy on port 81 which always prompts for basic auth, thus ensuring it happens.

Why the location with the huge regex? Those are protected projects that I want to require auth for even read access.

Why the listening on port 10240? I can't remember why that's there.

Oh, and I'm using wsgi with spawn-fcgi, which is why it's fastcgi_pass. This exact config probably won't be right for you but hopefully the same trick will solve your pushing woes.

Related Topic