Nginx Tornado Combination Causing 502 Bad Gateway Errors

nginxpythontornado

We are facing a problem with inconsistent 502 errors and tracking down the reasons has been a very frustrating exercise. We can reproduce the problem by sending several simultaneous requests quickly. The problem is that several is only in the range of 10 to 20 within a 5 seconds (not a typo). So clearly this type of load should be handled easily.

We really like the Nginx + Tornado approach but are considering going to a more traditional (e.g. threading) approach because this problem has been very difficult to solve. I was wondering if you a) know how to fix this issue and b) how we can tracked down the culprit(s).

The log files simply identify there being a connection refused. We have the same problem as this post:
https://stackoverflow.com/questions/2962439/how-do-i-debug-a-http-502-error

But there is no answer provided on how to solve the problem so I'm hoping you can help because this may be a common issue with this type of setup.

Thanks in advance,

Paul

Best Answer

By default nginx is not configured to retry connections to another upstream if one of them sends back a 502 error. You basically need to add this:

proxy_next_upstream error timeout http_502;

To your configuration. This will prevent the 502 errors from being sent directly back to the client and instead cause nginx to try and hunt for a better upstream. It will attempt all of the upstreams before failing back to the client according to this post:

http://forum.nginx.org/read.php?2,152071,152212

Here is more details on the configuration directive:

http://wiki.nginx.org/HttpProxyModule#proxy_next_upstream

Related Solutions

Nginx – 502 Bad Gateway error after failed requests using Passenger

I've finally found out what is the real problem. First, while investigating this issue I learned that Passanger log it's error messages in the nginx internal error log, not the ones in /var/log, on our server it's located at /usr/local/nginx/logs/error.log. So the actual error message I was getting is:

Exception ThreadError in application (deadlock; recursive locking) (process 6407, thread #<Thread:0x89e5ef0>):
    from /var/www/fantasy-sports/shared/bundle/ruby/1.9.1/gems/rack-1.3.2/lib/rack/lock.rb:14:in `lock'
...

There's more information about this issue there: https://github.com/rtomayko/rack-cache/issues/23

In the end I've resolved it by uncommenting the config.threadsafe! option in the environments/*.rb files.

Nginx error_page for 502 Bad Gateway errors

I did this for the whole vhost:

server {
         (...) 
         error_page 500 502 503 504 /5xx.html;
            location /5xx.html{
                    root /www/error_pages/;
         } 
}

This works perfectly for me.

Best Answer

Related Solutions

Nginx – 502 Bad Gateway error after failed requests using Passenger

Nginx error_page for 502 Bad Gateway errors

Related Topic