Nginx – After upgrade from ubuntu 14.04 to 16.04, nginx error 502: bad gateway until nginx is restarted

502-errornginxUbuntu

I have researched this problem, but in most cases the cause of the 502 error is an improperly configured nginx.conf or upstream service. I believe this is different..

As the title suggests, I upgraded ubuntu server 14.04 to 16.04. I use nginx as my web server, and am also running a java/tomcat server, setup in my nginx config as a proxy_pass.

Since the upgrade, every time the server starts up nginx displays error 502: Bad Gateway when attempting to connect to the proxy_pass site. All other sites specified in my config work as expected.

Is it possible that the order in which services are started could cause a persistent 502 error?

To resolve the issue, I must sudo systemctl restart nginx, after which, the proxy_pass service works as expected, until the next reboot.

From the error.log:

2018/01/24 11:33:20 [error] 1886#1886: *202 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: localhost, request: "GET /radio/rest2/savePlayQueue.view?u=user&p=enc:xxxxxxxx&v=2.0.0&c=DSub&id=0000&current=0000&position=0 HTTP/1.1", upstream: "http://[::1]:4040/radio/rest2/savePlayQueue.view?u=user&p=enc:xxxxxxxx&v=2.0.0&c=DSub&id=0000&current=0000&position=0", host: "www.myhostname.tld"

At the time when this error was generated by nginx, I was able to use lynx from that server to connect to localhost:4040/radio, and was served the appropriate content. Even after that the 502 error remains when connecting through nginx.

There is no defined upstream block for this, however the location block is:

location ^~ /radio/ {
        proxy_pass              http://localhost:4040;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version      1.1;
        proxy_set_header        Connection      "";
#        health_check;       # nginx: [emerg] unknown directive "health_check"
}

I don't want to have to restart nginx every time I boot up. How can I resolve this problem?

Best Answer

upstream: "http://[::1]:4040/…

Your upstream is probably only listening on IPv4 localhost (127.0.0.1:4040) whereas nginx is trying to connect to IPv6 localhost ([::1]:4040).

lynx works because it tries both.

GUESS: nginx may be failing because it tries both at the start, both fail, then it sticks with IPv6 from there on.

FIX: change upstream to use 127.0.0.1 explicitly or change the upstream to listen on both IPv4 and IPv6.