Nginx – Fix 502 Bad Gateway for Docker Swarm Reverse Proxy

centosnginxreverse-proxyselinux

I am running a docker swarm on "swarm.example.com". On the server, there's a container running that can be accessed on "swarm.example.com:3000".

On server "example.com" I'm running an nginx reverse proxy with following rules

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://swarm.example.com:3000;
    }
}

When I try to access app.example.com im getting 502 Bad Gatway error. Am I missing something ?

All servers are running CentOS 7.6

Thanks!

Best Answer

Trying to access the backend via the reverse proxy results in a 502 Bad Gateway error:

$ wget -S --spider http://nginxtest.example.com/
Spider mode enabled. Check if remote file exists.
--2019-05-18 10:12:11--  http://nginxtest.example.com/
Resolving nginxtest.example.com (nginxtest.example.com)... 192.168.15.20
Connecting to nginxtest.example.com (nginxtest.example.com)|192.168.15.20|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 502 Bad Gateway
  Server: nginx/1.12.2
  Date: Sat, 18 May 2019 08:12:11 GMT
  Content-Type: text/html
  Content-Length: 3693
  Connection: keep-alive
  ETag: "5a9e5ebd-e6d"
Remote file does not exist -- broken link!!!

This is most probably because selinux by default doesn't allow outgoing connections for webservers, because that's usually something they don't do.

You will find entries like this in /var/log/nginx/error.log:

2019/05/18 10:12:11 [crit] 1041#0: *5 connect() to 192.168.15.52:3000 failed (13: Permission denied) while connecting to upstream, client: 146.140.37.47, server: _, request: "HEAD / HTTP/1.1", upstream: "http://192.168.15.52:3000/", host: "nginxtest.example.com"

Additionally, you will find entries like this in /var/log/audit/audit.log:

type=AVC msg=audit(1558167131.910:463): avc: denied { name_connect } for pid=1041 comm="nginx" dest=3000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket permissive=0 type=SYSCALL msg=audit(1558167131.910:463): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=562671c4eef0 a2=10 a3=7ffcfbc72530 items=0 ppid=1006 pid=1041 auid=4294967295 uid=996 gid=994 euid=996 suid=996 fsuid=996 egid=994 sgid=994 fsgid=994 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)

Run the following command to allow nginx to connect to other hosts:

setsebool -P httpd_can_network_connect true

(The parameter -p makes the setting persistent. Otherwise it would be reset after the next reboot.)

And now the proxy works:

$ wget -S --spider http://nginxtest.example.com/
Spider mode enabled. Check if remote file exists.
--2019-05-18 10:15:14--  http://nginxtest.example.com/
Resolving nginxtest.example.com (nginxtest.example.com)... 192.168.15.20
Connecting to nginxtest.example.com (nginxtest.example.com)|192.168.15.20|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.12.2
  Date: Sat, 18 May 2019 08:15:15 GMT
  Content-Type: text/html
  Content-Length: 40
  Connection: keep-alive
  Last-Modified: Sat, 18 May 2019 08:08:16 GMT
  ETag: "5cdfbd70-28"
  Accept-Ranges: bytes
Length: 40 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

If you want to know more, there is a very detailed article about nginx and selinux on the nginx website.

Related Topic