Nginx – bind() to 0.0.0.0:80 failed (98: Address already in use)

amazon ec2nginxUbuntu

I've got a reverse proxy with 5 server configs set up, however since the start it's always been complaining about not being able to listen on the ports I specified so the server has been unable to start.

I've tried a few things:

  • Split config into multiple files inside sites-available and creating symlinks to sites-enabled
  • Delete the listen property and allowing it to fall back to defaults
  • Changing the ports to ipv6only=on – this caused an error
  • Changing the ports to [::]:80
  • Changing the ports so that there are no conflicts, ie, 80, 81, 82, etc.

Using netstat I can see that nothing is running on the ports I need, and when I run nginx -t I get a successful output saying syntax is ok and nginx.conf test is successful.

This is my nginx output:

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:81 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:82 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:83 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:84 failed (98: Address already in use)

Does anybody know why this is caused? Thanks in advance

Edit: lsof -Pni | grep LISTEN output:

sshd      1288     root    3u  IPv4  15712      0t0  TCP *:22 (LISTEN)
sshd      1288     root    4u  IPv6  15714      0t0  TCP *:22 (LISTEN)
nginx    14025     root    6u  IPv6 852470      0t0  TCP *:80 (LISTEN)
nginx    14025     root    7u  IPv6 852471      0t0  TCP *:443 (LISTEN)
nginx    14026 www-data    6u  IPv6 852470      0t0  TCP *:80 (LISTEN)
nginx    14026 www-data    7u  IPv6 852471      0t0  TCP *:443 (LISTEN)
dnsmasq  23894  dnsmasq    5u  IPv4 650875      0t0  TCP *:53 (LISTEN)
dnsmasq  23894  dnsmasq    7u  IPv6 650877      0t0  TCP *:53 (LISTEN)

Here's my server configs:

server {
        listen 80;
        server_name hac-staging-proxy.redant.cloud;

        location / {
                proxy_pass http://195.219.8.212/;
                proxy_set_header Host www.uat2prd.halfordsautocentres.com;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_buffering off;
        }
}

server {
        listen 81;
        server_name halfords-c1-staging-proxy.redant.cloud;

        location / {
                proxy_pass http://195.219.8.206/;
                proxy_set_header Host www.c1.uat2prd.halfordsautocentres.com;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_buffering off;
        }

}

server {
        listen 82;
        server_name halfords-c2-staging-proxy.redant.cloud;

        location / {
                proxy_pass http://195.219.8.206/;
                proxy_set_header Host www.c2.uat2prd.halfordsautocentres.com;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_buffering off;
        }

}

server {
        listen 83;
        server_name halfords-staging-proxy.redant.cloud;

        proxy_redirect http://www.uat2prd.halfords.com http://halfords-staging-proxy.redant.cloud;
        proxy_redirect https://www.uat2prd.halfords.com https://halfords-staging-proxy.redant.cloud;

        location / {
                add_header Set-Cookie Experiment=FH;
                resolver 127.0.0.1;
                proxy_pass http://www.uat2prd.halfords.com;
                # sub_filter_types *;
                sub_filter 'www.uat2prd.halfords.com' 'halfords-staging-proxy.redant.cloud';
                sub_filter 'www.c1.uat2prd.halfords.com' 'halfords-c1-staging-proxy.redant.cloud';
                sub_filter 'www.c2.uat2prd.halfords.com' 'halfords-c2-staging-proxy.redant.cloud';
                sub_filter_once off;
                proxy_set_header Set-Cookie Experiment=FH;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_buffering off;
        }
}

server {
        listen 84;
        server_name halfords-staging-proxy.redant.cloud2;

        proxy_redirect https://www.uat2prd.halfords.com https://halfords-staging-proxy.redant.cloud;
        proxy_redirect http://www.uat2prd.halfords.com http://halfords-staging-proxy.redant.cloud;

        ssl_certificate           /etc/nginx/domain.cert;
        ssl_certificate_key       /etc/nginx/domain.key;
        ssl_dhparam               /etc/ssl/certs/dhparam.pem;

        ssl on;
        ssl_session_cache  builtin:1000  shared:SSL:10m;
        ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "INTENTIONALLY REMOVED"
        ssl_prefer_server_ciphers on;

        location / {
                add_header Set-Cookie Experiment=FH;
                resolver 127.0.0.1;
                proxy_pass https://www.uat2prd.halfords.com;
                # proxy_ssl_session_reuse off;
                # sub_filter_types *;
                sub_filter 'www.uat2prd.halfords.com' 'halfords-staging-proxy.redant.cloud';
                sub_filter 'www.c1.uat2prd.halfords.com' 'halfords-c1-staging-proxy.redant.cloud';
                sub_filter 'www.c2.uat2prd.halfords.com' 'halfords-c2-staging-proxy.redant.cloud';
                sub_filter_once off;
                proxy_set_header Set-Cookie Experiment=FH;
                proxy_set_header Host www.uat2prd.halfords.com;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_buffering off;
        }
}

Best Answer

Don't have 'listen *:80' at the same time as 'listen [::]:80' (and similarly for port 443).

The reason being on a dual-stack IPv4/IPv6 host, if you listen on an IPv6 TCP port X, and there is nothing listening on IPv4 TCP port X, you will get both. If you see a connection come in as ::ffff:1.2.3.4 (where 1.2.3.4 is an IPv4 address), then this is what is happening. This is called an IPv4-mapped IPv6 addresses.

This, when nginx then goes to listen on the IPv4 port 80 (0.0.0.0:80), it will run into the fact that it also has effectively done so (via [::]:80).

There are plenty of implementation-specifics around this behaviour, but unless you are versed in network programming, I won't mention them here.

Hope that helps, Cameron