Nginx not spawning both ipv4 and ipv6 workers

ipv4ipv6nginx

Later edit After a LOT of troubleshooting the actual issue turned out to be a missing semicolon after the server_name directive. nginx -t -c /etc/nginx/nginx.conf wasn't catching it. Double-check for typos if you ever run into a case similar to this.

Original question follows:

I'm in the process of setting up a new server built on ubuntu 16.04 using nginx 1.10.0.

The specific issue is that while my new configuration essentially matches my old nginx configuration from an ubuntu 13.10 server using nginx 1.4.4, nginx 1.10.0 is only creating either ipv4 or ipv6 workers, but not both. This behavior is not present on the old server. Not sure what else to try at this point.

I've verified that my nginx installation was built with ipv6.

nginx version: nginx/1.10.0 (Ubuntu)
built with OpenSSL 1.0.2g-fips  1 Mar 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-stream --with-stream_ssl_module --with-mail --with-mail_ssl_module --with-threads

Below are my current configurations for the new server:

# /etc/nginx/nginx.conf
user www-data;
worker_rlimit_nofile 30000;
worker_processes 8;
pid /run/nginx.pid;

events {
  worker_connections 500000;
}

http {
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
  ssl_prefer_server_ciphers on;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

And I have a single site enabled at the moment for testing. I will have multiple vhosts configured eventually.

# /etc/nginx/sites-enabled/blog
server {
  server_name test.bloggyblog.com

  listen 80;
  listen [::]:80; 

  root /usr/local/apps/blog;
  index index.php;

  location / {
    try_files $uri $uri/ =404;
  }

  location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
  }
}

Lastly, the weird thing is whether the workers get bound to ipv4 or ipv6 entirely depends on the order in which the listen directives are placed. In the following data, I've switched the order and tried different configurations multiple times. After each change to /etc/nginx/sites-enabled/blog I did sudo service nginx stop; sudo service nginx start; sudo lsof -i; to get the data.

Also note that I changed the workers count to 8 after performing these steps. However while the number of workers increased, the same behavior was seen where all workers were either ipv4 or ipv6.

listen [::]:80; 
listen 80;
nginx    27675     root    6u  IPv4 204423      0t0  TCP *:http (LISTEN)
nginx    27676 www-data    6u  IPv4 204423      0t0  TCP *:http (LISTEN)

listen 80;
listen [::]:80;
nginx    27747     root    6u  IPv6 205134      0t0  TCP *:http (LISTEN)
nginx    27748 www-data    6u  IPv6 205134      0t0  TCP *:http (LISTEN)

listen 80;
listen [::]:80 default ipv6only=on;
nginx    27819     root    6u  IPv6 205849      0t0  TCP *:http (LISTEN)
nginx    27820 www-data    6u  IPv6 205849      0t0  TCP *:http (LISTEN)

listen 80;
listen [::]:80 default ipv6only=off;
nginx    27885     root    6u  IPv6 206495      0t0  TCP *:http (LISTEN)
nginx    27886 www-data    6u  IPv6 206495      0t0  TCP *:http (LISTEN)

listen 80;
listen [::]:80 default;
nginx    27953     root    6u  IPv6 207184      0t0  TCP *:http (LISTEN)
nginx    27954 www-data    6u  IPv6 207184      0t0  TCP *:http (LISTEN)

Best Answer

It looks like your default setting for ipv6only is different. On most operating systems you can create IPv6 sockets that also accept IPv4 connections so that you only need one socket (one listen directive).

It seems that on your old server it used ipv6only=on so you created both an IPv4 and an IPv6 socket. On your new server the default is ipv6only=off, which makes the IPv6 socket listen on IPv4 as well. And that creates a conflict with the separate IPv4 socket. If you remove the IPv4 listen line it will probably just work with both protocols.

To make things predictable it's probably best to explicitly set the ipv6only flag, and use one of these:

listen 80;
listen [::]:80 ipv6only=on;

or

listen [::]:80 ipv6only=off;