nginx http2 Module Not Working in Chrome on Ubuntu 18.04 – Fix

google-chromehttp2nginx

I'm trying to setup the http2 module in nginx on a fresh Ubuntu 18.04 vps.
Http2 works fine in Firefox (I checked the response headers).
Some third party websites indicate that my http2 configuration is ok:

https://tools.keycdn.com/http2-test
https://www.ssllabs.com/ssltest/analyze.html?d=example.com (I've got A+ on this site)

But when I try to open the website on Chrome, it won't even open (This site can’t be reached), although in the access log (/var/log/nginx/access.log) I get an "HTTP/2.0" 200 response from nginx, indicating that nginx behaves as if everything was ok.

From curl, I've this answer:

$ curl -I https://example.com/
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Sun, 19 May 2019 13:20:33 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options:: nosniff
X-XSS-Protection: 1;mode=block
Strict-Transport-Security: max-age=15768000

So, not http2, but at least I've a fallback to http1.1 with curl

So, apparently http2 works, but not in Chrome (blank page), and I don't know what's the problem.

If forgot to say that when I browse the page with Chrome, I don't get any error in the error log (/var/log/nginx/error.log), however, from time to time, an ssl related error appears (here is a sample):

2019/05/19 14:53:33 [crit] 11931#11931: *639 SSL_do_handshake() failed (SSL: error:1417D102:SSL routines:tls_process_client_hello:unsupported protocol) while SSL handshaking, client: 64.41.200.103, server: 0.0.0.0:443
2019/05/19 14:56:34 [notice] 12616#12616: signal process started
2019/05/19 15:06:52 [notice] 12638#12638: signal process started
2019/05/19 15:08:48 [notice] 12647#12647: signal process started
2019/05/19 15:10:02 [notice] 12724#12724: signal process started
2019/05/19 15:10:07 [crit] 12725#12725: *706 SSL_do_handshake() failed (SSL: error:1417D102:SSL routines:tls_process_client_hello:unsupported protocol) while SSL handshaking, client: 91.107.64.185, server: 0.0.0.0:443
2019/05/19 15:10:07 [crit] 12725#12725: *707 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 91.107.64.185, server: 0.0.0.0:443
2019/05/19 15:25:22 [crit] 12725#12725: *724 SSL_do_handshake() failed (SSL: error:1417D102:SSL routines:tls_process_client_hello:unsupported protocol) while SSL handshaking, client: 80.82.77.139, server: 0.0.0.0:443
2019/05/19 15:25:23 [crit] 12725#12725: *726 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 80.82.77.139, server: 0.0.0.0:443
2019/05/19 15:25:23 [crit] 12725#12725: *727 SSL_do_handshake() failed (SSL: error:1417D102:SSL routines:tls_process_client_hello:unsupported protocol) while SSL handshaking, client: 80.82.77.139, server: 0.0.0.0:443
2019/05/19 15:25:27 [error] 12725#12725: *735 open() "/home/me/example.com/www/sitemap.xml" failed (2: No such file or directory), client: 80.82.77.139, server: example.com, request: "GET /sitemap.xml HTTP/1.1", host: "123.456.789.123"
2019/05/19 15:37:22 [notice] 12745#12745: signal process started

Again, the errors are not provoked by me refreshing the page in Chrome (they are probably bots or other people accessing the website).

Here is my configuration:

In /etc/nginx/nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 1024;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;


        ##
        # Security settings
        ##

        # Avoid iframes for clickjacking attacks
        # add_header X-Frame-Options DENY;
        add_header X-Frame-Options SAMEORIGIN;

        # Avoid mime type sniffing
        add_header X-Content-Type-Options: nosniff;

        # Avoid certain type of XSS attacks (if browser understands it)
        add_header X-XSS-Protection "1;mode=block";

        ##
        # DoS and DDoS Protection Settings
        ##

        #Define limit connection zone called conn_limit_per_ip with memory size 15m based on the unique IP
        limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:15m;

        #Define limit request to 40/sec in zone called req_limit_per_ip memory size 15m based on IP
        limit_req_zone $binary_remote_addr zone=req_limit_per_ip:15m rate=40r/s;

        #Using the zone called conn_limit_per_ip with max 40 connections per IP
        limit_conn conn_limit_per_ip 40;

        #Using the zone req_limit_per_ip with an exceed queue of size 40 without delay for the 40 additonal
        limit_req zone=req_limit_per_ip burst=40 nodelay;

        #Do not wait for the client body or headers more than 5s (avoid slowloris attack)
        client_body_timeout 5s;
        client_header_timeout 5s;
        send_timeout 5s;

        #Establishing body and headers max size to avoid overloading the server I/O
        client_body_buffer_size 256k;
        client_header_buffer_size 2k;
        client_max_body_size 3m;
        large_client_header_buffers 2 2k;


        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        gzip_vary on;
        # gzip_proxied any;
        gzip_comp_level 4;
        gzip_buffers 16 8k;
        gzip_http_version 1.1;
        gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

And in (symlink) /etc/nginx/sites-enabled/example.com:

server {

        listen 80 default_server;
        listen [::]:80 default_server;

        # Prevent site from being displayed under a different domain (by creating another domain pointing to our server)
        return 301 https://example.com;
}


server {
    listen 80;
    listen [::]:80;

    server_name mysite.com;

    # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
    return 301 https://example.com$request_uri;
}


server {

        listen 443 ssl http2;
        listen [::]:443 ssl http2;



        # certs sent to the client in SERVER HELLO are concatenated in ssl_certificate
        ssl_certificate /etc/nginx/certs/mysite.com/fullchain.pem;
        ssl_certificate_key /etc/nginx/certs/mysite.com/key.pem;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;

        # modern configuration. tweak to your needs.
        ssl_protocols TLSv1.2;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;


        ##
        # Security settings
        ##

        # Avoid iframes for clickjacking attacks
        # add_header X-Frame-Options DENY;
        add_header X-Frame-Options SAMEORIGIN;

        # Avoid mime type sniffing
        add_header X-Content-Type-Options: nosniff;

        # Avoid certain type of XSS attacks (if browser understands it)
        add_header X-XSS-Protection "1;mode=block";

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;


        # OCSP Stapling ---
        # fetch OCSP records from URL in ssl_certificate and cache them
        ssl_stapling on;
        ssl_stapling_verify on;

        ## verify chain of trust of OCSP response using Root CA and Intermediate certs
        ssl_trusted_certificate /etc/nginx/certs/mysite.com/ca.pem;



        root /home/me/example.com/www;

        # Add index.php to the list if you are using PHP
        index index.php index.html;

        server_name mysite.com;
        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                #try_files $uri $uri/ =404;
                try_files $uri $uri/ /index.php?$query_string;
        }


        # pass PHP scripts to FastCGI server
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;

                # With php-fpm (or other unix sockets):
                fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;

        }

        location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
                access_log        off;
                add_header Cache-Control public;
                add_header Pragma public;
                add_header Vary Accept-Encoding;
                expires 365d;

        }


        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
                deny all;
        }

        location ~ /\.git {
                deny all;
        }
}

Here is the output of:

root@example.com:/etc/nginx# nginx -V
nginx version: nginx/1.14.0 (Ubuntu)
built with OpenSSL 1.1.0g  2 Nov 2017
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-FIJPpj/nginx-1.14.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

I've created the certificate using Certbot, and it worked fine.

What am I missing?

Best Answer

I suspect it’s this:

# Avoid mime type sniffing
add_header X-Content-Type-Options: nosniff;

HTTP/2 is a bit more strict about HTTP Headers than HTTP/1.1 was and in this header, unlike your others, you have included a colon in the header name which is a mistake. This leads to a double colon in the output:

X-Frame-Options: SAMEORIGIN
X-Content-Type-Options:: nosniff
X-XSS-Protection: 1;mode=block

Chrome rejects invalid headers like this. See this article on how to debug this to see if it was that: https://www.michalspacek.com/chrome-err_spdy_protocol_error-and-an-invalid-http-header But suspect when you correct that it will work.

Related Solutions

Nginx/Apache: set HSTS only if X-Forwarded-Proto is https

You could have multiple server blocks. So just add new server block for domains that need HSTS.

server {
    listen xx.xx.xxx.xxx:443 ssl default_server;

    # all ssl stuff
    # and other directives
}

server {
    listen xx.xx.xxx.xxx:443 ssl;
    server_name example.com other.example.com;

    # all ssl stuff
    # and other directives with HSTS enabled
}

Here first block will handle all https connections except example.com and other.example.com.

And you don't need ssl on directive if you have ssl flag in listen.

EDIT

There is another solution with only one server block:

map $scheme:$host $hsts_header {
    default "";
    https:example.com "max-age=31536000";
    https:other.example.com "max-age=31536000";
}

server {
    server_tokens off;

    listen xx.xx.xxx.xxx:80;
    listen xx.xx.xxx.xxx:443 ssl;

    ssl_certificate /etc/ssl/foo.crt;
    ssl_certificate_key /etc/ssl/private/foo.key;

    ssl_session_timeout 10m;

    # ... other ssl stuff

    location / {
        proxy_pass http://127.0.0.1:81;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header Host $host;
        add_header X-XSS-Protection "1; mode=block";
        add_header Strict-Transport-Security $hsts_header;
    }
}

We use map to define HSTS header value and use the fact, than nginx will not add header with empty value.

Nginx – OCSP validation – unable to get local issuer certificate

Those two errors was unrelated although the error message was same.

[...]SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 [...] Start Time: 1411583991 Timeout : 300 (sec) Verify return code: 20 (unable to get local issuer certificate)

Above error was issued openssl_client command. As explained by Florian Heigl, you get this error because the openssl_client need the Globalsign Root cert in /etc/ssl/certs.

OCSP_basic_verify() failed (SSL: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:Verify error:unable to get local issuer certificate) while requesting certificate status, responder: gv.symcd.com

For this error, it was issued by nginx ocsp routine, especially when you add ssl_stapling_verify on; line in nginx.conf.

Here some excerpt from the documentation of ssl_stapling_verify to explain why it throws the error

Syntax: ssl_stapling_verify on | off;

Enables or disables verification of OCSP responses by the server.

For verification to work, the certificate of the server certificate issuer, the root certificate, and all intermediate certificates should be configured as trusted using the ssl_trusted_certificate directive.

In other words, you need provide (2) Intermediate CA Bundle (RapidSSL SHA256 CA - G3) and (3) Intermediate CA Bundle (GeoTrust Global CA) to ssl_trusted_certificate directive.

cat GeoTrustGlobalCA.crt rapidsslG3.crt > ocsp-chain.crt

and add ocsp-chain.crt to ssl_trusted_certificate directive.

Best Answer

Related Solutions

Nginx/Apache: set HSTS only if X-Forwarded-Proto is https

Nginx – OCSP validation – unable to get local issuer certificate

Related Topic