Linux – Nginx reverse proxy SSL handshake error

linuxnginxPROXYreverse-proxyUbuntu

What I have:

  • Domain name *.posmete.com which points to Host server.
  • Host server running ubuntu 18.04 (Google cloud instance)
  • (inside host server) LXC Container 1 named "proxy" running ubuntu 18.04 and nginx 1.17 as a reverse proxy
  • (inside host server) LXC Container 2 named "web1" running ubuntu 18.04 and nginx 1.17 and php7.4-fpm as well as MySQL 8.0

Two lxc proxy devices with proxy_protocol=true

igorkovalenko@xprs-tst:~$ lxc config device show proxy
myport80:
  connect: tcp:127.0.0.1:80
  listen: tcp:0.0.0.0:80
  proxy_protocol: "true"
  type: proxy
myport443:
  connect: tcp:127.0.0.1:443
  listen: tcp:0.0.0.0:443
  proxy_protocol: "true"
  type: proxy

My containers list:

igorkovalenko@xprs-tst:~$ lxc list
+-------+---------+----------------------+----------------------------------------------+-----------+-----------+
| NAME  |  STATE  |         IPV4         |                     IPV6                     |   TYPE    | SNAPSHOTS |
+-------+---------+----------------------+----------------------------------------------+-----------+-----------+
| proxy | RUNNING | 10.130.126.23 (eth0) | fd42:f63:4839:fedc:216:3eff:fe26:523b (eth0) | CONTAINER | 0         |
+-------+---------+----------------------+----------------------------------------------+-----------+-----------+
| web1  | RUNNING | 10.130.126.41 (eth0) | fd42:f63:4839:fedc:216:3eff:fe18:2fa3 (eth0) | CONTAINER | 0         |
+-------+---------+----------------------+----------------------------------------------+-----------+-----------+

My Nginx Reverse Proxy Config (proxy container)

proxy_redirect              off;
proxy_set_header            Host            $http_host;
proxy_set_header            X-Real-IP       $remote_addr;
proxy_set_header            X-Forwared-For  $proxy_add_x_forwarded_for;
proxy_ssl_session_reuse on;
proxy_ssl_server_name on;

upstream backend {
      server web1:443;
}

server {

        server_name web1.posmete.com;

        location / {
#               proxy_set_header Host $host;
#               proxy_set_header X-Real-IP $remote_addr;
#               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#               proxy_set_header X-Forwarded-Proto https;
#               proxy_ssl_session_reuse off;
#               proxy_ssl_server_name on;
#               proxy_ssl_protocols TLSv1.2;
#               proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
#               proxy_ssl_certificate  /etc/letsencrypt/live/web1.posmete.com/fullchain.pem;
#               proxy_ssl_certificate_key /etc/letsencrypt/live/web1.posmete.com/privkey.pem;
                proxy_pass https://backend;
#               proxy_redirect http:// https://;

        }

        real_ip_header proxy_protocol;
        set_real_ip_from 127.0.0.1;

    listen [::]:443 ssl proxy_protocol; # managed by Certbot
    listen 443 ssl proxy_protocol; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/web1.posmete.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/web1.posmete.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
    if ($host = web1.posmete.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80 proxy_protocol;
        listen [::]:80 proxy_protocol;

        server_name web1.posmete.com;
    return 404; # managed by Certbot


}

My Nginx Web Server Config (web1 container)

server {
#    listen       80;
    listen 443;

    server_name  web1.posmete.com;

    root   /usr/share/nginx/html;
    index  index.php;

    location = /50x.html {
        root   /usr/share/nginx/html;
    }


    location / {
                try_files $uri $uri/ /index.php?$args;
        }

       location = /favicon.ico {
                log_not_found off;
                access_log off;
        }

        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }

    location ~* \.php$ {
        if ($uri !~ "^/uploads/") {
            fastcgi_pass unix:/run/php/php7.4-fpm.sock;
        }
        include                      fastcgi_params;
        fastcgi_intercept_errors     on;
        fastcgi_param                SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param                SCRIPT_NAME $fastcgi_script_name;
    }

     location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
            expires max;
            log_not_found off;
     }

    location ~ /\.ht {
        deny  all;
    }
}

What I'm trying to achieve:

I'm trying to install WordPress inside lxc container "web1" through WordPress installation script through https connection.

What problems I've faced:

web1.posmete.com 502 Bad Gateway

The errors from proxy server log:

2020/03/24 15:20:44 [error] 4199#4199: *42 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 169.197.108.42, server: web1.posmete.com, request: "GET /remote/login HTTP/1.1", upstream: "https://10.130.126.41:443/remote/login", host: "35.198.175.173"
2020/03/24 15:20:44 [warn] 4199#4199: *42 upstream server temporarily disabled while SSL handshaking to upstream, client: 169.197.108.42, server: web1.posmete.com, request: "GET /remote/login HTTP/1.1", upstream: "https://10.130.126.41:443/remote/login", host: "35.198.175.173"
2020/03/24 15:20:44 [error] 4199#4199: *42 connect() failed (111: Connection refused) while connecting to upstream, client: 169.197.108.42, server: web1.posmete.com, request: "GET /remote/login HTTP/1.1", upstream: "https://[fd42:f63:4839:fedc:216:3eff:fe18:2fa3]:443/remote/login", host: "35.198.175.173"
2020/03/24 15:20:44 [warn] 4199#4199: *42 upstream server temporarily disabled while connecting to upstream, client: 169.197.108.42, server: web1.posmete.com, request: "GET /remote/login HTTP/1.1", upstream: "https://[fd42:f63:4839:fedc:216:3eff:fe18:2fa3]:443/remote/login", host: "35.198.175.173"
2020/03/24 15:20:49 [info] 4199#4199: *42 client 127.0.0.1 closed keepalive connection

The errors from web server log:

10.130.126.23 - - [24/Mar/2020:15:20:44 +0000] "\x16\x03\x01\x00\xCD\x01\x00\x00\xC9\x03\x03\xB6j\x9Ef\x90\x08\x90O\x8A-VYw\xD8\x09rn\xD1\x10\xB1m\xCF\x0E\xDE\x95\x83]e\xC6J\x14\x06\x00\x008\xC0,\xC00\x00\x9F\xCC\xA9\xCC\xA8\xCC\xAA\xC0+\xC0/\x00\x9E\xC0$\xC0(\x00k\xC0#\xC0'\x00g\xC0" 400 157 "-" "-" "-"

I've tried almost all instructions through internet, but still 502 error.
Only one solution wich had a positive effect – was to duplicate SSL certs from proxy to web. But I want to avoid it and manage certs with certbot from one place.

So basically through HTTP everything is working ok, but through https I have 502 error

Best Answer

Without duplicating the private key and certificate, the handshake cannot complete:

A TLS (version 1.3) handshake is initiated by a ClientHello message, to which ServerHello, EncryptedExtensions, Certificate, and CertificateVerify messages are expected in response. The Certificate message contains the certificate and the CertificateVerify message contains a signature computed using the private key. Hence, the handshake cannot complete without duplicating the private key and certificate.

You need to duplicate.

When you think about the properties of SSL/TLS, the need for duplication should become intuitive, because SSL/TLS is used for server authentication (which requires the private key, otherwise authentication wouldn't be achieved).

Alternatively, you could use two distinct private keys and certificates. I'll elaborate if that's acceptable for you.

Related Topic