Nginx config using variable in ssl_certificate path throws permissions error

lets-encryptnginxopensslsslssl-certificate

The nginx configuration server block:

localhost:/etc/nginx$ cat nginx.conf | grep -B 3 -A 6 '$ssl_server_name'
  server {
    listen 443 ssl http2 default_server;

    ssl_certificate         /etc/letsencrypt/live/$ssl_server_name/fullchain.pem;
    ssl_certificate_key     /etc/letsencrypt/live/$ssl_server_name/privkey.pem;

    location / {
      include /etc/nginx/snippets/set-headers.conf;
      proxy_pass http://localhost:8080;
    }
  }

This is using the variable $ssl_server_name in the certificate directives which is supported since nginx 1.15.9. Relevant part of the nginx docs.

The configuration passes nginx -t and loads without issues, but page does not load in browser, and there is a permissions denied error opening the cert in error.log even though nginx is running as root:

localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/19 18:51:47 [error] 5676#5676: *251 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443
localhost:/etc/nginx$ ps -ef | grep nginx | grep -v grep
www-data  5676 24653  0 18:49 ?        00:00:00 nginx: worker process
root     24653     1  0 15:08 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
lrwxrwxrwx 1 root root 56 Apr 17 18:53 /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
-rw-r--r-- 1 root root 3591 Apr 17 18:53 /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/19 18:51:47 [error] 5676#5676: *251 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443
localhost:/etc/nginx$ ps -ef | grep nginx | grep -v grep
www-data  5676 24653  0 18:49 ?        00:00:00 nginx: worker process
root     24653     1  0 15:08 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
lrwxrwxrwx 1 root root 56 Apr 17 18:53 /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
-rw-r--r-- 1 root root 3591 Apr 17 18:53 /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ openssl
OpenSSL> version
OpenSSL 1.0.2g  1 Mar 2016
OpenSSL> ^C
localhost:/etc/nginx$ nginx -v
nginx version: nginx/1.17.0

When I replace $ssl_server_name with the domain name in the nginx configuration then there is no permissions error reading the very same cert file, and the page loads in the browser.

Why does using the variable in the cert path not work?

UPDATE:

I updated the archive folder group to www-data, still seing the permissions error:

localhost:/etc/nginx$ sudo chgrp -R www-data /etc/letsencrypt/archive
localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
f: /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwx------ root www-data archive
drwxr-xr-x root www-data [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data fullchain1.pem
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 07:18:58 [error] 4897#4897: *6 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

UPDATE 2:

Added group read and execute permissions to archive folder, still seing the permissions error:

localhost:/etc/nginx$ sudo chmod g+r /etc/letsencrypt/archive
localhost:/etc/nginx$ sudo chmod g+x /etc/letsencrypt/archive
localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/archive/ [DOMAIN NAME REDACTED]/fullchain1.pem
f: /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwxr-x--- root www-data archive
drwxr-xr-x root www-data  [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data fullchain1.pem
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 07:39:58 [error] 4897#4897: *22 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/ [DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

UPDATE 3:

Tried becoming www-data using sudo but got an error:

localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
This account is currently not available.

Update 4:

I also updated the permissions on the symlinked path live folder, still seing the permissions error:

localhost:/etc/nginx$ ll /etc/letsencrypt | grep live
drwx------   5 root root     4096 Apr 17 18:53 live/
localhost:/etc/nginx$ sudo chgrp www-data /etc/letsencrypt/live
localhost:/etc/nginx$ sudo chmod g+rx /etc/letsencrypt/live
localhost:/etc/nginx$ ll /etc/letsencrypt | grep live
drwxr-x---   5 root www-data 4096 Apr 17 18:53 live/
localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/live
f: /etc/letsencrypt/live
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwxr-x--- root www-data live
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 07:57:48 [error] 5104#5104: *17 cannot load certificate key "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Update 5:

Listing the permissions of all dirs in path including symlinks:

localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
f: /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwxr-x--- root www-data live
drwxr-xr-x root root     [DOMAIN NAME REDACTED]
lrwxrwxrwx root root     fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem
drwxr-x--- root www-data   ..
drwxr-xr-x root root       ..
drwxr-x--- root www-data   archive
drwxr-xr-x root www-data   [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data   fullchain1.pem

Update 6:

Tried temporarily changing the shell for www-data user, became www-data using sudo and tested reading the cert was possible, but the permission error is still happening:

localhost:/etc/nginx$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
localhost:/$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/bin/bash
localhost:/etc/nginx$ sudo vim /etc/passwd
localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
localhost:01:/$ whoami
www-data
localhost:/$ cat /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
-----BEGIN CERTIFICATE-----
[REDACTED CERT]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[REDACTED CERT]
-----END CERTIFICATE-----
localhost:/$ exit
logout
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 08:40:23 [error] 5259#5259: *14 cannot load certificate key "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Update 7:

Tried exporting the certs to another folder:

localhost:/etc/nginx$ mkdir /tmp/exported-certs
localhost:/etc/nginx$ sudo rsync -razL /etc/letsencrypt/live/ /tmp/exported-certs
localhost:/etc/nginx$ sudo ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
-rw-r--r-- 1 root www-data 3591 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
localhost:/etc/nginx$ sudo ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
-rw------- 1 root www-data 1704 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
localhost:/etc/nginx$ sudo namei -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
f: /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
drwxr-xr-x root root     /
drwxrwxrwt root root     tmp
drwxr-x--- root www-data exported-certs
drwxr-xr-x root root     [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data fullchain.pem
localhost:/etc/nginx$ sudo vim nginx.conf
localhost:/etc/nginx$ cat nginx.conf | grep -B 3 -A 6 '$ssl_server_name'
  server {
    listen 443 ssl http2 default_server;

    ssl_certificate /tmp/exported-certs/$ssl_server_name/fullchain.pem;
    ssl_certificate_key /tmp/exported-certs/$ssl_server_name/privkey.pem;

    location / {
      include /etc/nginx/snippets/set-headers.conf;
      proxy_pass http://localhost:8080;
    }
  }
localhost:/etc/nginx$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
localhost:/etc/nginx$ sudo nginx -s reload
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 10:52:48 [notice] 6250#6250: signal process started
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 10:53:08 [error] 6251#6251: *67 cannot load certificate key "/tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Then decided to check again as the www-data user because last time I checked it was when the certs were in the letsencrypt folder, also this time I remembered to check both cert and key:

localhost:/etc/nginx$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
localhost:/etc/nginx$ sudo vim /etc/passwd
localhost:/etc/nginx$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/bin/bash
localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
-----BEGIN CERTIFICATE-----
[CERT REDACTED]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[CERT REDACTED]
-----END CERTIFICATE-----
localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
cat: /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem: Permission denied   <---- THERE IT IS!
localhost:/$ ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
-rw------- 1 root www-data 1704 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
localhost:/$ exit
logout
localhost:/etc/nginx$ sudo chmod g+r /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
-----BEGIN PRIVATE KEY-----
[CERT REDACTED]
-----END PRIVATE KEY-----
localhost:/$ exit
logout
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/access.log
139.162.202.226 - [DOMAIN NAME REDACTED]:443 - [20/Jun/2019:11:04:08 +0100] "GET / HTTP/2.0" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

Once I addd the group read permission for www-data to the privkey.pem, the browser was able to load the page. 🙂

Thanks to all that commented on this question.

Best Answer

OK, thanks for feedback in comment. Let me share two ideas - not directly solution on what you have mentioned but possible workarounds....

prepare instead of linking

I am using in some specific case certbot (I guess the same as you) and haproxy. I have cron job running certbot with certonly and in case the cert is issued it is concatenating cert and key (one file is preferred) and copy out to location where haproxy expect it in configuration. The haproxy is restarted afterwards. In case you will not solve it and need to have it working may be try to "prepare" the cert for nginx somewhere instead of linking to the "original" location...

run certbot as www-data user

you can run certbot as user who is running nginx (www-data) so the output would fit for the purpose by the permission point of view. In that case you need to change the ownership first for the structure around - e.g. /etc/letsencrypt/account....

Related Topic