Nginx – Dockerized Nginx + Certbot + tls-sni challenge not working on renewal

I have an existing service running using docker containers:

• nginx (nginx:1.13.5)
• node-api (node container)
• node-website (node container)

I wanted to add SSL certificates for both api and website. So I've used certbot/certbot docker container to do so, without any problem.

I've generate the certificates using dns challenge, running the following command (from my local machine):

docker run --rm -it \
  --name certbot \
  -v $(pwd)/letsencrypt:/etc/letsencrypt \
  certbot/certbot \
    certonly --manual -d api.mydomain.com --preferred-challenges dns --renew-by-default --email xxx@xxxx --agree-tos --manual-public-ip-logging-ok

I was asked to enter TXT dns record for _acme-challenge.api.mydomain.com which I did, and got the certificates generated without any issues.

I then updated my Nginx configuration (/etc/nginx/nginx.conf/api.conf):

upstream api {
  server node-api:5000 max_fails=3;
}

map $http_upgrade $connection_upgrade {
  default       "upgrade";
  ""            "";
}

# Force HTTPS
server {
  listen 80;
  server_name api.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  server_name       api.domain.com;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_certificate   /etc/nginx/certificates/api.domain.com/fullchain.pem;
  ssl_certificate_key /etc/nginx/certificates/api.domain.com/privkey.pem;
  ssl_trusted_certificate /etc/nginx/certificates/api.domain.com/chain.pem;

  access_log /var/log/nginx/api-domain.access.log main;
  error_log  /var/log/nginx/api-domain.error.log error;

  location / {
    proxy_pass         http://api;
    proxy_redirect     off;

    proxy_set_header   Connection $connection_upgrade;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

And all good, restarted my server and now running perfectly with letsencrypt certificates! Awesome!

The problem

Now come the test of renewing the certificates… And here comes the troubles. First I just tried to run the renew --dry-run command without much of a success.

Attempting to renew cert (api.mydomain.com) from
/etc/letsencrypt/renewal/api.mydomain.com.conf produced an unexpected
error: The manual plugin is not working; there may be problems with
your existing configuration. The error was: PluginError('An
authentication script must be provided with –manual-auth-hook when
using the manual plugin non-interactively.',). Skipping.

So I've tried using the standalone option:

docker run --rm -it \
  --name certbot \
  -v $(pwd)/letsencrypt:/etc/letsencrypt \
  certbot/certbot \
    renew --dry-run --standalone --preferred-challenges dns

and got this error:

Attempting to renew cert (api.mydomain.com) from
/etc/letsencrypt/renewal/api.mydomain.com.conf produced an unexpected
error: None of the preferred challenges are supported by the selected
plugin. Skipping.

So I'm deducting that I cannot use dns challenges for the renewal of the certificate, fair enough.

Let's try with tls-sni that Nginx is supporting (I verified, it is enabled)

docker run --rm -it \
  --name certbot \
  -v $(pwd)/letsencrypt:/etc/letsencrypt \
  certbot/certbot \
    renew --dry-run --standalone --preferred-challenges tls-sni

And now I have this error:

Attempting to renew cert (api.mydomain.com) from
/etc/letsencrypt/renewal/api.mydomain.com.conf produced an unexpected
error: Failed authorization procedure. api.mydomain.com (tls-sni-01):
urn:acme:error:unauthorized :: The client lacks sufficient
authorization :: Incorrect validation certificate for tls-sni-01
challenge. Requested
e0fd03ddade6d902d5947028985253ba.63b177f317335bf9297f0bb963135fee.acme.invalid
from 13.210.106.2:443. Received 2 certificate(s), first certificate
had names "api.mydomain.com". Skipping.

I'm wondering if I need to enable something specific on Nginx side such as OCSP, ssl_stapling or ssl_dhparam

I've also noticed that if I'm trying to re-run the certonly on a different server than I got the certificates from, the dns challenge value is not the same.

So I have 2 questions:

• Can I easily fix the tls-sni verification challenge tuning the nginx configuration?

• If I re-run certonly with dns challenge on the actual server (where nginx container is running), updating the TXT record with the new ones generated. Can I renew certificates later on, using certonly command? Will the TXT record values change over time?

Thanks a lot in advance