Generate or renew letsencrypt certbot certificate for load balanced nginx server on Digital Ocean

certbotdigital-oceanload balancingssl-certificate

Does anybody know how to do it? Whether it is possible at all.

I followed this Digital Ocean tutorial which works but lacks the information how to apply this in a real world scenario with more than a single domain and multiple applications.

I know how to generate and renew certificates with certbot. Neither works though when the IP address is going through a load balancer.

Changing each and every IP address from the load balancer to the nginx server, renew and then changing a load of ip addresses back to the load balancer and copy paste the content of the certificates doesn't seem to be practical. Is there a better solution?

Edit. Instead of changing the ip address it is possible to just detach all but one nginx server from the load balancer and then renew on this droplet. However, the certbot needs to be a recent version. With a certbot install from 2016 this did not work.

Additional problem. After attaching the other nginx server back to the load balancer the certificates will have to be copied over manually.

Attaching certificates to the load balancer itself is also a manual task and there can be only one certificate to forward 443 to 80.

Best Answer

As pointed out by @ceejayoz

This sort of situation is why Let's Encrypt supports DNS-based challenge/auth

You should use the DNS feature of certbot. This will require that you add a new TXT record to the DNS records that certbot will then validate against. This removes the necessity to send HTTP requests to the load-balanced machines.

Alternatively, you can setup the same challenge on all LB systems so that no matter which host the certbot client checks against, it gets the challenge back. This obviously won't work if your having certbot do standalone mode.

Related Solutions

Nginx load balancer w/ https and lets encrypt cert renewal

Setting root to the default nginx webroot on Debian /var/www/html where certbot was originally configured to be pointed at for the .well-known folder resolved the issue.

upstream app_servers {
  server app-1.example.com;
  server app-2.example.com;
}

server {
  listen 80;
  listen 443 ssl;
  include snippets/ssl-app.example.com.conf;
  include snippets/ssl-params.conf;

  server_name app.example.com;

  root /var/www/html;

  location / {
    # Set proxy headers
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For 
    $proxy_add_x_forwarded_for;

    proxy_pass http://app_servers;
  }

  location ~ /.well-known {
    allow all;
  }
}

Nginx – certbot renew crashes nginx and all websites go offline

I think that I found a (hacky) workaround to fix the issue. The steps are the following:

Create an executable script in location /etc/letsencrypt/nginx_fix.sh with the following contents:

#!/bin/bash
for arg
do
  if [ "$arg" = "reload" ]
  then
    exec systemctl restart nginx
  fi
done
exec nginx "$@"

Add the line nginx_ctl = /etc/letsencrypt/nginx_fix.sh to the [renewalparams] block of each renewal configuration file that is managed by nginx authenticator, located at etc/letsencrypt/renewal/*.conf as follows:

# Options used in the renewal process
[renewalparams]
account = XXXXXX # Use your own account key
authenticator = nginx
installer = nginx
server = https://acme-v02.api.letsencrypt.org/directory
nginx_ctl = /etc/letsencrypt/nginx_fix.sh

Not marking this answer as accepted, because it is suboptimal in two ways:

You have to remember to edit the renewal configuration file for every domain that you add.

You still get some errors in the nginx logs as follows:

Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351182 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351186 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351227 (ruby) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351630 (ruby) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351655 (ruby) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351673 (ruby) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351692 (ruby) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351755 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351191 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351194 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351197 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351198 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351200 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351201 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351205 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351210 (PassengerAgent) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351618 (file_store.rb:*) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351651 (worker-1) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351656 (ruby) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351659 (connection_poo*) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351664 (n/a) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351665 (utils.rb:110) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351677 (n/a) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351682 (utils.rb:110) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351683 (n/a) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351696 (connection_poo*) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351701 (n/a) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351708 (worker-2) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351709 (worker-3) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351710 (worker-2) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351711 (worker-3) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351729 (io-worker-1) with signal SIGKILL.
Sep 02 09:37:24 phoenix.medialab.ntua.gr systemd[1]: nginx.service: Killing process 2351756 (PassengerAgent) with signal SIGKILL.

But at least it seems to be working, at least with the dry-run option.

Best Answer

Related Solutions

Nginx load balancer w/ https and lets encrypt cert renewal

Nginx – certbot renew crashes nginx and all websites go offline

Related Topic