How to Fix Certificate Chain with Let’s Encrypt / Certbot

certbotlets-encryptssl-certificate

I cannot wrap my head around the following problem. Verifying the certificates of the server with openssl fails, the chain is imcomplete.

Disclaimer: I am not an admin and did not work much with certificates yet.

Verifiy with OpenSSL

$ openssl verify -CAfile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem  /etc/letsencrypt/live/co2-avatar.com/cert.pem

# /etc/letsencrypt/live/co2-avatar.com/cert.pem: C = US, O = Internet Security Research Group, CN = ISRG Root X1
# error 2 at 2 depth lookup:unable to get issuer certificate

Check for one of the domains in the certificate

openssl s_client -connect co2avatar.org:443 -servername co2avatar.org
# CONNECTED(00000003)
# depth=0 CN = gitlab.sustainable-data-platform.org
# verify error:num=20:unable to get local issuer certificate
# verify return:1
# depth=0 CN = gitlab.sustainable-data-platform.org
# verify error:num=21:unable to verify the first certificate
# verify return:1
# ---
# Certificate chain
#  0 s:CN = gitlab.sustainable-data-platform.org
#    i:C = US, O = Let's Encrypt, CN = R3
# ---
# Server certificate
# -----BEGIN CERTIFICATE-----

Or run

curl -v https://co2avatar.org
# *   Trying 85.214.38.88:443...
# * TCP_NODELAY set
# * Connected to co2avatar.org (85.214.38.88) port 443 (#0)
# * ALPN, offering h2
# * ALPN, offering http/1.1
# * successfully set certificate verify locations:
# *   CAfile: /etc/ssl/certs/ca-certificates.crt
#   CApath: /etc/ssl/certs
# * TLSv1.3 (OUT), TLS handshake, Client hello (1):
# * TLSv1.3 (IN), TLS handshake, Server hello (2):
# * TLSv1.2 (IN), TLS handshake, Certificate (11):
# * TLSv1.2 (OUT), TLS alert, unknown CA (560):
# * SSL certificate problem: unable to get local issuer certificate
# * Closing connection 0
# curl: (60) SSL certificate problem: unable to get local issuer certificate

There might be both, a wrong configuration in my Apache VHost for the domain as well as a problem in the certificate chain itself. How can I check the last one (I've googled a lot, but most hits are about openssl verify with -CAfile or about different cert issuer)?

Do I need to check the root certifate bundle and how exactly?

Is there something like an -addtrust flag for certbot certonly?

Best Answer

Try openssl s_client and let you show the certs. The command is:

$ openssl s_client -connect co2avatar.org:443 -servername co2avatar.org -showcerts

You will find that your server returns a certificate for CN = gitlab.sustainable-data-platform.org and a subject alternative name which includes your domain DNS:co2-avatar.com. So the certificate itself is fine.

If you want to combine everything into one pipeline of commands to see the content of your certificate:

echo | openssl s_client -connect co2avatar.org:443 -servername co2avatar.org -showcerts 2>/dev/null |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -noout -text

Whats missing is the intermediate certificate. This should be sent by the server as well, but the first command shows you that it is not there - only the certificate is sent by your server.

So the failing openssl is correct, as indeed the intermediate certificate is missing.

So to solve it, you need to tweak your apache configuration. This is how your configuration could look like:

Filename should be similar to /etc/apache2/sites-enabled/co2-avatar.com-le-ssl.conf

<IfModule mod_ssl.c>
SSLStaplingCache shmcb:/var/run/apache2/stapling_cache(128000)
<VirtualHost *:443>
        ServerName co2-avatar.com
        ServerAlias www.co2-avatar.com
#... 
#... insert your other stuff here...
#...

SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/co2-avatar.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLUseStapling on
</VirtualHost>
</IfModule>

Based on your description, my best guess is that the following line is wrong in your config: SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/cert.pem. It should be replaced with SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem, in order to send also the intermediate(s).

Solution UPDATE (after discussion)

It turned out in discussion that the openssl and Apache version used on this CentOS server is just older, so some features are unsupported. (Apache 2.4.6, OpenSSL 1.0.2k, intermediate configuration, no HSTS, no OCSP)

According to Mozilla SSL Configuration Generator the following generic configuration could be used in this case:

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile      /path/to/signed_certificate
    SSLCertificateChainFile /path/to/intermediate_certificate
    SSLCertificateKeyFile   /path/to/private_key
</VirtualHost>

Translated to this specific case, a resulting working config would be as the following:

<VirtualHost *:443>
    ServerName  sustainable-data-platform.org
    ServerAlias co2-avatar.com
    ServerAlias ... <include all other SAN names here>
    
    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt/live/co2-avatar.com/cert.pem
    SSLCertificateChainFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    SSLCertificateKeyFile   /etc/letsencrypt/live/co2-avatar.com/privkey.pem

</VirtualHost>

As note for such old installations

Cross-Signed Let’s Encrypt R3 and DST Root CA X3, intermediate and root certificates will expire on Sep 29, 2021 and Sep 30, 2021 respectively. So since May 4, 2021, The newly issued certificates use a longer chain with cross-signed ISRG Root X1 as an intermediate certificate.

Unfortunately, due to the way certificate paths are built and verified, not all implementations of TLS can successfully verify the cross-sign. This is the case with OpenSSL 1.0.2. Hence, programs running on RHEL/CentOS 7 that use OpenSSL will likely fail to verify the new certificate chain or establish TLS connection. Upgrading to newer Openssl versions on such platforms is not straightforward.

There are a few options: either update the trust store (remove DST Root CA X3 root certificate - once it is removed, impact should be minimal) on the client side (or) change the certificate chain on the server side.

For Nginx

For Nginx there is only one parameter to specify the cert file. You should use the fullchain.pem provided by certbot to get it working correctly.

The right configuration in the server block for the given virtualhost would be as follow:

server {
  ...
  ssl_certificate /etc/letsencrypt/live/co2-avatar.com/fullchain.pem;  -> replaced cert.pem for fullchain.pem
  ssl_certificate_key /etc/letsencrypt/live/co2-avatar.com/privkey.pem;
}

References

DST Root CA X3 Expiration (September 2021)
Let’s Encrypt change affects OpenSSL 1.0.x and CentOS 7

Related Solutions

SSL Error: self signed certificate in certificate chain

You don't need the root certificate in the chain (though I don't believe having it hurts anything).

That error is more of a warning from openssl in this case I think. I believe what it means is just that openssl doesn't know that it should trust the root certificate of that chain. If you pull out just the root certificate from that bundle and point that openssl command at it with the -CAfile argument I expect that "error" should go away.

Inside the sf_bundle.crt file you should see two

-----BEGIN CERTIFICATE-----
....
-----END CERTIFICATE-----

blocks (possibly with plain text above each block showing what certificate the block contains). If you split each of those blocks into its own file so you end up with block1.crt and block2.crt you should be able to run openssl x509 -noout -subject -in <file.crt> on each of them to get their respective certificate subject lines.

Assuming that block2.crt has a subject line of /C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority you should then be able to run openssl s_client -CAfile block2.crt -connect imap.domain.ltd:465 and that should, hopefully, connect without giving you the self-signed certificate error.

How to debug certificate chains with OpenSSL

Be sure to include all the intermediate certificates and see that they are up to date. If test.crt is in fact a file containing only /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root, this is the correct approach. You can include the root as well, and most clients will accept such chains, but a few will choke.

In general it's best practice to include all the certificates from yours to the last one before the root, in case a client doesn't have the last intermediate.

It's also possible that test.crt contains things other than the correct chain. OpenSSL doesn't do partial chain validation by default (in older versions, it doesn't do it at all). When operating in this mode it doesn't care what is in /etc/ssl/certs.

Alternatively, you may be presenting an expired intermediary certificate. CAs often recertify their intermediates with the same key; if they do that, just download the updated intermediate CA certificate and replace the expired one in your chain.

Finally, with openssl s_client, you need to specify what it is validating against. For example, use the option -CApath /etc/ssl/certs or -CAfile your_ca.crt. For the first option use your system's trust store, and for the second option specify the root CA certificate.