Ubuntu – Debian issue with intermediate CA

certificate-authoritydebiansslssl-certificateUbuntu

Context

I have, for some reasons, the need to generate :

  • a root CA
  • an intermediate CA (signed by root CA)
  • a certificate (signed by intermediate CA)
  • the chain

I'm in a Dockerized environment, with 3 containers :

  • Container A : Nginx Reverse-proxy that decodes SSL (using chain + key) and forwards requests to Container B
  • Container B : Nginx (nothing special here, the responding app)
  • Container C : An application that needs to curl A (without insecure flags). I need to install intermediate CA on this one

It looks like an easy problem, but I'm stuck when Container C is based on Debian. It works like a charm on Ubuntu container for example. It also works with an Ubuntu desktop virtual machine.
My intermediate CA also works on my mac using OSX keychain store.

This looks like a Debian specific issue.

How did I test

I followed several steps :
– Generate the ca/certs/chain
– Install the intermediate CA on my mac
– curl -XGET https://service.local inside a terminal : OK

Then, using Docker, mount the intermediate CA on ubuntu, then call update-ca-certificates, and test using curl : OK

Then, using Docker, mount the intermediate CA on Debian, then call update-ca-certificates, and test using curl : KO

Scripting part

I've scripted the whole certs/ca/chain generation, no problem here, I won't go into detail unless you need them.

Docker-compose for Container C

I'm trying with different OS, that's why there are several services, and it shows I do the exact same thing between ubuntu and several versions of Debian.

I took some shortcuts by not calling update-ca-certificates ; instead I mount the file directly in /etc/ssl/certs since it's not doing much more really…
For those who are wondering : I also tried to mount the file elsewhere, then call update-ca-certificates, then curl, with another version of the entrypoint, but same result at the end, it works on Ubuntu and not on Debian.

version: '2'

services:
    ubuntu:
        image : "ubuntu:16.04"
        volumes :
            - './local_ca/LOCAL_CA.crt:/etc/ssl/certs/local_ca.pem:ro'
            - './entrypoint.sh:/entrypoint.sh'
        command : 'bash -c "/entrypoint.sh"'

    debian7:
        image : "debian:wheezy"
        volumes :
            - './local_ca/LOCAL_CA.crt:/etc/ssl/certs/local_ca.pem:ro'
            - './entrypoint.sh:/entrypoint.sh'
        command : 'bash -c "/entrypoint.sh"'

    debian8:
        image : "debian:jessie"
        volumes :
            - './local_ca/LOCAL_CA.crt:/etc/ssl/certs/local_ca.pem:ro'
            - './entrypoint.sh:/entrypoint.sh'
        command : 'bash -c "/entrypoint.sh"'

    debian9:
        image : "debian:stretch"
        volumes :
            - './local_ca/LOCAL_CA.crt:/etc/ssl/certs/local_ca.pem:ro'
            - './entrypoint.sh:/entrypoint.sh'
        command : 'bash -c "/entrypoint.sh"'

entrypoint.sh

For the sake of the example, assume the IP is known and replaced in the following template :

#!/bin/bash

apt-get update
apt-get install -y curl
echo '{{ip_of_reverse_proxy}} service.local' >> /etc/hosts
curl -XGET https://service.local

As I said before, I also did a version where I call update-ca-certificates instead of mounting directly, but it didn't make any difference, it works on Ubuntu and not on Debian

docker-compose logs

Here are the output logs, truncated so that we only get the interesting part.
As you can see, curl on ubuntu answers with a JSON whereas debian refuses the certificate verification since root ca is not trusted.

ubuntu_1   | {"_links":{"self":{"href":"\/"}}}

debian7_1  | curl performs SSL certificate verification by default, using a "bundle"
[...] (see debian9_1 logs)

debian8_1  | curl performs SSL certificate verification by default, using a "bundle"
[...] (see debian9_1 logs)

debian9_1  | curl: (60) SSL certificate problem: self signed certificate in certificate chain
debian9_1  | More details here: https://curl.haxx.se/docs/sslcerts.html
debian9_1  |
debian9_1  | curl performs SSL certificate verification by default, using a "bundle"
debian9_1  |  of Certificate Authority (CA) public keys (CA certs). If the default
debian9_1  |  bundle file isn't adequate, you can specify an alternate file
debian9_1  |  using the --cacert option.
debian9_1  | If this HTTPS server uses a certificate signed by a CA represented in
debian9_1  |  the bundle, the certificate verification probably failed due to a
debian9_1  |  problem with the certificate (it might be expired, or the name might
debian9_1  |  not match the domain name in the URL).
debian9_1  | If you'd like to turn off curl's verification of the certificate, use
debian9_1  |  the -k (or --insecure) option.

Questions

Question 1 : As I inject the exact same files and follow the exact same process, why does it work on ubuntu but not debian ? Aren't they sharing the same base ?

Question 2 : What's the right solution for Debian ? Do I really need to also inject and install the root CA ? (it works when I do that, but it looks weird to me since intermediate CA should be enough)

Best Answer

After further investigation, the biggest difference between ubuntu and debian is the SSL library used :

  • On Debian, curl is pre-compiled with OpenSSL
  • On Ubuntu, curl is pre-compiled with GnuTLS

I recompiled curl with GnuTLS on Debian and it works fine using only the intermediate CA.

I recompiled curl with OpenSSL on Ubuntu and now have to trust the root CA if I want my request to pass through.