Apache Extension Error

apache-2.4opensslssl-certificate

I have an Apache 2.4.18 with OpenSSL 1.0.2f configuration here with dual (RSA 4096 + ECC 384) certificate configuration. I also provide certificate transparency through TLS extension.

When using openssl s_client -serverinfo 18 -connect winpack.cf:443 to test my Signed Certificate Timestamps, openssl uses the EC-384 certificate and everything works (TLS extension 18 is correctly provided).

However, when running openssl s_client -cipher 'ECDHE-RSA-AES256-SHA' -serverinfo 18 -connect winpack.cf:443 to force use of RSA-4096 certificate, the extension is NOT provided and the output is as follows: (you can run it too, you will have the same results… I tried running it from 3 computers of different subnets)

CONNECTED(00000003)
140289703855760:error:140773E8:SSL routines:SSL23_GET_SERVER_HELLO:reason(1000):s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 152 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1455910331
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

What may be going on here?

Edit 1: Here is my mod_ssl configuration (obviously some certificate/private key paths have been removed)

<IfModule mod_ssl.c>

    # Pseudo Random Number Generator (PRNG):
    # Configure one or more sources to seed the PRNG of the SSL library.
    # The seed data should be of good random quality.
    # WARNING! On some platforms /dev/random blocks if not enough entropy
    # is available. This means you then cannot use the /dev/random device
    # because it would lead to very long connection times (as long as
    # it requires to make more entropy available). But usually those
    # platforms additionally provide a /dev/urandom device which doesn't
    # block. So, if available, use this one instead. Read the mod_ssl User
    # Manual for more details.
    #
    SSLRandomSeed startup builtin
    SSLRandomSeed startup file:/dev/urandom 512
    SSLRandomSeed connect builtin
    SSLRandomSeed connect file:/dev/urandom 512

    ##
    ##  SSL Global Context
    ##
    ##  All SSL configuration in this context applies both to
    ##  the main server and all SSL-enabled virtual hosts.
    ##

    #
    #   Some MIME-types for downloading Certificates and CRLs
    #
    AddType application/x-x509-ca-cert .crt
    AddType application/x-pkcs7-crl .crl

    #   Pass Phrase Dialog:
    #   Configure the pass phrase gathering process.
    #   The filtering dialog program (`builtin' is a internal
    #   terminal dialog) has to provide the pass phrase on stdout.
    SSLPassPhraseDialog exec:/usr/share/apache2/ask-for-passphrase

    #   Inter-Process Session Cache:
    #   Configure the SSL Session Cache: First the mechanism 
    #   to use and second the expiring timeout (in seconds).
    #   (The mechanism dbm has known memory leaks and should not be used).
    #SSLSessionCache         dbm:${APACHE_RUN_DIR}/ssl_scache
    SSLSessionCache     shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)
    SSLSessionCacheTimeout  300

    #   Semaphore:
    #   Configure the path to the mutual exclusion semaphore the
    #   SSL engine uses internally for inter-process synchronization. 
    #   (Disabled by default, the global Mutex directive consolidates by default
    #   this)
    #Mutex file:${APACHE_LOCK_DIR}/ssl_mutex ssl-cache


    #   SSL Cipher Suite:
    #   List the ciphers that the client is permitted to negotiate. See the
    #   ciphers(1) man page from the openssl package for list of all available
    #   options.
    #   Enable only secure ciphers:

    SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA256:ECDHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA

    #   Speed-optimized SSL Cipher configuration:
    #   If speed is your main concern (on busy HTTPS servers e.g.),
    #   you might want to force clients to specific, performance
    #   optimized ciphers. In this case, prepend those ciphers
    #   to the SSLCipherSuite list, and enable SSLHonorCipherOrder.
    #   Caveat: by giving precedence to RC4-SHA and AES128-SHA
    #   (as in the example below), most connections will no longer
    #   have perfect forward secrecy - if the server's key is
    #   compromised, captures of past or future traffic must be
    #   considered compromised, too.
    #SSLCipherSuite RC4-SHA:AES128-SHA:HIGH:MEDIUM:!aNULL:!MD5:!AES256-SHA:!AES128-SHA:!AES256-SHA256:!AES256-GCM-SHA384:!AES128-SHA256:!AES128-GCM-SHA256
    SSLHonorCipherOrder on

    #   The protocols to enable.
    #   Available values: all, SSLv3, TLSv1, TLSv1.1, TLSv1.2
    #   SSL v2  is no longer supported
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2 -SSLv2 -SSLv3

    #   Allow insecure renegotiation with clients which do not yet support the
    #   secure renegotiation protocol. Default: Off
    #SSLInsecureRenegotiation on

    #   Whether to forbid non-SNI clients to access name based virtual hosts.
    #   Default: Off
    #SSLStrictSNIVHostCheck On

    SSLUseStapling On
    SSLStaplingCache shmcb:${APACHE_RUN_DIR}/ssl_stapling(512000)

    SSLOpenSSLConfCmd DHParameters "[removed]"
    SSLOpenSSLConfCmd Curves P-521:P-384:P-256
    SSLOpenSSLConfCmd ServerInfoFile "[removed]_SCTs.pem"

</IfModule>

Edit 2: My SSL configuration file:

<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerAdmin root@winpack.cf

        DocumentRoot /var/www

        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".
        #Include conf-available/serve-cgi-bin.conf

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        #   A self-signed (snakeoil) certificate can be created by installing
        #   the ssl-cert package. See
        #   /usr/share/doc/apache2/README.Debian.gz for more info.
        #   If both key and certificate are stored in the same file, only the
        #   SSLCertificateFile directive is needed.

        SSLCertificateFile [removed]_RSA.crt
        SSLCertificateFile [removed]_EC.crt

        SSLCertificateKeyFile [removed]_RSA.key
        SSLCertificateKeyFile [removed]_EC.key

        #   Server Certificate Chain:
        #   Point SSLCertificateChainFile at a file containing the
        #   concatenation of PEM encoded CA certificates which form the
        #   certificate chain for the server certificate. Alternatively
        #   the referenced file can be the same as SSLCertificateFile
        #   when the CA certificates are directly appended to the server
        #   certificate for convinience.

        #   Certificate Authority (CA):
        #   Set the CA certificate verification path where to find CA
        #   certificates for client authentication or alternatively one
        #   huge file containing all of them (file must be PEM encoded)
        #   Note: Inside SSLCACertificatePath you need hash symlinks
        #        to point to the certificate files. Use the provided
        #        Makefile to update the hash symlinks after changes.
        SSLCACertificatePath /etc/ssl/certs/
        #SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt

        #   Certificate Revocation Lists (CRL):
        #   Set the CA revocation path where to find CA CRLs for client
        #   authentication or alternatively one huge file containing all
        #   of them (file must be PEM encoded)
        #   Note: Inside SSLCARevocationPath you need hash symlinks
        #        to point to the certificate files. Use the provided
        #        Makefile to update the hash symlinks after changes.
        #SSLCARevocationPath /etc/apache2/ssl.crl/
        #SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl

        #   Client Authentication (Type):
        #   Client certificate verification type and depth.  Types are
        #   none, optional, require and optional_no_ca.  Depth is a
        #   number which specifies how deeply to verify the certificate
        #   issuer chain before deciding the certificate is not valid.
        #SSLVerifyClient require
        #SSLVerifyDepth  10

        #   SSL Engine Options:
        #   Set various options for the SSL engine.
        #   o FakeBasicAuth:
        #    Translate the client X.509 into a Basic Authorisation.  This means that
        #    the standard Auth/DBMAuth methods can be used for access control.  The
        #    user name is the `one line' version of the client's X.509 certificate.
        #    Note that no password is obtained from the user. Every entry in the user
        #    file needs this password: `xxj31ZMTZzkVA'.
        #   o ExportCertData:
        #    This exports two additional environment variables: SSL_CLIENT_CERT and
        #    SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
        #    server (always existing) and the client (only existing when client
        #    authentication is used). This can be used to import the certificates
        #    into CGI scripts.
        #   o StdEnvVars:
        #    This exports the standard SSL/TLS related `SSL_*' environment variables.
        #    Per default this exportation is switched off for performance reasons,
        #    because the extraction step is an expensive operation and is usually
        #    useless for serving static content. So one usually enables the
        #    exportation for CGI and SSI requests only.
        #   o OptRenegotiate:
        #    This enables optimized SSL connection renegotiation handling when SSL
        #    directives are used in per-directory context.
        #SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
        <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
        </Directory>

        #   SSL Protocol Adjustments:
        #   The safe and default but still SSL/TLS standard compliant shutdown
        #   approach is that mod_ssl sends the close notify alert but doesn't wait for
        #   the close notify alert from client. When you need a different shutdown
        #   approach you can use one of the following variables:
        #   o ssl-unclean-shutdown:
        #    This forces an unclean shutdown when the connection is closed, i.e. no
        #    SSL close notify alert is send or allowed to received.  This violates
        #    the SSL/TLS standard but is needed for some brain-dead browsers. Use
        #    this when you receive I/O errors because of the standard approach where
        #    mod_ssl sends the close notify alert.
        #   o ssl-accurate-shutdown:
        #    This forces an accurate shutdown when the connection is closed, i.e. a
        #    SSL close notify alert is send and mod_ssl waits for the close notify
        #    alert of the client. This is 100% SSL/TLS standard compliant, but in
        #    practice often causes hanging connections with brain-dead browsers. Use
        #    this only for browsers where you know that their SSL implementation
        #    works correctly.
        #   Notice: Most problems of broken clients are also related to the HTTP
        #   keep-alive facility, so you usually additionally want to disable
        #   keep-alive for those clients, too. Use variable "nokeepalive" for this.
        #   Similarly, one has to force some clients to use HTTP/1.0 to workaround
        #   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
        #   "force-response-1.0" for this.
        BrowserMatch "MSIE [2-6]" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
        # MSIE 7 and newer should be able to use keepalive
        BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown

        [headers removed]

        #H2Direct On

        Protocols h2 http/1.1

    </VirtualHost>
</IfModule>

Best Answer

For the record, the above issue was caused by an OpenSSL bug.

When SSL_CTX_use_serverinfo_file() is used to configure custom TLS extension data for an SSL_CTX, e.g. SCT data, and that SSL_CTX has been configured with multiple certificates via SSL_CTX_use_certificate(), strange things can happen.

To illustrate, assume an SSL_CTX which has been configured with a serverinfo file for TLS extension type 18, and two server certificates: first an RSA certificate, then an EC certificate. With such a server setup, using openssl s_client to request the configured TLS extension might work:

$ openssl s_client -connect 127.0.0.1:8443 -cipher ECDSA -serverinfo 18
CONNECTED(00000003)
-----BEGIN SERVERINFO FOR EXTENSION 18-----
...

or it might not:

$ openssl s_client -connect 127.0.0.1:8443 -cipher RSA -serverinfo 18
CONNECTED(00000003)
140735089771344:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:

It turns out that by requesting the TLS extension type 18 in the handshake, the handshake fails for the first configured server certificate, but succeeds for the last configured server certificate.

Once this bug is fixed, another unexpected behavior surfaces: the TLS extension data is only returned when the last configured server certificate is used. It looks like this happens because SSL_CTX_use_serverinfo_file(), meant to configure the TLS extension data for the entire SSL_CTX, only copies that data into the currently configured cert (i.e. ctx->cert->key->serverinfo). But when TLS extension data is retrieved, it's done using the cert->pkeys array. Adding another server certificate adds another entry to the cert->pkeys array and changes ctx->cert->key, but does not copy/preserve any configured serverinfo data.