OpenSSL Handshake Failure – Troubleshooting Guide

centos5linuxopenssl

We were recently forced to migrate our production cloud servers from GoDaddy to Azure because GoDaddy is ending thier cloud server service.

One of our servers was a CentOS 5.7 running a JasperReports Bitnami stack. During the migration process I upgrade all servers to the most recent distribution, and rebuilt Jasper from the Azure Bitnami Jasper image on Ubuntu 12.04LTS

A have the SSL Certificate installed on the JasperServer and working correctly

All the new servers are performing beautifully, now heres where the problem comes in.

We also have a dedicated CentOS 5.8 virtual server on GoDaddy which is staying there(for now), there are a collection of sites on said server which serve up reports from Jasper via Soap.

However it is getting handshake failures when attempting to connect

#openssl s_client -connect newjasperserver.com:443
CONNECTED(00000003)
9092:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:583:

and:

#openssl version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

the new server is running:

#openssl version
OpenSSL 1.0.1c 10 May 2012

Now after a lot of research, it appears that there is an incompatibility between OpenSSL < 0.9.8k and OpenSSL 1.0.1.

The options I've identified are:

  1. Migrate the server to a CentOS 6.4 server on Azure (Ideal, but politically difficult, dont ask why)

  2. Upgrade the server in-place(Unsupported, and I don't want to try it on a production server)

  3. Wipe the server and rebuild it with 6.4(possibility, though if I do that, I will force option 1)

  4. Remove OpenSSL from the server and install a newer version (once again, something I'm not comfortable with on a production server)

  5. Install a second instance of OpenSSL (my #2 option, but I'm unsure how to proceed)

  6. Install an alternative to OpenSSL (havent even begun to look into this)

  7. Disable enforced encryption on the Jasper Server and allow connection via http (this is looking like my best temporary fix until I can force that server to be migrated to Azure)

Are there any options I have missed? Is there a way on the Jasper side to allow connections from the older OpenSSL?

Best Answer

The incompatibility that you ran into is this:

The version of OpenSSL on RHEL5 (and its derivatives) doesn't advertise support for TLS at all. It only does SSLv3 and SSLv2.

The version of OpenSSL on RHEL6 (and its derivatives) supports TLS all the way up to TLSv1.2. It also does SSLv3, but it wants to negotiate TLS.

They should still be able to negotiate a session, as there is a (small) list of common ciphers to both of them, but depending on what you have selected for cipher settings on the server (e.g. to mitigate BEAST, eliminate low-security ciphers, etc.) there may not be any common ciphers that the client and server can use to communicate.

The cipher suites for the server would be set in <Connector ciphers= in your Tomcat server.xml or in Apache's SSLCipherSuite if you have it sitting behind Apache. The client would use whatever it was configured to use, or DEFAULT otherwise.

The resolution is to check the cipher suites on the server, e.g. with openssl ciphers -v STRING where STRING is whatever you configured on the server, and repeating the same on the client, and then adjusting one or both until a cipher suite is available that both will negotiate.

Related Topic