Java – SSLHandshakeException: Received fatal alert: handshake_failure

The handshake failure could have occurred due to various reasons:

• Incompatible cipher suites in use by the client and the server. This would require the client to use (or enable) a cipher suite that is supported by the server.
• Incompatible versions of SSL in use (the server might accept only TLS v1, while the client is capable of only using SSL v3). Again, the client might have to ensure that it uses a compatible version of the SSL/TLS protocol.
• Incomplete trust path for the server certificate; the server's certificate is probably not trusted by the client. This would usually result in a more verbose error, but it is quite possible. Usually the fix is to import the server's CA certificate into the client's trust store.
• The cerificate is issued for a different domain. Again, this would have resulted in a more verbose message, but I'll state the fix here in case this is the cause. The resolution in this case would be get the server (it does not appear to be yours) to use the correct certificate.

Since, the underlying failure cannot be pinpointed, it is better to switch on the -Djavax.net.debug=all flag to enable debugging of the SSL connection established. With the debug switched on, you can pinpoint what activity in the handshake has failed.

Update

Based on the details now available, it appears that the problem is due to an incomplete certificate trust path between the certificate issued to the server, and a root CA. In most cases, this is because the root CA's certificate is absent in the trust store, leading to the situation where a certificate trust path cannot exist; the certificate is essentially untrusted by the client. Browsers can present a warning so that users may ignore this, but the same is not the case for SSL clients (like the HttpsURLConnection class, or any HTTP Client library like Apache HttpComponents Client).

Most these client classes/libraries would rely on the trust store used by the JVM for certificate validation. In most cases, this will be the cacerts file in the JRE_HOME/lib/security directory. If the location of the trust store has been specified using the JVM system property javax.net.ssl.trustStore, then the store in that path is usually the one used by the client library. If you are in doubt, take a look at your Merchant class, and figure out the class/library it is using to make the connection.

Adding the server's certificate issuing CA to this trust store ought to resolve the problem. You can refer to my answer on a related question on getting tools for this purpose, but the Java keytool utility is sufficient for this purpose.

Warning: The trust store is essentially the list of all CAs that you trust. If you put in an certificate that does not belong to a CA that you do not trust, then SSL/TLS connections to sites having certificates issued by that entity can be decrypted if the private key is available.

Update #2: Understanding the output of the JSSE trace

The keystore and the truststores used by the JVM are usually listed at the very beginning, somewhat like the following:

keyStore is : 
keyStore type is : jks
keyStore provider is : 
init keystore
init keymanager of type SunX509
trustStore is: C:\Java\jdk1.6.0_21\jre\lib\security\cacerts
trustStore type is : jks
trustStore provider is : 

If the wrong truststore is used, then you'll need to re-import the server's certificate to the right one, or reconfigure the server to use the one listed (not recommended if you have multiple JVMs, and all of them are used for different needs).

If you want to verify if the list of trust certs contains the required certs, then there is a section for the same, that starts as:

adding as trusted cert:
  Subject: CN=blah, O=blah, C=blah
  Issuer:  CN=biggerblah, O=biggerblah, C=biggerblah
  Algorithm: RSA; Serial number: yadda
  Valid from SomeDate until SomeDate

You'll need to look for if the server's CA is a subject.

The handshake process will have a few salient entries (you'll need to know SSL to understand them in detail, but for the purpose of debugging the current problem, it will suffice to know that a handshake_failure is usually reported in the ServerHello.

1. ClientHello

A series of entries will be reported when the connection is being initialized. The first message sent by the client in a SSL/TLS connection setup is the ClientHello message, usually reported in the logs as:

*** ClientHello, TLSv1
RandomCookie:  GMT: 1291302508 bytes = { some byte array }
Session ID:  {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA]
Compression Methods:  { 0 }
***

Note the cipher suites used. This might have to agree with the entry in your merchant.properties file, for the same convention might be employed by the bank's library. If the convention used is different, there is no cause of worry, for the ServerHello will state so, if the cipher suite is incompatible.

2. ServerHello

The server responds with a ServerHello, that will indicate if the connection setup can proceed. Entries in the logs are usually of the following type:

*** ServerHello, TLSv1
RandomCookie:  GMT: 1291302499 bytes = { some byte array}
Cipher Suite: SSL_RSA_WITH_RC4_128_SHA
Compression Method: 0
***

Note the cipher suite that it has chosen; this is best suite available to both the server and the client. Usually the cipher suite is not specified if there is an error. The certificate of the server (and optionally the entire chain) is sent by the server, and would be found in the entries as:

*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=server, O=server's org, L=server's location, ST =Server's state, C=Server's country
  Signature Algorithm: SHA1withRSA, OID = some identifer

.... the rest of the certificate
***

If the verification of the certificate has succeeded, you'll find an entry similar to:

Found trusted certificate:
[
[
  Version: V1
  Subject: OU=Server's CA, O="Server's CA's company name", C=CA's country
  Signature Algorithm: SHA1withRSA, OID = some identifier

One of the above steps would not have succeeded, resulting in the handshake_failure, for the handshake is typically complete at this stage (not really, but the subsequent stages of the handshake typically do not cause a handshake failure). You'll need to figure out which step has failed, and post the appropriate message as an update to the question (unless you've already understood the message, and you know what to do to resolve it).

Impossible to answer without seeing a stack trace and possibly the output produced when running client or server with -Djavax.net.debug=ssl,handshake. However:

String[] cipherSuites = sslSocket.getSupportedCipherSuites(); sslSocket.setEnabledCipherSuites(cipherSuites);

Don't do that. It is radically insecure, and it just conceals the real problem, which is almost always a certificate trust chain problem: either the client doesn't trust the server's certificate or vice versa.