Linux – If I get an invalid public key error does that mean I have an incorrectly installed certificate

linuxnginxpublic-keyssl-certificate

I have created an API (running on an nginx server, let me know if there are any details that you think are relevant) for a client and they received this error when trying to call my service:

Notice: SSL certificate problem: self signed certificate in certificate chain 
in /usr/local/zend/apache2/htdocs/PefcSCS/components/ \
com_pefcscs/views/simplesearchs/tmpl/default.php on line 9

I cannot reproduce this error and do not have a self-signed certificate. Looking in the logs I have:

$ sudo tail -n 100 /var/log/nginx/error.log
2014/05/04 07:17:09 [crit] 24027#0: *844 SSL_do_handshake() failed 
(SSL: error:05066066:Diffie-Hellman routines:COMPUTE_KEY:invalid 
public key error:1408B005:SSL routines:SSL3_GET_CLIENT_KEY_EXCHANGE:DH lib) 
while SSL handshaking, client: xx.169.xx.141, server: 0.x.0.x:443

Which says: invalid public key error. Does this mean that they provided an invalid public key? Could someone explain the meaning of the error and how to solve this problem? Thanks.

EDIT: P.S. I'm only assuming that these two points are related, it may be a coincidence and I'm inferring a connection between a botched public key exchange and my clients strange error stating that we have a self-signed certificate.

Best Answer

Probably not an answer yet, but more than I can make readable in comments:

That error code (and string) are about a Diffie-Hellman public key. Unless you are using static DH, which would be very unusual -- I've never even seen a public CA issue a DH cert -- this must be the client's ephemeral DH key for a DHE ciphersuite. For it to be invalid suggests something pretty weird in the client SSL stack, an "attack" (or at least damage) on the session, or the DH parameters in your server getting clobbered -- all of which shouldn't happen. If the client claims 'self-signed cert' and you aren't using one, that is a different problem that shouldn't be related, but might possibly be through some weird bug.

The client appears to be something in PHP, which I'm not familiar with, but I'll guess is likely using openssl to make client SSL connections (in this case to you) (just as nginx AFAIK uses openssl on the server side). If you configured your server to provide a full cert chain including root, and openssl client does not have that root in the truststore it is using (which may be configurable), it may confusingly report 'self-signed cert' rather than a more exact 'untrusted root'.

If I'm right it's openssl, can (you ask) the client people try a connection to you using commandline openssl s_client, with the same truststore their program is using, and the same cipherlist (possibly default) -- and see if that gets any error and if so the (much more verbose) error info s_client gives? In other words the command would be something like

  openssl s_client -connect $yourhost:$yourport -CAfile $file_if_any -CApath $dir_if_any -cipher $string_if_any

(If it does connect, just control-D to exit without sending anything.) If s_client says something like verify error: ... self-signed, they should fix the truststore they are using to include the root cert of the chain you are using. (That might mean changing to use a different truststore, if they can't or don't want to modify the one they currently use.)

In the meantime what (if anything) is configured in your server for ssl_dhparam ? Try openssl dhparam -check -noout on that file just in case, although anything wrong there should have failed at an earlier step in the handshake.

When other (client) connnections succeed, can you check if they negotiate DHE or not? Remeber that an RSA cert, the most common by far, can support plain-RSA DHE-RSA and ECDHE-RSA ciphers. Some clients can display the selected cipher, in particular most browsers if your server accepts browser https requests. I don't use nginx myself, but http://osdir.com/ml/nginx/2010-07/msg00033.html indicates it can log this. s_client always tells you cipher, along with much other stuff.

If the problem still occurs, can you, or they or someone, get a network capture -- using wireshark/tshark, tcpdump, or similar -- of a failed connection? That will confirm the ciphersuite and other options used, and show the DHE exchange up to the failure, which should at least narrow down the problem. Easiest way to look (IMO) is to display in wireshark (even if you captured with something else), expand the ServerHello and get the CipherSuite, and expand the ServerKeyExchange and ClientKeyExchange and get all the details.

Related Topic