Centos – Connection to URL using OpenSSL client works, but curl fails

centoscurlopensslssl

I have a CentOS 5.9 server, from which I need to make SSL connections to another server. The remote server has a certificate ultimately signed by the GeoTrust Global CA. At the time of writing, this certificate is the second one listed on GeoTrust's download page. I am getting inconsistent results, depending on whether I use OpenSSL or curl to make the connection:

openssl s_client -connect <server>:443 -CAfile /path/to/GeoTrustCA.pem

works fine, but

curl --cacert /path/to/GeoTrustCA.pem https://<server>/

fails with the standard "could not verify certificate" error.

Here are details of the tools I'm using:

$ curl --version 
curl 7.15.5 (i386-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 Protocols: tftp ftp telnet dict
ldap http file https ftps  Features: GSS-Negotiate IDN IPv6 Largefile
NTLM SSL libz

and

$ openssl version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

I'm stumped. The server I'm connecting to has been working with no apparent problems for years: I've never heard of anyone or any system being unable to connect to it like this.

Best Answer

I was curious about this so I did some testing. From what I can tell, there appears to be a fundamental difference in what openssl and curl take into consideration when using their root CA switches (-CAfile and --cacert respectively).

When the --cacert switch is used in curl, it appears to ONLY use the admin-specified root during verification. For example, I download the GeoTrust PEM file you mentioned earlier and tried using it to fetch a page from yahoo:

[foo@foobox tmp]# curl --cacert /tmp/geotest.pem https://info.yahoo.com/
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Now, I tried the same test with openssl:

[foo@foobox tmp]# openssl s_client -connect info.yahoo.com:443 -CAfile /tmp/geotest.pem
CONNECTED(00000003)
depth=3 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
verify return:1
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
verify return:1
depth=0 /C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./CN=www.yahoo.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./CN=www.yahoo.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
...
...
...
...
...
...
SSL handshake has read 5342 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 6CAE87314ED66784B25C0FB36197D822CC73032FBFF30AD9E37CFF3D1678EBCC
    Session-ID-ctx:
    Master-Key: 6B9135F16512A251AB6DBEF62C6B261EC31DB90A0076C33DD67B27EAAB83A0333D50B1B7F10727DE47AB051A9C3A0499
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1415842989
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

You'll note that there are no errors in the initial chain description and that it shows Verify return code: 0 (ok) at the bottom.

So even though GeoTrust isn't mentioned at all in the certificate chain, OpenSSL is somehow able to validate/verify the root.

Hmmm...

That got me to thinking about where the default truststore for openssl is... for some reason I had difficulty finding information about it online (I'm sure it's well documented and I'm just blind). Upon poking around on my system, I came across /etc/pki/tls/certs/ca-bundle.crt

I searched in ca-bundle.crt and removed the root that yahoo uses (/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority) and ran the same exact command again:

[foo@foobox tmp]# openssl s_client -connect info.yahoo.com:443 -CAfile /tmp/geotest.pem
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./CN=www.yahoo.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
...
...
...
...
...
...
...
...
---
SSL handshake has read 5342 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 09D998B153574D5C785BFF191B99CAB8BFCEF4DAC482F75A601886E668BF9CE6
    Session-ID-ctx:
    Master-Key: 1F98289FEB5926B8814D5E3B163FB40CC03BBC5C2D8A0045C0DFF0532458F18F722D5FD53155327B0A78627E3FE909E5
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1415843859
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

This time, we get verification errors.

Soooooo, with all that said, I'm inclined to suspect the following:

The root certificate you're using may not actually match the root that you need to use.
curl ONLY uses the root you tell it to use.
openssl uses the root you point it to ANNNND whatever roots are in its default truststore.

As for why openssl does this? No idea. The documentation for that switch doesn't mention this workflow/behavior:

-CAfile file

A file containing trusted certificates to use during server authentication and to use when attempting to build the client certificate chain.

Perhaps someone else can examine the code for openssl and elaborate further.

Related Solutions

Ubuntu 11.10, using wget/curl fails with ssl

So it turns out that installing the ca-certificates package didn't install the one that I needed. I found this post about certificates being presented out of order. This seems to be the case with my request to sagepay.

The solution ended up being to install another CA certificate from Verisign. I'm not sure why this fixes the issue with it being out of order but it does, but I suspect the out of order issue really isn't a problem at all and it was infact because I was missing a certificate all along. The additional certificate is available in that post but I didn't want to blindly trust it. I've looked at the list of CA certificates from cURL's site and it is listed there so I do trust it.

The certificate:

Verisign Class 3 Public Primary Certification Authority
=======================================================
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCEHC65B0Q2Sk0tjjKewPMur8wDQYJKoZIhvcNAQECBQAwXzELMAkGA1UEBhMCVVMx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmltYXJ5
IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2MDEyOTAwMDAwMFoXDTI4MDgwMTIzNTk1OVow
XzELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAz
IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhEBarsAx94
f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/isI19wKTakyYbnsZogy1Ol
hec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0GCSqGSIb3DQEBAgUAA4GBALtMEivPLCYA
TxQT3ab7/AoRhIzzKBxnki98tsX63/Dolbwdj2wsqFHMc9ikwFPwTtYmwHYBV4GSXiHx0bH/59Ah
WM1pF+NEHJwZRDmJXNycAA9WjQKZ7aKQRUzkuxCkPfAyAw7xzvjoyVGM5mKf5p/AfbdynMk2Omuf
Tqj/ZA1k
-----END CERTIFICATE-----

I put this in a file in:

/usr/share/ca-certificates/curl/Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.crt

I then modified the /etc/ca-certificates.conf and added the following line at the end:

curl/Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.crt

After that I ran the command:

sudo update-ca-certificates

Looking into the /etc/ssl/certs directory I see it correctly linked:

ls -al | grep cURL
lrwxrwxrwx 1 root root     69 2012-03-27 16:03 415660c1.0 -> Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.pem
lrwxrwxrwx 1 root root     69 2012-03-27 16:03 7651b327.0 -> Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.pem
lrwxrwxrwx 1 root root    101 2012-03-27 16:03 Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.pem -> /usr/share/ca-certificates/curl/Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.crt

And everything works!

curl  -I https://test.sagepay.com
HTTP/1.1 200 OK...

SSL – Fix ‘Unable to Locally Verify the Issuer’s Authority’ for GeoTrust SSL CA

The server is not configured properly — it does not send the required intermediate certificate. Note that there is only one certificate in the certificate chain:

---
Certificate chain
 0 s:/serialNumber=8z3ZNMMt8GMi9Qumrn0xficRkxAYJZQq/C=AU/ST=New South Wales/L=North Sydney/O=SERCO GROUP PTY LIMITED/CN=*.131500.com.au
   i:/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA
---

There should be a second certificate after this with the subject s:/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA.

In Apache intermediate certificates are configured using the SSLCertificateChainFile option.

As for why this site seems to work for you in Chrome, there are several possible explanations:

Different browsers may use separate certificate stores, and your Chrome may have the GeoTrust SSL CA certificate trusted directly (however, this is unlikely to be the case if the CA intended to use that certificate as intermediate).
Browsers often cache intermediate certificates in their certificate stores, therefore if you previously had visited another site which had the GeoTrust SSL CA intermediate certificate properly configured, you may then be able to access a site which uses the same intermediate certificate, but does not properly send it to clients, without security warnings, because the browser can get the required intermediate certificate from its cache and is able to verify the certificate chain.
The end entity certificate in question contains an HTTP URL which could be used to fetch the intermediate certificate:
```
    Authority Information Access: 
        OCSP - URI:http://gtssl-ocsp.geotrust.com
        CA Issuers - URI:http://gtssl-aia.geotrust.com/gtssl.crt
```
(the CA Issuers link here points to the issuer certificate in the DER format). Some systems may be able to use such links to fetch the intermediate certificate even if it is not returned by the server. According to the Mozilla Bug 399324, Firefox (and other software based on Mozilla) is not currently able to follow such AIA links; however, Internet Explorer is able to use them.

Best Answer

Related Solutions

Ubuntu 11.10, using wget/curl fails with ssl

SSL – Fix ‘Unable to Locally Verify the Issuer’s Authority’ for GeoTrust SSL CA

Related Topic