Ubuntu – Unknown SSL protocol error in connection to rubygems.org:443 from Ubuntu

I ran into this problem on two Ubuntu 12.04 instances using kernel 3.2.0-54 after upgrading to the latest OpenSSL with the Heartbleed fix. Connections to some servers over HTTPS that never timed out before are doing so now. Particularly, I am having issues connecting to Rackspace Cloud Files and Rubygems.

The OpenSSL version is OpenSSL 1.0.1 14 Mar 2012, built on: Mon Apr 7 20:33:29 UTC 2014. I think I have the latest ca-certificates package and running update-ca-certificates neither adds nor removes any certificates.

Here is the verbose output from a curl attempt to rubygems.org. This query returns successfully when run from servers on the same network that have not been upgraded to the latest OpenSSL and from my OS X laptop:

>> curl https://rubygems.org/api/v1/versions/rubygems-update.json --verbose --trace-time
16:16:18.985045 * About to connect() to rubygems.org port 443 (#0)
16:16:18.985181 *   Trying 54.245.255.174... connected
16:16:19.049839 * successfully set certificate verify locations:
16:16:19.049881 *   CAfile: none
  CApath: /etc/ssl/certs
16:16:19.050177 * SSLv3, TLS handshake, Client hello (1):
16:17:28.642516 * Unknown SSL protocol error in connection to rubygems.org:443 
16:17:28.642584 * Closing connection #0
curl: (35) Unknown SSL protocol error in connection to rubygems.org:443 

Also, here is the output from testing the connection using openssl (it's the same from both servers that have been upgraded):

>> openssl s_client -host rubygems.org -port 443 -debug
CONNECTED(00000003)
write to 0x1967540 [0x19675c0] (225 bytes => 225 (0xE1))
0000 - 16 03 01 00 dc 01 00 00-d8 03 02 53 4e 8c b5 e5   ...........SN...
0010 - a6 af b3 98 8a 0d 32 e5-fc c9 41 af eb e4 63 f9   ......2...A...c.
0020 - 19 3e b1 41 9f 62 65 23-a6 d2 f9 00 00 66 c0 14   .>.A.be#.....f..
0030 - c0 0a c0 22 c0 21 00 39-00 38 00 88 00 87 c0 0f   ...".!.9.8......
0040 - c0 05 00 35 00 84 c0 12-c0 08 c0 1c c0 1b 00 16   ...5............
0050 - 00 13 c0 0d c0 03 00 0a-c0 13 c0 09 c0 1f c0 1e   ................
0060 - 00 33 00 32 00 9a 00 99-00 45 00 44 c0 0e c0 04   .3.2.....E.D....
0070 - 00 2f 00 96 00 41 c0 11-c0 07 c0 0c c0 02 00 05   ./...A..........
0080 - 00 04 00 15 00 12 00 09-00 14 00 11 00 08 00 06   ................
0090 - 00 03 00 ff 01 00 00 49-00 0b 00 04 03 00 01 02   .......I........
00a0 - 00 0a 00 34 00 32 00 0e-00 0d 00 19 00 0b 00 0c   ...4.2..........
00b0 - 00 18 00 09 00 0a 00 16-00 17 00 08 00 06 00 07   ................
00c0 - 00 14 00 15 00 04 00 05-00 12 00 13 00 01 00 02   ................
00d0 - 00 03 00 0f 00 10 00 11-00 23 00 00 00 0f 00 01   .........#......
00e0 - 01                                                .
read from 0x1967540 [0x196cb20] (7 bytes => -1 (0xFFFFFFFFFFFFFFFF))
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 225 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

Could this be related to the Ubuntu OpenSSL package? I'm not sure what to try at this point or how else to diagnose what might be going on.

Thanks in advance.

More Information

In order to ssh into one of these servers after the OpenSSL upgrade, I had to change my local OSX ssh_config file as shown here: https://superuser.com/a/597956 . Not sure if this is related but it seems like it might be.

Packet Capture

Here is the packet capture sequence for a failed request:

1   0.000000    10.242.0.14 54.245.255.174  TCP 74  49596 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=89903699 TSecr=0 WS=128
2   0.094645    54.245.255.174  10.242.0.14 TCP 66  https > 49596 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1380 SACK_PERM=1 WS=128
3   0.094675    10.242.0.14 54.245.255.174  TCP 54  49596 > https [ACK] Seq=1 Ack=1 Win=14720 Len=0
4   0.095539    10.242.0.14 54.245.255.174  SSL 296 Client Hello
5   0.189818    54.245.255.174  10.242.0.14 TCP 54  https > 49596 [ACK] Seq=1 Ack=243 Win=15744 Len=0
6   0.196661    54.245.255.174  10.242.0.14 SSL 227 [TCP Previous segment not captured] Continuation Data
7   0.196681    10.242.0.14 54.245.255.174  TCP 66  [TCP Dup ACK 4#1] 49596 > https [ACK] Seq=243 Ack=1 Win=14720 Len=0 SLE=2761 SRE=2934
8   60.191971   54.245.255.174  10.242.0.14 TCP 54  https > 49596 [FIN, ACK] Seq=2934 Ack=243 Win=15744 Len=0
9   60.191995   10.242.0.14 54.245.255.174  TCP 66  [TCP Dup ACK 4#2] 49596 > https [ACK] Seq=243 Ack=1 Win=14720 Len=0 SLE=2761 SRE=2935
10  69.800750   54.245.255.174  10.242.0.14 TCP 54  https > 49596 [RST, ACK] Seq=2935 Ack=243 Win=0 Len=0
11  69.800767   10.242.0.14 54.245.255.174  TCP 66  [TCP Dup ACK 4#3] 49596 > https [ACK] Seq=243 Ack=1 Win=14720 Len=0 SLE=2761 SRE=2935
12  69.801646   54.245.255.174  10.242.0.14 TCP 54  https > 49596 [RST, ACK] Seq=1 Ack=243 Win=14720 Len=0

When I compared this against a successful request, one thing that stood out was that the failed request was using SSL while the successful one was using TLSv1.1.

Here is another packet capture sequence for another failed request. This one fails due to a timeout but is repeatable and does not result in a timeout from other machines.

1   0.000000    10.242.0.14 174.143.184.158 TCP 74  60061 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=126050664 TSecr=0 WS=128
2   0.001096    174.143.184.158 10.242.0.14 TCP 66  https > 60061 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1380 SACK_PERM=1 WS=1024
3   0.001128    10.242.0.14 174.143.184.158 TCP 54  60061 > https [ACK] Seq=1 Ack=1 Win=14720 Len=0
4   0.002482    10.242.0.14 174.143.184.158 SSL 314 Client Hello
5   0.003636    174.143.184.158 10.242.0.14 SSL 975 [TCP Previous segment not captured] Continuation Data
6   0.003659    10.242.0.14 174.143.184.158 TCP 66  [TCP Dup ACK 4#1] 60061 > https [ACK] Seq=261 Ack=1 Win=14720 Len=0 SLE=2761 SRE=3682
7   300.096643  10.242.0.14 174.143.184.158 TCP 66  60061 > https [FIN, ACK] Seq=261 Ack=1 Win=14720 Len=0 SLE=2761 SRE=3682
8   300.297854  10.242.0.14 174.143.184.158 TCP 66  [TCP Retransmission] 60061 > https [FIN, ACK] Seq=261 Ack=1 Win=14720 Len=0 SLE=2761 SRE=3682
....

As per the discussion below, the Continuation Data packet in both scenarios, when viewed in wireshark, does not display anything in the SSL section even though there are bytes visible. For the first packet capture, the number of bytes is fewer than that for the second capture. In fact, in the second capture, there is visible text which leads me to believe it is part of a certificate.