TLS 1.2 client hello triggers TCP Reset from 2012 R2

iis-8.5pkitlswindows-server-2012-r2

Struggling with a PKI implementation in my lab (ADCS 2012 R2) and cannot for the life of me figure out where I have gone wrong.

Got all the way to the point of being able to generate SSL/TLS certs for an IIS server and make the binding. Also used the NARTAC tool to shut down SSLv2/3 + enable TLS 1.0-1.2. Using testssl.sh I can verify that SSL is disabled and can get a handshake for TLS 1.0 and 1.1, but every time the client hello for TLS 1.2 is received the server sends a TCP Reset. The system logs show the pair of SCHANNEL 36874 and 36888 error IDs that correspond with the resets. Error 36874 suggests that the client hello is presenting cipher suites that aren't supported by the server.

This is where I hit the brick wall, and my searching the interwebs is not yielding a winner. I can all but guarantee I've overlooked something glaringly obvious. However, the blinders of frustration are preventing me from seeing it.

Thoughts?

Client hello details:

Frame 136: 377 bytes on wire (3016 bits), 377 bytes captured (3016 bits) on interface 0 Ethernet II, Src: CiscoInc_07:be:7f
(fc:5b:39:07:be:7f), Dst: Vmware_01:02:14 (00:50:56:01:02:14) Internet
Protocol Version 4, Src: 192.168.6.75, Dst: 10.22.163.219 Transmission
Control Protocol, Src Port: 35836 (35836), Dst Port: 443 (443), Seq:
1, Ack: 1, Len: 311 Secure Sockets Layer
SSL Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 306
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 302
Version: TLS 1.2 (0x0303)
Random
Session ID Length: 0
Cipher Suites Length: 176
Cipher Suites (88 suites)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_DH_DSS_WITH_AES_256_GCM_SHA384 (0x00a5)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 (0x00a3)
Cipher Suite: TLS_DH_RSA_WITH_AES_256_GCM_SHA384 (0x00a1)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
Cipher Suite: TLS_DH_RSA_WITH_AES_256_CBC_SHA256 (0x0069)
Cipher Suite: TLS_DH_DSS_WITH_AES_256_CBC_SHA256 (0x0068)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_DH_RSA_WITH_AES_256_CBC_SHA (0x0037)
Cipher Suite: TLS_DH_DSS_WITH_AES_256_CBC_SHA (0x0036)
Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0088)
Cipher Suite: TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA (0x0087)
Cipher Suite: TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0086)
Cipher Suite: TLS_DH_DSS_WITH_CAMELLIA_256_CBC_SHA (0x0085)
Cipher Suite: TLS_GOSTR341001_WITH_28147_CNT_IMIT (0x0081)
Cipher Suite: TLS_GOSTR341094_WITH_28147_CNT_IMIT (0x0080)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 (0xc032)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02e)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0084)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_DH_DSS_WITH_AES_128_GCM_SHA256 (0x00a4)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 (0x00a2)
Cipher Suite: TLS_DH_RSA_WITH_AES_128_GCM_SHA256 (0x00a0)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
Cipher Suite: TLS_DH_RSA_WITH_AES_128_CBC_SHA256 (0x003f)
Cipher Suite: TLS_DH_DSS_WITH_AES_128_CBC_SHA256 (0x003e)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
Cipher Suite: TLS_DH_RSA_WITH_AES_128_CBC_SHA (0x0031)
Cipher Suite: TLS_DH_DSS_WITH_AES_128_CBC_SHA (0x0030)
Cipher Suite: TLS_DHE_RSA_WITH_SEED_CBC_SHA (0x009a)
Cipher Suite: TLS_DHE_DSS_WITH_SEED_CBC_SHA (0x0099)
Cipher Suite: TLS_DH_RSA_WITH_SEED_CBC_SHA (0x0098)
Cipher Suite: TLS_DH_DSS_WITH_SEED_CBC_SHA (0x0097)
Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0045)
Cipher Suite: TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA (0x0044)
Cipher Suite: TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0043)
Cipher Suite: TLS_DH_DSS_WITH_CAMELLIA_128_CBC_SHA (0x0042)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 (0xc031)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02d)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_SEED_CBC_SHA (0x0096)
Cipher Suite: TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0041)
Cipher Suite: TLS_RSA_WITH_IDEA_CBC_SHA (0x0007)
Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007)
Cipher Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA (0xc00c)
Cipher Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA (0xc002)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA (0x0010)
Cipher Suite: TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA (0x000d)
Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d)
Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
Compression Methods Length: 1
Compression Methods (1 method)
Extensions Length: 85
Extension: ec_point_formats
Extension: elliptic_curves
Extension: SessionTicket TLS
Extension: signature_algorithms
Type: signature_algorithms (0x000d)
Length: 32
Signature Hash Algorithms Length: 30
Signature Hash Algorithms (15 algorithms)
Extension: Heartbeat

And the TCP reset:

137 2.086290 10.22.163.219 192.168.6.75 TCP 54 443 → 35836 [RST, ACK]
Seq=1 Ack=312 Win=0 Len=0 Frame 137: 54 bytes on wire (432 bits), 54
bytes captured (432 bits) on interface 0 Ethernet II, Src:
Vmware_01:02:14 (00:50:56:01:02:14), Dst: IETF-VRRP-VRID_19
(00:00:5e:00:01:19) Internet Protocol Version 4, Src: 10.22.163.219,
Dst: 192.168.6.75 Transmission Control Protocol, Src Port: 443 (443),
Dst Port: 35836 (35836), Seq: 1, Ack: 312, Len: 0
Source Port: 443
Destination Port: 35836
[Stream index: 3]
[TCP Segment Len: 0]
Sequence number: 1 (relative sequence number)
Acknowledgment number: 312 (relative ack number)
Header Length: 20 bytes
Flags: 0x014 (RST, ACK)
Window size value: 0
[Calculated window size: 0]
[Window size scaling factor: 256]
Checksum: 0x74ff [validation disabled]
Urgent pointer: 0
[SEQ/ACK analysis]

SCHANNEL 36874

Log Name: System Source: Schannel Date: 5/4/2016
9:48:36 AM Event ID: 36874 Task Category: None Level:
Error Keywords: User: SYSTEM Computer:
OCSP.corp.contoso.com Description: An TLS 1.2 connection request was
received from a remote client application, but none of the cipher
suites supported by the client application are supported by the
server. The SSL connection request has failed.

SCHANNEL 36888:

Log Name: System Source: Schannel Date: 5/4/2016
9:48:36 AM Event ID: 36888 Task Category: None Level:
Error Keywords: User: SYSTEM Computer:
OCSP.corp.contoso.com Description: A fatal alert was generated and
sent to the remote endpoint. This may result in termination of the
connection. The TLS protocol defined fatal error code is 40. The
Windows SChannel error state is 1205.

EDIT 1

Found that I can successfully get a non-domain joined 2012 R2 IIS server to respond to a TLS 1.2 hello packet. So my issue appears to be related to being a domain joined system. Still haven't solved that part of the equation.

Best Answer

Certain ciphers seem to have interoperability issues. We had the same problem. The problematic ciphers we found are:

  • TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
  • TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
  • TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
  • TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
  • TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
  • TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)

See also these links.