Can Windows 7 access an RD farm that uses SHA512/4096-bit certificates

certificaterdpwindows 7windows-server-2012

I have a Server 2012 RD farm that works fine if configured with the self-signed certs generated by Server Manager, but not with certs from our internal CA.

With the self-signed certs, our remote clients can connect but of course get security warnings due to the untrusted certificates. The clients do trust our root CA, so we should be able to eliminate those by using certs from our internal CA instead.

However, when I do configure the farm to use certs from our internal CA (which runs AD CS on Server 2008 R2,) clients can log into the RD Web site but cannot open RDP sessions. They get errors such as the following on Windows 7 (none of the clients are newer than 7 so I haven't tried 8) :

Your computer can't connect to the remote computer because an error
occurred on the remote computer that you want to connect to. Contact
your network administrator for assistance.

or

Your computer can't connect to the remote computer because the Remote
Desktop Gateway and the remote computer are unable to exchange
policies. This could happen due to one of the following reasons:

  1. The remote computer is not capable of exchanging policies with the Remote Desktop Gateway.
  2. The remote computer's configuration does not permit a new connection.
  3. The connection between the Remote Desktop Gateway and the remote computer ended.

Contact your network administrator for assistance.

In turn these will appear in the server logs:

(Source: Schannel; Event ID: 36874)
[sic] An TLS 1.2 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.

(Source: Schannel; Event ID: 36888)
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 1205.

Our root CA and issuing CA are both using SHA512 hashes and 4096-bit public keys. I notice that the self-signed certs from Server Manager that do work are using SHA256 and 2048-bit keys, so I'm wondering if the stronger encryption is not supported by RDP in Windows 7.

(I can't test this easily as I can't make our CA hand out a cert using SHA256, I'm guessing because the CA's own public key is too big. Even if it did, the client would still need SHA512 to validate the issuing CA against the root CA.)

The strange thing is that it does work with our certs except for "RD Connection Broker – Enable Single Sign On." If I leave that one set to the self-signed cert but use ours for the other three, everything works basically fine (apart from users having to enter their password three times.)

In that case, Internet Explorer on the client trusts one of our certs without issue even though it has SHA512. This makes it seem rather odd that the stronger crypto would trip up RDP – I would have assumed they'd both use a provider built into Windows.

Best Answer

Tentatively, it looks to me like the answer is, no, Windows 7 cannot access an RD farm (at least a Server 2012 one) if the "RD Connection Broker - Enable Single Sign On" certificate is using a certificate signed using an SHA512 hash.

I set up a new root authority and issuing authority in a very similar manner to how our existing ones were set up, except I gave both 2048-bit keys and SHA256 hashes. I applied a cert from this new issuing authority as the "RD Connection Broker - Enable Single Sign On" certificate, and the problem went away.

The only real difference in CA setups other than the crypto strength was that the new issuing authority was set up in Server 2012 instead of Server 2008 R2, but I expect that wouldn't make any difference.

I'd still welcome a more definitive answer.