Firefox does not like our internal certificates. I'm trying to figure out why…
tl;dr – Our internal IIS servers, signed with our internal CA, present a "Secure Connection Failed" page, with technical details that say "Connection Not Encrypted". The certificate is installed in Firefox's internal certificate store.
Here are our certificates https://www.highlands.edu/site/is-certification-authority
Unfortunately, we can't expose said internal servers for you to see the exact error page. Here are screenshots though. https://imgur.com/a/dmMdG
The weird part of all this is that our internal certificates work fine on Apache (suggesting that the problem is IIS). However, our IIS servers work fine with any other certificates, such as third party (GeoTrust) or self signed (suggesting that the problem is the cert).
If I add an exception, such as someinternal.highlands.edu under the about:config page to the "security.tls.insecure_fallback_hosts" then the site works.
If I look at IIS error logs I see the following two errors over and over when I hit it with FireFox (but not Chrome, IE, Safari, etc)
An TLS 1.2 connection request was received from a remote client
application, but none of the cipher suites supported by the client
application are supported by the server. The SSL connection request
has failed.A fatal alert was generated and sent to the remote endpoint. This may
result in termination of the connection. The TLS protocol defined
fatal error code is 40. The Windows SChannel error state is 1205.
Now, in the album I posted above (https://imgur.com/a/dmMdG), the last two screenshots show a packet capture from Wireshark. It appears that Firefox does not support SHA512, which is kind of supported by this article (http://blogs.technet.com/b/silvana/archive/2014/03/14/schannel-errors-on-scom-agent.aspx). I'm not exactly sure this is true, and it seems like a silly thing for Firefox to drop support though (this previously worked), especially if every other browser in the world supports this.
So there's everything we've found, and some of my assumptions. Does anyone know what is actually going on with Firefox? I can't find anything to validate my assumptions and I'm still not sure if this is something we've misconfigured in IIS, our internal CA, or just a bug with Firefox.
Best Answer
Firstly, do you have this update installed on the server?
https://support.microsoft.com/en-us/kb/2973337
SHA512 as a hash algorithm on TLS 1.2 is disabled by default in Windows. The above patch will enable it and hopefully fix your problem.
The reason your third party certificates work now is because they won't be using the SHA512 hash (in my experience this is rare to see, people are only just migrating away from SHA-1 and usually move to the SHA256 variant of SHA-2).
If that fails then...
Can you run another Wireshark trace when attempting to connect with Firefox and post the output of the TLS client hello cipher suite list.
Can you then post the list of cipher suites your IIS server is currently configured to accept.
I suspect that there are no cipher suites that appear in both lists.
If it turns out that this is the case, then you will want to reconfigure your server to accept a wider list of cipher suites. The best tool for the job in this case is IIS Crypto. I always recommend using the 'Best Practices' button, it's a one click setup to reconfigure your server according to current best practice. Although, as always, test it first!
https://www.nartac.com/Products/IISCrypto/
It's also curious that Firefox doesn't advertise the SHA512 hash algorithm in the client hello. I've just verified the same behaviour on my workstation using Firefox 37. I'll do some more digging around this point as I'm curious...