The question is old, but it will be found during searching. I spent some time to find solution to the same problem. Thus I decide to write the answer to share my results with others.
Short answer: you should not use IIS Crypto to specify the order of Cipher Suites. I recommend you to click on "Defaults" buttons to remove previously set order and then use Group Policy ("Computer Configuration" \ "Administrative Templates" \ "Network" \ "SSL Configuration Settings") to configure Cipher Suites via local policy.
The reason of the error "protocol or cipher suite mismatch" can be one from below:
- your server support some "bad cipher suite"
- your server don't support some cipher suite, which MUST be supported corresponds to TLS specification.
- your server supports HTTP/2 and it has some protocols from the the black list above the other protocols, which are not in the list. It's typically enough the change the order of the cipher suites to fix the problem.
The exact black list could be different on different systems. You can find in Internet some blacklist. For example, Appendix A of RFC 7540 (Hypertext Transfer Protocol Version 2 (HTTP/2)) contains one list. The cipher suites are TLS_RSA_WITH_AES_128_CBC_SHA for TLS 1.2 (see here) and TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 for TLS 1.3 (see here). The TLS_ECDHE_ECDSA_* are important only is you use certificate with elliptic curves. Other very good cipher suite TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 is not yet implemented by Microsoft. Additionally you can consider to add at least TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA to support connection from old systems and TLS_RSA_WITH_AES_128_CBC_SHA to support very old systems (Android 2.3.7, , Java 6u45, OpenSSL 0.9.8y) and TLS_RSA_WITH_3DES_EDE_CBC_SHA only if you need support IE 8 / XP. Thus you can use today, for example
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
with disabled TLS 1.0, TLS 1.1 to have better security or just
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
if you need to have good security and the best performance.
You can set the following short set of cipher suite for example to solve your problem:
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Below I includes an example of configuration on Windows 10. I configured IIS 10 to have A+ rating from Qualys SSL Labs with RSA 2048 key and free SSL certificate from Let's Encrypt.
I disabled DES 56/56, RC2 128/128, RC2 40/128, RC2 56/128, RC4 128/128, RC4 40/128, RC4 56/128, RC4 64/128, Triple DES 168/168, NULL, MD5, Multi-Protocol Unified Hello, PCT 1.0, SSL 2.0, SSL 3.0 and TLS 1.0/1.1 manually in the registry (see KB245030). I disabled TLS 1.0 and TLS 1.1 protocols only because TLS_FALLBACK_SCSV (Downgrade attack) can't be prevented in IIS till now, which makes impossible to get A+ rating of www.ssllabs.com. I see it as an disadvantage, but TLS 1.2 is supported currently very wide. By the way you can use DisabledByDefault: 1, but Enabled: 1 for TLS 1.0 and TLS 1.1. It could be helpful if you would run SQL Server 2008/2012 on the computer. The web server will not use TLS 1.0 and TLS 1.1, but SQL Server do use.
The most important step, which get many my time and which is your main problem was configuration of the the Cipher Suites. I made it using gpedit.msc. I chosen "Computer Configuration" \ "Administrative Templates" \ "Network" \ "SSL Configuration Settings" and configured "SSL Cipher Suite Order" value to the following
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
The above order could be not optimal and I'm not sure, that all above protocols are supported on IIS 7.5 (I used IIS 10.0 from Windows 10). Nevertheless I'm sure that your problem is related with the list of the Cipher Suite, because I had exactly the same problem like you described during my experiments with the list of Cipher Suite.
In any way, after configure the above settings in Group Polity and rebooting the computer (gpupdate /force /target:computer was not enough in my tests) I get A+ ratings and the following list of testing results of the "Handshake Simulation" part:
One can see that iOS is successfuly supported for the following clients:
Safari 6 / iOS 6.0.1
Safari 7 / iOS 7.1
Safari 8 / iOS 8.4
Safari 9 / iOS 9
The clients which don't supports TLS 1.2 seems for me now not so important and I think that the above configuration is a good compromise between the support of legacy clients and the usage of safe protocols.
Best Answer
It means that the server is configured with a certificate having a certain key strength (perhaps 2048-bit RSA), but the particular cipher suite in use is configured to use different key exchange material of lower strength. This is most likely a Diffie-Hellman (DH) parameter with 1028-bit strength.
To fix this, you can either remove DH-related key exchange ciphersuites from your configuration, or generate and configure stronger DH parameters. More direct guidance is available at weakdh.org