I've found all sorts of documentation on what IPSEC is and how to (in very non-specific ways) configure it, but I can't seem to configure a Windows Firewall With Advanced Security rule that both A) requires all SQL Server traffic to be encrypted, and B) works.
As an exercise, I created an inbound rule that allows all traffic the default MSSQLSERVER service when it is connected to the domain, and tested that I can connect to the SQL Server instance with that rule.
When I change the Action from "Allow the connection" to "Allow the connection if it is secure", using the "Require the connections to be encrypted" option, I can no longer establish a connection. I can't seem to find any information on how to instruct the client to use that encrypted connection, or whether there are other prerequisites that must be met. I also can't seem to find much on using IPSEC with SQL Server in general.
I know how to encrypt SQL Server connections via SSL, but the primary application that uses this server will not connect using SSL until some future release.
How do I get the client operating system to connect using IPSEC from the Windows Firewall with Advanced Security settings in Windows 7?
Best Answer
Here are the 2 papers you need to read in order to determine the best route you want to go in order to encrypt your server traffic. The first route is server isolation,which is setting up secure communications between specfic domain hosts. The second is domain isolation which is slightly more relaxed in that the only real requirement for communications is that the hosts all be domain members (note these are simplified explanations). These papers all seem to be writen for the vista/2008 timeframe but the concepts still apply. To get to the settings use the windows firewall control panel and choose conenction security rules -> new rule. Look here for more details on connection security rules.