Windows 7/8 Strongswan IKEv2 Wrong Gateway

Figured this out. @ecdsa pointed me in the right direction, and I finally was able to solve the problem by following this guide.

ipsec pki --gen --type rsa --size 4096 --outform pem > vpnca.key.pem
ipsec pki --self --flag serverAuth --in vpnca.key.pem --type rsa --digest sha1 \
    --dn "C=US, O=Example Company, CN=Example VPN CA" --ca > vpnca.crt.der
ipsec pki --gen --type rsa --size 4096 --outform pem > vpn.example.com.key.pem
ipsec pki --pub --in vpn.example.com.key.pem --type rsa > vpn.example.com.csr
ipsec pki --issue --cacert vpnca.crt.der --cakey vpnca.key.pem --digest sha1 \
    --dn "C=US, O=Example Company, CN=vpn.example.com" \
    --san "vpn.example.com" --flag serverAuth --outform pem \
    < vpn.example.com.csr > vpn.example.com.crt.pem 
openssl rsa -in vpn.example.com.key.pem -out vpn.example.com.key.der -outform DER

cp vpnca.crt.der /etc/ipsec.d/cacerts
cp vpn.example.com.crt.pem /etc/ipsec.d/certs
cp vpn.example.com.key.der /etc/ipsec.d/private

About the error

The error message was "Error 13801: IKE authentication credentials are unacceptable", which sounded like my user credentials weren't working. However, this is a message about authenticating the server, which is done (per my configuration) by the server's SSL certificate. Microsoft has published documentation on Troubleshooting IKEv2 VPN Connections that lists possible causes for this error:

• The certificate is expired.
• The trusted root for the certificate is not present on the client.
• The subject name of the certificate does not match the remote computer.
• The certificate does not have the required Enhanced Key Usage (EKU) values assigned.

In my case, my problem had to do with the EKU values. Following the guide I linked at the top, I was able to generate a certificate with the correct EKU values, and it worked great.

To troubleshoot this, you can disable EKU checking on your Windows client (of course, this should only be done for testing):

• Launch regedit
• Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\RasMan\Parameters
• Add a DWORD called DisableIKENameEkuCheck, and set its value to 1
• Microsoft documentation instructs you to reboot after doing this, but I didn't need to in order for this to take effect.

You can do exactly that with the rightca option. Just configure the distinguished name of the CA for which you want to accept client certificates.

You actually don't even have to set that option as strongSwan accepts all client certificates for which it can successfully verify the trust chain to a trusted CA certificate (i.e. the option is mainly to restrict clients to a specific CA if there are multiple trusted CAs).