ipsecjunipersonicwallvpn
I have implemented a Site-to-Site IPsec tunnel between a Juniper SSG firewall and a SonicWall.
However, any traffic that I send over the tunnel gets dropped on the SonicWall with an error saying sonicwall ipsec vpn decryption failed selector check error.
I have tried multiple things with no luck including disabling replay protection and enabling fragmented packets handling.
Does anyone know what this error message means? What is the selector check?
You've asked a great question. The question seems very simple, but in fact the answer is somewhat more complex. I'll do my best to answer it in a succinct manner. Also, since you mentioned ISAKMP, I am going to assume you are interested in IKEv1. Things change a little for IKEv2 (well, a lot), but I did want to mention the answer below only correlates to IKEv1.
Phase 1 can be accomplished in two different mods: Main Mode and Aggressive Mode. In either mode, the first message is sent by the Initiator, and the second message is sent by the Responder. Both of these messages include what is known in the cryptography world as a Nonce. A Nonce is simply a randomly generated number to use in key generation. (the term Nonce comes from _N_umber used _Once_). So, after message 1 and message 2, both sides know each other's Nonces.
The Nonce's are combined with the Pre-Shared-Key to create a Seed value for generating secret keys. The relative part of the IKE RFC is here:
For pre-shared keys: SKEYID = prf(pre-shared-key, Ni_b | Nr_b)
SKEYID is the Seed value that will later be used to generate additional secret keys. The Pre-Shared-Key and both Nonce values (Ni_b is the Initiator's Nonce, and Nr_B is the Responder's Nonce) is combined by using a PRF, or Psuedo Random Function. A PRF is like a hashing algorithm, except that the result can be as many bits as you need.
Now, if you were initially doing Main Mode, messages 3 and 4 share the Initiator's and Responder's (respectively) Diffie-Hellman public keys. They both use the these values to generate the Diffie-Hellman shared secret. If you were doing Aggressive mode, these DH Public values are also included in Message 1 and Message 2 (along with the Nonces).
The Seed value is then combined with the DH Shared Secret (and a few other values) to create three Session Keys: a Derevative key, an Authentication key, and an Encryption key. The RFC states it as such:
The result of either Main Mode or Aggressive Mode is three groups of authenticated keying material:
SKEYID_d = prf(SKEYID, g^xy | CKY-I | CKY-R | 0) SKEYID_a = prf(SKEYID, SKEYID_d | g^xy | CKY-I | CKY-R | 1) SKEYID_e = prf(SKEYID, SKEYID_a | g^xy | CKY-I | CKY-R | 2)
SKEYID_d, _a, and _e are the three Session keys mentioned above. SKEYID is the Seed value. g^xy is the DH Shared Secret. CKY-I and CKI-R are the Initiator and Responder Cookies, these are just additional randomly generated values that are used later to identify this particular ISAKMP exchange and security association. 0 1 2 are the literal numbers 0, 1, and 2. The Pipe symbol | represents concatenation.
In any case, all these values are combined together using a PRF which creates three Session keys:
The Authentication Key and Encryption Key are used to secure/encrypt the ensuing Phase2 negotiation. In Main Mode, messages 5 and 6 of Phase1 are also protected by these keys. Furthermore, any future ISAKMP informational exchanges (DPD, Rekey events, Delete messages, etc) are also protected by these two keys.
The Derivative Key is handed to IPsec, and IPsec generates its own Keying material from this Key. If you recall, IPsec does not innately include a Key Exchange mechanism, so the only way for it to acquire secret keys is to either set them manually (which is archaic, and never really done anymore), OR to depend on an external service to provide the keying material, like ISAKMP.
The RFC says it like so:
SKEYID_e is the keying material used by the ISAKMP SA to protect the confidentiality of its messages.
SKEYID_a is the keying material used by the ISAKMP SA to authenticate its messages.
SKEYID_d is the keying material used to derive keys for non-ISAKMP security associations.
With all that said, we can refer back to your question: What key is used to secure the Pre-Shared-Key?
In Main Mode, the Pre-Shared-Key (PSK) is verified in Messages 5 and 6. Message 5 and 6 are Protected by the Session keys ISAKMP generates, described above.
In Aggressive Mode, none of the messages in the negotiation are encrypted. And the PSK is verified in Messages 2 and 3. Notice, I said in both cases the PSK is verified, and I never said the PSK is exchanged. Obviously, if nothing in Aggressive mode is Encrypted, and you simply sent the Pre-Shared-Key across the wire unencrypted, there would be a huge gaping security vulnerability.
Lucky for us, the writers of ISAKMP already thought of that. And as a result, they created a special method for verifying that each party has the correct PSK, without actually sharing it across the wire. There are two items that are use to validate to each Peer that they both have the same PSK: the Identity Method and the Identity Hash.
VPN Peers can choose to identify themselves by various methods; most commonly, peers will simply use their source IP address. But they have the option to use a FQDN or Hostname. Each of these, along with the correlating value for the chosen method, are what make up the Identity Method. So for example, if I had the IP 5.5.5.5, and I wanted to use my IP address to identify myself, my ID Method would effectively be [IP Address, 5.5.5.5]. (Note: BOTH values make up the entire ID Method)
The ID Method is then combined (using a PRF) with the Seed value we discussed earlier (SKEYID), and a few other values, to create the Identity Hash. Recall, that what went into creating SKEYID in the first place was the Pre-Shared-Key.
The ID Method and ID Hash are then sent across the wire, and the other party attempts to re-create the ID Hash using the same formula. If the receiver is able to re-create the same ID Hash, it proves to the receiver that the sender must have had the correct pre-shared-key.
This is described in the RFC here:
To authenticate either exchange the initiator of the protocol generates HASH_I and the responder generates HASH_R where:
HASH_I = prf(SKEYID, g^xi | g^xr | CKY-I | CKY-R | SAi_b | IDii_b ) HASH_R = prf(SKEYID, g^xr | g^xi | CKY-R | CKY-I | SAi_b | IDir_b )
IDii and IDir are the ID Method. And HASH_I and HASH_R are the Initiator and Responder hash's. Both of these are shared in Phase1. From the RFC:
When doing a pre-shared key authentication, Main Mode is defined as follows:
Initiator Responder ---------- ----------- HDR, SA --> <-- HDR, SA HDR, KE, Ni --> <-- HDR, KE, Nr HDR*, IDii, HASH_I --> <-- HDR*, IDir, HASH_RAggressive mode with a pre-shared key is described as follows:
Initiator Responder ----------- ----------- HDR, SA, KE, Ni, IDii --> <-- HDR, SA, KE, Nr, IDir, HASH_R HDR, HASH_I -->
And now, we can finally answer your question fully:
The Pre-Shared-Key is combined using a PRF with Nonces, and a bunch of other values known to anyone else in the negotiation. The result is a value that can only be mutually attained by two parties if both parties started with the same values -- aka, the same pre-shared-key. The resulting value is what is shared on the wire, with allows two parties to verify they have matching Pre-Shared-Keys without actually exposing any information about the Pre-Shared-Key itself.
Main Mode goes a step more secure by also encrypting the exchange of the "resulting value" described above, making it even more difficult to extract any useful information about what the Pre-Shared-Key was.
(it seems I failed miserably at answering this succinctly)
Make sure you save the config to the firebox using the admin (read/write) password. I had a similar issue where I configured IPSec and tried to vpn immediately right after using an iphone and got the same error, once I saved the config to the firebox, I started getting different types of errors that I was able to eventually resolve by following carefully their IPSec vpn from iOS devices guide
Best Answer
This sounds like an issue with traffic-selectors - if you are using policy-based VPN on both sides, you need to make sure the policy (eg: traffic you permit over the tunnel) is the same but reversed on each side. eg:
Side A:
From Side A Network to Side B Network then tunnel
Side B:
From Side B Network to Side A Network then tunnel