I have two offices (Victoria at IP 1.2.3.4 and Toronto at IP 5.6.7.8) each with pfSense running Strongswan, and each with an IKEv2 IPSec tunnel back to a Cisco ASA 5512 at IP 9.8.7.6. I recently updated software on the ASA from 9.4.2(11) to 9.4.3(4) without issue. Both tunnels came back up and worked fine for 1 day and 17 hours, but (without any configuration changes on either side) the Victoria tunnel has now stopped passing traffic.
The tunnel is established without a problem, but show ipsec sa tells me no traffic is passing. Restarting the tunnel does not make a difference.
ASA1# show ipsec sa peer 1.2.3.4
peer address: 1.2.3.4
Crypto map tag: OUTSIDE_map, seq num: 1, local addr: 9.8.7.6
access-list OUTSIDE_cryptomap_2 extended permit ip 192.168.242.0 255.255.255.0 192.168.244.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.242.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.244.0/255.255.255.0/0/0)
current_peer: 1.2.3.4
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 1428, #pkts decrypt: 1428, #pkts verify: 1428
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0
local crypto endpt.: 9.8.7.6/500, remote crypto endpt.: 1.2.3.4/500
path mtu 1500, ipsec overhead 55(36), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: CB3A6309
current inbound spi : 5E3D8A13
inbound esp sas:
spi: 0x5E3D8A13 (1581091347)
transform: esp-aes-gcm-256 esp-null-hmac no compression
in use settings ={L2L, Tunnel, PFS Group 14, IKEv2, }
slot: 0, conn_id: 167936, crypto-map: OUTSIDE_map
sa timing: remaining key lifetime (sec): 2676
IV size: 8 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xCB3A6309 (3409601289)
transform: esp-aes-gcm-256 esp-null-hmac no compression
in use settings ={L2L, Tunnel, PFS Group 14, IKEv2, }
slot: 0, conn_id: 167936, crypto-map: OUTSIDE_map
sa timing: remaining key lifetime (sec): 2676
IV size: 8 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001
A packet capture looking for dropped packets gives me this, but doesn't tell me what rule is being applied.
939: 20:11:44.438591 0023.ab3f.8255 24e9.b315.cddf 0x0800 Length: 89
192.168.244.114.51353 > 192.168.242.200.53: [no cksum] udp 47 [tos 0x10] (ttl 63, id 8826) Drop-reason: (acl-drop) Flow is denied by configured rule
Here is some of the relevant configuration; as far as I can see both offices are identical. If anyone has any suggestions as to what could be happening here, I'd appreciate it!
ASA1# show running-config crypto map
crypto map OUTSIDE_map 1 match address OUTSIDE_cryptomap_2
crypto map OUTSIDE_map 1 set pfs group14
crypto map OUTSIDE_map 1 set peer 1.2.3.4
crypto map OUTSIDE_map 1 set ikev2 ipsec-proposal AESGCM
crypto map OUTSIDE_map 1 set security-association lifetime seconds 3600
crypto map OUTSIDE_map 1 set nat-t-disable
crypto map OUTSIDE_map 2 match address OUTSIDE_cryptomap_3
crypto map OUTSIDE_map 2 set pfs group14
crypto map OUTSIDE_map 2 set peer 5.6.7.8
crypto map OUTSIDE_map 2 set ikev2 ipsec-proposal AESGCM
crypto map OUTSIDE_map 2 set security-association lifetime seconds 3600
crypto map OUTSIDE_map 2 set nat-t-disable
crypto map OUTSIDE_map interface OUTSIDE
ASA1# show running-config access-list OUTSIDE_cryptomap_2
access-list OUTSIDE_cryptomap_2 extended permit ip object NOC-network object Victoria-network
ASA1# show running-config access-list OUTSIDE_cryptomap_3
access-list OUTSIDE_cryptomap_3 extended permit ip object NOC-network object Toronto-network
ASA1# show running-config nat
nat (INSIDE,OUTSIDE) source static NOC-network NOC-network destination static Victoria-network Victoria-network no-proxy-arp route-lookup
nat (INSIDE,OUTSIDE) source static NOC-network NOC-network destination static Toronto-network Toronto-network no-proxy-arp route-lookup
ASA1# show running-config tunnel-group
tunnel-group 1.2.3.4 type ipsec-l2l
tunnel-group 1.2.3.4 general-attributes
default-group-policy GroupPolicy_Victoria
tunnel-group 1.2.3.4 ipsec-attributes
ikev1 pre-shared-key *****
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
tunnel-group 5.6.7.8 type ipsec-l2l
tunnel-group 5.6.7.8 general-attributes
default-group-policy GroupPolicy_Toronto
tunnel-group 5.6.7.8 ipsec-attributes
ikev1 pre-shared-key *****
isakmp keepalive threshold 15 retry 2
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
ASA1# show running-config crypto ikev2
crypto ikev2 policy 2
encryption aes-gcm-256
integrity null
group 21 24
prf sha512
lifetime seconds 28800
crypto ikev2 policy 3
encryption aes-256
integrity sha512
group 21 24
prf sha512
lifetime seconds 28800
crypto ikev2 enable OUTSIDE
ASA1# show running-config crypto ipsec
crypto ipsec ikev2 ipsec-proposal AES256-SHA512
protocol esp encryption aes-256
protocol esp integrity sha-512
crypto ipsec ikev2 ipsec-proposal AESGCM
protocol esp encryption aes-gcm-256
protocol esp integrity sha-512
crypto ipsec ikev2 sa-strength-enforcement
crypto ipsec security-association pmtu-aging infinite
#ASA1 show running-config all sysopt
no sysopt traffic detailed-statistics
no sysopt connection timewait
sysopt connection tcpmss 1380
sysopt connection tcpmss minimum 0
sysopt connection permit-vpn
sysopt connection reclassify-vpn
no sysopt connection preserve-vpn-flows
no sysopt radius ignore-secret
no sysopt noproxyarp OUTSIDE
no sysopt noproxyarp INSIDE
no sysopt noproxyarp DMZ1
no sysopt noproxyarp management
Best Answer
Your scenario looks very similar to something you'll see if you have a stale SPI on your ASA that is "trumping" the valid active SPI. The bad news is, if this is the case...you can only correct this by rebooting the ASA itself.
Something you can try is running a packet tracer to simulate traffic and check that you see a "VPN Encrypt Phase" in the output. If you do, but the traffic fails to encrypt then it could be match against a stale SPI that isn't active.
You can also use
show asp table classify cryptoand look for matches on your encryption domain. You do not want to see multiple matches with the samecs_id. See this example: (notice how they networks are exactly the same but they have differentcs_idand only one has hitscounts?)The above example is something I've taken directly from troubleshooting exactly the same problem you are describing in which case a reboot corrected the issue. I also had an issue where the same firewall stopped passing traffic on an active tunnel with encaps and decaps for only a few hosts. Bouncing the tunnel corrected the problem.
These issues only started after upgrading their VPNs to IKEv2. I suspect if you re-key you're tunnels to often on IKEv2 it seems to be very likely you're have issues with SPIs. You may want to try increasing the lifetime of your phase two tunnels since you have it set to 3600 right now (My customer was hitting the default data lifetime on the Phase 2 tunnels every 4 minutes and rekeying, so I set the data lifetime to unlimited)