Cisco ASA – Resolving NAT/ACL Issues Across EZVPN

aclcisco-asanat;vpn

I have an EZVPN running between two locations, Location A has a 5520 and is the EZVPN server, Location B has a 5506 and is a EZVPN client. Currently I'm trying to set NAT and ACL(s) so that hosts on the Location B inside network can access a few servers in Location A's DMZ. Below are my packet traces from both locations. Attached are sanitized configs from both locations.

*

*LocationA-Firewall# packet-tracer input inside tcp <Location B inside ip> 443 <Location A DMZ server ip> 443 detailed
Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   <Location B inside ip>  255.255.255.0   outside
Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (dmz,outside) source static DMZ_Servers DMZ_Servers destination static (location B)-remote_network (location B)-remote_network no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface outside
Untranslate <Location B inside ip>/443 to <Location B inside ip>/443
Phase: 3
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x73cb5f60, priority=11, domain=permit, deny=true
      hits=343658, user_data=0x5, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
      src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
      dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
      input_ifc=dmz, output_ifc=any
Result:
input-interface: dmz
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule*

*

===========================================================

LocationB-Firewall# packet-tracer input inside tcp <Location B inside ip> 443 <Location A DMZ server ip> 443

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop <internet next hop> using egress ifc  outside

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Remote_DMZ Remote_DMZ no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface outside
Untranslate <Location A DMZ server ip>/443 to <Location A DMZ server ip>/443

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group OWL_inside in interface inside
access-list OWL_inside extended permit ip any4 any4
Additional Information:

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:      
nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Remote_DMZ Remote_DMZ no-proxy-arp route-lookup
Additional Information:
Static translate <Location B inside ip>/443 to <Location B inside ip>/443

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: ACCESS-LIST
Subtype: aaa-user
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Remote_DMZ Remote_DMZ no-proxy-arp route-lookup
Additional Information:

Phase: 10
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 568767, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

On those packet traces I'm testing HTTPS access since it's one of the ports I need open across the two zones. Since it looks like the problem with Location A's ASA, I've tried the following NAT and ACL additions (outside_access_in is the ACL applied to the outside interface, dmz_access_in is the ACL applied to the DMZ):

access-list dmz_access_in extended permit tcp object (location B)-remote_network object-group DMZ_Servers eq https

nat (dmz,outside) source static DMZ_Servers DMZ_Servers destination static (location B)-remote_network (location B)-remote_network no-proxy-arp route-lookup

access-list outside_access_in extended permit tcp object (location B)-remote_network object-group DMZ_Servers eq https

None of those worked. Besides those ACL entries I have no entries in my ACLs concerning traffic from Location B. ASP packet captures have not helped either. Sanitized configs for both locations are as follows:

    :
    ASA Version 9.5(2) 
    !
    hostname LocationB-Firewall
    domain-name company.com

    names
    !
    interface GigabitEthernet1/1
     nameif outside
     security-level 0
     ip address dhcp setroute 
    !
    interface GigabitEthernet1/2
     nameif inside
     security-level 100
     ip address (location B inside) 255.255.255.0 
    !
    interface Management1/1
     management-only
     no nameif
     no security-level
     no ip address
    !
    boot system disk0:/asa952-lfbff-k8.SPA
    ftp mode passive
    dns server-group DefaultDNS
     domain-name oscarwinski.com
    object network obj_any
     subnet 0.0.0.0 0.0.0.0
    object-group network Location_A_Networks
     network-object (Location A network).0 255.255.255.0

    object-group network Location_B_Networks
     network-object (Location B network).0 255.255.255.0

    object-group network Remote_DMZ
     network-object host <Location A dmz server IP>


    access-list LocationB_inside extended permit ip any4 any4 
  access-list outside_access_in extended permit tcp host <internet IP> object-group Location_B_Networks eq https 


    no pager
    logging console emergencies
    logging buffered debugging
    logging asdm informational
    mtu outside 1500
    mtu inside 1500
    icmp unreachable rate-limit 1 burst-size 1
    e
    arp timeout 14400
    no arp permit-nonconnected
    nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Location_A_Networks Location_A_Networks no-proxy-arp route-lookup
    nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Remote_DMZ Remote_DMZ no-proxy-arp route-lookup

    !
    object network obj_any
     nat (any,outside) dynamic interface

    access-group outside_access_in in interface outside
    access-group LocationB_inside in interface inside

    timeout xlate 3:00:00
    timeout pat-xlate 0:00:30
    timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
    timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
    timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
    timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
    timeout tcp-proxy-reassembly 0:01:00
    timeout floating-conn 0:00:00
    user-identity default-domain LOCAL
    aaa authentication ssh console LOCAL 
    http server enable
    http 192.168.1.0 255.255.255.0 inside
    http 0.0.0.0 0.0.0.0 inside
    no snmp-server location
    no snmp-server contact
    service sw-reset-button
    crypto ipsec security-association pmtu-aging infinite
    crypto ca trustpool policy
    telnet timeout 5
    no ssh stricthostkeycheck
    ssh (Location A outside interface) 255.255.255.255 outside
    ssh 0.0.0.0 0.0.0.0 inside
    ssh timeout 5
    ssh key-exchange group dh-group1-sha1
    console timeout 0
    vpnclient server (outside interface)
    vpnclient mode network-extension-mode
    vpnclient nem-st-autoconnect
    vpnclient vpngroup vpn password *****
    vpnclient username ezvpn password *****
    vpnclient enable

    !
    dhcpd address (location B network)-(location B network) inside
    dhcpd enable inside
    !
    threat-detection basic-threat
    threat-detection statistics access-list
    no threat-detection statistics tcp-intercept

    dynamic-access-policy-record DfltAccessPolicy

    !
    class-map inspection_default
     match default-inspection-traffic
    !
    !
    policy-map type inspect dns preset_dns_map
     parameters
      message-length maximum client auto
      message-length maximum 512
    policy-map global_policy
     class inspection_default
      inspect dns preset_dns_map 
      inspect ftp 
      inspect h323 h225 
      inspect h323 ras 
      inspect rsh 
      inspect rtsp 
      inspect esmtp 
      inspect sqlnet 
      inspect skinny  
      inspect sunrpc 
      inspect xdmcp 
      inspect sip  
      inspect netbios 
      inspect tftp 
      inspect ip-options 
      inspect pptp
    !
    service-policy global_policy global
    prompt hostname context 

    : end

Location A below:

    ASA Version 9.1(3) 
    !
    hostname LocationA-Firewall
    domain-name company.com
    interface GigabitEthernet0/0
     nameif outside
     security-level 0
     ip address (location A outside) 255.255.255.240 
    !
    interface GigabitEthernet0/1
     nameif inside
     security-level 100
     ip address (location A inside) 255.255.255.0 
    !
    !
    object network (location B)-remote_network
     subnet (location B) 255.255.255.0
    object-group network (Location A)_Networks
     network-object (Location A) 255.255.255.0
    object-group network DMZ_Servers
     network-object <DMZ servers IPs>

    access-list ezvpn_split extended permit tcp object-group (Location A)_Networks object (location B)-remote_network 


    nat (dmz,outside) source static DMZ_Servers DMZ_Servers destination static (location B)-remote_network (location B)-remote_network no-proxy-arp route-lookup

    nat (inside,outside) source static (Location A)_Networks (Location A)_Networks destination static (location B)-remote_network (location B)-remote_network no-proxy-arp route-lookup


access-group outside_access_in in interface outside
access-group inside in interface inside
access-group dmz_access_in in interface dmz
    !
    crypto ipsec ikev1 transform-set TRANS_ESP_3DES_SHA esp-3des esp-sha-hmac 
    crypto ipsec ikev1 transform-set TRANS_ESP_3DES_SHA mode transport
    crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac 
    crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac 
    crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
    crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac 
    crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac 
    crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
    crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac 
    crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac 
    crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac 
    crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
    crypto ipsec ikev1 transform-set ESP-AES128-SHA esp-aes esp-sha-hmac 
    crypto ipsec ikev1 transform-set aes256set esp-aes-256 esp-sha-hmac 
    crypto ipsec security-association pmtu-aging infinite
    crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev1 transform-set TRANS_ESP_3DES_SHA
    crypto dynamic-map vpn_dyn_map 1 set ikev1 transform-set aes256set
    crypto dynamic-map vpn_dyn_map 1 set reverse-route
    crypto dynamic-map vpn_dyn_map 500 set ikev1 transform-set ESP-3DES-MD5
    crypto map VPN 65535 ipsec-isakmp dynamic vpn_dyn_map
    crypto map VPN interface outside
    crypto isakmp identity address 
    crypto isakmp nat-traversal 10
    crypto ikev1 enable outside
    crypto ikev1 policy 5
     authentication pre-share
     encryption aes
     hash sha
     group 2
     lifetime 86400
    crypto ikev1 policy 10
     authentication pre-share
     encryption 3des
     hash sha
     group 2
     lifetime 86400
    crypto ikev1 policy 20
     authentication pre-share
     encryption 3des
     hash md5
     group 2
     lifetime 86400
    crypto ikev1 policy 90
     authentication pre-share
     encryption aes-256
     hash sha
     group 5
     lifetime 86400
    ssh 0.0.0.0 0.0.0.0 inside
    management-access inside
    no vpn-addr-assign aaa
    no vpn-addr-assign dhcp
    group-policy DfltGrpPolicy attributes
     vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-client ssl-clientless
    !username ezvpn password <removed>
     vpn-group-policy DefaultRAGroup
    tunnel-group DefaultL2LGroup ipsec-attributes
     ikev1 pre-shared-key <removed>
    tunnel-group DefaultRAGroup ppp-attributes
     no authentication chap
     authentication ms-chap-v2
    tunnel-group ezvpn type remote-access
    tunnel-group ezvpn general-attributes
     default-group-policy ezvpnpolicy
    tunnel-group ezvpn ipsec-attributes
     ikev1 pre-shared-key <removed>
    tunnel-group (location A outside) type ipsec-l2l
    tunnel-group (location A outside) ipsec-attributes
     ikev1 pre-shared-key <removed>
    !
    class-map preset_dns_map
    class-map global-class
     match access-list global_mpc
    class-map apple_class
     match access-list dmz_access_in
    class-map inspection_default
     match default-inspection-traffic
    !
    policy-map type inspect dns preset_dns_map
     parameters
      message-length maximum 512
    policy-map type inspect http apple_map
     parameters
     match not request header host regex apple_domain
    policy-map global_policy
     description Internet_Netflow
     class inspection_default
      inspect dns preset_dns_map 
      inspect ftp 
      inspect h323 h225 
      inspect h323 ras 
      inspect netbios 
      inspect rsh 
      inspect skinny  
      inspect sqlnet 
      inspect sunrpc 
      inspect tftp 
      inspect xdmcp 
      inspect http 
      inspect icmp 
      inspect rtsp 
      inspect sip  
     class global-class
      flow-export event-type all destination 10.1.1.25
     class apple_class
      inspect http apple_map 
    !
    service-policy global_policy global
    prompt hostname context 
    : end

EDIT: Here's what I see in my logging after running packet traces in both directions:

Location A

Feb  4 10:05:08 10.255.1.1 %ASA-5-111008: User 'enable_15' executed the 'packet-tracer input dmz tcp <DMZ Server IP> 443 <Location B Inside IP> 443' command.
Feb  4 10:05:08 10.255.1.1 %ASA-5-111010: User 'enable_15', running 'CLI' from IP x.x.x.x, executed 'packet-tracer input dmz tcp <DMZ Server IP> 443 <Location B Inside IP> 443'

Location B

Feb  4 08:50:30 <Location B Inside IP> %ASA-5-111008: User 'enable_15' executed the 'packet-tracer input inside tcp <Location B Inside IP> 443 <DMZ Server IP> 443' command.
Feb  4 08:50:30 <Location B Inside IP> %ASA-5-111010: User 'enable_15', running 'CLI' from IP x.x.x.x, executed 'packet-tracer input inside tcp <Location B Inside IP> 443 <DMZ Server IP> 443'
Feb  4 08:51:15 <Location B Inside IP> %ASA-7-609001: Built local-host outside:<DMZ Server IP>
Feb  4 08:51:15 <Location B Inside IP> %ASA-6-302013: Built outbound TCP connection 936480 for outside:<DMZ Server IP>/443 (<DMZ Server IP>/443) to inside:<Location B Inside IP>45/50378 (<Location B Inside IP>45/50378)
Feb  4 08:51:45 <Location B Inside IP> %ASA-6-302014: Teardown TCP connection 936480 for outside:<DMZ Server IP>/443 to inside:<Location B Inside IP>45/50378 duration 0:00:30 bytes 0 SYN Timeout
Feb  4 08:51:45 <Location B Inside IP> %ASA-7-609002: Teardown local-host outside:<DMZ Server IP> duration 0:00:30

Both locations have the following logging options turned on:

logging enable
logging timestamp
logging standby
logging buffer-size 1048576
logging console emergencies
logging monitor debugging
logging buffered debugging
logging trap debugging
logging asdm notifications
logging facility 23
logging host inside <syslog server IP>

I also thought possibly adding the dmz network I wanted to get in into the split tunnel list on the EZVPN would work, it did not:

access-list ezvpn_split extended permit tcp object-group DMZ_Servers object (location B)-remote_network 

Edit: Added ACL statements to both configs

Best Answer

Here's the solution after a lot of pounding my head against this problem: 1) Take out the nat statements at Location B since they interfere with the VPN tunnel:

nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Location_A_Networks Location_A_Networks no-proxy-arp route-lookup
nat (inside,outside) source static Location_B_Networks Location_B_Networks destination static Remote_DMZ Remote_DMZ no-proxy-arp route-lookup

2) Keep the DMZ statement in the split tunnel ACL:

access-list ezvpn_split extended permit tcp object-group DMZ_Servers object (location B)-remote_network 

3) I had the access list entry switch around for no good reason. It was this:

access-list dmz_access_in extended permit tcp object (location B)-remote_network object-group DMZ_Servers eq https

when it should have been this:

access-list dmz_access_in extended permit tcp object-group DMZ_Servers object (location B)-remote_network eq https