Cisco ASA 5506-X – Site-to-Site VPN Tunnel – Return traffic dropped

ciscocisco-asaipsecnat;vpn

I've configured a Cisco ASA 5506-X for a customer of mine and I'm having trouble successfully passing traffic round-trip to the remote network. The VPN tunnel connects successfully according to 'show crypto ipsec sa'. Below is a copy of the scrubbed configuration I'm using currently:

: 
: Serial Number: XXXXXXXXXXX
: Hardware:   ASA5506, 4096 MB RAM, CPU Atom C2000 series 1250 MHz, 1 CPU (4 cores)
:
ASA Version 9.4(1) 
!
hostname ciscoasa01
enable password XXXXXXXXXXXXXXX encrypted
names
!
interface GigabitEthernet1/1
 nameif outside
 security-level 0
 ip address 172.16.10.163 255.255.255.248 
!
interface GigabitEthernet1/2
 nameif inside
 security-level 100
 ip address 192.168.2.1 255.255.255.0 
!
interface GigabitEthernet1/3
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/4
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/5
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/6
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/7
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/8
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Management1/1
 management-only
 nameif management
 security-level 100
 no ip address
!
ftp mode passive
clock timezone CST -6
clock summer-time CDT recurring
dns domain-lookup outside
dns domain-lookup inside
dns server-group DefaultDNS
 name-server 8.8.8.8
 name-server 4.2.2.2
same-security-traffic permit inter-interface
object network obj_any
 subnet 0.0.0.0 0.0.0.0
object network Datacenter
 subnet 10.10.185.0 255.255.255.0
object network Internal
 subnet 192.168.2.0 255.255.255.0
access-list outside_cryptomap extended permit ip object Internal object Datacenter 
access-list outside_cryptomap extended permit icmp object Internal object Datacenter
access-list internet_access extended permit ip object Internal any 
access-list outside_access_in extended permit ip any any
pager lines 24
logging enable
logging asdm informational
mtu outside 1500
mtu inside 1500
mtu management 1500
no failover
no monitor-interface service-module 
icmp unreachable rate-limit 1 burst-size 1
icmp permit any outside
icmp permit any inside
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
nat (inside,outside) source static Internal Internal destination static Datacenter Datacenter no-proxy-arp route-lookup
nat (outside,outside) source static Internal Internal destination static Datacenter Datacenter no-proxy-arp route-lookup
!
object network obj_any
 nat (any,outside) dynamic interface
!
nat (inside,outside) after-auto source dynamic any interface
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 172.16.10.161 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL 
http server enable
http 192.168.2.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
service sw-reset-button
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
crypto ipsec ikev2 ipsec-proposal 3DES
 protocol esp encryption 3des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES256
 protocol esp encryption aes-256
 protocol esp integrity sha-1
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set peer 20.30.40.185 
crypto map outside_map 1 set ikev1 transform-set ESP-3DES-SHA ESP-3DES-SHA-TRANS ESP-AES-256-SHA ESP-AES-256-SHA-TRANS
crypto map outside_map 1 set security-association lifetime seconds 3600
crypto map outside_map interface outside
crypto ca trustpool policy
crypto ikev2 policy 1
 encryption aes-256
 integrity sha
 group 2
 prf sha
 lifetime seconds 3600
crypto ikev1 enable outside
crypto ikev1 policy 120
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 3600
telnet timeout 5
no ssh stricthostkeycheck
ssh 192.168.2.0 255.255.255.0 inside
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
management-access management
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ntp server 24.56.178.140 source outside prefer
group-policy DfltGrpPolicy attributes
 vpn-tunnel-protocol ikev2 l2tp-ipsec 
group-policy GroupPolicy_20.30.40.185 internal
group-policy GroupPolicy_20.30.40.185 attributes
 vpn-tunnel-protocol ikev1 
dynamic-access-policy-record DfltAccessPolicy
username admin password XXXXXXXXXXXXXX encrypted privilege 15
tunnel-group 20.30.40.185 type ipsec-l2l
tunnel-group 20.30.40.185 general-attributes
 default-group-policy GroupPolicy_20.30.40.185
tunnel-group 20.30.40.185 ipsec-attributes
 ikev1 pre-shared-key *****
 peer-id-validate nocheck
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****
!
class-map filtered-class
 match any
class-map global-class
 match default-inspection-traffic
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
 description Filtered Traffic
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ip-options 
 class filtered-class
  sfr fail-open
policy-map global-policy
 class global-class
  inspect icmp 
!
service-policy global_policy global
prompt hostname context 
no call-home reporting anonymous
Cryptochecksum:3ed383cb9ad07574a579a99ea71c2946
: end

When I perform a packet trace from this Cisco ASA to the remote network, it works just fine however when I perform a packet trace from the remote network back to the LAN behind this ASA I get the following:

# packet-tracer input outside tcp 10.10.185.2 3389 192.168.2.5 3389 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.2.5 using egress ifc  inside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit ip any any 
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fffde35d0a0, priority=13, domain=permit, deny=false
    hits=24, user_data=0x7fffcfffed00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
    src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
    dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
    input_ifc=outside, output_ifc=any

Phase: 3
Type: NAT     
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fffdd3fd8a0, priority=0, domain=nat-per-session, deny=false
    hits=470, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
    src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
    dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
    input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fffddb1e950, priority=0, domain=inspect-ip-options, deny=true
    hits=2051, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
    src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
    dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
    input_ifc=outside, output_ifc=any

Phase: 5
Type: SFR
Subtype: 
Result: ALLOW
Config:
class-map filtered-class
 match any
policy-map global_policy
 description Filtered Traffic
 class filtered-class
  sfr fail-open
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fffde1e96f0, priority=71, domain=sfr, deny=false
    hits=25, user_data=0x7fffde1e90a0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
    src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
    dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
    input_ifc=outside, output_ifc=any

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP  
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fffde312c90, priority=70, domain=ipsec-tunnel-flow, deny=false
    hits=1, user_data=0x197c4, cs_id=0x7fffddbdbc70, reverse, flags=0x0, protocol=0
    src ip/id=10.10.185.0, mask=255.255.255.0, port=0, tag=any
    dst ip/id=192.168.2.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
    input_ifc=outside, output_ifc=any

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

I can't seem to locate what's holding up traffic from returning properly. I also cannot seem to get traffic sourced from the LAN behind the Cisco ASA to the Internet and back even though I have NAT rules that should take care of that but I'll look to resolve one thing at a time starting with the VPN tunnel traffic.

Any ideas?

EDIT

Adding some info per dareuja's request.

show run nat output:

# sho run nat
nat (outside,any) source static any any destination static interface Win_Svr service RDP RDP no-proxy-arp
nat (inside,outside) source dynamic obj_any interface
nat (inside,outside) source static Internal Internal destination static Datacenter Datacenter no-proxy-arp route-lookup
!
object network obj_any
 nat (any,outside) dynamic interface

Packet tracer output sourcing from inside IP of 192.168.2.5 to outside at 8.8.8.8

# packet-tracer input inside tcp 192.168.2.5 53 8.8.8.8 53

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 172.16.10.161 using egress ifc  outside

Phase: 2
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (inside,outside) source dynamic obj_any interface
Additional Information:
Dynamic translate 192.168.2.5/53 to 172.16.10.163/53

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4      
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: SFR
Subtype: 
Result: ALLOW
Config:
class-map filtered-class
 match any
policy-map global_policy
 description Filtered Traffic
 class filtered-class
  sfr fail-open
service-policy global_policy global
Additional Information:

Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW 
Config:
nat (inside,outside) source dynamic obj_any interface
Additional Information:

Phase: 7
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: FLOW-CREATION
Subtype: 
Result: ALLOW
Config:
Additional Information:
New flow created with id 13956, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Output of 'show crypto ipsec sa peer 20.30.40.185'

# sh crypto ipsec sa peer 20.30.40.185 detail   
peer address: 20.30.40.185
    Crypto map tag: outside_map, seq num: 1, local addr: 172.16.10.163

      access-list outside_cryptomap extended permit ip 192.168.2.0 255.255.255.0 10.10.185.0 255.255.255.0 
      local ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (10.10.185.0/255.255.255.0/0/0)
      current_peer: 20.30.40.185


      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 11043, #pkts decrypt: 11043, #pkts verify: 11043
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #pkts no sa (send): 0, #pkts invalid sa (rcv): 0
      #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
      #pkts invalid prot (rcv): 0, #pkts verify failed: 0
      #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 0
      #pkts invalid pad (rcv): 0,
      #pkts invalid ip version (rcv): 0,
      #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
      #pkts replay failed (rcv): 0
      #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
      #pkts internal err (send): 0, #pkts internal err (rcv): 0

      local crypto endpt.: 172.16.10.163/4500, remote crypto endpt.: 20.30.40.185/4500
      path mtu 1500, ipsec overhead 82(52), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: CDCAAA61
      current inbound spi : 337E6914

    inbound esp sas:
      spi: 0x337E6914 (863922452)
         transform: esp-aes-256 esp-sha-hmac no compression 
         in use settings ={L2L, Tunnel,  NAT-T-Encaps, IKEv1, }
         slot: 0, conn_id: 12288, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373985/2388)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap: 
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xCDCAAA61 (3452611169)
         transform: esp-aes-256 esp-sha-hmac no compression 
         in use settings ={L2L, Tunnel,  NAT-T-Encaps, IKEv1, }
         slot: 0, conn_id: 12288, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4374000/2386)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap: 
          0x00000000 0x00000001

Best Answer

You won't be able to test a VPN using packet-tracer from remote to local, a drop is expected if you do so. I believe it's due to the expectation that the traffic would be encrypted, receiving an unencrypted packet (even through simulation) is dropped per security. (if I can get it set up on my lab, I'll test this out as well.) EDIT tested in lab as well, sourcing from remote will drop when testing VPN. If anyone else has had a different experience, please let me know.