Cisco ASA 5510 – Troubleshooting ACL-Drop Packets

ciscocisco-asanat;

I am trying to troubleshoot the cause of "Flow is denied by configured rule (acl-drop)" packets on "show asp drop" on an ASA 8.4

I have done packet captures and packet traces, but I am not able to use the information to proceed further. Here is a random entry from my Cisco capture, the 188 address is our external IP:

116: 11:31:20.695917 54.67.53.97.443 > 188.188.188.188.11585: FP 2560128298:2560128712(414) ack 2362777837 win 233 <nop,nop,timestamp 246272938 1558888826

Here is the packet trace:

packet-trac input outside tcp 52.216.128.211 https 188.188.188.188 11585

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   188.188.188.188   255.255.255.255 identity

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 4
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: NP Identity Ifc
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Now, I understand why packets to 188.188.188.188 11585 drop, there is no ACL pointing it to anything, but this appears to be returning traffic to a computer on the LAN, so why is it triggering a drop? This is what my NAT looks like:

object network INSIDE-HOSTS

 subnet 10.10.14.0 255.255.254.0

 nat (inside,outside) dynamic interface

I am trying to troubleshoot ISP connection issues, some speed tests show a decent amount of retransmit packets, so that prompted me to look at asp drop.

Thank you for any tips or pointers. I've searched google for about a week, but am stuck.

: Saved
:
ASA Version 8.4(2)8 
!
hostname asa5510
enable password xxx encrypted
passwd xxx encrypted
names
!
interface Ethernet0/0
 speed 100
 duplex full
 nameif outside
 security-level 0
 ip address 188.188.188.188 255.255.255.224 
!
interface Ethernet0/1
 nameif inside
 security-level 100
 ip address 10.10.15.1 255.255.254.0 
!
interface Ethernet0/1.20
 description VLAN 20
 vlan 20
 nameif VLAN20
 security-level 90
 ip address 192.168.5.2 255.255.255.0 
!
interface Ethernet0/1.30
 description VLAN 30
 vlan 30
 nameif VLAN30
 security-level 90
 ip address 192.168.10.2 255.255.255.0 
!
interface Ethernet0/2
 nameif DMZ
 security-level 50
 ip address 10.10.10.1 255.255.255.0 
!
interface Ethernet0/3
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Management0/0
 shutdown
 nameif management
 security-level 100
 ip address 192.168.15.1 255.255.255.0 
 management-only
!
regex domainlist1 "\.pandora\.com"
regex domainlist2 "\.windowsupdate\.com"
regex domainlist3 "\.xxx\.com"
regex contenttype "Content-Type"
regex applicationheader "application/.*"
boot system disk0:/asa842-8-k8.bin
no ftp mode passive
same-security-traffic permit intra-interface
object network SMTP_SERVER
 host 10.10.15.13
object network INSIDE-HOSTS
 subnet 10.10.14.0 255.255.254.0
object network VPN-HOSTS
 subnet 10.10.5.0 255.255.255.0
object network ADMIN-SRV
 host 10.10.15.10
object network TC_RESTRICTED
 subnet 10.10.6.0 255.255.255.0
object network OWA
 host 10.10.15.11
object network SECURITY_2
 host 10.10.15.16
object network 5315
 subnet 10.10.16.0 255.255.255.0
object network DT_COLO
 subnet 10.10.1.0 255.255.255.0
object network HQ
 subnet 10.10.14.0 255.255.254.0
object network SECURITY_1
 host 10.10.15.54
object network KAYAKO
 host 10.10.15.25
object network MAGENTODEV
 host 10.10.15.20
object network VLAN20
 subnet 192.168.5.0 255.255.255.0
object network VLAN30
 subnet 192.168.10.0 255.255.255.0
object network OWA-outside
 host 188.188.188.165
object network RTMP
 host 10.10.15.89
object network DMZ
 subnet 10.10.10.0 255.255.255.0
object-group network MAILPROTECTOR
 network-object host 52.0.70.91
 network-object host 52.0.74.211
 network-object host 52.0.31.31
 network-object host 52.1.23.3
 network-object host 52.1.140.110
 network-object host 52.1.182.179
 network-object host 54.152.160.187
 network-object host 54.152.160.142
object-group network BLOCK_WEB
 network-object host 10.10.14.1
object-group network SAFE_ALL
 network-object host 10.10.15.12
 network-object host 10.10.15.13
access-list INCOMING extended deny ip object-group BLOCK_WEB any 
access-list INCOMING extended permit tcp any object OWA eq https 
access-list INCOMING extended permit tcp object VPN-HOSTS 10.10.15.0 255.255.255.0 
access-list INCOMING extended permit tcp any object KAYAKO eq www 
access-list INCOMING extended permit tcp 14.141.67.0 255.255.255.0 object KAYAKO 
access-list INCOMING extended permit tcp 14.141.58.0 255.255.255.0 object KAYAKO 
access-list INCOMING extended permit tcp object-group MAILPROTECTOR object ADMIN-SRV eq ldap 
access-list INCOMING extended permit tcp object-group MAILPROTECTOR object SMTP_SERVER eq smtp 
access-list INCOMING extended permit tcp any object OWA eq 993 
access-list INCOMING extended permit tcp any object OWA eq 587 
access-list INCOMING extended permit tcp host 113.190.242.147 object MAGENTODEV 
access-list INCOMING extended permit tcp host 101.99.23.40 object MAGENTODEV 
access-list INCOMING extended permit tcp any object SECURITY_1 eq www 
access-list INCOMING extended permit tcp any object SECURITY_2 eq www 
access-list INCOMING extended permit tcp any object SECURITY_1 eq 6036 
access-list INCOMING extended permit tcp any object SECURITY_2 eq 6036 
access-list INCOMING extended permit tcp host 118.70.109.213 object MAGENTODEV 
access-list SPLIT_VPN standard permit 10.10.1.0 255.255.255.0 
access-list SPLIT_VPN standard permit 10.10.14.0 255.255.254.0 
access-list TC_RESTRICTED standard permit host 10.10.15.26 
access-list ACCOUNTING standard permit host 10.10.15.86 
access-list ACCOUNTING standard permit host 10.10.15.81 
access-list ACCOUNTING standard permit host 10.10.15.87 
access-list ACCOUNTING standard permit host 10.10.14.44 
access-list ACCOUNTING standard permit host 10.10.15.83 
access-list ACCOUNTING standard permit host 10.10.14.30 
access-list SHAREPOINT standard permit host 10.10.15.26 
access-list SHAREPOINT standard permit host 10.10.15.75 
access-list SHAREPOINT standard permit host 10.10.15.28 
access-list SHAREPOINT standard permit host 10.10.15.84 
access-list SHAREPOINT standard permit host 10.10.15.41 
access-list SHAREPOINT standard permit host 10.10.15.82 
access-list SHAREPOINT standard permit host 10.10.15.25 
access-list SHAREPOINT standard permit host 10.10.15.85 
access-list SHAREPOINT standard permit host 10.10.14.66 
access-list SHAREPOINT standard permit host 10.10.15.66 
access-list SHAREPOINT standard permit host 10.10.15.19 
access-list SHAREPOINT standard permit host 10.10.15.98 
access-list SHAREPOINT standard permit host 10.10.15.88 
access-list SHAREPOINT standard permit host 10.10.14.197 
access-list SHAREPOINT standard permit host 10.10.15.94 
access-list SHAREPOINT standard permit host 10.10.14.3 
access-list HQ_5315 extended permit ip 10.10.14.0 255.255.254.0 10.10.16.0 255.255.255.0 
access-list netflow-export extended permit ip any any 
access-list inside_mpc extended permit tcp any any eq www 
access-list HQ_DT extended permit ip 10.10.14.0 255.255.254.0 10.10.1.0 255.255.255.0 
access-list HQ_DT extended permit ip 10.10.1.0 255.255.255.0 10.10.14.0 255.255.254.0 
access-list OUTGOING extended deny ip object-group BLOCK_WEB any 
access-list OUTGOING extended permit ip object-group SAFE_ALL any 
access-list OUTGOING extended deny tcp any any eq smtp 
access-list OUTGOING extended permit ip any any 
access-list VLAN20 extended permit tcp any host 10.10.15.11 
access-list VLAN20 extended deny ip any object INSIDE-HOSTS 
access-list VLAN20 extended permit ip any any 
pager lines 24
logging enable
logging list url_logging message 304001
logging list ASP_DROP message 101000-107000
logging trap url_logging
logging asdm informational
logging device-id hostname
logging host inside 10.10.15.22
flow-export destination inside 10.10.15.22 9996
flow-export template timeout-rate 1
flow-export delay flow-create 60
mtu outside 1500
mtu inside 1500
mtu VLAN20 1500
mtu VLAN30 1500
mtu DMZ 1500
mtu management 1500
ip local pool Restricted 10.10.6.10-10.10.6.100 mask 255.255.255.0
ip local pool VPNClients 10.10.5.10-10.10.5.100 mask 255.255.255.0
ip verify reverse-path interface outside
no failover
no monitor-interface inside
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-645-106.bin
no asdm history enable
arp timeout 14400
nat (inside,outside) source static any any destination static TC_RESTRICTED TC_RESTRICTED
nat (inside,outside) source static any any destination static VPN-HOSTS VPN-HOSTS
nat (inside,outside) source static HQ HQ destination static DT_COLO DT_COLO route-lookup
nat (inside,outside) source static HQ HQ destination static 5315 5315 route-lookup
!
object network SMTP_SERVER
 nat (inside,outside) static interface service tcp smtp smtp 
object network INSIDE-HOSTS
 nat (inside,outside) dynamic interface
object network ADMIN-SRV
 nat (inside,outside) static interface service tcp ldap ldap 
object network OWA
 nat (inside,any) static 188.188.188.165
object network SECURITY_2
 nat (inside,outside) static 188.188.188.176
object network SECURITY_1
 nat (inside,outside) static 188.188.188.177
object network KAYAKO
 nat (inside,outside) static 188.188.188.180
object network MAGENTODEV
 nat (inside,outside) static 188.188.188.171
object network VLAN20
 nat (VLAN20,outside) dynamic 188.188.188.163
object network VLAN30
 nat (VLAN30,outside) dynamic interface
object network RTMP
 nat (inside,outside) static 188.188.188.181
object network DMZ
 nat (DMZ,outside) dynamic 188.188.188.164
access-group INCOMING in interface outside
access-group OUTGOING out interface outside
access-group VLAN20 in interface VLAN20
route outside 0.0.0.0 0.0.0.0 188.188.188.161 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
aaa-server DNS01 protocol ldap
aaa-server DNS01 (inside) host 10.10.15.9
 ldap-base-dn DC=acmecorp,DC=domain
 ldap-scope subtree
 ldap-login-password *****
 ldap-login-dn admin
 ldap-over-ssl enable
 server-type microsoft
 group-search-timeout 300
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL 
aaa authentication http console LOCAL 
http server enable
http 0.0.0.0 0.0.0.0 inside
snmp-server group public v3 auth 
snmp-server user admin public v3 encrypted auth md5 xxx 
snmp-server host inside 10.10.15.22 community *****
no snmp-server location
no snmp-server contact
snmp-server community *****
crypto ipsec ikev2 ipsec-proposal aes-3des-des-sha1
 protocol esp encryption aes 3des des
 protocol esp integrity sha-1
crypto map IPSECMAP 10 match address HQ_DT
crypto map IPSECMAP 10 set pfs 
crypto map IPSECMAP 10 set peer 66.128.157.196 
crypto map IPSECMAP 10 set ikev2 ipsec-proposal aes-3des-des-sha1
crypto map IPSECMAP 15 match address HQ_5315
crypto map IPSECMAP 15 set pfs 
crypto map IPSECMAP 15 set peer 76.79.165.138 
crypto map IPSECMAP 15 set ikev2 ipsec-proposal aes-3des-des-sha1
crypto map IPSECMAP interface outside
crypto ca trustpoint my.acmecorp.trustpoint
 enrollment terminal
 fqdn vpn.acmecorp.com
 subject-name CN=vpn.acmecorp.com,OU=Remote_Access,O=acmecorp,C=US,St=California,L=City
 serial-number
 keypair my.acmecorp.key
 crl configure
crypto ca certificate chain my.acmecorp.trustpoint
 certificate ca 0301
xxx
  quit
 certificate 2b1803127e1caf
xxx
  quit
crypto isakmp identity address 
no crypto isakmp nat-traversal
crypto ikev2 policy 1
 encryption 3des
 integrity sha
 group 2
 prf sha
 lifetime seconds 86400
crypto ikev2 enable outside
telnet 10.10.14.0 255.255.254.0 inside
telnet timeout 1440
ssh 0.0.0.0 0.0.0.0 outside
ssh 0.0.0.0 0.0.0.0 inside
ssh 0.0.0.0 0.0.0.0 DMZ
ssh timeout 60
console timeout 0
management-access inside
dhcpd address 10.10.15.100-10.10.15.250 inside
dhcpd dns 10.10.15.10 10.10.1.9 interface inside
dhcpd domain acmecorp.domain interface inside
!
priority-queue outside
  queue-limit   785
  tx-ring-limit 9
threat-detection basic-threat
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
ssl trust-point my.acmecorp.trustpoint outside
webvpn
 enable outside
 anyconnect-essentials
 anyconnect image disk0:/anyconnect-win-3.1.09013-k9.pkg 5
 anyconnect image disk0:/anyconnect-macosx-i386-3.1.09013-k9.pkg 12
 anyconnect profiles Accounting disk0:/accounting_anyconnect_profile.xml
 anyconnect profiles AdminVPN disk0:/adminvpn_anyconnect_profile.xml
 anyconnect profiles Timeclock disk0:/timeclock_anyconnect_profile.xml
 anyconnect enable
 tunnel-group-list enable
group-policy Accounting internal
group-policy Accounting attributes
 vpn-idle-timeout 1440
 vpn-tunnel-protocol ssl-client 
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value ACCOUNTING
 webvpn
  anyconnect keep-installer installed
  anyconnect profiles value Accounting type user
group-policy AdminVPN internal
group-policy AdminVPN attributes
 dns-server value 10.10.15.10
 vpn-idle-timeout 2880
 vpn-tunnel-protocol ssl-client 
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value SPLIT_VPN
 default-domain value acmecorp.domain
 webvpn
  anyconnect keep-installer installed
  anyconnect ssl keepalive 300
  anyconnect ssl rekey time 300
  anyconnect ssl rekey method new-tunnel
  anyconnect profiles value AdminVPN type user
  anyconnect ask enable
  always-on-vpn profile-setting
group-policy Sharepoint internal
group-policy Sharepoint attributes
 dns-server value 10.10.15.10
 vpn-idle-timeout 2880
 vpn-tunnel-protocol ssl-client 
 group-lock value SHAREPOINT
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value SHAREPOINT
 default-domain value acmecorp.domain
group-policy Timeclock internal
group-policy Timeclock attributes
 vpn-idle-timeout 360
 vpn-tunnel-protocol ssl-client 
 password-storage enable
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value TC_RESTRICTED
 address-pools value RESTRICTED
username george password xxx encrypted
username george attributes
 group-lock value ACCOUNTING
 service-type remote-access
username greg password xxx encrypted
username greg attributes
 group-lock value SHAREPOINT
username dasia password xxx encrypted
username dasia attributes
 group-lock value ACCOUNTING
username admin password xxx encrypted
username admin attributes
 service-type admin
username gary password xxx encrypted
username gary attributes
 group-lock value SHAREPOINT
username niraj password xxx encrypted
username niraj attributes
 group-lock value SHAREPOINT
username leighann password xxx encrypted
username leighann attributes
 group-lock value SHAREPOINT
username chrisw password xxx encrypted
username chrisw attributes
 group-lock value SHAREPOINT
username dorene password xxx encrypted
username dorene attributes
 group-lock value ACCOUNTING
username glover password xxx encrypted
username glover attributes
 group-lock value SHAREPOINT
username natasha password xxx encrypted
username natasha attributes
 group-lock value SHAREPOINT
username anya password xxx encrypted
username anya attributes
 group-lock value ACCOUNTING
tunnel-group TIMECLOCK type remote-access
tunnel-group TIMECLOCK general-attributes
 address-pool RESTRICTED
 default-group-policy Timeclock
tunnel-group TIMECLOCK webvpn-attributes
 group-alias Timeclock enable
tunnel-group ACCOUNTING type remote-access
tunnel-group ACCOUNTING general-attributes
 address-pool VPNClients
 default-group-policy Accounting
tunnel-group ACCOUNTING webvpn-attributes
 group-alias Accounting enable
tunnel-group AdminVPN type remote-access
tunnel-group AdminVPN general-attributes
 address-pool VPNClients
 default-group-policy AdminVPN
tunnel-group AdminVPN webvpn-attributes
 group-alias AdminVPN enable
tunnel-group SHAREPOINT type remote-access
tunnel-group SHAREPOINT general-attributes
 address-pool Restricted
 default-group-policy Sharepoint
tunnel-group SHAREPOINT webvpn-attributes
 group-alias Sharepoint enable
tunnel-group 123123123 type ipsec-l2l
tunnel-group 123123123 ipsec-attributes
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****
tunnel-group 123123123 type ipsec-l2l
tunnel-group 123123123 ipsec-attributes
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****
!
class-map netflow-export-class
 match access-list netflow-export
class-map global-class
 match any
class-map type regex match-any DomainBlockList
 match regex domainlist2
 match regex domainlist3
 match regex domainlist1
class-map priority_voip
 match dscp ef 
 match tunnel-group 123123123
class-map type inspect http match-any BlockDomainsClass
 match request header host regex class DomainBlockList
class-map inspection_default
 match default-inspection-traffic
class-map httptraffic
 match access-list inside_mpc
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 1500
policy-map type inspect http http_inspection_policy
 parameters
  protocol-violation action drop-connection
 class BlockDomainsClass
  reset log
 match request method connect
  drop-connection log
policy-map inside_policy
 class httptraffic
  inspect http http_inspection_policy 
policy-map vlan20_policy
 class httptraffic
  inspect http http_inspection_policy 
policy-map global_policy
 class inspection_default
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect netbios 
  inspect tftp 
  inspect ip-options 
  inspect dns preset_dns_map 
  inspect icmp 
 class netflow-export-class
  flow-export event-type all destination 10.10.15.22
 class class-default
  user-statistics accounting
policy-map outside_policy
 class class-default
policy-map inside-policy
 class httptraffic
  inspect http http_inspection_policy 
!
service-policy outside_policy interface outside
service-policy inside_policy interface inside
service-policy vlan20_policy interface VLAN20
prompt hostname context 
no call-home reporting anonymous
hpm topN enable
Cryptochecksum:xxx
: end

Best Answer

Now, I understand why packets to 188.188.188.188 11585 drop, there is no ACL pointing it to anything, but this appears to be returning traffic to a computer on the LAN, so why is it triggering a drop?

As I see, you were trying to apply the information from your traffic capture to test with packet-tracer.

ONLY IF your current connection/traffic flow is still ACTIVE, the packet-tracer will show Action: allow at the end with something like Type: FLOW-LOOKUP and Found flow with id 144799389, using existing flow, otherwise it will drop.

What packet-tracer does is to make up (fake) traffic and verify against your rules/NAT... If there is already an active connection, then it will what I wrote above.

#

To test/check your ISP connection, try to check your Firewall's outside/inside interface counter errors, then you can try to bypass Firewall and connect your laptop directly to ISP device (with public IP address setting on Laptop) or put a switch in between your Firewall and ISP device, and connect a laptop on that switch (again, with public IP address setting on it)

Updated answer

Okay, now we know that packet-tracer data was from capture test type asp-drop acl-drop.

As I mentioned earlier, if the packet is not a part of currently active flow/connection, firewall will drop it (acl-drop reason). That packet could be a delayed one hitting firewall after the flow/connection completely closed, thus got denied.

To verify that, you can create two other captures:

  • capture HTTP interface outside match tcp host 188.188.188.188 host 52.216.128.211 eq 80
  • capture HTTPS interface outside match tcp host 188.188.188.188 host 52.216.128.211 eq 443

Compare the outputs (time, source port, sequence...) from these two HTTP and HTTPS captures with the dropped packets from capture test. I hope it would help us to understand why.