Cisco Switch ARP Issue: How to Fix Wrong Dynamic ARP Entry – Unpingable IP Address

arpciscomac addressswitch

Situation:
Our Stacked Switch (two WS-C2960X-48LPS-L – 15.2(2)E3 – 192.168.10.19 – VLAN 10) wont be reachable via ICMP or SNMP by the Cisco Prime Infrastructure (172.16.2.103 – VLAN1[nativ]) and back.

I found out following facts:

 show arp on 192.168.10.19:
 Protocol  Address          Age (min)  Hardware Addr   Type   Interface
 Internet  172.16.2.103            5   0050.5698.af6a  ARPA   Vlan1
 Internet  192.168.10.254        151   0000.0c9f.f00a  ARPA   Vlan10
  • For general there wont be any arp entry for the 172.16.2.103 because the traffic will first go to the default gateway…
  • It is just for this address (as far as I know) all other devices could ping the switch
  • If I delete the arp entry it will work for a few minutes after that the switch generates the entry seen a few lines above
  • there arent any ACLs as the standard – permit line
  • I already searched for a bug report at the cisco website but havn't found anything about this.
  • we had another IOS version bevor this one and it havn't worked eather

As a workaround I could make a static arp entry with the MAC address of the default gateway or the Prime and it will work stable.

Have you any guesses why this happens or how to solve it, so I havn't to change the entry if the Server or the Default Gateway (redundant ip) changes.

enter image description here

Running Config of 192.168.10.19:
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname SW_19-POE
!
boot-start-marker
boot-end-marker
!
enable secret 5 *******
!
username **** privilege 15 secret 5 ********

no aaa new-model
clock timezone cet 1 0
clock summer-time MEZ recurring last Sun Mar 2:00 last Sun Oct 3:00
switch 1 provision ws-c2960x-48lps-l
switch 2 provision ws-c2960x-48lps-l
!
!
no ip domain-lookup
ip domain-name *****.local
ip device tracking probe delay 10
!
udld aggressive

authentication mac-move permit
!
mls qos map cos-dscp 0 8 16 26 32 46 48 56
mls qos queue-set output 1 threshold 2 3200 3200 100 3200
mls qos queue-set output 1 threshold 3 3200 3200 100 3200
mls qos
!
!
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
interface Port-channel1
 description UPLINK TO 192.168.10.18
 switchport mode trunk
 ip device tracking maximum 0
 storm-control broadcast level pps 500
 nmsp attachment suppress
!
interface Port-channel2
 switchport mode trunk
 storm-control broadcast level pps 500
!
interface Port-channel3
 switchport mode trunk
 storm-control broadcast level pps 500
!
interface Port-channel4
 switchport mode trunk
 storm-control broadcast level pps 500
!
interface range GigabitEthernet1/0/1-47
 switchport mode access
 switchport voice vlan 56
 switchport port-security maximum 2
 switchport port-security violation  restrict
 switchport port-security
 no logging event link-status
 priority-queue out
 no snmp trap link-status
 mls qos trust dscp
 storm-control broadcast level pps 500
 spanning-tree portfast
 spanning-tree bpduguard enable
!

interface GigabitEthernet1/0/48
 switchport mode access
 switchport voice vlan 56
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 spanning-tree portfast
 spanning-tree bpduguard enable
!
!
interface GigabitEthernet1/0/49
 description UPLINK 192.168.10.18
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 1 mode on
!
interface GigabitEthernet1/0/50
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 2 mode on
!
interface GigabitEthernet1/0/51
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 3 mode on
!
interface GigabitEthernet1/0/52
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 4 mode on
!
interface range GigabitEthernet2/0/1-47
 switchport mode access
 switchport voice vlan 56
 switchport port-security maximum 2
 switchport port-security violation  restrict
 switchport port-security
 no logging event link-status
 priority-queue out
 no snmp trap link-status
 mls qos trust dscp
 storm-control broadcast level pps 500
 spanning-tree portfast
 spanning-tree bpduguard enable
!
interface GigabitEthernet2/0/48
 switchport mode access
 switchport voice vlan 56
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 spanning-tree portfast
 spanning-tree bpduguard enable
!
interface GigabitEthernet2/0/49
 description UPLINK 192.168.10.18
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 1 mode on
!
interface GigabitEthernet2/0/50
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 2 mode on
!
interface GigabitEthernet2/0/51
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 3 mode on
!
interface GigabitEthernet2/0/52
 switchport mode trunk
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level pps 500
 channel-group 4 mode on
!
interface Vlan1
 no ip address
 no ip route-cache
 shutdown
!
interface Vlan10
 ip address 192.168.10.19 255.255.255.0
 no ip redirects
 no ip route-cache
!
ip default-gateway 192.168.10.254
no ip http server
no ip http secure-server
!
!
ip access-list standard permit_line
 permit 172.16.2.104
 permit 172.16.2.103

ip access-list standard permit_snmp
 permit 172.16.2.103
 deny   any
!
logging host 172.16.2.103
!
snmp-server community ****** RW permit_snmp
snmp-server community ****** RO permit_snmp
snmp-server community ****** RW permit_snmp
snmp-server community ****** RO permit_snmp
snmp-server location ******
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps port-security
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server host 172.16.2.103 ******
!


Switch Core:

boot-start-marker
boot-end-marker
!
!
!
clock timezone cet 1
clock summer-time MEZ recurring last Sun Mar 2:00 last Sun Oct 3:00
switch 1 provision ws-c3750g-12s
switch 2 provision ws-c3750g-12s
switch 3 provision ws-c3750x-24
switch 4 provision ws-c3750x-12s
stack-mac persistent timer 0
system mtu routing 1500
udld enable

ip routing

!
mls qos map cos-dscp 0 8 16 26 32 46 48 56
mls qos queue-set output 1 threshold 2 3200 3200 100 3200
mls qos queue-set output 1 threshold 3 3200 3200 100 3200
mls qos
!
!
license boot level ipservices switch 3
!
spanning-tree mode rapid-pvst
spanning-tree portfast bpduguard default
spanning-tree extend system-id
spanning-tree vlan 1-168,170-1000 priority 8192
spanning-tree vlan 169,1001-1005 priority 0
!
vlan internal allocation policy ascending

!
interface rage  Port-channel1-22
switchport trunk encapsulation dot1q
switchport mode trunk
storm-control broadcast level pps 500
!
!
interface Port-channel33
switchport mode access
!

interface range GigabitEthernet1/0/1-7
description some other switches
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 1 mode on
!
!
interface GigabitEthernet1/0/8
description Channel_Link_to_SW_18+19-POE
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 8 mode on
!
interface range GigabitEthernet1/0/9-12
description some other switches or routers
switchport trunk encapsulation dot1q
switchport mode trunk
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 9 mode on
!
!
interface range GigabitEthernet2/0/1-7
description some other switches or routers
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 11 mode active
!
!
interface GigabitEthernet2/0/8
description Channel_Link_to_SW_18+19-POE
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 8 mode on
!
interface range GigabitEthernet2/0/9-12
description some other switches or routers
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 9 mode on
!
!
interface range GigabitEthernet3/0/1-4
!
!
interface GigabitEthernet3/0/5
switchport access vlan 43
switchport mode access
no logging event link-status
speed 100
duplex full
no snmp trap link-status
no cdp enable
spanning-tree portfast
spanning-tree bpduguard enable
spanning-tree guard root
!
interface GigabitEthernet3/0/6
switchport trunk encapsulation dot1q
switchport mode access
no logging event link-status
no snmp trap link-status
spanning-tree portfast
spanning-tree bpduguard disable
!
interface range GigabitEthernet3/0/7-10
description switches
switchport access vlan 180
!
!
interface GigabitEthernet3/0/11
switchport access vlan 180
!
interface GigabitEthernet3/0/12
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1
switchport mode trunk
storm-control broadcast level pps 500
storm-control multicast level pps 500
spanning-tree portfast trunk
spanning-tree bpduguard disable
spanning-tree guard root
!
interface GigabitEthernet3/0/13
switchport access vlan 15
!
interface GigabitEthernet3/0/14
switchport access vlan 15
!
interface GigabitEthernet3/0/15
!
interface GigabitEthernet3/0/16
!
interface GigabitEthernet3/0/17
description HP-BLADE04-18
switchport access vlan 180
!
interface GigabitEthernet3/0/18
switchport access vlan 66
!
interface range GigabitEthernet3/0/19-22
switchport access vlan 56
!
!
interface GigabitEthernet3/0/23
switchport trunk encapsulation dot1q
switchport mode trunk
no logging event link-status
no snmp trap link-status
!
interface GigabitEthernet3/0/24
switchport access vlan 541
switchport mode access
spanning-tree portfast
!
iterface GigabitEthernet3/1/4
!
interface range TenGigabitEthernet3/1/1-2
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 31 mode active
!

!
interface range GigabitEthernet4/0/1-12
description Channel_Link_to_SW_1
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 1 mode on
!
!
interface range TenGigabitEthernet4/1/1-2
switchport trunk encapsulation dot1q
switchport mode trunk
priority-queue out 
mls qos trust dscp
storm-control broadcast level pps 500
channel-group 31 mode active
!
!
interface Vlan1
ip address 172.16.3.68 255.255.0.0
!
interface Vlan2
no ip address
!
interface Vlan10
ip address 192.168.10.252 255.255.255.0
no ip proxy-arp
!
interface Vlan180
ip address 192.168.180.2 255.255.255.0
standby 180 ip 192.168.180.1
standby 180 timers 2 6
standby 180 priority 150
standby 180 preempt
!
interface Vlan181
ip address 192.168.181.2 255.255.255.0
!
interface Vlan254
ip address 158.158.254.3 255.255.255.240
!
interface Vlan700
no ip address
!
!
router eigrp 100
network 158.158.254.3 0.0.0.0
network 172.16.3.68 0.0.0.0
network 192.168.180.2 0.0.0.0
network 192.168.181.2 0.0.0.0
passive-interface Vlan180
passive-interface Vlan181
!
ip classless
ip route 0.0.0.0 0.0.0.0 172.16.2.101
no ip http server
no ip http secure-server
!
!
ip access-list standard permit_line
permit 172.16.2.104
permit 172.16.2.103

ip access-list standard permit_snmp
permit 172.16.2.103
!
ip sla enable reaction-alerts
logging trap debugging
logging 172.16.2.103
!
snmp-server community ******** RW 1 
snmp-server community ******** RO permit_snmp 
snmp-server community ******** RW permit_snmp
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server host 172.16.2.103 ******** 
!
!
line con 0
logging synchronous
stopbits 1
line vty 0 4
logging synchronous
line vty 5 15
access-class permit_line in
logging synchronous
!
ntp clock-period 36029501
ntp server 130.149.17.21 prefer
mac address-table aging-time 14400
end

Switch 18:


clock timezone cet 1 0
clock summer-time MEZ recurring last Sun Mar 2:00 last Sun Oct 3:00
switch 1 provision ws-c2960s-48ts-l
authentication mac-move permit
!
!
ip domain-name ******.local
udld aggressive
!
spanning-tree mode rapid-pvst
spanning-tree portfast bpduguard default
spanning-tree extend system-id
!
!
vlan internal allocation policy ascending
!
!
interface Port-channel1
description UPLINK to SW_Core
switchport mode trunk
storm-control broadcast level pps 500
!
interface Port-channel2
description UPLINK TO SW_19-POE
switchport mode trunk
storm-control broadcast level pps 500
!
interface FastEthernet0
no ip address
!
interface range GigabitEthernet1/0/1-48
switchport mode access
no logging event link-status
priority-queue out 
no snmp trap link-status
mls qos trust dscp
storm-control broadcast level pps 500
no cdp enable
spanning-tree portfast
spanning-tree bpduguard enable
!
!
interface GigabitEthernet1/0/49
description UPLINK TO SW_Core
switchport mode trunk
priority-queue out 
storm-control broadcast level pps 500
channel-group 1 mode on
!
interface GigabitEthernet1/0/50
description UPLINK TO SW_Core
switchport mode trunk
priority-queue out 
storm-control broadcast level pps 500
channel-group 1 mode on
!
interface GigabitEthernet1/0/51
description UPLINK TO SW_19-POE
switchport mode trunk
priority-queue out 
storm-control broadcast level pps 500
channel-group 2 mode on
!
interface GigabitEthernet1/0/52
description UPLINK TO SW_19-POE
switchport mode trunk
priority-queue out 
storm-control broadcast level pps 500
channel-group 2 mode on
!
interface Vlan1
no ip address
no ip route-cache
shutdown
!
interface Vlan10
ip address 192.168.40.18 255.255.255.0
no ip redirects
no ip route-cache
!
ip default-gateway 192.168.10.254
no ip http server
no ip http secure-server
!
!
ip access-list standard permit_line
permit 172.16.2.104
permit 172.16.2.103
!
ip access-list standard permit_snmp
permit 172.16.2.103
deny any
logging host 172.16.2.103
!
snmp-server community ******** RW permit_snmp 
snmp-server community ******** RO permit_snmp 
snmp-server community ******** RW permit_snmp 
snmp-server community ******** RO permit_snmp
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server host 172.16.2.103 ********

Best Answer

A duplicate ip on the network can cause this behavior. If you have 2 machines with different MACs competing for the same IP, ARP will sometime resolve to one MAC and sometimes resolve to the other MAC. The result is unpredictable and incorrect arp resolution.

Related Topic