Linux ARP – Fix Weird Issue with Two Interfaces Sharing Same Subnet

arplinuxsubnet

I have two linux machines connected through a switch.

bothe machines run linux RHEL 7.3 kernel 3.10.0-327.el7.x86_64

one of the machines is connected with both ports to the switch and the other with one port.

both ports share same subnet.

ifconfig machine 1
ens3f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 6.6.6.2  netmask 255.255.255.0  broadcast 6.6.6.255
        ether 34:9a:17:aa:28:1b  txqueuelen 1000  (Ethernet)

ifconfig machine 2

ens1f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 6.6.6.11  netmask 255.255.255.0  broadcast 6.6.6.255
    ether 34:9a:17:65:55:5d  txqueuelen 1000  (Ethernet)

ens3f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 6.6.6.12  netmask 255.255.255.0  broadcast 6.6.6.255
    ether 34:9a:17:aa:26:1b  txqueuelen 1000  (Ethernet)

i flushed the arp tables using 

ip -s neigh flush

then i run a tcp server on machine 1 (with one port), and a tcp client that binds to a specific device (device ens3f1 with ip 6.6.6.12) on machine 2. i see this in the arp table of the server (machine1)

ip neigh show
6.6.6.12 dev ens3f1 lladdr 34:9a:17:65:55:5d REACHABLE
10.224.12.254 dev eno1 lladdr 00:00:5e:00:01:01 REACHABLE
6.6.6.11 dev ens3f1  FAILED

as you can see in the arp table i see a connection between ip 6.6.6.12 and mac 34:9a:17:65:55:5d but in the ifconfig you see this data is not correct.
after running agian the clinet with bind to the other port (6.6.6.11) i see this arp table

6.6.6.12 dev ens3f1 lladdr 34:9a:17:65:55:5d REACHABLE
10.224.12.254 dev eno1 lladdr 00:00:5e:00:01:01 REACHABLE
6.6.6.11 dev ens3f1 lladdr 34:9a:17:65:55:5d REACHABLE

here you can see both ip's has the same mac!!
doe's anyone know how to solve this issue?
this is the tcpdump in the two machines 

tcpdump: listening on ens1f1, link-type EN10MB (Ethernet), capture size    65535 bytes
17:36:39.309925 34:9a:17:aa:28:1b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has ***** (Broadcast) tell 6.6.6.2, length 46
17:36:39.309931 34:9a:17:65:55:5d (oui Unknown) > 34:9a:17:aa:28:1b (oui Unknown), ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply ****** is-at 34:9a:17:65:65:5d (oui Unknown), length 28

tcpdump -i ens3f1 -e -vv
tcpdump: listening on ens3f1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:36:39.309941 34:9a:17:aa:28:1b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has r-aa-nitro01.rdmz.labs.

Thanks!!

Best Answer

This is due to the default configuration for the arp_filter option in the network stack. The possible values are documented in https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt:

arp_filter - BOOLEAN

1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface (therefore you must use source based routing for this to work). In other words it allows control of which cards (usually 1) will respond to an arp request.

0 - (default) The kernel can respond to arp requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. Only for more complex setups like load- balancing, does this behaviour cause problems.

arp_filter for the interface will be enabled if at least one of conf/{all,interface}/arp_filter is set to TRUE, it will be disabled otherwise

Also arp_ignore is relevant here:

arp_ignore - INTEGER Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:

0 - (default): reply for any local target IP address, configured on any interface

1 - reply only if the target IP address is local address configured on the incoming interface

2 - reply only if the target IP address is local address configured on the incoming interface and both with the sender's IP address are part from same subnet on this interface

So, after setting arp_filter to 1 and arp_ignore to 2, you should get the behavior you want.

Related Topic