ARP Ignore Setting – Why It Breaks ARP on Point-to-Point Interfaces in KVM

bridgekvm-virtualizationlinux-networking

If I add the following settings on a kvm guest:

net.ipv4.conf.all.arp_ignore=1

Networking of the guest stops (highly intermittent).

The kvm host uses bridge-networking:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto {{ vbox_interface }}
iface {{ vbox_interface }} inet static
  address {{ vbox_main_ipv4 }}
  netmask {{ vbox_ipv4_netmask }}
  gateway {{ vbox_ipv4_gw }}
  pointopoint {{ vbox_ipv4_gw }}

iface {{ vbox_interface }} inet6 static
  address {{ vbox_main_ipv6 }}
  netmask 128
  gateway {{ vbox_ipv6_gw }}

auto  br0
iface br0 inet static
 address   {{ vbox_main_ipv4 }}
 netmask   255.255.255.255
 bridge_ports none
 bridge_stp off
 bridge_fd 0
 bridge_maxwait 0
{% for ip in vbox_addtl_ipv4 %}
 up route add -host {{ ip }} dev br0
{% endfor %}

iface br0 inet6 static
  address {{ vbox_main_ipv6 }}
  netmask 64

Why is that?

As Michael is pointing out in the comments, some context:

  • I do understand the basic gist of arp_ignore (http://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_to_disable_ARP) – I mainly want to keep that setting as I am in a untrusted network
  • I do understand that the setup of network is maybe not standard, this is due to me following this: https://wiki.hetzner.de/index.php/KVM_mit_Nutzung_aller_IPs_-_the_easy_way/en – which allows me to make usage of all IPs of my given subnet
  • I do understand that setting arp_ignore=0 on the guest VM will solve the issue. However I would like to know why (because I do not need to do that on the host itself) and I can not accept that as answer because I do not believe setting it should cause the network to go offline (e.g. I want to solve the issue without making that setting a dependency)
  • I am assuming (thanks Peter Zhabin) that with netmask 255.255.255.255 it is safe enough that nobody within the untrusted network can take advantage of arp?

So my question I suppose is:

Why does arp_ignore=1 break the kvm-guest <> kvm-host networking when the kvm-guest uses pointopoint/255.255.255.255 ?

Edit 23/10/2018: guest interfaces config

# The loopback network interface
auto lo
iface lo inet loopback

auto ens7
iface ens7 inet static
  address IP from the pool {{ vbox_addtl_ipv4 }}
  netmask 255.255.255.255
  gateway {{ vbox_main_ipv4 }}
  pointopoint {{ vbox_main_ipv4 }}

iface ens7 inet6 static
  address xxx::xxx
  netmask 64
  gateway {{ vbox_main_ipv6 }}

Best Answer

I have verified this behavior on Ubuntu 18.04 without any bridges, KVMs or Hetzner whatsoever and I think this is actually a kernel bug with regard to handling of arp_ignore on point-to-point Ethernet interfaces. Steps to verify:

  1. Make sure netplan won't interfere by disabling it altogether.

  2. Set up two systems with Ethernet interfaces connected to each other and IP addresses assigned like this:

    Sever A:

    ip addr add 192.168.100.1/32 peer 192.168.100.2 dev ens33

    ip link set ens33 up

    Server B:

    ip addr add 192.168.100.2/32 peer 192.168.100.1 dev ens33

    ip link set ens33 up

  3. Observe a normal ping from Server A to Server B and ip addr show output that contains this line:

    inet 192.168.100.1 peer 192.168.100.2/32 scope global ens33

  4. Enable arp_ignore=1 with sysctl net.ipv4.conf.all.arp_ignore=1 on Server A and see pings stopping as soon as ARP entry times out. After certain short amount of time pings will resume for a while and then end again. This process will continue indefinitely in this manner.

  5. Run a tcpdump on Server B and observe inbound pings and outbound endless who-has requests for the Server A IP address coming from Server B with no response. Occasionally, Server A will issue who-has request for the Server B IP, will get a reply, and Server B will temporarily cache source MAC-address obtained from this ARP request and this is when pings resume.

Now, here's what's going on. There's an in-kernel difference between point-to-point and standard broadcast interfaces in terms of how iproute2 (and similar tools) populate in_ifaddr structure defined in include/linux/inetdevice.h.

For normal interfaces it's ifa_address field gets populated with local interface address; for point-to-point interfaces ifa_address gets populated with remote peer address and ifa_local gets populated with local interface address.

This is normal and expected behavior, as in old-school days any point-to-point interface was considered an attached remote device itself; a local endpoint address corresponding to it may not even exist. So nothing is broken here.

What's really broken is that arp_ignore() handler defined in net/ipv4/arp.c indirectly uses confirm_addr_indev() function defined in net/ipv4/devinet.c that iteratively tries to match target IP address of the ARP request (that is, local interface address) with ifa_address field of all configured IP address records on an interface.

This works well on standard interfaces as they do have their local IP address recorded in ifa_address field, however with point-to-point interfaces it fails, 'cause they have peer IP address recorded in this field instead.

Now, the real question is does this issue really need a fix, as it has been pointed out already that there's no sense in using arp_ignore on p2p links. I think it does, as there might exist a situation where the host in question has more than one interface (and these interfaces are not p2p ones) and someone might enable this stuff with conf.all prefix only to see his p2p links going down for no apparent reason.

Related Topic