Linux kernel 4.9 not responding to neighbour solicitation

ipv6linux-networking

I have this small network where Linux kernel 4.9 acts as a v6 router. There are two other servers connected to this router and both have ipv6 address configured on interfaces connecting to this router. Address prefix is fc::/8.

But when I try to ping from one server to other it fails with error “Address unreachable”

So, I tried to ping the local v6 address on the router, that also fails with error “Address unreachable”

When I check the packet captures, I see server trying to get the MAC address using neighbor solicitation, but Linux Router is not responding to that. I have v6 forwarding enabled.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:53:02.828354 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:03.869313 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:04.907996 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:05.947761 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:06.987722 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:08.027434 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:09.067203 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:10.107501 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:11.146856 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:12.186701 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:13.226875 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:14.266411 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:15.306211 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32
17:53:16.346467 IP6 fdcd:dead:beef:babe::2 > ff02::1:ff00:3: ICMP6, neighbor solicitation, who has fdcd:dead:beef:cafe::3, length 32

Router side interfaces:

3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:67:09:97:85 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fdcd:dead:beef:babe::3/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::2e0:67ff:fe09:9785/64 scope link
       valid_lft forever preferred_lft forever

4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:67:09:97:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fdcd:dead:beef:cafe::3/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::2e0:67ff:fe09:9786/64 scope link
       valid_lft forever preferred_lft forever

Router side routing config:

::1 dev lo proto kernel metric 256  pref medium
0:0:0:a::/64 dev enp4s0 proto kernel metric 256  expires 86348sec pref medium
fdcd:dead:beef:babe::/64 dev enp2s0 proto kernel metric 256  pref medium
fdcd:dead:beef:cafe::/64 dev enp3s0 proto kernel metric 256  pref medium
fe80::/64 dev enp2s0 proto kernel metric 256  pref medium
fe80::/64 dev enp3s0 proto kernel metric 256  pref medium
fe80::/64 dev enp4s0 proto kernel metric 256  pref medium
fe80::/64 dev enp1s0 proto kernel metric 256  pref medium
default via fe80::6238:e0ff:fed0:1db9 dev enp4s0 proto ra metric 1024  expires 1748sec hoplimit 64 pref medium

Host 1 interface:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:0d:e9:58 brd ff:ff:ff:ff:ff:ff
    inet6 fdcd:dead:beef:babe::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fe0d:e958/64 scope link
       valid_lft forever preferred_lft forever

Routing config:

fdcd:dead:beef:babe::/64 dev eth0 proto kernel metric 256  pref medium
fdcd:dead:beef:cafe::/64 dev eth0 metric 1024  pref medium
fe80::/64 dev eth1 proto kernel metric 256  pref medium
fe80::/64 dev eth0 proto kernel metric 256  pref medium
default via fdcd:dead:beef:babe::3 dev eth0 metric 1024  pref medium

Host 2 interface:

3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:67:09:97:81 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.2/24 brd 192.168.100.255 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fdcd:dead:beef:cafe::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::2e0:67ff:fe09:9781/64 scope link
       valid_lft forever preferred_lft forever

Routing config:

fdcd:dead:beef:babe::/64 dev enp2s0 metric 1024  pref medium
fdcd:dead:beef:cafe::/64 dev enp2s0 proto kernel metric 256  pref medium
fe80::/64 dev enp2s0 proto kernel metric 256  pref medium
fe80::/64 dev enp1s0 proto kernel metric 256  pref medium

Network:

Host A (eth0) <----> L2 switch A <---> Linux v6 router (enp2s0)

Host B (enp2s0) <----> L2 switch B <---> Linux v6 router (enp3s0)

Firewall config:

root@XXXXX:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

Any help is much appreciated

Best Answer

I just had this problem and when running tcpdump on the target machine I wasn't seeing the neighbour solicitation packets arrive at all, so the machine wasn't responding because it wasn't seeing the requests in the first place. The fact that tcpdump on one machine was telling me the packets were being sent, but tcpdump on the target machine was telling me the packets weren't arriving, told me the problem was not with either machine but with whatever was in between them, in this case a network switch. ip6tables is not the problem in this case, as tcpdump monitors raw network traffic before it gets processed by ip6tables, so even if you have a rule that blocks traffic, those packets will still be visible to tcpdump.

The problem turned out to be buggy IPv6 multicast firmware in the Cisco 2975 switch I was using. I had noticed that my IPv6 multicast traffic was being forwarded to every single port on the switch, causing it to be received by machines that weren't interested in it, using up some of their network bandwidth. By enabling MLD snooping, the switch monitors multicast subscription messages and only forwards multicast traffic to the ports that have subscribed to it.

The neighbour solicitation messages are also sent via multicast, and the switch is supposed to forward these messages to all ports even though they haven't subscribed to them. But because of an apparent bug, the switch was treating them like any other multicast traffic and not forwarding them on as no machine had subscribed to those addresses (which you're not supposed to have to do). The switch was correctly forwarding the simpler broadcasts (like ff02::1) however not the more complex neighbour solicitation ones which contain part of the MAC address in order to narrow down the number of recipients.

The instant I disabled MLD snooping on the switch (config; no ipv6 mld snooping) everything immediately started working again.

As this switch is long out of support, I guess my only choices are to live with the extra network traffic or replace the switch with a more recent one that has better IPv6 support.

Related Topic