Multiple ipv6 routers on the same physical network, how to get it working

ipv6networkingrouting

I have multiple Internet routers from the same provider (I'll name “boxes” the Internet routers), all linked to the same hardware network switch forming a physical network (I'll name “NML” that physical network for “No Man's Land”) on which are connected two routers (I'll name them “routers”) doing routing and firewall tasks for a private LAN:

_____________________________________________ WAN (Internet)
    |           |           |           |
    |           |           |           |
 ___|___     ___|___     ___|___     ___|___
[       ]   [       ]   [       ]   [       ]
[ Box 1 ]   [ Box 2 ]   [ Box 3 ]   [ Box 4 ]
[ 0:A:1 ]   [ 0:B:1 ]   [ 0:C:1 ]   [ 0:D:1 ]
[_______]   [_______]   [_______]   [_______]
    |           |           |           |
    |           |           |           |
____|___________|___________|___________|____ NML (A physical network
          |                       |                for boxes and routers)
          |                       |
      ____|____               ____|____ 
     [         ]             [         ]
     [ Router1 ]             [ Router2 ]
     [  0:A:2  ]             [  0:A:3  ]
     [  0:B:2  ]             [  0:B:3  ]
     [  0:C:2  ]             [  0:C:3  ]
     [  0:D:2  ]             [  0:D:3  ]
     [_________]             [_________]
          |                       |
          |                       |
__________|_______________________|__________ LAN (Where people are working)

On the ipv4 side, NML is a local network using an ipv4 private class, it works.

On the ipv6 side, each boxes share the same /64 prefix and each router get an auto-configured ipv6 address from each box (each router get four ipv6, one per box).

To make it simple you can imagine ipv6 addresses with 3 chars:

  • 0:A:1: 0 prefix from provider network prefix, A prefix from first box network prefix, 1 suffix from first box address;
  • 0:A:2: 0 prefix from provider network prefix, A prefix from first box network prefix, 1 suffix from first router getting address from first box;
  • 0:C:1: 0 prefix from provider network prefix, C prefix from third box network prefix, 1 suffix from third box address;
  • 0:C:3: 0 prefix from provider network prefix, C prefix from third box network prefix, 3 suffix from second router getting address from third box.

So each router get four ipv6 addresses, one per box.

  • From router 1 I can ping6 every boxes using their ipv6 addresses (0:A:1, 0:B:1, 0:C:1, 0:D:1);
  • From router 2 I can ping6 every router2 ipv6 addresses (0:A:3, 0:B:3, 0:C:3, 0:D:3), the opposite is true;
  • From both router 1 and router 2 I can't ping6 an ipv6 address, even when I add an explicit route to one box for that ipv6 address (ip -6 route add something via 0:A:1), either using box's ipv6 address from provider prefix either using box's link-local address.

At first only one box had ipv6 activated and at this time I was able to query the Internet using ipv6, but since I activated ipv6 on all the boxes I'm not able to query the Internet using ipv6 anymore. If I do a traceroute6 to an Internet address from a router, it never goes beyond the box.

Note that I don't need to do ipv6 stuff from LAN through the routers at this time, I only need routers to be able to do Internet stuff using ipv6 (mainly to build some VPN over ipv6).

The Internet routers (named “boxes”) are property of ISP and the only option I have is an "enable IPv6" checkbox on the customer page, I don't have any access to router configuration itself and no one other option than enabling or disabling ipv6. The routers between LAN and NML (named “routers”) are standard Debian systems running on some x86-based networking hardware. On a Debian point of view you can imagine it's like a PC : I can do whatever a standard Debian can do.

So, two questions:

  • Is the setup correct and expected to work? If yes, where can I look for to find the problem?
  • Is the setup incorrect and not expected to work at all? If yes, what can I do to fix it?

I removed some "expires" information and stuff like that to remove some verbosity, plus I sometime added some leading 0 to align addresses for easier reading, then replaced some bits to use the ipv6 example class.

# my boxes' ipv6 addresses
box0    2001:db8:ee84:2180::1
box1    2001:db8:ee84:21c0::1
box2    2001:db8:2f13:1ea0::1
box3    2001:db8:399a:08f0::1
box4    2001:db8:399a:39e0::1

# my router1's ipv6 addresses
# ip -6 addr show dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2001:db8:ee84:2180:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
    inet6 2001:db8:ee84:21c0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
    inet6 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:08f0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:39e0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic 
    inet6 fe80::200:24ff:fed1:3d9e/64 scope link 

# my router2's ipv6 addresses
ip -6 addr show dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2001:db8:ee84:2180:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:ee84:21c0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:2f13:1ea0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:08f0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:39e0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 fe80::200:24ff:fed1:6336/64 scope link 

# default ipv6 routes on router1
# ip -6 route
2001:db8:ee84:2180::/64 dev eth2  proto kernel  metric 256
2001:db8:ee84:21c0::/64 dev eth2  proto kernel  metric 256
2001:db8:2f13:1ea0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:08f0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:39e0::/64 dev eth2  proto kernel  metric 256
fe80::/64 dev eth2  proto kernel  metric 256 
default via fe80::e69e:12ff:fe04:286f dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe03:8b35 dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe02:10de dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::0224:d4ff:fea7:f258 dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::0224:d4ff:febb:af9e dev eth2  proto ra  metric 1024 hoplimit 64

# default ipv6 routes on router2
# ip -6 route
2001:db8:ee84:2180::/64 dev eth2  proto kernel  metric 256
2001:db8:ee84:21c0::/64 dev eth2  proto kernel  metric 256
2001:db8:2f13:1ea0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:08f0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:39e0::/64 dev eth2  proto kernel  metric 256
fe80::/64 dev eth2  proto kernel  metric 256 
default via fe80::e69e:12ff:fe03:8b35 dev eth2  proto ra  metric 1024 mtu 1480 hoplimit 64
default via fe80::0224:d4ff:febb:af9e dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe02:10de dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe04:286f dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::0224:d4ff:fea7:f258 dev eth2  proto ra  metric 1024 hoplimit 64

# neigh from router1
# ip -6 neigh | grep eth2
fe80::224:d4ff:febb:af9e dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
fe80::200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
fe80::e69e:12ff:fe04:286f dev eth2 lladdr e4:9e:12:04:28:6f router REACHABLE
2001:db8:2f13:1ea0::1 dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
fe80::e69e:12ff:fe02:10de dev eth2 lladdr e4:9e:12:02:10:de router STALE
2001:db8:399a:39e0:200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
2001:db8:399a:8f0:200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
2001:db8:399a:39e0::1 dev eth2 lladdr 00:24:d4:a7:f2:58 router STALE
2001:db8:399a:8f0::1 dev eth2 lladdr e4:9e:12:02:10:de router STALE
2001:db8:399a:39e0:: dev eth2  FAILED
fe80::213:46ff:fe8f:1e4a dev eth2 lladdr 00:13:46:8f:1e:4a STALE
2001:db8:399a:8f0:: dev eth2  FAILED
fe80::e69e:12ff:fe03:8b35 dev eth2 lladdr e4:9e:12:03:8b:35 router STALE
fe80::224:d4ff:fea7:f258 dev eth2 lladdr 00:24:d4:a7:f2:58 router STALE
fe80::8226:89ff:fe2d:b3d3 dev eth2 lladdr 80:26:89:2d:b3:d3 STALE
fe80::20a:f7ff:fe12:e77 dev eth2 lladdr 00:0a:f7:12:0e:77 STALE
2001:db8:ee84:2180::1 dev eth2 lladdr e4:9e:12:03:8b:35 router STALE
fe80::21d:9ff:fe2c:628d dev eth2 lladdr 00:1d:09:2c:62:8d STALE

# get from router1
# ip -6 route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via fe80::e69e:12ff:fe04:286f dev eth2  proto ra  src 2001:db8:399a:39e0:200:24ff:fed1:3d9e  metric 1024  hoplimit 64

# neigh from router2
# ip -6 neigh | grep eth2
2001:db8:399a:8f0:200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
2001:db8:399a:8f0::1 dev eth2 lladdr e4:9e:12:02:10:de router STALE
fe80::e69e:12ff:fe04:286f dev eth2 lladdr e4:9e:12:04:28:6f router STALE
fe80::224:d4ff:fea7:f258 dev eth2 lladdr 00:24:d4:a7:f2:58 router DELAY
fe80::e69e:12ff:fe02:10de dev eth2 lladdr e4:9e:12:02:10:de router REACHABLE
fe80::200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
fe80::224:d4ff:febb:af9e dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
2001:db8:399a:39e0:200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
fe80::e69e:12ff:fe03:8b35 dev eth2 lladdr e4:9e:12:03:8b:35 router REACHABLE

# get from router2
ip -6 route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via 2001:db8:399a:8f0::1 dev eth2  src 2001:db8:399a:39e0:200:24ff:fed1:6336  metric 1024

Some traceroute example:

# traceroute from router1
# traceroute 2001:4860:4860::8888
traceroute to 2001:4860:4860::8888 (2001:4860:4860::8888), 30 hops max, 80 byte packets
 1  2001:db8:ee84:21c0::1 (2001:db8:ee84:21c0::1)  26.146 ms  27.728 ms  28.507 ms

Best Answer

I'm transposing this ipv4 example to ipv6.

The trouble is that unless you actively ping each box's ip with the right source, it won't be easy to match (using the mac address) the correct public ip with the correct link local ip. Here's what I could infer from neighbour tables from both routers and from the fact that the link local's end is the mac's end:

box0 2001:db8:ee84:2180::1 <=> fe80::e69e:12ff:fe03:8b35
box2 2001:db8:2f13:1ea0::1 <=> fe80::0224:d4ff:febb:af9e
box3 2001:db8:399a:08f0::1 <=> fe80::e69e:12ff:fe02:10de
box4 2001:db8:399a:39e0::1 <=> fe80::0224:d4ff:fea7:f258

and the remaining one that couldn't be found by the data has to be:

box1 2001:db8:ee84:21c0::1 <=> fe80::e69e:12ff:fe04:286f

With this it becomes clear that both ip -6 route get examples in the question show the wrong source was used for the chosen router. It appears to be a linux' limitation in this case (having multiple LANs overlayed on a single physical LAN).

So here is what has to be done for router1. Also, by trial and error "src" still has to be stated for the default route in the table or an other source might still be chosen. Note that the public IP or the link local IP can (should) be used interchangeably as gateway, so better use the public IP, even if the RA appears to prefer the link ip, it's easier to use the public in a script, and finding the correspondance between the two kinds of IP like above doesn't have to be done anymore.

# ip -6 route add 2001:db8:ee84:2180::/64 dev eth2 src 2001:db8:ee84:2180:200:24ff:fed1:3d9e table 100
# ip -6 route add default via 2001:db8:ee84:2180::1 dev eth2 src 2001:db8:ee84:2180:200:24ff:fed1:3d9e table 100

# ip -6 route add 2001:db8:ee84:21c0::/64 dev eth2 src 2001:db8:ee84:21c0:200:24ff:fed1:3d9e table 101
# ip -6 route add default via 2001:db8:ee84:21c0::1 dev eth2 src 2001:db8:ee84:21c0:200:24ff:fed1:3d9e table 101

# ip -6 route add 2001:db8:2f13:1ea0::/64 dev eth2 src 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e table 102
# ip -6 route add default via 2001:db8:2f13:1ea0::1 dev eth2 src 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e table 102

# ip -6 route add 2001:db8:399a:08f0::/64 dev eth2 src 2001:db8:399a:08f0:200:24ff:fed1:3d9e table 103
# ip -6 route add default via 2001:db8:399a:08f0::1 dev eth2 src 2001:db8:399a:08f0:200:24ff:fed1:3d9e table 103

# ip -6 route add 2001:db8:399a:39e0::/64 dev eth2 src 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104
# ip -6 route add default via 2001:db8:399a:39e0::1 dev eth2 src 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104

At least a "default" default route has to be put in the main (the usual) table, else whatever is done later, there will be an error like "no route to host" or "network is unreachable". Since you already got 5 of these from router advertisement, it's not needed.

Now let's add the rules that bind each source IP to its matching table, thus selecting the right source for any of the (box) routers:

# ip -6 rule add from 2001:db8:ee84:2180:200:24ff:fed1:3d9e table 100
# ip -6 rule add from 2001:db8:ee84:21c0:200:24ff:fed1:3d9e table 101
# ip -6 rule add from 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e table 102
# ip -6 rule add from 2001:db8:399a:08f0:200:24ff:fed1:3d9e table 103
# ip -6 rule add from 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104

It's also possible to force (or even loadbalance with the statistic --mode nth match) a choice using for example ip6tables ... -j MARK ... with ip -6 rule add fwmark ... table 10x

Now it would be better to script this...

There might be an other issue. If a box-router goes offline, the corresponding default route will disappear quickly on the linux system because it's not advertised anymore, but the IP will stay. I don't know then if this IP might still be wrongly chosen to be used on a remaining default route. If this happens, the ip rules added might override the default route to use the now offline box-router, thus defeating the fail-over mechanism of having multiple boxes as routers. I don't have the means to test this.

Related Topic