I have multiple Internet routers from the same provider (I'll name “boxes” the Internet routers), all linked to the same hardware network switch forming a physical network (I'll name “NML” that physical network for “No Man's Land”) on which are connected two routers (I'll name them “routers”) doing routing and firewall tasks for a private LAN:
_____________________________________________ WAN (Internet)
| | | |
| | | |
___|___ ___|___ ___|___ ___|___
[ ] [ ] [ ] [ ]
[ Box 1 ] [ Box 2 ] [ Box 3 ] [ Box 4 ]
[ 0:A:1 ] [ 0:B:1 ] [ 0:C:1 ] [ 0:D:1 ]
[_______] [_______] [_______] [_______]
| | | |
| | | |
____|___________|___________|___________|____ NML (A physical network
| | for boxes and routers)
| |
____|____ ____|____
[ ] [ ]
[ Router1 ] [ Router2 ]
[ 0:A:2 ] [ 0:A:3 ]
[ 0:B:2 ] [ 0:B:3 ]
[ 0:C:2 ] [ 0:C:3 ]
[ 0:D:2 ] [ 0:D:3 ]
[_________] [_________]
| |
| |
__________|_______________________|__________ LAN (Where people are working)
On the ipv4 side, NML is a local network using an ipv4 private class, it works.
On the ipv6 side, each boxes share the same /64 prefix and each router get an auto-configured ipv6 address from each box (each router get four ipv6, one per box).
To make it simple you can imagine ipv6 addresses with 3 chars:
0:A:1
:0
prefix from provider network prefix,A
prefix from first box network prefix,1
suffix from first box address;0:A:2
:0
prefix from provider network prefix,A
prefix from first box network prefix,1
suffix from first router getting address from first box;0:C:1
:0
prefix from provider network prefix,C
prefix from third box network prefix,1
suffix from third box address;0:C:3
:0
prefix from provider network prefix,C
prefix from third box network prefix,3
suffix from second router getting address from third box.
So each router get four ipv6 addresses, one per box.
- From router 1 I can
ping6
every boxes using their ipv6 addresses (0:A:1
,0:B:1
,0:C:1
,0:D:1
); - From router 2 I can
ping6
every router2 ipv6 addresses (0:A:3
,0:B:3
,0:C:3
,0:D:3
), the opposite is true; - From both router 1 and router 2 I can't
ping6
an ipv6 address, even when I add an explicit route to one box for that ipv6 address (ip -6 route add something via 0:A:1
), either using box's ipv6 address from provider prefix either using box's link-local address.
At first only one box had ipv6 activated and at this time I was able to query the Internet using ipv6, but since I activated ipv6 on all the boxes I'm not able to query the Internet using ipv6 anymore. If I do a traceroute6
to an Internet address from a router, it never goes beyond the box.
Note that I don't need to do ipv6 stuff from LAN through the routers at this time, I only need routers to be able to do Internet stuff using ipv6 (mainly to build some VPN over ipv6).
The Internet routers (named “boxes”) are property of ISP and the only option I have is an "enable IPv6" checkbox on the customer page, I don't have any access to router configuration itself and no one other option than enabling or disabling ipv6. The routers between LAN and NML (named “routers”) are standard Debian systems running on some x86-based networking hardware. On a Debian point of view you can imagine it's like a PC : I can do whatever a standard Debian can do.
So, two questions:
- Is the setup correct and expected to work? If yes, where can I look for to find the problem?
- Is the setup incorrect and not expected to work at all? If yes, what can I do to fix it?
I removed some "expires" information and stuff like that to remove some verbosity, plus I sometime added some leading 0
to align addresses for easier reading, then replaced some bits to use the ipv6 example class.
# my boxes' ipv6 addresses
box0 2001:db8:ee84:2180::1
box1 2001:db8:ee84:21c0::1
box2 2001:db8:2f13:1ea0::1
box3 2001:db8:399a:08f0::1
box4 2001:db8:399a:39e0::1
# my router1's ipv6 addresses
# ip -6 addr show dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2001:db8:ee84:2180:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
inet6 2001:db8:ee84:21c0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
inet6 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
inet6 2001:db8:399a:08f0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
inet6 2001:db8:399a:39e0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
inet6 fe80::200:24ff:fed1:3d9e/64 scope link
# my router2's ipv6 addresses
ip -6 addr show dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2001:db8:ee84:2180:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic
inet6 2001:db8:ee84:21c0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic
inet6 2001:db8:2f13:1ea0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic
inet6 2001:db8:399a:08f0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic
inet6 2001:db8:399a:39e0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic
inet6 fe80::200:24ff:fed1:6336/64 scope link
# default ipv6 routes on router1
# ip -6 route
2001:db8:ee84:2180::/64 dev eth2 proto kernel metric 256
2001:db8:ee84:21c0::/64 dev eth2 proto kernel metric 256
2001:db8:2f13:1ea0::/64 dev eth2 proto kernel metric 256
2001:db8:399a:08f0::/64 dev eth2 proto kernel metric 256
2001:db8:399a:39e0::/64 dev eth2 proto kernel metric 256
fe80::/64 dev eth2 proto kernel metric 256
default via fe80::e69e:12ff:fe04:286f dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe03:8b35 dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe02:10de dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::0224:d4ff:fea7:f258 dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::0224:d4ff:febb:af9e dev eth2 proto ra metric 1024 hoplimit 64
# default ipv6 routes on router2
# ip -6 route
2001:db8:ee84:2180::/64 dev eth2 proto kernel metric 256
2001:db8:ee84:21c0::/64 dev eth2 proto kernel metric 256
2001:db8:2f13:1ea0::/64 dev eth2 proto kernel metric 256
2001:db8:399a:08f0::/64 dev eth2 proto kernel metric 256
2001:db8:399a:39e0::/64 dev eth2 proto kernel metric 256
fe80::/64 dev eth2 proto kernel metric 256
default via fe80::e69e:12ff:fe03:8b35 dev eth2 proto ra metric 1024 mtu 1480 hoplimit 64
default via fe80::0224:d4ff:febb:af9e dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe02:10de dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe04:286f dev eth2 proto ra metric 1024 hoplimit 64
default via fe80::0224:d4ff:fea7:f258 dev eth2 proto ra metric 1024 hoplimit 64
# neigh from router1
# ip -6 neigh | grep eth2
fe80::224:d4ff:febb:af9e dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
fe80::200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
fe80::e69e:12ff:fe04:286f dev eth2 lladdr e4:9e:12:04:28:6f router REACHABLE
2001:db8:2f13:1ea0::1 dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
fe80::e69e:12ff:fe02:10de dev eth2 lladdr e4:9e:12:02:10:de router STALE
2001:db8:399a:39e0:200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
2001:db8:399a:8f0:200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
2001:db8:399a:39e0::1 dev eth2 lladdr 00:24:d4:a7:f2:58 router STALE
2001:db8:399a:8f0::1 dev eth2 lladdr e4:9e:12:02:10:de router STALE
2001:db8:399a:39e0:: dev eth2 FAILED
fe80::213:46ff:fe8f:1e4a dev eth2 lladdr 00:13:46:8f:1e:4a STALE
2001:db8:399a:8f0:: dev eth2 FAILED
fe80::e69e:12ff:fe03:8b35 dev eth2 lladdr e4:9e:12:03:8b:35 router STALE
fe80::224:d4ff:fea7:f258 dev eth2 lladdr 00:24:d4:a7:f2:58 router STALE
fe80::8226:89ff:fe2d:b3d3 dev eth2 lladdr 80:26:89:2d:b3:d3 STALE
fe80::20a:f7ff:fe12:e77 dev eth2 lladdr 00:0a:f7:12:0e:77 STALE
2001:db8:ee84:2180::1 dev eth2 lladdr e4:9e:12:03:8b:35 router STALE
fe80::21d:9ff:fe2c:628d dev eth2 lladdr 00:1d:09:2c:62:8d STALE
# get from router1
# ip -6 route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via fe80::e69e:12ff:fe04:286f dev eth2 proto ra src 2001:db8:399a:39e0:200:24ff:fed1:3d9e metric 1024 hoplimit 64
# neigh from router2
# ip -6 neigh | grep eth2
2001:db8:399a:8f0:200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
2001:db8:399a:8f0::1 dev eth2 lladdr e4:9e:12:02:10:de router STALE
fe80::e69e:12ff:fe04:286f dev eth2 lladdr e4:9e:12:04:28:6f router STALE
fe80::224:d4ff:fea7:f258 dev eth2 lladdr 00:24:d4:a7:f2:58 router DELAY
fe80::e69e:12ff:fe02:10de dev eth2 lladdr e4:9e:12:02:10:de router REACHABLE
fe80::200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
fe80::224:d4ff:febb:af9e dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
2001:db8:399a:39e0:200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
fe80::e69e:12ff:fe03:8b35 dev eth2 lladdr e4:9e:12:03:8b:35 router REACHABLE
# get from router2
ip -6 route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via 2001:db8:399a:8f0::1 dev eth2 src 2001:db8:399a:39e0:200:24ff:fed1:6336 metric 1024
Some traceroute example:
# traceroute from router1
# traceroute 2001:4860:4860::8888
traceroute to 2001:4860:4860::8888 (2001:4860:4860::8888), 30 hops max, 80 byte packets
1 2001:db8:ee84:21c0::1 (2001:db8:ee84:21c0::1) 26.146 ms 27.728 ms 28.507 ms
Best Answer
I'm transposing this ipv4 example to ipv6.
The trouble is that unless you actively ping each box's ip with the right source, it won't be easy to match (using the mac address) the correct public ip with the correct link local ip. Here's what I could infer from neighbour tables from both routers and from the fact that the link local's end is the mac's end:
and the remaining one that couldn't be found by the data has to be:
With this it becomes clear that both
ip -6 route get
examples in the question show the wrong source was used for the chosen router. It appears to be a linux' limitation in this case (having multiple LANs overlayed on a single physical LAN).So here is what has to be done for router1. Also, by trial and error "src" still has to be stated for the default route in the table or an other source might still be chosen. Note that the public IP or the link local IP can (should) be used interchangeably as gateway, so better use the public IP, even if the RA appears to prefer the link ip, it's easier to use the public in a script, and finding the correspondance between the two kinds of IP like above doesn't have to be done anymore.
At least a "default" default route has to be put in the main (the usual) table, else whatever is done later, there will be an error like "no route to host" or "network is unreachable". Since you already got 5 of these from router advertisement, it's not needed.
Now let's add the rules that bind each source IP to its matching table, thus selecting the right source for any of the (box) routers:
It's also possible to force (or even loadbalance with the
statistic --mode nth
match) a choice using for exampleip6tables ... -j MARK ...
withip -6 rule add fwmark ... table 10x
Now it would be better to script this...
There might be an other issue. If a box-router goes offline, the corresponding default route will disappear quickly on the linux system because it's not advertised anymore, but the IP will stay. I don't know then if this IP might still be wrongly chosen to be used on a remaining default route. If this happens, the ip rules added might override the default route to use the now offline box-router, thus defeating the fail-over mechanism of having multiple boxes as routers. I don't have the means to test this.