Networking – Using OVH VRack, 2 PVEs Can’t Fully Communicate

arpnetworkingovhproxmoxrouting

In OVH, I have 2 ProxMox servers, each one containing a Firewall and a few other hosts. I am trying to use OVH vRack for private communication between them, but it is not working.

Here's a summary of my network:

VRack Configuration

The goal is accessing PRD1FRM206 from PRD2FRM201 and vice-versa.

Hosts

  • PRD1FRM206 – Host in PVE01 Server
  • PRD1FWL100 – Firewall in PVE01 Server
  • PRD2FRM201 – Host in PVE02 Server
  • PRD2FWL100 – Firewall in PVE02 Server
  • PVE01 and PVE02 – ProxMox dedicated servers, both hosted in OVH, interconnected by OVH VRack

PVE01 Network configuration:

# Server pag-01
# network interfaces
#
# Author:       Gilberto Martins
# Creation:     03/19/2021
# ================================
    auto lo
    iface lo inet loopback

    auto enp5s0f0
    iface enp5s0f0 inet manual
    auto enp5s0f1
    iface enp5s0f1 inet manual

    # Internet Interface
    auto vmbr0
    iface vmbr0 inet dhcp
      # Internet Interface
      bridge-ports enp5s0f0
      bridge-stp off
      bridge-fd 0

    # Tools Network
    auto vmbr1
    iface vmbr1 inet manual
      # Rede Tools - 172.21.10.0/27
      bridge-ports dummy1
      bridge-stp off
      bridge-fd 0

    # WebPRD Network
    auto vmbr2
    iface vmbr2 inet manual
      # Rede WebPRD - 172.21.20.0/27
      bridge-ports dummy2
      bridge-stp off
      bridge-fd 0

    # WebHML Network
    auto vmbr3
    iface vmbr3 inet manual
      # Rede WebHML - 172.21.30.0/27
      bridge-ports dummy3
      bridge-stp off
      bridge-fd 0

    # Interface PrivateNetwork
#    auto vmbr4
#    iface vmbr4 inet static
      # Rede VRack - NAO USAR
#      address 192.168.0.10/31
#      bridge-ports enp5s0f1
#      bridge-stp off
#      bridge-fd 0

    # WebSites Network
    auto vmbr5
    iface vmbr5 inet manual
      # Rede WebSites - 172.21.40.0/27
      bridge-ports dummy4
      bridge-stp off
      bridge-fd 0

PVE01 current interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether KK:KK:KK:KK:KK:KK brd ff:ff:ff:ff:ff:ff
3: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr4 state UP group default qlen 1000
    link/ether YY:YY:YY:YY:YY:YY brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether UU:UU:UU:UU:UU:UU brd ff:ff:ff:ff:ff:ff
    inet 9.9.9.9/24 brd 9.9.9.255 scope global dynamic vmbr0
       valid_lft 56089sec preferred_lft 56089sec
    inet6 zz99::zz22:zzbb:zzhh:zzkk/64 scope link 
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2a:30:fb:a2:d2:f1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30c0:14ff:fea4:abfd/64 scope link 
       valid_lft forever preferred_lft forever
6: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:b3:67:f5:c3:cd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a849:97ff:fe6c:14e9/64 scope link 
       valid_lft forever preferred_lft forever
7: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5e:99:bd:90:12:24 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e033:5fff:fe6d:222a/64 scope link 
       valid_lft forever preferred_lft forever
8: vmbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether AA:AA:AA:AA:AA:AA brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a242:3fff:fe47:3cfb/64 scope link 
       valid_lft forever preferred_lft forever
9: tap201i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 2a:30:fb:a2:d2:f1 brd ff:ff:ff:ff:ff:ff
10: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 1a:61:72:52:5b:a0 brd ff:ff:ff:ff:ff:ff
11: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 56:16:5b:14:ce:e3 brd ff:ff:ff:ff:ff:ff
12: tap100i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 96:b3:67:f5:c3:cd brd ff:ff:ff:ff:ff:ff
13: tap100i3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether 5e:99:bd:90:12:24 brd ff:ff:ff:ff:ff:ff
14: tap100i4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr4 state UNKNOWN group default qlen 1000
    link/ether ae:84:54:57:7f:46 brd ff:ff:ff:ff:ff:ff
15: tap203i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether aa:dd:66:e9:fd:74 brd ff:ff:ff:ff:ff:ff
17: tap204i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether ce:6b:9e:cb:ca:25 brd ff:ff:ff:ff:ff:ff
18: tap205i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether f2:76:a3:12:48:da brd ff:ff:ff:ff:ff:ff
19: tap206i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether be:92:f0:2e:54:2b brd ff:ff:ff:ff:ff:ff
21: tap402i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 5a:4b:71:1c:b1:6e brd ff:ff:ff:ff:ff:ff
22: tap403i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether ba:0a:25:76:01:6e brd ff:ff:ff:ff:ff:ff
23: tap301i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether 9e:2c:dd:7b:fb:8a brd ff:ff:ff:ff:ff:ff
24: tap302i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether 6e:50:73:30:67:ae brd ff:ff:ff:ff:ff:ff
25: tap303i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether ae:96:60:a4:bc:21 brd ff:ff:ff:ff:ff:ff
26: veth900i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether fe:92:fa:19:f1:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0
29: tap304i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether f2:14:af:70:17:42 brd ff:ff:ff:ff:ff:ff
31: tap404i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 8e:3e:76:76:fb:29 brd ff:ff:ff:ff:ff:ff
32: tap401i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether e2:af:68:37:ed:7e brd ff:ff:ff:ff:ff:ff
33: dummy4: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr5 state UNKNOWN group default qlen 1000
    link/ether c2:7e:27:1c:0c:af brd ff:ff:ff:ff:ff:ff
34: vmbr5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c2:7e:27:1c:0c:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c07e:27ff:fe1c:caf/64 scope link 
       valid_lft forever preferred_lft forever
35: tap100i5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr5 state UNKNOWN group default qlen 1000
    link/ether 92:cb:02:fe:5f:86 brd ff:ff:ff:ff:ff:ff
42: tap501i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr5 state UNKNOWN group default qlen 1000
    link/ether 8a:80:41:55:95:0c brd ff:ff:ff:ff:ff:ff
49: tap202i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether c6:2e:7c:40:b8:02 brd ff:ff:ff:ff:ff:ff

PVE02 Network configuration:

# Server pag-02
# network interfaces
#
# Author:       Gilberto Martins
# Creation:     06/08/2021
# ================================

    auto lo
    iface lo inet loopback
    auto eno1
    iface eno1 inet manual
    auto eno2
    iface eno2 inet manual
    
    # Internet Interface 
    auto vmbr0
    iface vmbr0 inet dhcp
      # Interface externa - NAO USAR
      bridge-ports eno1
      bridge-stp off
      bridge-fd 0
    
    # Tools Network
    auto vmbr1
    iface vmbr1 inet manual
      # Tools Network - 172.22.10.0/27
      bridge-ports dummy1
      bridge-stp off
      bridge-fd 0
    
    # DataBase Network
    auto vmbr2
    iface vmbr2 inet manual
      # DataBase Network - 172.22.20.0/27
      bridge-ports dummy2
      bridge-stp off
      bridge-fd 0

    # VRack Network
#    auto vmbr3
#    iface vmbr3 inet static
      # VRack Network
#      address 192.168.0.11/31
#      bridge-ports eno2
#      bridge-stp off
#      bridge-fd 0

PVE02 current interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether d0:50:99:fb:24:13 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr3 state UP group default qlen 1000
    link/ether d0:50:99:fb:24:12 brd ff:ff:ff:ff:ff:ff
4: enp0s20f0u8u3c2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 26:fc:24:e9:66:dc brd ff:ff:ff:ff:ff:ff
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether RR:RR:RR:RR:RR:RR brd ff:ff:ff:ff:ff:ff
    inet 4.4.4.4/24 brd 4.4.4.255 scope global dynamic vmbr0
       valid_lft 73446sec preferred_lft 73446sec
    inet6 fe80::d250:99ff:fefb:2413/64 scope link 
       valid_lft forever preferred_lft forever
6: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:32:c1:5c:c7:77 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ccf5:5bff:fead:bf80/64 scope link 
       valid_lft forever preferred_lft forever
7: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 46:c7:8c:94:01:4b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::58d2:51ff:fe31:6516/64 scope link 
       valid_lft forever preferred_lft forever
8: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d0:50:99:fb:24:12 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d250:99ff:fefb:2412/64 scope link 
       valid_lft forever preferred_lft forever
13: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 9a:de:c5:ba:40:80 brd ff:ff:ff:ff:ff:ff
14: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether ba:32:c1:5c:c7:77 brd ff:ff:ff:ff:ff:ff
15: tap100i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 46:c7:8c:94:01:4b brd ff:ff:ff:ff:ff:ff
16: tap100i3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether a2:e9:f1:ba:f1:a9 brd ff:ff:ff:ff:ff:ff
17: tap301i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 66:ba:b1:22:e8:22 brd ff:ff:ff:ff:ff:ff
18: tap302i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether e2:f8:74:ad:e4:77 brd ff:ff:ff:ff:ff:ff
19: tap303i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 3e:b1:f0:42:8d:75 brd ff:ff:ff:ff:ff:ff
20: tap304i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 52:7a:ec:b5:46:4b brd ff:ff:ff:ff:ff:ff
21: veth201i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr201i0 state UP group default qlen 1000
    link/ether fe:0c:f2:09:62:fe brd ff:ff:ff:ff:ff:ff link-netnsid 0
22: fwbr201i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:fd:8d:06:38:c5 brd ff:ff:ff:ff:ff:ff
23: fwpr201p0@fwln201i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 52:58:a1:6d:db:00 brd ff:ff:ff:ff:ff:ff
24: fwln201i0@fwpr201p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr201i0 state UP group default qlen 1000
    link/ether ae:fd:8d:06:38:c5 brd ff:ff:ff:ff:ff:ff

PRD1FWL100 Network configuration:

# This is the network config written by 'subiquity'
#
# Author:       Gilberto Martins
# Modified:     03/19/2021
# ===============================

network:
  ethernets:
    # External IP
    ens18:
      # IP and Gateway have been intentionally changed
      addresses:
      - 1.1.1.1/32
      gateway4: 1.1.1.254
      # OVH mandatory routes
      routes:
      - to: 1.1.1.154/32
        via: 1.1.1.1
      - to: 0.0.0.0/0
        via: 1.1.1.1
      nameservers:
        addresses:
          - 172.21.10.2
        search:
          - kprd1
    # Tools Network
    ens19:
      addresses:
      - 172.21.10.1/27
    # WebPrd Network
    ens20:
      addresses:
      - 172.21.20.1/27
    # WebHml Network
    ens21:
      addresses:
      - 172.21.30.1/27
    # Vrack Network (RFC 3021)
    ens22:
      addresses:
      - 172.30.0.0/31
      routes:
        # Tools network at kprd2
      - to: 172.22.10.0/27
        via: 172.30.0.0
        # Database network at kprd2
      - to: 172.22.20.0/27
        via: 172.30.0.0
        # VRack <-> VRack 
      - to: 172.30.0.1
        via: 172.30.0.0
    # WebServer Network
    ens23:
      addresses:
      - 172.21.50.1/27
  version: 2

PRD1FWL100 current interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XS:XS:XS:XS:XS:XS brd ff:ff:ff:ff:ff:ff
    inet 9.9.9.9/32 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::ff:fe41:b0ec/64 scope link 
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 22:a9:69:cd:9a:08 brd ff:ff:ff:ff:ff:ff
    inet 172.21.10.1/27 brd 172.21.10.31 scope global ens19
       valid_lft forever preferred_lft forever
    inet6 fe80::20a9:69ff:fecd:9a08/64 scope link 
       valid_lft forever preferred_lft forever
4: ens20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:c5:9a:8e:13:0d brd ff:ff:ff:ff:ff:ff
    inet 172.21.20.1/27 brd 172.21.20.31 scope global ens20
       valid_lft forever preferred_lft forever
    inet6 fe80::94c5:9aff:fe8e:130d/64 scope link 
       valid_lft forever preferred_lft forever
5: ens21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 36:b2:5a:cc:a4:91 brd ff:ff:ff:ff:ff:ff
    inet 172.21.30.1/27 brd 172.21.30.31 scope global ens21
       valid_lft forever preferred_lft forever
    inet6 fe80::34b2:5aff:fecc:a491/64 scope link 
       valid_lft forever preferred_lft forever
6: ens22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 92:5b:ab:3c:75:2f brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.0/31 scope global ens22
       valid_lft forever preferred_lft forever
    inet6 fe80::905b:abff:fe3c:752f/64 scope link 
       valid_lft forever preferred_lft forever
7: ens23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 9a:a2:c1:97:59:54 brd ff:ff:ff:ff:ff:ff
    inet 172.21.50.1/27 brd 172.21.50.31 scope global ens23
       valid_lft forever preferred_lft forever
    inet6 fe80::98a2:c1ff:fe97:5954/64 scope link 
       valid_lft forever preferred_lft forever
8: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none 
    inet 10.10.1.1/29 brd 10.10.1.7 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::ece8:6abc:f8bd:d5f4/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

PRD1FWL100 current routing table

Note: External adresses have been cloaked

user@prd1fwl100:~$ ip route 
default via 9.9.9.9 dev ens18 proto static 
10.10.1.0/29 dev tun0 proto kernel scope link src 10.10.1.1 
9.9.9.9 via 8.8.8.8 dev ens18 proto static 
172.21.10.0/27 dev ens19 proto kernel scope link src 172.21.10.1 
172.21.20.0/27 dev ens20 proto kernel scope link src 172.21.20.1 
172.21.30.0/27 dev ens21 proto kernel scope link src 172.21.30.1 
172.21.50.0/27 dev ens23 proto kernel scope link src 172.21.50.1 
172.22.10.0/27 via 172.30.0.0 dev ens22 proto static 
172.22.20.0/27 via 172.30.0.0 dev ens22 proto static 
172.30.0.1 via 172.30.0.0 dev ens22 proto static 

user@prd1fwl100:~$ ip route show table local
broadcast 10.10.1.0 dev tun0 proto kernel scope link src 10.10.1.1 
local 10.10.1.1 dev tun0 proto kernel scope host src 10.10.1.1 
broadcast 10.10.1.7 dev tun0 proto kernel scope link src 10.10.1.1 
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 
local 9.9.9.9 dev ens18 proto kernel scope host src 9.9.9.9
broadcast 172.21.10.0 dev ens19 proto kernel scope link src 172.21.10.1 
local 172.21.10.1 dev ens19 proto kernel scope host src 172.21.10.1 
broadcast 172.21.10.31 dev ens19 proto kernel scope link src 172.21.10.1 
broadcast 172.21.20.0 dev ens20 proto kernel scope link src 172.21.20.1 
local 172.21.20.1 dev ens20 proto kernel scope host src 172.21.20.1 
broadcast 172.21.20.31 dev ens20 proto kernel scope link src 172.21.20.1 
broadcast 172.21.30.0 dev ens21 proto kernel scope link src 172.21.30.1 
local 172.21.30.1 dev ens21 proto kernel scope host src 172.21.30.1 
broadcast 172.21.30.31 dev ens21 proto kernel scope link src 172.21.30.1 
broadcast 172.21.50.0 dev ens23 proto kernel scope link src 172.21.50.1 
local 172.21.50.1 dev ens23 proto kernel scope host src 172.21.50.1 
broadcast 172.21.50.31 dev ens23 proto kernel scope link src 172.21.50.1 
local 172.30.0.0 dev ens22 proto kernel scope host src 172.30.0.0

PRD2FWL100 Network configuration:

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        # Internet interface
        eth0:
            # Sensitive addressing information have been intentionally changed
            addresses:
            - 3.3.3.3/32
            gateway4: 3.3.3.254
            match:
              macaddress: XX:XX:XX:XX:XX:XX
            # OVH mandatory routes
            routes:
            - to: 3.3.3.3/32
              via: 3.3.3.8
            - to: 0.0.0.0/0
              via: 3.3.3.8
            nameservers:
              addresses:
                - 172.22.10.2
              search:
                - kprd2
            set-name: eth0
        # Tools interface
        eth1:
            addresses:
            - 172.22.10.1/27
            match:
                macaddress: 6a:6d:d1:0a:de:10
            nameservers:
                addresses:
                - 172.22.10.2
                search:
                - kprd2
            set-name: eth1
        # Database interface
        eth2:
            addresses:
            - 172.22.20.1/27
            match:
                macaddress: aa:89:70:41:ed:22
            set-name: eth2
        # VRack Network
        eth3:
            addresses:
            - 172.30.0.1/31
            match:
                macaddress: ZZ:ZZ:ZZ:ZZ:ZZ:ZZ
            routes:
              # Tools network at kprd1
            - to: 172.21.10.0/27
              via: 172.30.0.1
              # WebPrd network at kprd1
            - to: 172.21.20.0/27
              via: 172.30.0.1
              # WebHml network at kprd1
            - to: 172.21.30.0/27
              via: 172.30.0.1
              # WebServer network at kprd1
            - to: 172.21.50.0/27
              via: 172.30.0.1
              # VRack <-> VRack 
            - to: 172.30.0.0
              via: 172.30.0.1
            set-name: eth3

PRD2FWL100 current interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether FE:FE:FE:FE:FE brd ff:ff:ff:ff:ff:ff
    inet 7.7.7.7/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ff:fe92:ec0/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 6a:6d:d1:0a:de:10 brd ff:ff:ff:ff:ff:ff
    inet 172.22.10.1/27 brd 172.22.10.31 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::686d:d1ff:fe0a:de10/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether aa:89:70:41:ed:22 brd ff:ff:ff:ff:ff:ff
    inet 172.22.20.1/27 brd 172.22.20.31 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::a889:70ff:fe41:ed22/64 scope link 
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d6:9f:c5:e4:93:9d brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.1/31 scope global eth3
       valid_lft forever preferred_lft forever
    inet6 fe80::d49f:c5ff:fee4:939d/64 scope link 
       valid_lft forever preferred_lft forever
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.10.2.1/29 brd 10.10.2.7 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::d63:c98b:2e1:ad3d/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

PRD2FWL100 routing table

Note: External addresses have been cloaked

user@prd2fwl100:~$ ip route
default via 144.217.125.8 dev eth0 proto static 
10.10.2.0/29 dev tun0 proto kernel scope link src 10.10.2.1 
9.9.9.9 via 8.8.8.8 dev eth0 proto static 
172.21.10.0/27 via 172.30.0.1 dev eth3 proto static 
172.21.20.0/27 via 172.30.0.1 dev eth3 proto static 
172.21.30.0/27 via 172.30.0.1 dev eth3 proto static 
172.21.50.0/27 via 172.30.0.1 dev eth3 proto static 
172.22.10.0/27 dev eth1 proto kernel scope link src 172.22.10.1 
172.22.20.0/27 dev eth2 proto kernel scope link src 172.22.20.1 
172.30.0.0 via 172.30.0.1 dev eth3 proto static 

user@prd2fwl100:~$ ip route show table local
broadcast 10.10.2.0 dev tun0 proto kernel scope link src 10.10.2.1 
local 10.10.2.1 dev tun0 proto kernel scope host src 10.10.2.1 
broadcast 10.10.2.7 dev tun0 proto kernel scope link src 10.10.2.1 
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 
local 8.8.8.8 dev eth0 proto kernel scope host src 8.8.8.8 
broadcast 172.22.10.0 dev eth1 proto kernel scope link src 172.22.10.1 
local 172.22.10.1 dev eth1 proto kernel scope host src 172.22.10.1 
broadcast 172.22.10.31 dev eth1 proto kernel scope link src 172.22.10.1 
broadcast 172.22.20.0 dev eth2 proto kernel scope link src 172.22.20.1 
local 172.22.20.1 dev eth2 proto kernel scope host src 172.22.20.1 
broadcast 172.22.20.31 dev eth2 proto kernel scope link src 172.22.20.1 
local 172.30.0.1 dev eth3 proto kernel scope host src 172.30.0.1

PRD1FRM206 Network configuration:

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - 172.21.10.7/27
            gateway4: 172.21.10.1
            match:
                macaddress: ca:7a:03:34:a0:43
            nameservers:
                addresses:
                - 172.21.10.2
                search:
                - kprd1
            set-name: eth0

PRD2FRM201 Network configuration:

PRD2FRM201 is a LXC host with the following configuration at ProxMox:

  • IP 172.22.10.2/27
  • Gateway 172.22.10.1
  • Bridge vmbr1

Comunication tests:

From PRD2FWL100, I can ping all hops before PRD1FRM206:

user@prd2fwl100:~$ ping 172.30.0.0 -c1
PING 172.30.0.0 (172.30.0.0) 56(84) bytes of data.
64 bytes from 172.30.0.0: icmp_seq=1 ttl=64 time=0.671 ms

--- 172.30.0.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.671/0.671/0.671/0.000 ms

user@prd2fwl100:~$ ping 172.21.10.1 -c1
PING 172.21.10.1 (172.21.10.1) 56(84) bytes of data.
64 bytes from 172.21.10.1: icmp_seq=1 ttl=64 time=0.822 ms

--- 172.21.10.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.822/0.822/0.822/0.000 ms

But I cannot ping or arping PRD1FRM206:

user@prd2fwl100:~$ ping 172.21.10.7 -c1
PING 172.21.10.7 (172.21.10.7) 56(84) bytes of data.
From 172.30.0.1 icmp_seq=1 Destination Host Unreachable

--- 172.21.10.7 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

user@prd2fwl100:~$ arping 172.21.10.7 -c1
ARPING 172.21.10.7 from 172.30.0.1 eth3
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

Next, I will try to ping all IPs from PRD2FRM201 to PRD1FRM206:

user@PRD2FRM201:~$ sudo ping 172.22.10.1 -c1
PING 172.22.10.1 (172.22.10.1) 56(84) bytes of data.
64 bytes from 172.22.10.1: icmp_seq=1 ttl=64 time=0.134 ms

--- 172.22.10.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.134/0.134/0.134/0.000 ms

user@PRD2FRM201:~$ sudo ping 172.30.0.1 -c1
PING 172.30.0.1 (172.30.0.1) 56(84) bytes of data.
64 bytes from 172.30.0.1: icmp_seq=1 ttl=64 time=0.159 ms

--- 172.30.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.159/0.159/0.159/0.000 ms

Likewise, there is a spot I cannot go further:

user@PRD2FRM201:~$ sudo ping 172.30.0.0 -c1
PING 172.30.0.0 (172.30.0.0) 56(84) bytes of data.

--- 172.30.0.0 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

user@PRD2FRM201:~$ sudo arping 172.30.0.0 -c1
ARPING 172.30.0.0 from 172.22.10.2 eth0
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

What I have to do so I can have this issue fixed?

Best Answer

Some routes are simply wrong. I show the low level correction, the higher level networking configuration should be changed accordingly.

PRD1FWL100

172.30.0.1 via 172.30.0.0 dev ens22 proto static

while it's technically working, using one own's IP address as gateway is the same as not using a gateway: as should be on two systems bridged in the same Ethernet broadcast LAN. Do not use a gateway at all here. Replace with... actually the LAN route should have been set by the kernel, but the higher level tool might have chosen to override it (by declaring addresses with noprefixroute). The src hint below is probably optional. Put either this (should be preferred since that would be the default set by the kernel if not overridden):

172.30.0.0/31 dev ens22 src 172.30.0.0

or else this:

172.30.0.1/32 dev ens22 src 172.30.0.0

172.22.10.0/27 via 172.30.0.0 dev ens22 proto static 
172.22.20.0/27 via 172.30.0.0 dev ens22 proto static

These routes are really wrong: if the router declares itself as the gateway, again that means it considers theses addresses to be reachable directly in the same LAN. Instead of trying to route these packets to the next-hop (172.30.0.1) it will emit ARP broadcasts on ens22 but there won't be any reply...

... except for the peer PRD2FRM201's addresses when pinging its IP addresses 172.22.x.1, since Linux following the weak host model will answer ARP to any local address on any interface. This apparently half working part might have lead to believe that the issue was elsewhere. This can be checked on PRD1FWL100 with ip neigh show dev ens22 which will show an ARP table polluted with addresses from other IP networks. Those belonging to the peer router will be resolved, but the others will have a FAILED state (=> no route to host).

Replace with:

172.22.10.0/27 via 172.30.0.1 dev ens22 
172.22.20.0/27 via 172.30.0.1 dev ens22

PRD2FWL100

This is exactly the same problem with reversed addresses.

172.30.0.1 via 172.30.0.0 dev ens22 proto static

to be replaced with:

172.30.0.0/31 dev ens22 src 172.30.0.1

or

172.30.0.0/32 dev ens22 src 172.30.0.1

172.21.10.0/27 via 172.30.0.1 dev eth3 proto static 
172.21.20.0/27 via 172.30.0.1 dev eth3 proto static 
172.21.30.0/27 via 172.30.0.1 dev eth3 proto static 
172.21.50.0/27 via 172.30.0.1 dev eth3 proto static

to be replaced with:

172.21.10.0/27 via 172.30.0.0 dev eth3 
172.21.20.0/27 via 172.30.0.0 dev eth3 
172.21.30.0/27 via 172.30.0.0 dev eth3 
172.21.50.0/27 via 172.30.0.0 dev eth3
Related Topic