Ubuntu – OpenVpn & KVM: can’t access LAN behind server

kvm-virtualizationopenvpnrouteUbuntu

i just set up an OpenVPN Server on a rented root Server. I can ping the OpenVPN Server IP, but can't connect or ping to other Machines (VMs) on the OpenVPN Server Network.
According to the official OpenVPN doku i added "push route" to the server.conf to enable access to other Machines on the Server Network.

server.conf:

port 1194
proto udp
dev tun
ca ca.crt
cert ex1.crt
key ex1.key  # This file should be kept secret
dh dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 10.10.10.0 255.255.255.0"
keepalive 10 120
tls-auth ta.key 0 # This file is secret
cipher AES-256-CBC-HMAC-SHA1
cipher AES-256-CBC
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3

ifconfig from the openvpn server:

br0       Link encap:Ethernet  HWaddr 62:e6:d9:07:46:c2  
          inet addr:148.251.139.133  Bcast:148.251.139.133  Mask:255.255.255.255
          inet6 addr: fe80::60e6:d9ff:fe07:46c2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:69028 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:2899368 (2.8 MB)

eth0      Link encap:Ethernet  HWaddr 44:8a:5b:9b:a0:7d  
          inet addr:148.251.139.133  Bcast:148.251.139.159  Mask:255.255.255.224
          inet6 addr: 2a01:4f8:210:4384::2/64 Scope:Global
          inet6 addr: fe80::468a:5bff:fe9b:a07d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:78430 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82129 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6908929 (6.9 MB)  TX bytes:52446354 (52.4 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:21482 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21482 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:47236296 (47.2 MB)  TX bytes:47236296 (47.2 MB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:672 (672.0 B)  TX bytes:728 (728.0 B)

virbr0    Link encap:Ethernet  HWaddr 52:54:00:76:9b:d5  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:277 errors:0 dropped:0 overruns:0 frame:0
          TX packets:269 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:41733 (41.7 KB)  TX bytes:45532 (45.5 KB)

virbr1    Link encap:Ethernet  HWaddr 52:54:00:4c:37:b5  
          inet addr:10.10.10.1  Bcast:10.10.10.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2258 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2446 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:370903 (370.9 KB)  TX bytes:197319 (197.3 KB)

vnet0     Link encap:Ethernet  HWaddr fe:54:00:15:93:50  
          inet6 addr: fe80::fc54:ff:fe15:9350/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:277 errors:0 dropped:0 overruns:0 frame:0
          TX packets:75139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:45611 (45.6 KB)  TX bytes:3938924 (3.9 MB)

vnet1     Link encap:Ethernet  HWaddr fe:54:00:ff:e6:3e  
          inet6 addr: fe80::fc54:ff:feff:e63e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2258 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77316 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:402515 (402.5 KB)  TX bytes:4090711 (4.0 MB)

The machine on the server network which i try to connect is a KVM VM and has got the IP: 10.10.10.2. I can ping 10.10.10.1 (IP of the virbr1).
Ipv4 forwarding on the ovpn/kvm Server is enabled. While testing i also deactivate the ufw firewall of the ovpn server and of the VM.

If i ping 10.10.10.2 from the ovpn client i get the following tcpdump(tun0) on the server:

11:54:46.002533 IP 10.8.0.6 > 10.10.10.2: ICMP echo request, id 17629,
seq 0, length 64 11:54:46.002602 IP 10.8.0.1 > 10.8.0.6: ICMP
10.10.10.2 protocol 1 port 10088 unreachable, length 92

while tcpdump of the virbr1 bridge(network of the vm: 10.10.10.2) doesn't show anything at all.
I'm assuming, that the openvpn server doesn't forward the packets to virbr1.

routes of the openvpn server:

Kernel IP routing table Destination     Gateway         Genmask       
Flags Metric Ref    Use Iface
0.0.0.0         148.251.139.129 0.0.0.0         UG    0      0        0 eth0
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 virbr1
148.251.139.128 148.251.139.129 255.255.255.224 UG    0      0        0 eth0
148.251.139.128 0.0.0.0         255.255.255.224 U     0      0        0 eth0
148.251.164.152 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.153 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.154 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.155 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.156 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.157 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.158 0.0.0.0         255.255.255.255 UH    0      0        0 br0
148.251.164.159 0.0.0.0         255.255.255.255 UH    0      0        0 br0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

routes on the vm client:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.122.1   0.0.0.0         UG    0      0        0 eth0
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

I'm really stuck with this problem and it would be really nice to be able to understand the problem and then be able to solve it.

Best Answer

Ok problem solved:

The problem was a misconfiguration in libvirt. The virtual bridge was configured in isolated mode, after switching to routed virtual network and adding the route: 10.8.0.0/24 gw 10.10.10.1 to the VM guests everything worked as expected.