Linux Networking – Bridge with Veths Not Sending Outbound Packets

bridgelinuxnetworking

I've been running into an issue where a Linux networking bridge I create on Ubuntu 18.04 cannot access the Internet. I have a network namespace in Linux that I want to run an application in. I want this application to be able to send outbound packets to the Internet. Therefore, I setup a veth pair and put the peer inside of the network namespace. Veth1 is the veth on the host machine/default network namespace and veth2 is the veth inside the custom network namespace (test). I then setup a Linux bridge on the host and added veth1 to it. Here are commands I've ran to achieve this:

# Create namespace.
ip netns add test

# Put up loopback interface.
ip netns exec test ip link set lo up

# Create veth pair.
ip link add veth1 type veth peer name veth2

# Put veth2 inside namespace.
ip link set veth2 netns test

# Add IP address to veth2 inside namespace.
ip netns exec test ip addr add 172.20.0.2/16 dev veth2

# Put veth2 up.
ip netns exec test ip link set veth2 up

# Delete default route in namespace.
ip netns exec test ip route delete default

# Add veth2 to default route in namespace.
ip netns exec test ip route add default dev veth2

# Create bridge br0.
ip link add br0 type bridge

# Add veth1 to bridge (I've also tried 'brctl addif br0 veth1').
ip link set veth1 master br0

# Add IP to br0.
ip addr add 172.20.0.1/16 dev br0

# Put br0 up.
ip link set br0 up

Initially, I was trying to get this to work for an application I didn't create. The application was sending outbound packets through the veth2 interface inside of the network namespace since that's the default route. However, all it sent was ARP requests (who-has) and it never received any sort of response. Therefore, I decided to create my own C program that uses AF_PACKET sockets. Here is the code for anyone wondering. All it does is bind to a specific interface and sends an empty UDP packet to a destination specified in the command line. I also made it so you can set the source IP in the command line. One other thing I'd like to note is the program retrieves the MAC address of the gateway and uses that as the destination MAC for the Ethernet header (I wasn't sure what to set the destination MAC to and read setting it to the gateway MAC address should work since ARP requests shouldn't go to IPs outside of the network).

When executing the program inside the network namespace like this:

ip netns exec test ./test_veth veth2 10.50.0.11 10.50.0.3

Traffic never reaches 10.50.0.3. I can see the traffic on veth1 and br0 via tcpdump. Here's an example of br0:

root@netvm02:/home/roy# tcpdump -i br0 -nne
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:29:13.928570 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:14.928741 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:15.928957 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:16.929181 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:17.929412 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0

When I run the program inside the default network namespace and attached to veth, I never end up seeing the traffic on br0. This might be because of my program setting the destination MAC to the gateway, though:

root@netvm02:/home/roy# tcpdump -i veth1 -nne
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:30:58.397476 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:30:59.397707 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:31:00.398022 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:31:01.398295 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:31:02.398544 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0

I tried also attaching the program to br0 and 10.50.0.3 still doesn't see the traffic. Therefore, I'm assuming there's something wrong with the bridge.

If I attach it to the main interface (ens18 in this case), I can see traffic on 10.50.0.3:

root@test02:/home/roy# tcpdump -i any host 10.50.0.11 and udp -nne
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
22:17:59.964569  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:00.964726  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:01.965059  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:02.965271  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:03.965544  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0

I've also tried adding the physical interface (ens18) to the bridge via brctl (bridge-utils):

brctl addif br0 ens18

This results in the VM not able to send any packets outbound and connection to the VM is lost.

I've tried masquerading both 172.20.0.0/16 and the br0 interface via:

iptables -t nat -A POSTROUTING -s 172.20.0.0/16 -j MASQUERADE
iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE

Unfortunately, neither of these worked. What's weird is when running the program, I'm not seeing any packets being processed by these rules when running iptables -t nat -L -n -v:

Chain POSTROUTING (policy ACCEPT 5 packets, 355 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       172.20.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      br0     0.0.0.0/0            0.0.0.0/0

I also tried setting the source IP of the program to 172.20.0.2 to see if the first rule would process the packets. Sadly, it didn't.

I've also tried setting net.ipv4.ip_forward to 1 via sysctl net.ipv4.ip_forward=1. I had no luck with this as well, though.

Here are the forwarding rules I tried in IPTables:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  A      A       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    br0     0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    !br0    0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  A      br0     0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    A       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  ens18  br0     0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    ens18   0.0.0.0/0            0.0.0.0/0

I know a lot of these are probably useless, but I was just trying things to see if they made any difference.

Here is additional information including a full ifconfig and more:

root@netvm02:/home/roy# ifconfig
veth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 02:a2:0f:2a:7b:bf  txqueuelen 1000  (Ethernet)
        RX packets 3655  bytes 154906 (154.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2380  bytes 101548 (101.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::185a:96ff:fe62:d174  prefixlen 64  scopeid 0x20<link>
        ether 02:a2:0f:2a:7b:bf  txqueuelen 1000  (Ethernet)
        RX packets 726  bytes 55088 (55.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 276  bytes 12624 (12.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.50.0.11  netmask 255.255.255.0  broadcast 10.50.0.255
        inet6 fe80::e087:deff:fe1f:d504  prefixlen 64  scopeid 0x20<link>
        ether e2:87:de:1f:d5:04  txqueuelen 1000  (Ethernet)
        RX packets 1423812  bytes 306465717 (306.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1694988587  bytes 2103526747383 (2.1 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2436  bytes 223919 (223.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2436  bytes 223919 (223.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@netvm02:/home/roy# ip netns exec test ifconfig
veth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.0.2  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::407d:2aff:fe5e:8c78  prefixlen 64  scopeid 0x20<link>
        ether 42:7d:2a:5e:8c:78  txqueuelen 1000  (Ethernet)
        RX packets 2380  bytes 101548 (101.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3677  bytes 155830 (155.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@netvm02:/home/roy# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether e2:87:de:1f:d5:04 brd ff:ff:ff:ff:ff:ff
    inet 10.50.0.11/24 brd 10.50.0.255 scope global dynamic ens18
       valid_lft 80490sec preferred_lft 80490sec
    inet6 fe80::e087:deff:fe1f:d504/64 scope link
       valid_lft forever preferred_lft forever
4: veth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 02:a2:0f:2a:7b:bf brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:a2:0f:2a:7b:bf brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::185a:96ff:fe62:d174/64 scope link
       valid_lft forever preferred_lft forever

root@netvm02:/home/roy# ip netns exec test ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: veth2@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:7d:2a:5e:8c:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.20.0.2/16 scope global veth2
       valid_lft forever preferred_lft forever
    inet6 fe80::407d:2aff:fe5e:8c78/64 scope link
       valid_lft forever preferred_lft forever


root@netvm02:/home/roy# ip route
default via 10.50.0.1 dev ens18 proto dhcp src 10.50.0.11 metric 100
10.50.0.0/24 dev ens18 proto kernel scope link src 10.50.0.11
10.50.0.1 dev ens18 proto dhcp scope link src 10.50.0.11 metric 100
172.20.0.0/16 dev br0 proto kernel scope link src 172.20.0.1

root@netvm02:/home/roy# ip netns exec test ip route
default dev veth2 scope link
172.20.0.0/16 dev veth2 proto kernel scope link src 172.20.0.2

root@netvm02:/home/roy# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.02a20f2a7bbf       no              veth1

Additionally, both 10.50.0.11 and 10.50.0.3 are VMs on my home server running ProxMox. They're using DHCP on the main interfaces (ens18), but have static IP mappings from my Edge Router.

I haven't messed with bridges or veths much before this, so there is probably something I'm missing.

I just want traffic from br0 to be able to reach the Internet. In the above, I am testing connections on my local network, but the application I plan to run will be sending packets to IPs outside of the network.

If you need any additional information, please let me know!

Any help is highly appreciated and thank you for your time!

Best Answer

You have to treat a separate network namespace as though it were a different host and the connection between the veth pair as the line where the external packets come in. So you MUST activate routing. The iptables in the main namespace will see the packets in PREROUTING and POSTROUTING and INPUT and OUTPUT.

So to setup the outbound functions (replace eth0 with your outward interface):

# Activate router functions
# Has side effects: e.g. net.ipv4.conf.all.accept_redirects=0,secure_redirects=1
# Resets ipv4 kernel interface 'all' config values to default for HOST or ROUTER
# https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
echo 1 > /proc/sys/net/ipv4/ip_forward
# Set a gateway for the 'inside' namespace
# You have to specify an ip which will be the next hop
# This ip must be on the network segment of the main namespace veth
ip netns exec test ip route add default via 172.20.0.1
# Masquerade outgoing connections (you can limit to tcp with `-p tcp`)
iptables -t nat -A POSTROUTING -s 172.20.0.1 -o eth0 -j MASQUERADE
# If default FORWARD policy is DROP
# Let packets move from the outward interface
# to the virtual ethernet pair and vice versa
iptables -A FORWARD -i eth0 -o br0 -j ACCEPT
iptables -A FORWARD -o eth0 -i br0 -j ACCEPT
# Setup a resolver (replace with your own DNS, does not work with a loopback resolver)
mkdir -p /etc/netns/test
echo nameserver dns-ip > /etc/netns/test/resolv.conf
# Maybe give it its own hosts file, to do edits
cp /etc/hosts /etc/netns/test/hosts

Now you can test with ip netns exec test ping example.com

Related Topic