Linux Networking – Linux Server Ignoring Packets from Gateway

debianlinux-networkingnetworkingproxmoxrouting

I have a Proxmox Linux server, which is able to send and receive packets to hosts on the local network, but will not process packets from the gateway. This causes internet traffic to fail, so I can't run apt to update packages. All protocols appear to be affected.

VMs running on the server can access the gateway just fine.

My /etc/network/interfaces file contains:

auto lo
iface lo inet loopback

iface enp10s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.0.1.200/24
    gateway 10.0.1.1
    bridge_ports enp10s0
    bridge_stp off
    bridge_fd 0

auto wlp7s0
iface wlp7s0 inet static
    hostapd /etc/hostapd/hostapd.conf
    address 10.0.2.1
    netmask 255.255.255.0

auto vmbr1
iface vmbr1 inet static
    address 10.1.2.1
    netmask 255.255.255.0
    bridge_ports none
    bridge-stp off
    bridge-fd 0

    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.1.2.0/24' -o wlp7s0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.1.2.0/24' -o wlp7s0 -j MASQUERADE

wlp7s0 and vmbr1 are NAT'd to allow a VM to access wireless IOT devices that should not have access to the general network/internet.

My route table:

$ ip route
default via 10.0.1.200 dev vmbr0 metric 100 
10.0.1.0/24 dev vmbr0 proto kernel scope link src 10.0.1.200 
10.0.2.0/24 dev wlp7s0 proto kernel scope link src 10.0.2.1 
10.1.2.0/24 dev vmbr1 proto kernel scope link src 10.1.2.1

After some reading, I tried changing rp_filter, but changing the value from 2 to 0 didn't help. The default settings (with VM interfaces removed):

$ sysctl -a | grep \\.rp_filter
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.enp10s0.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.vmbr0.rp_filter = 0
net.ipv4.conf.vmbr1.rp_filter = 0
net.ipv4.conf.wlp7s0.rp_filter = 0

ip_forward is set:

$ cat /proc/sys/net/ipv4/ip_forward
1

I have verified via tcpdump that packets are being received from the gateway when I try pinging from the server to the gateway, or from the server to the gateway. This example uses ping:

# tcpdump -n -i vmbr0 host 10.0.1.1 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:42:37.136341 IP 10.0.1.200 > 10.0.1.1: ICMP echo request, id 22073, seq 1, length 64
22:42:37.136478 IP 10.0.1.1 > 10.0.1.200: ICMP echo reply, id 22073, seq 1, length 64
22:42:38.142240 IP 10.0.1.200 > 10.0.1.1: ICMP echo request, id 22073, seq 2, length 64
22:42:38.142429 IP 10.0.1.1 > 10.0.1.200: ICMP echo reply, id 22073, seq 2, length 64

The output of ping -v is simply empty:

$ ping -v 10.0.1.1
PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
^C
--- 10.0.1.1 ping statistics ---
22 packets transmitted, 0 received, 100% packet loss, time 511ms

The only entry in ip tables is the NAT:

# iptables-save -c
# Generated by iptables-save v1.8.2 on Sat Jan 29 22:45:46 2022
*raw
:PREROUTING ACCEPT [1828405583:1847667077335]
:OUTPUT ACCEPT [10762322:981310704]
COMMIT
# Completed on Sat Jan 29 22:45:46 2022
# Generated by iptables-save v1.8.2 on Sat Jan 29 22:45:46 2022
*filter
:INPUT ACCEPT [10597558:1212589593]
:FORWARD ACCEPT [1782904005:1841102268241]
:OUTPUT ACCEPT [10762351:981313827]
COMMIT
# Completed on Sat Jan 29 22:45:46 2022
# Generated by iptables-save v1.8.2 on Sat Jan 29 22:45:46 2022
*nat
:PREROUTING ACCEPT [29808561:4940456833]
:INPUT ACCEPT [2456738:231340403]
:OUTPUT ACCEPT [1168080:75403202]
:POSTROUTING ACCEPT [2829337:181352732]
[190:11400] -A POSTROUTING -s 10.1.2.0/24 -o wlp7s0 -j MASQUERADE
COMMIT
# Completed on Sat Jan 29 22:45:46 2022

Best Answer

With thanks to Nikita Kipriyanov for pointing me in the right direction, I have solved the problem.

When running

tcpdump -en host 10.0.1.1

I noticed that ping responses were coming in on an interface that didn't exist on the system:

# tcpdump -en host 10.0.1.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp10s0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:01:49.233889 24:4b:fe:4a:11:96 > 02:29:9a:f3:0a:00, ethertype IPv4 (0x0800), length 98: 10.0.1.200 > 10.0.1.1: ICMP echo request, id 5544, seq 1, length 64
10:01:49.234053 02:29:9a:f3:0a:00 > 00:26:18:e2:68:f9, ethertype IPv4 (0x0800), length 98: 10.0.1.1 > 10.0.1.200: ICMP echo reply, id 5544, seq 1, length 64

This means that the system was seeing the incoming packets with the right IP address under tcpdump, but correctly ignoring them for processing based on the MAC address.

I checked the configuration on the gateway router, and noticed it had a fixed ARP entry with that bad address (probably an old configuration). Changing the setting the gateway to enp10s0's MAC address fixed the problem.