Linux – Can’t ping multihomed Linux machine on non-default interface

icmplinuxnetworkingrouting

I have a multihomed Ubuntu server with a set of interfaces that includes:

eth2: 10.10.0.131/24
eth3: 10.20.0.2/24

The default interface is eth2, with a gateway of 10.10.0.1. Here's what the routing table looks like:

root@c220-1:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.10.0.1       0.0.0.0         UG        0 0          0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth2
10.20.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth3
10.30.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.40.0.0       0.0.0.0         255.255.0.0     U         0 0          0 eth1

From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.

If I ping 10.10.0.131 from 192.168.3.5, the packets reach the machine, but it doesn't send any replies:

c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 0, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 1, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 2, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 3, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 4, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 5, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 6, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 7, length 64

If I ping from the router (10.20.0.1) on the same network, the server replies properly:

c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80

Note that, as per the answer in this similar question, I have rp_filter off on all of the interfaces, but it doesn't resolve the issue:

$ for i in eth0 eth1 eth2 eth3 all default
> do
> cat /proc/sys/net/ipv4/conf/$i/rp_filter
> done
0
0
0
0
0
0

Best Answer

The problem is that since the default route is through eth2, the ping responses get sent through eth2 even though the requests were received on eth3. (If you tcpdump eth2 you should see the responses being sent.) There's then probably some device which is dropping the packets because they have an invalid source IP for the network they're on. You need some source policy routing in order to get the responses to be sent out the interface they were received on.

  1. Create a new routing table (only needs to be done once):

    echo 13 eth3 >> /etc/iproute2/rt_tables

  2. Add a default route to this new table going out eth3:

    ip route add default via 10.20.0.1 table eth3

  3. Add a policy rule to use this new table for packets with source address of eth3's IP:

    ip rule add from 10.20.0.2 lookup eth3
Related Topic