Ubuntu – Multihomed ubuntu inbound connectivity issue

linux-networkingmulti-homedUbuntu

I have a multi-homed system (Ubuntu server, 11.04) where eth0 is lan and eth1 is wan. There is internet connectivity through both, but via different routes.

When the system boots, I cannot connect to e.g. apache(TCP 443) on the WAN interface dispite the fact it listens on *.443.

If I bring down eth0 I can, however. After bringing eth0 back up I can still access apache.

There are no IPTables rules active (Policy: accept).

Information summary:

System & Kernel

Ununtu Server 11.04
$ uname -r
Linux myserver 2.6.38-11-server #50-Ubuntu SMP Mon Sep 12 21:34:27 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

Network interfaces:

eth0 is lan, dhcp assigned.
eth1 is wan, static ip.

IPTables

No rules set.
Policy is ACCEPT

eth1 outbound connectivity works.

I have tested outbound connectivity on eth1 using nmap, and this works:

$ nmap -sT -P0 -p80 -eeth1 google.com)
port is open.

Test from WAN there shows no access.

 # nmap -sS -P0 -p443 myhost 

 states the port is filtered

Test from LAN succeeds using Chrome.

Apache is listening correctly:

$ sudo lsof -i :443
COMMAND PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
apache2 822     root    4u  IPv4   7965      0t0  TCP *:https (LISTEN)
apache2 857 www-data    4u  IPv4   7965      0t0  TCP *:https (LISTEN)
apache2 858 www-data    4u  IPv4   7965      0t0  TCP *:https (LISTEN)

The following will enable inbound connections:

# ifconfig eth0 down

# ifconfig eth0 up

Contents of /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback


# The primary network interface
auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
    address xx.yy.zz.ww
    netmask 255.255.255.128
    gateway xx.yy.zz.1
    nameserver 8.8.8.8

Question

I'd like to simply ask "What is wrong?", but I' not expecting there to be enough info in this question for that to work.

Instead, I'll ask: Where do I start looking for the cause of the problem ? What information do I need to fetch ?

Edit: netstat -rvn output

Just after reboot

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface 
xx.yy.zz.0      0.0.0.0         255.255.255.128 U         0 0          0 eth1
10.10.53.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
0.0.0.0         10.10.53.1      0.0.0.0         UG        0 0          0 eth0
0.0.0.0         xx.yy.zz.1      0.0.0.0         UG        0 0          0 eth1

eth0 down

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
xx.yy.zz.0      0.0.0.0         255.255.255.128 U         0 0          0 eth1
0.0.0.0         xx.yy.zz.1      0.0.0.0         UG        0 0          0 eth1

eth0 back up

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
xx.yy.zz.0      0.0.0.0         255.255.255.128 U         0 0          0 eth1
10.10.53.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
0.0.0.0         xx.yy.zz.1      0.0.0.0         UG        0 0          0 eth1

Best Answer

Based on that I'd suggest that your default route out of eth0 is taking precedence. Routing only cares about the best way to send a given packet, it doesn't care about general data flow. A client will attempt to connect to your server on xx.yy.zz.1 via eth0, but the response will be routed out of eth1 on 10.10.53.1. Even if the reply packet successfully reaches it's destination it will have the wrong IP address as a source and the client won't know what to do with it. You'll be able to verify if that's the case just by running packet captures on your interfaces as you make requests to the server.

Check out this guide on how to route packets back out of the interface they arrived on

Related Topic