Linux – Changing the default gateway, multiple NICs

linuxnetworkingUbuntu

I have Ubuntu 13.10 (kernel 3.11.0-19-generic). On startup, the network configurations are as follows:

dor@ubuntu:~$ sudo ifconfig -a
em1       Link encap:Ethernet  HWaddr 44:37:e6:4c:ca:0a  
          inet addr:10.1.0.102  Bcast:10.1.7.255  Mask:255.255.248.0
          inet6 addr: fe80::4637:e6ff:fe4c:ca0a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:732 errors:0 dropped:0 overruns:0 frame:0
          TX packets:79 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:68789 (68.7 KB)  TX bytes:12907 (12.9 KB)
          Interrupt:20 Memory:fe600000-fe620000 

eth5      Link encap:Ethernet  HWaddr 00:0a:5e:50:cf:1e  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:20 Base address:0xa000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8485 (8.4 KB)  TX bytes:8485 (8.4 KB)

p4p1      Link encap:Ethernet  HWaddr 00:15:17:0b:6d:1c  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fe0b:6d1c/64 Scope:Link
          inet6 addr: fc00::215:17ff:fe0b:6d1c/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41 errors:0 dropped:0 overruns:0 frame:0
          TX packets:100 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4062 (4.0 KB)  TX bytes:15981 (15.9 KB)
          Interrupt:16 Memory:fe540000-fe560000 

dor@ubuntu:~$ sudo route -n
[sudo] password for dor: 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 p4p1
10.1.0.0        0.0.0.0         255.255.248.0   U     0      0        0 em1
192.168.1.0     0.0.0.0         255.255.255.0   U     1      0        0 p4p1

The gateway to the internet is 10.1.0.1. So I manually execute the following:

dor@ubuntu:~$ sudo route del default
dor@ubuntu:~$ sudo route add default gw 10.1.0.1

This is how I manage to get to the internet.
But instead of doing this everytime manually, how do I make this permanent?

The content of /etc/network/interfaces (deleted some of the commented lines):

dor@ubuntu:~$ cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto em1

iface eth5 inet static
     address 10.90.90.91
     netmask 255.255.255.0
     gateway 10.90.90.90

iface em1 inet static
     address 10.1.0.102
     netmask 255.255.248.0
     gateway 10.1.0.1

I've read the manual of interfaces and route for so many times and many google results but can't figure this out. Thank you.


Edit #1, 17 Apr 2014 12:11 UTC:
(response to the responder Sobrique)

I've edit the file /etc/network/interfaces to have the following content:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto em1
#iface em1 inet dhcp

iface eth5 inet static
     address 10.90.90.91
     netmask 255.255.255.0
#     gateway 10.90.90.90

iface p4p1 inet static
     address 192.168.1.32
     netmask 255.255.255.0
#     gateway 192.168.1.22

iface em1 inet static
     address 10.1.0.102
     netmask 255.255.248.0
     gateway 10.1.0.1

Now there's an internet access immediately upon startup, without having to execute any command.
The problem is that I don't have access to the network 192.168.1.X.
Earlier there was an access. How can I fix this? Some data:

dor@ubuntu:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.0.1        0.0.0.0         UG    0      0        0 em1
10.1.0.0        0.0.0.0         255.255.248.0   U     0      0        0 em1
dor@ubuntu:~$ ifconfig -a
em1       Link encap:Ethernet  HWaddr 44:37:e6:4c:ca:0a
          inet addr:10.1.0.102  Bcast:10.1.7.255  Mask:255.255.248.0
          inet6 addr: fe80::4637:e6ff:fe4c:ca0a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30356 errors:0 dropped:0 overruns:0 frame:0
          TX packets:329 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2752089 (2.7 MB)  TX bytes:43904 (43.9 KB)
          Interrupt:20 Memory:fe600000-fe620000

eth5      Link encap:Ethernet  HWaddr 00:0a:5e:50:cf:1e
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:20

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:97 errors:0 dropped:0 overruns:0 frame:0
          TX packets:97 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8463 (8.4 KB)  TX bytes:8463 (8.4 KB)

p4p1      Link encap:Ethernet  HWaddr 00:15:17:0b:6d:1c
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:16 Memory:fe540000-fe560000

Edit #2, 20 Apr 2014 08:05 UTC:

Eventually I've edit the file /etc/network/interfaces to have the following configurations:

auto em1
iface em1 inet dhcp

auto eth5
iface eth5 inet static
     address 10.90.90.91
     netmask 255.255.255.0
     up route add -net 10.90.90.0/24 dev eth5

auto p4p1
iface p4p1 inet static
     address 192.168.1.32
     netmask 255.255.255.0
     up route add -net 192.168.1.0/24 dev p4p1

And it worked! See:

dor@ubuntu:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.0.1        0.0.0.0         UG    0      0        0 em1
10.1.0.0        0.0.0.0         255.255.248.0   U     0      0        0 em1
10.90.90.0      0.0.0.0         255.255.255.0   U     0      0        0 eth5
10.90.90.0      0.0.0.0         255.255.255.0   U     0      0        0 eth5
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 p4p1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 p4p1

The only problem is having those duplicates. But it doesn't interfere with anything, it still works.


Edit #3, 20 Apr 2014 09:57 UTC:

Indeed as I suspected, the duplicates were caused from the commands such as:

up route add -net 192.168.1.0/24 dev p4p1

So I removed them. Now everything's perfect.

Best Answer

I think the mistake here, is that you've got multiple default gateways configured in your config file. It doesn't work that way - it's a default, in that any traffic which doesn't have a known route gets sent there for passing on (in the expectation that it'll know, or have it's own default). But it's not tied to a particular interface explicitly - implicitly there will be interfaces that can reach the default gateway, but others that cannot.

So I would imagine what you need to do is remove your 'gateway' lines, and add in the single one you want - ideally in the right interface stanza, but I don't actually think it should matter.