I am setting up a cloud environment on Ubuntu and I am running into an interesting problem. Out of the box Ubuntu (all linux really) doesn't allow you to specify more than one gateway in /etc/network/interfaces
Since I am not using DHCP for any of the NICs and I am using vLANs on my switch and router (reference image below) I need each NIC on this server to not just have it's own IP but have it's own gateway specified. Since vLAN 2 which is 10.0.1.0/24 cannot access 10.0.0.1 which is the default gateway but inaccessible due to netmask 255.255.255.0.
All my research has lead me to understand that I need to configure the routing tables by hand, however, when I try to add routing for more than the first eth0 NIC I get the following error:
# ip route add default via 10.0.1.1 dev eth1 table eth1
RTNETLINK answers: File exists
At this point I am lost for things to try… I cannot add the routes to the new route tables, and without explicit routes each NIC card tries to use 10.0.0.1 as the gateway since it's the default gateway for 10.0.0.0/24
Best Answer
So here's the answer after taking Matt's suggestion and going the trial and error route.
The command I used had the following pattern ip route add via dev
The final command actually was
After adding these I had no issues pinging between the IP's on the vLAN.. Fantastic...
Essentially the difference here is that I added a static route so that interaction next hop is sent to the vLAN gateway ID instead of going to the default gateway.
Also so these are permanent I wound up adding them to the /etc/network/interfaces config file as post-up configs. So I wound up with this line for eth1 and eth2, but I left eth0 alone since it can use the default gateway