How to add an static route on google compute engine

google-compute-enginelinux-networkingrouting

I am building a cluster in the Google Compute Engine and currently I need to add a static route to another machine in the same network, but something is going wrong and I get an error "RTNETLINK answers: Network is unreachable".

Important note: I am not a network expert and I am probably doing some basic mistakes in the process.

I have a machine A and a machine B, where A has a subnet 11.10.0.0/16 with other machines B cannot reach, so A will be B's gateway to them. Both have the flag IP forwarding active and are in the same network (using eth0 on both) and can reach the other directly.

The command and error (executed on B) are:

B:~$ sudo ip route add 11.10.0.0/16 via 10.240.0.8 dev eth0
RTNETLINK answers: Network is unreachable

Host A

A:~$ ip route list
default via 10.240.0.1 dev eth0 
10.240.0.1 dev eth0  scope link 
11.10.0.0/16 via 11.10.0.2 dev tun0 
11.10.0.2 dev tun0  proto kernel  scope link  src 11.10.0.1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1

A:~$ ip addr
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 42:01:0a:f0:00:08 brd ff:ff:ff:ff:ff:ff
    inet 10.240.0.8/32 brd 10.240.0.8 scope global eth0
       valid_lft forever preferred_lft forever
3: docker0:  mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:7d:6d:9b:0b brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
7: tun0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 100
    link/none 
    inet 11.10.0.1 peer 11.10.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever

A:~$ sudo iptables --list -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination

A:~$ sudo iptables --list -t filter
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination

Host B

B:~$ ip route
default via 10.240.0.1 dev eth0 
10.240.0.1 dev eth0  scope link 
11.11.0.0/16 via 11.11.0.2 dev tun0 
11.11.0.2 dev tun0  proto kernel  scope link  src 11.11.0.1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1

B:~$ ip addr
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 42:01:0a:f0:00:07 brd ff:ff:ff:ff:ff:ff
    inet 10.240.0.7/32 brd 10.240.0.7 scope global eth0
       valid_lft forever preferred_lft forever
3: docker0:  mtu 1460 qdisc noqueue state DOWN group default 
    link/ether 02:42:b0:25:d5:57 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
17: tun0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 100
    link/none 
    inet 11.11.0.1 peer 11.11.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever

B:~$ sudo iptables --list -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination

B:~$ sudo iptables --list -t filter
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination

I hope I've given enough information about the issue.

Best Answer

The virtual network you get on a GCE instance is effectively a /32 network to which only the instance itself and the gateway are attached (the latter being configured using a host route). This means that all outgoing traffic is sent to the gateway.

That is the reason why you get the following error:

B:~$ sudo ip route add 11.10.0.0/16 via 10.240.0.8 dev eth0
RTNETLINK answers: Network is unreachable

The error simply tells you that there is no host or network route that matches 10.240.0.8 (other than the default route which uses a gateway itself).

There is no way to set up your desired configuration using routing configuration on the hosts. Instead you need to configure routing in GCE itself, as described here. Conceptually you can think of this as configuring the routing table on the gateway. You don't need any additional configuration on the hosts because as explained above, they send all outgoing packets to the gateway.

Related Solutions

Ubuntu – KVM Ubuntu Guest cannot connect to the internet on bridged networking

I think you are missing a iptable rule for the masquerade

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

Linux – How to configure dual homed server in order for both network segments to communicate

There are two problems with this setup:

The hosts on LAN1 know nothing about the LAN2 segment. When you ping a host on LAN1 (let's call it host1) from SRV-02, the packet will be routed through SRV-01 and will reach host1. However, host1 will send the reply to it's default gateway (ISP router) as it doesn't have a specific route to LAN2. (The ISP router will either a) also send it to it's default gateway as it also doesn't know about LAN2, or b) drop the packet as it comes from an unknown source not it's local LAN.)
When trying to reach WAN from LAN2, the packets will be routed through SRV-02 to ISP router where two situations are possible:
- The router will not NAT translate the packet as the source of the packet (LAN2) is not it's local LAN (this is the more probable situation), or
- The router will NAT translate the packet and send it to the Internet. However, when the reply comes and the destination is translated back to the LAN2 address, the packet will not be delivered as the ISP router doesn't have a route for that network. The packet will be sent incorrectly to the default gateway (ISP).

These issues could be fixed by adding a static route to LAN2 to ISP router and adding a source NAT configuration for LAN2 on SRV-01. However, that is not possible due to no admin access to the ISP router.

There are two solutions that get around it:

A. Make SRV-01 a full router for LAN1 and LAN2 hosts

Add another network adapter to SRV-01 (making it 3 in total)
Change the topology as follows:

WAN -> ISP router -> LAN1 -> SRV-01 +-> LAN3 (for hosts originally in LAN1)
                                    +-> LAN2 -> SRV-02

Basically, we're making SRV-01 a router for both LAN segments.

This will require moving hosts originally in LAN1 to a new subnet LAN3 - let's say we use 10.0.1.0/24
The network configuration of SRV-01 will need to be changed as follows:

/etc/network/interfaces:

# LAN1 - to ISP router
auto eth0
iface eth0 inet dhcp
# we can even use dhcp as the IP address is not really important
# - there are no more hosts on LAN1 apart from ISP router and SRV-01

# LAN3 - for hosts originally in LAN1
iface eth1
    address 10.0.1.1
    netmask 255.255.255.0

# LAN2
iface eth2
    address 10.0.2.1
    netmask 255.255.255.0

iptables rules to make WAN access work:

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.1.0/24 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 -j MASQUERADE

Alternatively, if you choose to keep the static IP address on SRV-01 on eth0 the rules could be changed (although MASQUERADE would still work):

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.1.0/24 -j SNAT --to-source 192.168.5.8
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 -j SNAT --to-source 192.168.5.8

DHCP will need to be configured on SRV-01 on eth1 (LAN3, for hosts originally on LAN1), and possibly on eth2 (LAN2) as well if required. (In both cases the gateway will be the local address of eth1 or eth2 respectively, but that goes without saying :)

This will make communication possible between LAN3 and LAN2 (via SRV-01 which is the default gateway for both). WAN access will also work from both LAN3 and LAN2 thanks to the double source NAT.

B. Make SRV-01 a DHCP server for LAN1

This approach is not as clean as above but is slightly simpler. It assumes you are able to disable DHCP on ISP router

Disable DHCP on ISP router
Set up DHCP for LAN1 on SRV-01 and make SRV-01 (192.168.5.8) the default gateway for LAN1
Set up source NAT translation for LAN2 on SRV-01 so that the WAN access works from LAN2:

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 -d 192.168.5.4 -j SNAT --to-source 192.168.5.8
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 ! -d 192.168.5.0/24 -j SNAT --to-source 192.168.5.8

The first line enables SNAT so that LAN2 hosts can access the ISP router itself and the second line disables SNAT for LAN2-LAN1 access.

Again, this approach is not as clean as the one above as there are two routers in the same subnet (SRV-01, ISP router). When I used this approach myself I noticed my second router (SRV-01 in this scenario) would send ICMP redirects to the ISP router as it would see that the client (host on LAN1) and the upstream router (ISP router) are on the same LAN. This might not be desired as network policies implemented on SRV-01 could be circumvented.

Hope that helps.