Different gateways for different IPs on same interface on Linux

clustergatewaylinux-networkingpolicy-routingrhel7

I have the following question:

considering the attached image:
Schema

I have a server (RHEL 7) with 3 IP addresses (on the same physical interface):

192.168.10.1 (eth0, server address on network 192.168.10.0/24)
192.168.10.100 (eth0:1, a cluster resource's virtual IP on network 192.168.10.0/24)
192.168.20.100 (eth0:2, a cluster resource's virtual IP on network 192.168.20.0/24)

Since the two cluster resources are on two different networks, I must be sure that, for each client request to one of the resources, the path of the responses remains exactly the same. This means using the two different gateways on the basis on the resources' IP addresses.

Is source policy routing with iproute2 the right way to achieve this?

Thanks in advance,

Best Answer

Right. Basically, it should look like this (from the working system with two ethernet links):

admin@mamba:~$ ip route show table all
default via 192.168.10.254 dev eth1  table admin
192.168.10.0/24 dev eth1  table admin  scope link  src 192.168.10.100
default via 192.168.20.254 dev eth0
192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.100
192.168.10.0/24 dev eth1  proto kernel  scope link  src 192.168.10.100

admin@mamba:~$ ip rule show
0:      from all lookup local
32764:  from all to 192.168.10.100 lookup admin
32765:  from 192.168.10.100 lookup admin
32766:  from all lookup main
32767:  from all lookup default

Related Solutions

Linux – Selecting gateway on application level on Linux

I can think of one way to do it which is as follows.

Define two IP addresses on your NIC like 10.0.0.1 & 10.0.0.2. This should be easy and straight-forward. This can be done using ifconfig.
Configure source based routing such that each source IP is routed to one of the default gateways. This link can help you.
Finally, you can tell your application to use bind one of the addresses 10.0.0.1 or 10.0.0.2. This way you can choose one of the gateways depending on the source IP selected. To test this, you can use telnet with -b source_ip option.

Here are the commands I used previously to enable source-based routing:

$ sudo ip rule add from 10.0.0.2 tab 1 priority 500
$ sudo ip route add default via 10.0.0.101 dev eth0 tab 1
$ sudo ip route flush cache

If you have the default gateway set to 10.0.0.100, then this should work for you. The packets sent from 10.0.0.1 should be sent to default gateway, and packets sent from 10.0.0.2 should be sent to 2nd gateway 10.0.0.101 as instructed above.

Sql-server – How to network this Windows Failover Cluster and MongoDB Replica Set? (diagram inside)

There are two questions here, one for MS clustering, and another one for Mongo.

MS Clustering

The decision of where to put the public, heart-beat, inter-node communication, and quorum drive is significant. Also cluster architecture makes a difference; you pick different quroum options if the two nodes are in adjacent racks than if they were in completely different datacenters.

Put the heartbeat on the same interface/subnet as the public interface

This theory holds that if you lose your public interface, you want the heartbeat to fail because this node is effectively down to users.

Put the heartbeat on it's own private interface/subnet

This theory holds that something outside of the cluster is arbiting who is doing what role, and unnecessary node-death is to be avoided.

Put the WFS on the heartbeat network

If the two nodes are in the same overall network (the same set of switches is supporting the non-public networks for both nodes) then putting the WFS on the heartbeat network doesn't introduce any new vulnerabilities.

If the two nodes are in different network fault domains (such as different datacenters), this is a bad idea. The heartbeat network provides the 'node majority' quorum option, and the WFS provides the 'File Share Majority' quorum option. You really want both options to be in separate fault domains.

Your revised diagram makes sense if both nodes are in the same data-center, though I myself would but the heartbeat on the public side.

MongoDB

MongoDB is a bit simpler. With even numbers of nodes, you absolutely want a third node to act as tie-breaker. They're pretty clear about that. However, your diagram states:

Up to 12 replica members (7 can vote).

7 is an odd number. You don't require an Arbiter.

Unlike Microsoft clusters, Mongo's cluster voting doesn't care about multiple avenues of network to break voting deadlocks. Because of this, separate arbitration and cluster-internal networks do not provide any meaningful increase in robustness. The only reason you'd want a separate arbitration network is if replication traffic was expected to be so heavy that election-packets (the heartbeat, actually) would get pushed so far down the stack that it would miss the 10 second timeout.