linux – Round Robin Usage of Multiple IPs for Outgoing Connections on Single Interface

linuxlinux-networkingload balancingnetworking

my issue:
I have a process creating more than 10k TCP-Connection to the same destination IP&Port per second. After a short period of time, no new connections can be created any more, as there are no more source ports available.

I already tried to mitigate this by setting net.ipv4.ip_local_port_range and net.ipv4.tcp_fin_timeout but it did not solve the issue.

Now my thought was: If I could tell the kernel to round robin source IP-Addresses, then I could easily add more IPs to my one interface and therefore have a higher limit of outgoing connections.

I tried adding multiple routes to the same destination via the same interface but specify different source IPs:

ip route add default via 10.1.1.1 dev eth0 src 10.1.1.10
ip route add default via 10.1.1.1 dev eth0 src 10.1.1.11

But i get RTNETLINK answers: File exists.

Am I on the right track here?
Is adding another source IP "they way to go" in those cases?

Is there a way to round robin / load balance the usage of source IPs through the kernel somehow?

My hope was, that in case I manage to place two routes with the same weight but different src-addresses towards the same network, the kernel would round robin that.

(Background: I am running HAproxy on that machine and have to load balance more than 10k connections towards a single backend server)

Best Answer

Not easily via the kernel, no.

I am running HAproxy on that machine and have to load balance more than 10k connections towards a single backend server

You can tell haproxy to use a specific source IP address for outgoing connections to a server, for example:

  server app1_s1 10.0.1.1:80 source 10.1.1.10
  server app1_s2 10.0.1.1:80 source 10.1.1.11
  server app2_s1 10.0.1.2:80 source 10.1.1.10
  server app2_s2 10.0.1.2:80 source 10.1.1.11

Related Solutions

Linux – iptables NAT/Forwarding with external ADSL router; PCs on the network can’t access the internet

I would replace

iptables -A FORWARD -i eth1 -p tcp ! --syn -j ACCEPT

with

iptables -A FORWARD -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT

Also I don't see any rules for your gateways traffic, except icmp (INPUT and OUTPUT).

Linux: force source address for packets to locally bound IPs

The local table (table 255) is consulted before main, and contains all local routes (hence the name). It is maintained by the kernel (hence proto kernel). On your machine, it likely looks something like this.

local 10.1.1.1 dev eth0 proto kernel scope host src 10.1.1.1
broadcast 10.2.2.0 dev eth0 proto kernel scope link src 10.2.2.2
local 10.2.2.2 dev eth0 proto kernel scope host src 10.2.2.2
broadcast 10.2.2.255 dev eth0 proto kernel scope link src 10.2.2.2
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope host src 127.0.0.1

The source addresses are specified therein.

Linux supports more than one routing table, and routing involves consulting the use of the routing policy database (RPDB) to decide when and which table to consult. If a table doesn't contain the answer, or a route is of type throw, the next RPDB rule is consulted.

$ ip ru
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

You could try removing the rule for the local table lookup, and place it later (ip rule del pref 0; ip rule add from all lookup local pref 1), which isn't recommended. You'd then prepend a rule to consult a custom routing table (pick any number of your fancy above 255), in which there would be a sole route for 10.1.1.1 with the source address set to 10.2.2.2. The better way to do this is to just call bind() if you are the author of the programs in question that'll be talking to 10.1.1.1.

Related Topic