Linux – IPTables + Limit module: Why doesn’t limit-burst get completely used

iptableslinuxlinux-networkingrate-limiting

Long time reader, first time poster.. yada yada yada..

Anyways, I am hopeful someone out there has some extensive iptables/netfilter LIMIT or HASHLIMIT module experience and explain the behavior I'm witnessing.

Background:
We have a webserver and want to limit how many connections a customer can have over the course of a month (HTTP keepalives are off, btw). So, I am trying to use the iptables LIMIT module to limit the number of their new connections to a set number per month (let's say 500). iptables LIMIT module uses a "token bucket" algorithm so I should be able to set the limit-burst (bucket size) to 500 and the limit (refill rate) to 500 divided by 28 days or about 18/day. This will make sure the bucket gets refilled in a month's time (4 weeks) if it is every completely emptied. (I understand this will actually grant more than exactly 500 but it should be close enough for our needs).

Here are my iptables rules (We group IPs using ipset. LimBurTest4 contains my source testing machines)

Chain INPUT (policy DROP 2316 packets, 186K bytes)
 pkts bytes target     prot opt in     out     source               destination
2952K  626M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0      state RELATED,ESTABLISHED /* Accept outgoing return traffic */
  379 13702 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0      icmptype 8 state NEW limit: avg 1/sec burst 1
  377 30868 DROP       icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0      tcp flags:0x3F/0x00 /* Block NULL packets */
   73 14728 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0      tcp flags:!0x17/0x02 state NEW /* Block SYN flood */
    0     0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0      tcp flags:0x3F/0x3F /* Block XMAS packets */
   24   120 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0      match-set LimBurTest4 src tcp dpt:443 limit: avg 18/day burst 500 /* LimitBurst Test */
   76 30180 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0      match-set LimBurTest4 src tcp dpt:443 /* LimitBurst Testing */ LOG flags 0 level 4 prefix "LimBurTest Over Quota "
 2522  138K REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 /* Reject queries */ reject-with tcp-reset

I add the LIMIT rule like this:

iptables -I INPUT 7 -m set --match-set LimBurTest4 src -p tcp --dport 443 -m limit --limit 18/day --limit-burst 500 -m comment --comment "Limit Burst Test" -j ACCEPT

Testing:
Then I created a simple shell script to do requests to my webserver one after the other using curl. Each successful request takes about 0.20 ms. Output of the script looks like this:

./limBurTest.sh
1  - 200   -  0.257 ms 
2  - 200   -  0.193 ms 
3  - 200   -  0.155 ms 
etc.
etc.

Outcome:
My expectation of this configuration would be to quickly (in a couple seconds) use up all 500 connections before I start seeing rejected connections. However, that is not happening at all. Instead, my test script made 24 successful connections and then the rest were rejected. For example, in the above iptables output, I ran my shell script in a loop 100 times and you can see 24 ACCEPT rule matches and 76 LOG rule matches after the ACCEPT rule. I have tested this on CentOS 6.8 and Ubuntu 16.04 and both behave this way, but this seems contrary to the documentation. Why can I not use up all 500 connections specified by limit-burst?

And, yes, of course, I have done extensive googling and have seen lots of examples of people saying that the LIMIT module is supposed to work exactly as I've described. And I've read the netfilter docs numerous times.

What am I missing here?

Thank you in advance.

Best Answer

The limit module does not limit based on number of connections but rather on the number of ip packets (even jumbo-packets). So if a single connection takes roughly 20.8 packets then you would see a burst of 500/20.8~=24 connections.

see the output of the command man 8 iptables-extensions|grep -e '^ *limit$' -A 18 for more info, specifically the phrase "Maximum initial number of packets to match: [...]".

Related Topic