Nginx: How to limit request rate based on user agent

nginxrate-limiting

Trying to get the following behavior working in nginx

A default rate limit of 1r/s for each ip when using a browser.
A rate limit of 10r/s for bing and google spiders.
Reject bad bots.

Unfortunately google doesn't publish ip addresses for googlebot so I'm limited to useragent.

So far this gets close:

http { 
  # Rate limits
  map $http_user_agent $uatype {
    default 'user';
    ~*(google|bing|msnbot) 'okbot';
    ~*(slurp|nastybot) 'badbot';
  }

  limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
  limit_req_zone $binary_remote_addr zone=two:10m rate=10r/s;

  ...

  server {
    ...

    location / {
      if ($uatype == 'badbot) {
        return 403;
      }

      limit_req zone=one burst=5 nodelay;
      if ($uatype != 'user') {
        limit_req zone=two burst=10 nodelay;
      }

      ...
    }

  ...
  }
}

BUT – 'if' isn't allowed to do this.

$ nginx -t

nginx: [emerg] "limit_req" directive is not allowed here in /etc/nginx/nginx.conf
nginx: configuration file /etc/nginx/nginx.conf test failed

There are so many untested suggestions on nginx forums, most do not even pass configtest.

One that looks promising is Nginx Rate Limiting by Referrer? — Downside of that version is that all of the configuration is repeated for each different limit (I have many rewrite rules)

Anyone got something good?

Best Answer

Unfortunately you can't dynamize this way, limit request module doesn't support this.

The link you found is probably the only way to achieve this. Use include directive to "avoid" repeating your configuration.

But what if a thirdparty crawler suddenly impersonate a goodbot user agent ?

Related Solutions

NGINX Redirect – How to Rewrite All HTTP Requests to HTTPS While Maintaining Sub-Domain

Correct way in new versions of nginx

Turn out my first answer to this question was correct at certain time, but it turned into another pitfall - to stay up to date please check Taxing rewrite pitfalls

I have been corrected by many SE users, so the credit goes to them, but more importantly, here is the correct code:

server {
       listen         80;
       server_name    my.domain.com;
       return         301 https://$server_name$request_uri;
}

server {
       listen         443 ssl;
       server_name    my.domain.com;
       # add Strict-Transport-Security to prevent man in the middle attacks
       add_header Strict-Transport-Security "max-age=31536000" always; 

       [....]
}

Nginx – How to rate-limit in nginx, but including/excluding certain IP addresses

It is really better to avoid using the "if" directive. When the key in limit_req_zone (and limit_conn_zone) is empty the limits are not applied. You can use this in conjunction with the map and geo modules to create a whitelist of IPs where the throttle limits are not applied.

This example shows how to configure a limit for both concurrent requests and request rate from a single IP.

http {
    geo $whitelist {
       default 0;
       # CIDR in the list below are not limited
       1.2.3.0/24 1;
       9.10.11.12/32 1;
       127.0.0.1/32 1;
    }

    map $whitelist $limit {
        0     $binary_remote_addr;
        1     "";
    }

    # The directives below limit concurrent connections from a 
    # non-whitelisted IP address to five

    limit_conn_zone      $limit    zone=connlimit:10m;

    limit_conn           connlimit 5;
    limit_conn_log_level warn;   # logging level when threshold exceeded
    limit_conn_status    503;    # the error code to return

    # The code below limits the number requests from a non-whitelisted IP
    # to one every two seconds with up to 3 requests per IP delayed 
    # until the average time between responses reaches the threshold. 
    # Further requests over and above this limit will result 
    # in an immediate 503 error.

    limit_req_zone       $limit   zone=one:10m  rate=30r/m;

    limit_req            zone=one burst=3;
    limit_req_log_level  warn;
    limit_req_status     503;

The zone directives must be placed at the http level, however the other directives can be placed further down, e.g. at the server or the location level to limit their scope or further tailor the limits.

For futher information refer to the Nginx documentation ngx_http_limit_req_module and ngx_http_limit_conn_module

Related Topic