Nginx Rate Limiting – How to Log Rate-Limited IPs to a Specific File

nginxrate-limitingsynchronization

I'm looking for an approach to synchronize rate-limited IPs between nginx nodes.
I want to log these IPs and after that pushing them into a database and developing an agent to update blocked IPs in nginx config files.

My challenge is to find a way to have IPs nginx limited with 429 status code.

So, Is it possible to log rate-limited IPs into an specific file in nginx or do you suggest any other approach to synchronize rate-limited IPs between nodes?

Best Answer

Yes, you can do that, and a similar example is even in the nginx documentation.

The access_log directive also takes an optional if= parameter which evaluates variables given to it, and logs only if the result is not 0 or an empty string. Combined with the fact that you can have more than one access_log in a level, you can log differently based on your needs.

First, though, you will need a map to map the HTTP response status you are interested in to a variable. Remember that map must be outside the server block.

map $status $rate_limited {
    default 0;
    429     1;
}

Then in the relevant server block you will declare your access_log.

access_log /var/log/nginx/rate_limited.log combined if=$rate_limited;

Remember that any appearance of access_log in one level overrides all others from higher levels, so you will want to copy (or better, include) the access_log directives from higher levels that you also want to use.

Related Solutions

Nginx – How to rate-limit in nginx, but including/excluding certain IP addresses

It is really better to avoid using the "if" directive. When the key in limit_req_zone (and limit_conn_zone) is empty the limits are not applied. You can use this in conjunction with the map and geo modules to create a whitelist of IPs where the throttle limits are not applied.

This example shows how to configure a limit for both concurrent requests and request rate from a single IP.

http {
    geo $whitelist {
       default 0;
       # CIDR in the list below are not limited
       1.2.3.0/24 1;
       9.10.11.12/32 1;
       127.0.0.1/32 1;
    }

    map $whitelist $limit {
        0     $binary_remote_addr;
        1     "";
    }

    # The directives below limit concurrent connections from a 
    # non-whitelisted IP address to five

    limit_conn_zone      $limit    zone=connlimit:10m;

    limit_conn           connlimit 5;
    limit_conn_log_level warn;   # logging level when threshold exceeded
    limit_conn_status    503;    # the error code to return

    # The code below limits the number requests from a non-whitelisted IP
    # to one every two seconds with up to 3 requests per IP delayed 
    # until the average time between responses reaches the threshold. 
    # Further requests over and above this limit will result 
    # in an immediate 503 error.

    limit_req_zone       $limit   zone=one:10m  rate=30r/m;

    limit_req            zone=one burst=3;
    limit_req_log_level  warn;
    limit_req_status     503;

The zone directives must be placed at the http level, however the other directives can be placed further down, e.g. at the server or the location level to limit their scope or further tailor the limits.

For futher information refer to the Nginx documentation ngx_http_limit_req_module and ngx_http_limit_conn_module

nginx – Rate Limiting with X-Forwarded-For Header

Yes, typical rate-limiting configuration definition string looks like:

 limit_req_zone  $binary_remote_addr zone=zone:16m rate=1r/s;

where $binary_remote_addr is the unique key for limiter. You should try changing it to $http_x_forwarded_for variable which gets the value of X-Forwarded-For header. Although this will increase memory consumption because $binary_remote_addr is using compressed binary format for storing IP addresses and $http_x_forwarded_for is not.

 limit_req_zone  $http_x_forwarded_for zone=zone:16m rate=1r/s;

Best Answer

Related Solutions

Nginx – How to rate-limit in nginx, but including/excluding certain IP addresses

nginx – Rate Limiting with X-Forwarded-For Header

Related Topic