Ratelimit POST requests

apache-2.2cacheperformance-tuningvarnish

I'm running a large WordPress multiuser site. I have a varnish cache in front of the WordPress application server. As it makes no sense to cache POST requests, I am vulnerable to DDoS using lots of POSTs against the varnish cache server.

I have tried setting op a firewall rules that only accepted 20 simultaneously connections from each client, but that had impact on the users sitting behind shared proxies and on schools with lots of users behind the same gateway.

Varnish does not have the option to rate-limit the numbers of POSTs, and I want to do the rate-limit BEFORE it hits the application server. Is there a small transparent proxy that would do the job?

Currently Varnish is receiving about 100-150 hits/second, the proxy should at least be able to handle this load.

Best Answer

I am going to put this down as an answer only because it's going to be long. As you quite accurately said blocking more than 20 concurrent requests from 1 IP won't do the trick. You have to set "smarter" criteria. I will go as far as to say that putting another proxy between a proxy and an application server is not elegant nor useful.

I don't know why you want to do the rate-limit before hitting apache because you are missing out on fail2ban, mod_qos, mod-antiloris (highly specific) and other solutions. Moreover I don't know if POST requests are your only problem in terms of a DDoS.

Caching POST request replies is possible and makes sense as well. Unless you are serving dynamic content every single time. That of course doesn't mean that you can cache authenticated pages.

You can apply request time-outs for POST if the request has taken more that 5s. Although users with slow connections won't be able to POST anything. You can also apply a rule on a per URL basis combined with the above rule. This makes sense, since it is reasonable for a user to POST 1000kb on a file uploading page but not on the login page. As I said create "smarter" criteria. They might be long and it might take some time to formulate them but they will provide you with a sustainable solution as I don't know of a one-size fits all in this kind of situation.

Another solution which you can combine is an application firewall. Might be more than you need but it can as well keep you safe from many other things. Here is a owasp page with recommendations and the relevant wiki

EDIT: I have to admit, I don't have any experience with that configuration (varnish and antiloris). Sooner or later there's so much that varnish can cache (although it is highly "programmable"). The main thing you can do is know the pattern of use and when it deviates from the norm. If you just want to prevent that very specific type of attack then writing better rules for Varnish should do it. However requests hitting apache is not a bad thing, as long as you have the proper mods/confs in place, Apache can tell when a request is legit and thus process it and when not to. Blocking X number of connections from each client is NOT a good thing to do, unless you can definitely blacklist that IP. You can do that on apache either through fail2ban(regex) or mod_qos

Related Solutions

Varnish POST problem “9 FetchError c backend write error: 11” for application/x-www-form-urlencoded content

This bug report seems to suggest a workaround to your problem; you should give it a try.

https://www.varnish-cache.org/trac/ticket/849

Nginx cache reverse proxy: how to keep app server alive during the 5 second window when cache expires

If you can post your nginx config, I may be able to help you better way!

Generally, I wil use Nginx's fastcgi_cache / proxy_cache with fastcgi_cache_use_stale/proxy_cache_use_stale

I said both option because if u can run backend app using Nginx's fastcgi or other module then its better to do that way.

If Apache on 8080 cannot be removed, better use proxy_cache with proxy_cache_use_stale updating line.

Please provide your config so we can try to improve it.

Added a sample config based on yours (very raw, most likely will need tweaking)

#IMPORTANT outside server{..} block
proxy_cache_path /var/run/nginx-cache levels=1:2 keys_zone=GROVE:500m inactive=60m;
proxy_cache_key "$scheme$request_method$host$request_uri";
fastcgi_cache_use_stale updating;

server {

        #other stuff

        set $no_cache 0;

        # POST requests and urls with a query string should always go to PHP
        if ($request_method = POST) {
                set $no_cache 1;
        }   

    # grove urls
    # ----------
    location / {
        default_type text/html;
        client_max_body_size 50m;

        proxy_pass http://127.0.0.1:8080;
        proxy_redirect off;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;


        client_body_buffer_size    128k;

        proxy_connect_timeout      60; # time to connect to upstream server
        proxy_send_timeout         300; # time to wait for upstream to accept data
        proxy_read_timeout         300; # time to wait for upstream to return data

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;


        proxy_cache_bypass $no_cache;
            proxy_no_cache $no_cache;

            proxy_cache GROVE;
            proxy_cache_valid  60m;
}

Best Answer

Related Solutions

Varnish POST problem “9 FetchError c backend write error: 11” for application/x-www-form-urlencoded content

Nginx cache reverse proxy: how to keep app server alive during the 5 second window when cache expires

Related Topic