Nginx – Load Balancer –> Varnish –> Nginx

nginxvarnish

Been scratching my head for a few hours now and wanted to see if anyone can help.

1) I have a load balancer with 6 servers in the back end.

2) The back end servers are Nginx and to get the real IP addresses of the visitors, all I have to do is the following in each Nginx install and I am able to get the real client IP address of each visitor.

 set_real_ip_from 192.168.255.0/24; <-- to handle the load balancer IP
 real_ip_header X-Forwarded-For;

3) Now I have installed Varnish in front of each Nginx running on 127.0.0.1 doing caching and for some reason now Nginx doesn't see the real client Ip addresses anymore coming from LoadBalancer –> Varnish –> Nginx

It's printing the following:

IP address:
192.168.255.9 <– this should be the real client IP address and not the 192.168 (assuming the load balancer IP address is being printed)

More detailed host address:
192.168.255.9

Many thank if you can help.

Dave

UPDATE:

Without Varnish in the equation, I have the following LB –> NGINX and within NGINX the
following existis

set_real_ip_from  192.168.255.0/24;
real_ip_header X-Forwarded-For;

When NGINX logs the remote_addr, the first entry below prints the real client IP address

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
              '$status $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for"';


213.205.234.x - - [05/Sep/2012:09:42:08 -0700] "GET /2011/10/28/chicken-and-apples-in- honey-mustard-sauce/ HTTP/1.1" 200 18283 "-" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; GT-I9100 Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" "213.205.234.x"

With Varnish in the equation LB –> Varnish –> NGINX

And within NGINX I switched the set_real_ip_from to point to 127.0.0.1

set_real_ip_from  127.0.0.1;
real_ip_header X-Forwarded-For;

$remote_addr in NGINX doesn't print the real client IP address:

192.168.255.9 - - [05/Sep/2012:09:46:41 -0700] "GET /2012/09/03/stuffed-baked-potatoes-deconstructed/ HTTP/1.1" 200 18159 "-" "Mozilla/5.0 (Linux; U; Android 2.2.1; en-us; ADR6400L Build/FRG83D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1" "69.255.125.x, 192.168.255.9"

As you can see from above the $remote_addr being printed is the Load Balancer's IP address: 192.168.255.9 instead of the client's remote_addr. Although the "$http_x_forwarded_for"' is printing the correct address I guess: "69.255.125.x, 192.168.255.9". My goal is to have the $remote_addr hold the correct IP address instead

Thanks
Dave

UPDATE:

Here's my default.vcl from Varnish, commented out the part mentioned by Shane, here's the current output from the access log from NGINX

127.0.0.1 - - [05/Sep/2012:11:16:43 -0700] "GET /wp-content/plugins/wp-pagenavi/pagenavi-css.css?ver=2.70 HTTP/1.1" 304 0 "http://mobilefoodblog.com/2011/10/28/chicken-and-apples-in-honey-mustard-sauce/" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; SCH-I405 Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1" "67.168.209.192, 192.168.255.9"


    # This is a basic VCL configuration file for varnish.  See the vcl(7)
    # man page for details on VCL syntax and semantics.
    # 
    # Default backend definition.  Set this to point to your content
    # server.
    # 

    backend default {
         .host = "localhost";
         .port = "8080";
    }

    sub detect_device {
      # Define the desktop device
      set req.http.X-Device = "desktop";

      if (req.http.User-Agent ~ "iP(hone|od)" || req.http.User-Agent ~ "Android" || req.http.User-Agent ~ "iPad") {
        # Define smartphones and tablets
        set req.http.X-Device = "smart";
      }

      elseif (req.http.User-Agent ~ "SymbianOS" || req.http.User-Agent ~ "^BlackBerry" || req.http.User-Agent ~ "^SonyEricsson" || req.http.User-Agent ~ "^Nokia" || req.http.User-Agent ~ "^SAMSUNG" || req.http.User-Agent ~ "^LG") {
        # Define every other mobile device
        set req.http.X-Device = "other";
      }
    }

    acl purge {
            "localhost";
    }

    sub vcl_recv {
       call detect_device;

         # if (req.restarts == 0) {
         #  if (req.http.x-forwarded-for) {
         #      set req.http.X-Forwarded-For =
         #          req.http.X-Forwarded-For ", " client.ip;
         #  } else {
         #      set req.http.X-Forwarded-For = client.ip;
         #  }
         # }

       if (req.request == "PURGE") {
            if (!client.ip ~ purge) {
                 error 405 "Not allowed.";
            }
            return(lookup);
       }

    if (req.url ~ "^/$") {
          unset req.http.cookie;
        }
    }

    sub vcl_hit {
            if (req.request == "PURGE") {
                    set obj.ttl = 0s;
                    error 200 "Purged.";
            }
    }

    sub vcl_miss {
        if (req.request == "PURGE") {
                    error 404 "Not in cache.";
        }

        if (!(req.url ~ "wp-(login|admin)")) {
                            unset req.http.cookie;
        }

        if (req.url ~ "^/[^?]+.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.|)$") {
               unset req.http.cookie;
               set req.url = regsub(req.url, "\?.$", "");
        }

        if (req.url ~ "^/$") {
               unset req.http.cookie;
        }
    }

    sub vcl_fetch {
            if (req.url ~ "^/$") {
               unset beresp.http.set-cookie;
            }

        if (!(req.url ~ "wp-(login|admin)")) {
               unset beresp.http.set-cookie;
        }
    }

    sub vcl_hash {
         set req.hash += req.url;
         if (req.http.host) {
             set req.hash += req.http.host;
         } else {
             set req.hash += server.ip;
         }

         # And then add the device to the hash (if its a mobile device)
         if (req.http.X-Device ~ "smart" || req.http.X-Device ~ "other") {
           set req.hash += req.http.X-Device; 
         }

         return (hash);
    }

Best Answer

Since Varnish is running on each server that's running nginx, the source of the connection from the perspective of nginx is 127.0.0.1, not the load balancer anymore.

set_real_ip_from 192.168.255.0/24;

That's the problem; nginx won't 'trust' the X-Forwarded-For header when the connection originates from 127.0.0.1 (the Varnish process); all it trusts is the entire 192.168.255.0/24 network. Add an authorization to trust the header when Varnish sends it.

set_real_ip_from 127.0.0.1;

Edit:

nginx behaves badly when parsing the X-Forwarded-For header for the "real" client IP; it looks for the last entry in the header, which is never the real client IP when there's more than one entry. See this question for more info on this problem.

I'd recommend getting Varnish to stop adding its own X-Forwarded-For header. You'll want to strip this part of the vcl_recv function:

if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For =
            req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
    }
}

Provide your current vcl config if you need assistance what needs to change for you, as this may be explicitly configured or being appended by the defaults (or both).

Edit 2:

Swap this in for the vcl_recv function in your Varnish config; it combines the customizations you've configured with the default behavior while removing the X-Forwarded-For trickery that's present by default.

sub vcl_recv {
    call detect_device;
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        return(lookup);
    }
    if (req.url ~ "^/$") {
        unset req.http.cookie;
    }
    # Default logic follows; it's normally appended.
    # It'll still be appended, but having the return(lookup)
    # prevents its use. X-Forward-For header behavior removed.
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    }
    return (lookup);
}

Related Solutions

Nginx – Configure php5-fpm for many concurrent users

I suspect your varnish cache is not caching anywhere near enough of the hits

here's what I would do in your situation:

Lower php max children to 100 or even 50 (if varnish does its job properly you don't need them) also remove the max requests line to allow the php processes not to respawn too quickly and thus prevent APC from being cleared too quickly which is also bad

also IF is not good according to nginx - http://wiki.nginx.org/IfIsEvil

I would change this line:

            if (!-e $request_filename) {
                    rewrite ^(.+)$ /index.php?q=$1 last;
            }

to:

try_files $uri $uri/ /index.php?$args;

If your version of nginx supports it (pretty certain if your nginx version is > 0.7.51 then it supports it)

you should also look at inserting the w3tc nginx rules direct into your vhost file to enabled proper disk enhanced caching of pages (which is faster than APC caching with nginx)

Take a look at the following varnish vcl which I use for sites - you will need to read through and edit a few things for your website - it also assumes that its only WP sites on the server and only 1 site on the server, it can easily be modified for more sites (take a look at the cookie section)

generic vcl: https://gist.github.com/b7332971a848bcb7ecef

With this config I would argue to remove fastcgi_cache to prevent any possible issues with a cache-chain occurring whereby trying locate any stray stale cache entries is more difficult

also tell w3tc that varnish is at 127.0.0.1 and it will purge it for you ;)

I deployed this to a server on Wednesday evening (with a few domain specific modifications) that was handling 2500 active site visitors it reduced load to less than 1 and the approx number of running php children was around 10-20 (this number does depend on number of logged in users and other factors like cookies) this server did have much more ram but the principle is the same, you should be able to easily handle the number of visitors you get at peaks

Nginx+apache2.4+php-fpm – SERVER[“REMOTE_ADDR”] and php_admin_value

This can be set with:

fastcgi_param PHP_VALUE 'sendmail_path "/usr/sbin/sendmail -t -i -f owner@example.com"';

If you have multiple values to pass, they must all be in the same PHP_VALUE, separated by \n.

fastcgi_param PHP_VALUE 'allow_url_fopen 0\nmemory_limit 64M'

Best Answer

Related Solutions

Nginx – Configure php5-fpm for many concurrent users

Nginx+apache2.4+php-fpm – SERVER[“REMOTE_ADDR”] and php_admin_value

Related Topic