Varnish X-Forward-IP

Ok, I will answer my own question to help people who may have the same problem:

First add the following lines into varnish configuration file (default.vcl)

sub vcl_recv {
   if (req.http.host == "myDomain.net") {
       set req.http.host = "myDomain.net";
       set req.backend = myBackend;
       # Compatiblity with Apache log
       remove req.http.X-Forwarded-For;
       set req.http.X-Forwarded-For = client.ip;
       # No cache for POST requests
       if (req.request == "POST") {
           return(pipe);
       }   
       return(lookup);
   }
}

Then add personalized logs format for apache while configuring your vhost

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined
 ...
 CustomLog      ${APACHE_LOG_DIR}/access.log varnishcombined

That's it!

This is absolutely possible, but it takes a few steps to get it working nicely! The way we do it here when we need that kind of configuration is:

• Create a VPC. You need to do this in VPC because you need to make subnets in them.

• Create one subnet in each Availability Zone that you will have instances that are registered with the ELB. You should subnet so that you have a small number of IP addresses in each subnet, as every IP address will become overhead. (We currently use subnets that are a /26)

• Begin creating a DNS Director backend in your varnish VCL. Add in the 3 subnets you created above. (https://www.varnish-cache.org/docs/3.0/reference/vcl.html#the-dns-director)

• Set the host setting in the DNS Director backend to be the host name that varnish should expect to see. (for instance, if your front end service is called, say, front-end-service.subdomain.example.com, put front-end-service.example.com as the Host setting in the VCL.)

• Set the suffix setting in the DNS Director to something that you can resolve. Continuing the example above, you could easily use '-varnish.example.com' for your suffix. When a request hits varnish, varnish will look at the HTTP Host header, and if the name matches what varnish has in the VCL's DNS Director Backend for the Host header setting, varnish will append the suffix and perform a DNS lookup on the hostname that is the result of concatenating the contents of the Host header with the suffix. Thus, in this example, a DNS lookup will be performed by varnish for the host named "front-end-service.subdomain.example.com-varnish.example.com"

• Create your loadbalancer backend and make attach it to each subnet that you have created.

• Set a DNS record for the result of concatenation to be a CNAME for the DNS name that amazon provides your for your loadbalancer.

• Start varnish, optionally look at varnishstat to verify the number of backends.

• Test your setup by issuing a

  curl -H "Host: front-end-service.subdomain.example.com" http://varnish-server-hostname.example.com/whatever-path

• Watch the request come in with varnishlog to verify that everything is working.

It may be useful to note that AWS recommends that you should have a subnet with at least 20 unused IP addresses in it if you are going to place a loadbalancer in that subnet. (See http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/UserScenariosForVPC.html)

We have done this for a recent project that needed ELBs for a requirement spec, but we are concerned about how this scales with respect to ease of management and are looking into service discovery based approaches along with something like an automated VCL update, plus automated VCL deploy via something like Varnish Agent (https://github.com/varnish/vagent2)

However, if you don't mind managing your VPC subnets, the above description works very well.

Cheers!