Varnish multi proxy via DNS config and regular expression

You don't need to enable IP Forwarding on the varnish server for it to work. Varnish will not forward the client connection but, as a proxy, create a new connection on the user's behalf.

If you can

telnet apache2 80

or

curl -x apache2:80 http://yoursite.com/yourpage

from the varnish server then your network setup is ok. For the latter example please add the code below to your vcl_recv config:

# Normalise requests sent via curl's -X mode and LWP.  Must do before
# backend selection.
if (req.url ~ "^http://") {
  set req.url = regsub(req.url, "http://[^/]*", "");
}

It would help if you had posted your varnish configuration but the original one should work out of the box (even if it's not caching a great deal of pages).

I assume you created a director for your 3 apache backends and that this director is the default backend for all incoming connections to varnish.

If so, run

varnishlog | grep _health

and make sure your backends are not sick. If so, adjust your backend probe (health check).

As a rule of thumb, varnish does not care what virtualhost configuration is being used in the backends. I suggest you go back to the original configuration and define a single backend to start with. Then move on to a director. Only then further customise your varnish config.

good luck

Do you anticipate using Edge Side Includes (ESI)? If so, the Nginx ESI module is broken and has some open bugs. If you use Varnish, output isn't compressed, so you're somewhat stuck using Nginx to do compression of ESI enabled pages. While I work with Python frameworks, Rails is similar.

With your current setup, you could do something like:

Nginx -> Apache -> Passenger -> Rails Varnish -> Apache -> Passenger -> Rails

Both would drop in front of your existing system. With Nginx, you could also give it direct access to the static files and allow it to serve those without having to proxy through Apache. Using the Location directive, you can slice off portions of your webspace and prevent that from having to go through the proxy.

However, if you wanted to move completely to Nginx, your infrastructure becomes:

nginx -> passenger -> rails (nginx -> uwsgi -> python)

If you add Varnish, you end up with:

varnish -> nginx -> passenger -> rails

unless you use ESI, in which case you end up with:

nginx -> varnish -> nginx -> passenger -> rails

At some point, removing Varnish from the mix becomes quite intriguing. However, recent Varnish releases are still faster than Nginx's caching and you have a lot of control over how you can cache. While both Nginx and Varnish give you quite a bit of control, Varnish's VCL allows you to write C code to do things that neither does out of the box, without touching the daemon's source code. Whether that is useful to you is up to you.

Since you are using Apache currently, I would be more inclined to put Varnish in front unless you are going to migrate to Nginx and remove Apache completely. Varnish in your case is more of a drop-in solution. If you decide that you're going to use ESI in the future, you would need to run both.