How much RAM required by Varnish

You can run Varnish in a few different modes. One uses the disk as the cache repository and in this mode, you can safely restart Varnish without losing your cache. This also means you can support a very large cache and let the OS's IO cache keep the most requested content in the server's memory. This mode is configured with the flag -s file,/var/lib/varnish/varnish_storage.bin,1G

The other mode is memory-only - Varnish keeps all cached data only in memory, never writing to disk. This can deliver extremely fast response times, but the cache size is limited to how much memory your server has. Also, if you restart Varnish you lose the cache and suddenly requests will start flooding your backend. Note, however, that Varnish is extremely stable and we only ever restart Varnish when an update is released (every 6 months or so). This mode is configured like -s malloc,3200M

So to answer your question, Varnish performs much better with more RAM. We have a Amazon EC2 instance with 3.5GB RAM and no swap and give Varnish 3GB of RAM for caching. Currently we have almost the entire backend site in cache, including images and PDF files etc. I can definitely recommend this setup.

Yeah, it's valid. My personal approach would be to use Varnish up front and use VCL to split the traffic between static NGINX requests and your heavy lifting (whether that be Apache or Passenger or... it doesn't matter). This is especially true if it is on the same machine as you don't need the extra overhead. It doesn't necessarily buy you anything.