Caching huge files with varnish. Is it viable

cachesquidvarnish

I have one server that provides access to some hundreds of files. All of them are really big (some have more than 10GB). These files won't change and are read only, but I'll need to provide access to them via a WAN.

I want to make the access faster by using a reverse proxy server running near my users. For example:

  • User X wants to access something on server A.
  • User X access a reverse proxy server on his LAN which causes a cache
    miss. The file is downloaded to this proxy server.
  • The next time user X wants the same file, he doesn't need to go get it in my main server.

I know that this situation is trivial, but I won't have a lot of traffic, my problem is more related to the file sizes themselves. I read something about Squid having problems caching files larger than 2GB. Does Varnish face the same kind of problems?

Thank you for any thoughts on that.

Best Answer

It's technically possible, but for what you want to achieve - I'd consider simply mirror the backend server periodically with rsync.

One of the reasons for this, is how Varnish delivers objects from its cache. If the object is not in Varnishs cache, it will fetch it from the backend. When this happens, the user will have to wait - without any transferring to the client occurring.

When the file is stored in the Varnish cache, transfer to the user can start.

Depending on how the users access the servers in the first place, this can cause significant amount of extra waiting.

Related Topic