How to increase the limit of the varnish log

loggingvarnish

i want to increase the log limit of the varnish cache server , as i see when i write the log to a file it only give me about 13k of requests , i want to increase it to something like 1 million for a high traffic site , so i can save the log every 10 mins .

i know varnish can run as Demon and write the logs on run time , but i don't need this

so is there anyway to increase the varnish log limit to a higher value ?

Best Answer

You can set the size of Varnish shared memory log buffer with the -l shmlogsize parameter:

-l shmlogsize Specify size of shmlog file. Scaling suffixes like 'k', 'm' can be used up to (e)tabytes. Default is 80 Megabytes. Specifying less than 8 Megabytes is unwise.

You can read about Varnish startup parameters in the official varnishd documentation. Depending on your OS or Linux distribution you will either have to edit the /etc/init.d/varnish script or distribution specific config files (e.g. /etc/default/varnish on Debian).

Bear in mind that writing varnish logs to disk on a high traffic site will eventually produce a large amount of disk I/O, even if you write a huge log buffer every couple of minutes, this will produce load too. If you do not need full logging you can use a daemonized varnishlog process with a filter that selects only the requests you are interested in, thus reducing the need file I/O.

Related Solutions

How to utilize Varnish for A/B Testing and Feature Rollout

A/B Testing - How do you test two "versions" of each page and compare? I mean, how does varnish know which page to serve up? If and how do you save seperate versions on each page?

You have several choices:

Simply expose them at different URLs.
Bypass the cache for the specific URL. You could do this by returning pass in vcl_recv. Something like this:
```
sub vcl_recv {
    if (req.url ~ "^/path/to/document") {
        return (pass);
    }
}
```
Explicitly purge the cache when you expose a new version.

Feature rollout - how would you set up a simple feature rollout mechanism? Let's say i want to open a new feature/page to just 10% of the traffic.. and then later increase that to 20%?

I'm not sure there's a "simple" way to do this. Since you can put arbitrary C code in your .vcl files you could probably add some logic to pick a random number and then select the appropriate backend path based on the result.

How do you handle code deployments? Do you purge your entire varnish cache every deployment? (We have deployments on a daily basis). Or do you just let it slowly expire (using TTL)?

For major changes we just purge the cache, and for smaller changes we just let things expire.

Varnish X-Forward-IP

If you're just interested in the "real" client IP, then take the req.http.X-Forwarded-For = client.ip setting out of Varnish completely.

Varnish will then have no interaction with the header - it should then pass the header as set by the load balancer through to Apache, allowing Apache to see and log the "real" client IP.

Best Answer

Related Solutions

How to utilize Varnish for A/B Testing and Feature Rollout

Varnish X-Forward-IP

Related Topic