How to utilize Varnish for A/B Testing and Feature Rollout

abcachereverse-proxytestingvarnish

Today we have our web layer exposed to the world. We would like to add Varnish in front of our web layer to accelerate the site and reduce calls to the backend.
However, we have some concerns and i was wondering how most people approach them:

  1. A/B Testing – How do you test two "versions" of each page and compare? I mean, how does varnish know which page to serve up? If and how do you save seperate versions on each page?

  2. Feature rollout – how would you set up a simple feature rollout mechanism? Let's say i want to open a new feature/page to just 10% of the traffic.. and then later increase that to 20%?

  3. How do you handle code deployments? Do you purge your entire varnish cache every deployment? (We have deployments on a daily basis). Or do you just let it slowly expire (using TTL)?

Any ideas and examples regarding these issues is greatly appreciated!

Best Answer

A/B Testing - How do you test two "versions" of each page and compare? I mean, how does varnish know which page to serve up? If and how do you save seperate versions on each page?

You have several choices:

  • Simply expose them at different URLs.

  • Bypass the cache for the specific URL. You could do this by returning pass in vcl_recv. Something like this:

    sub vcl_recv {
    if (req.url ~ "^/path/to/document") {
        return (pass);
    }
}

  • Explicitly purge the cache when you expose a new version.

Feature rollout - how would you set up a simple feature rollout mechanism? Let's say i want to open a new feature/page to just 10% of the traffic.. and then later increase that to 20%?

I'm not sure there's a "simple" way to do this. Since you can put arbitrary C code in your .vcl files you could probably add some logic to pick a random number and then select the appropriate backend path based on the result.

How do you handle code deployments? Do you purge your entire varnish cache every deployment? (We have deployments on a daily basis). Or do you just let it slowly expire (using TTL)?

For major changes we just purge the cache, and for smaller changes we just let things expire.

Related Topic