How to keep Varnish cached populate after backend down for an extended period

apache-2.2varnish

We have Varnish 3.0.2 running on Amazon's Linux and it works great. We have a ttl of 48 hours for most content pages and much longer for images, PDFs etc.

This weekend we've taken the backend down for some maintenance, so I upped the ttl to 5 days earlier in the week. I had assumed that anything in cache would continue to be served for up to 5 days, but much to our disappointment we checked varnishstat this morning and the cache was almost completely empty and varnish was serving "page not found" messages.

I know that this is not what Varnish is designed to do, but why would it reset its cache when the backend is down? And how can I prevent it for next time?

Update 2012-06-11: After looking in the /var/log/messages I see every 3 hours or so:

Jun  9 03:56:31 idea-varnish varnishd[1128]: Manager got SIGINT
Jun  9 03:56:33 idea-varnish varnishd[6708]: Platform: Linux,3.2.18-1.26.6.amzn1.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Jun  9 03:56:33 idea-varnish varnishd[6708]: child (6709) Started
Jun  9 03:56:33 idea-varnish varnishd[6708]: Child (6709) said Child starts

I guess this is the server crashing and wiping all the objects in memory. I have only just now installed the -debuginfo rpm but not sure that will actually show anything more.

I supposed we could have switched back to disk-based storage during the scheduled downtime? or would a crash like this wipe that anyway?

Best Answer

Did the varnish process maybe restart? There's an uptime counter in varnishstat. Under certain circumstances the varnish worker thread can die, but it gets restarted immediately. When everything is working fine, this might go unnoticed, but with (planned) backend down time it can be quite inconvient.

Related Solutions

Nginx – Varnish 3.0.2 to Apache2 sometimes return error 503

I was running into this with Apache, and the solution was a combination of the following (note that I'm using Apache/2.4.7 (Ubuntu) + varnish 3.0.5-2 on Ubuntu 14.04 LTS in AWS EC2):

Please keep in mind that this was made for an M3.Medium instance on Amazon EC2 (1x Intel Xeon E5-2670 core + 3.75GB RAM). Adjust as necessary for your hardware!

In /etc/default/varnish, edit your start-up options:

 DAEMON_OPTS="-a :80 \
     -T localhost:6082 \
     -f /etc/varnish/default.vcl \
     -S /etc/varnish/secret \
     -p thread_pools=2 \
     -p thread_pool_max=600 \
     -p listen_depth=1024 \
     -p lru_interval=900 \
     -p connect_timeout=600 \
     -p max_restarts=6 \
     -s malloc,1G"

In /etc/varnish/default.vcl or whatever your VCL is, change the back-end timeouts (note that we're also setting these in /etc/default/varnish):

backend default {
    .host = "127.0.0.1";
    .port = "8000";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
}

Disable KeepAlives. This page has more information (varies depending on back-end web server software): http://www.feedthebot.com/pagespeed/keep-alive.html

For Apache, all I had to do was change line 92 in /etc/apache2/apache2.conf to the following:

    KeepAlive Off

What I think is going on here is that the KeepAlives, as implemented in the back-end web server software, are sending explicit connection resets, which Varnish doesn't work well with. There is probably more to this story, and I encourage you to dig into this and post your findings here for future generations to learn from.

Additional reading: - https://www.varnish-cache.org/trac/wiki/Future_Feature#Keepalivetimeoutonbackendconnections ( and a few more, but can't post the links. Some Googling for "varnish keepalive backend timeout" should surface what you want)

More debugging help: If you're still stuck, try doing the following: - start varnishlog -w err.log on your Varnish server - On your client, get Siege: http://www.joedog.org/siege-home/ and load it up with some of the URLs you've seen 503 (hint: urls.txt, use -i -b -c500 -r10 and it should be enough to trigger the 503s) - start varnishlog -r temp -c -m 'TxStatus:503' > err-parsed.txt. This will grab all the Varnish log entries where Varnish returned a 503. FWIW, here's the full text of one of my errors. TL;DR the error Varnish was reporting was FetchError c http first read error: -1 0 (Success) :

936 SessionOpen c 10.8.226.98 51895 :80 936 ReqStart c 10.8.226.98 51895 357447130 936 RxRequest c GET 936 RxURL c /ip/69.120.68.54 936 RxProtocol c HTTP/1.1 936 RxHeader c Host: 10.201.81.157 936 RxHeader c Accept: */* 936 RxHeader c Accept-Encoding: gzip 936 RxHeader c User-Agent: Mozilla/5.0 (apple-x86_64-darwin11.4.2) Siege/3.0.5 936 RxHeader c Connection: close 936 VCL_call c recv lookup 936 VCL_call c hash 936 Hash c /ip/69.120.68.54 936 Hash c 10.201.81.157 936 VCL_return c hash 936 HitPass c 357445183 936 VCL_call c pass pass 936 Backend c 103 default default 936 FetchError c http first read error: -1 0 (Success) 936 Backend c 269 default default 936 FetchError c http first read error: -1 0 (Success) 936 VCL_call c error deliver 936 VCL_call c deliver deliver 936 TxProtocol c HTTP/1.1 936 TxStatus c 503 936 TxResponse c Service Unavailable 936 TxHeader c Server: Varnish 936 TxHeader c Content-Type: text/html; charset=utf-8 936 TxHeader c Retry-After: 5 936 TxHeader c Content-Length: 418 936 TxHeader c Accept-Ranges: bytes 936 TxHeader c Date: Thu, 05 Jun 2014 23:05:48 GMT 936 TxHeader c X-Varnish: 357447130 936 TxHeader c Age: 0 936 TxHeader c Via: 1.1 varnish 936 TxHeader c Connection: close 936 Length c 418

Hope this helps!

Nginx – Varnish stops listening on port 80 after few hours

Increase the cli_timeout parameter in Varnish to 60s.

This controls how long the monitoring parent will wait for the child to respond to a health check. If the operating system is busy paging data to/from disk, the 10s default may be to low. Increase it to one minute (default from 4.0 on) and see if the problem goes away.

If that doesn't help, my next guess would be over-eager log rotation scripts that kills more than they should.

Best Answer

Related Solutions

Nginx – Varnish 3.0.2 to Apache2 sometimes return error 503

Nginx – Varnish stops listening on port 80 after few hours

Related Topic