Varnish does not start properly (crashes after startup) with no error messages

reverse-proxyvarnish

I am running Varnish (2.0.4 from the Ubuntu unstable apt repository, though I have also used the standard repository) in a test environment (Virtual Machines) on Ubuntu 9.10, soon to be 10.04.

When I have a working configuration and the server starts successfully it seems like everything is fine, however if, for whatever reason, I stop and then restart the varnish daemon it doesn't always startup properly, and there are no errors going into syslog or messages to indicate what might be wrong.

If I run varnish in debug mode (-d) and issue start when prompted then 7 times out of time it will run, but occasionally it will just shut down 'silently'.

My startup command is (the $1 allows for me to pass -d to the script this lives in):

varnishd -a :80 $1 \
-T 127.0.0.1:6082 \
-s malloc,1GB \
-f /home/deploy/mysite.vcl \
-u deploy \
-g deploy \
-p obj_workspace=4096 \
-p sess_workspace=262144 \
-p listen_depth=2048 \
-p overflow_max=2000 \
-p ping_interval=2 \
-p log_hashstring=off \
-h classic,5000009 \
-p thread_pool_max=1000 \
-p lru_interval=60 \
-p esi_syntax=0x00000003 \
-p sess_timeout=10 \
-p thread_pools=1 \
-p thread_pool_min=100 \
-p shm_workspace=32768 \
-p thread_pool_add_delay=1

and the VCL looks like this:

# nginx/passenger server, HTTP:81
backend default {
.host = "127.0.0.1";
.port = "81";
}

sub vcl_recv {
  # Don't cache the /useradmin or /admin path
  if (req.url ~ "^/(useradmin|admin|session|sessions|login|members|logout|forgot_password)") {
    pipe;
  }

  # If cache is 'regenerating' then allow for old cache to be served
  set req.grace = 2m;

  # Forward to cache lookup
  lookup;
}

# This should be obvious
sub vcl_hit {
  deliver;
}

sub vcl_fetch {
  # See link #16, allow for old cache serving
  set obj.grace = 2m;

  if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {
    deliver;
  }

  remove obj.http.Set-Cookie;
  remove obj.http.Etag;

  set obj.http.Cache-Control = "no-cache";
  set obj.ttl = 7d;
  deliver;
}

Any suggestions would be greatly appreciated, this is driving me absolutely crazy, especially because its such an inconsistent behaviour.

Best Answer

When debug mode and error files are not helping, strace is our best friend.

Type strace -f -o strace.out before your varnish start command, so that strace can trace all your system calls.

Then, take a look at strace.out, especially the last lines. In this particular situations, strace is always a good tool to enlighten the way.

Hope this helps.

Related Solutions

Custom 503 Error Page With Varnish

The Varnish FAQ suggests using vcl_error for this (and it's how I've done it):

This is the default VCL for the error page:

sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";

    synthetic {"
        <?xml version="1.0" encoding="utf-8"?>
        <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
            "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
        <html>
            <head>
                <title>"} obj.status " " obj.response {"</title>
            </head>
            <body>
                <h1>Error "} obj.status " " obj.response {"</h1>
                <p>"} obj.response {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} req.xid {"</p>
                <address><a href="http://www.varnish-cache.org/">Varnish</a></address>
            </body>
        </html>
    "};
    return(deliver);
}

if you want a custom version, simply override the function in your config and replace the markup in the synthetic statement.

If you want to have different markup for different error codes you can do that fairly easily too:

sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";
    if (obj.status == 404) {
        synthetic {"
            <!-- Markup for the 404 page goes here -->
        "};
    } else if (obj.status == 500) {
        synthetic {"
            <!-- Markup for the 500 page goes here -->
        "};
    } else {
        synthetic {"
            <!-- Markup for a generic error page goes here -->
        "};
    }
}

Nginx – Configure php5-fpm for many concurrent users

I suspect your varnish cache is not caching anywhere near enough of the hits

here's what I would do in your situation:

Lower php max children to 100 or even 50 (if varnish does its job properly you don't need them) also remove the max requests line to allow the php processes not to respawn too quickly and thus prevent APC from being cleared too quickly which is also bad

also IF is not good according to nginx - http://wiki.nginx.org/IfIsEvil

I would change this line:

            if (!-e $request_filename) {
                    rewrite ^(.+)$ /index.php?q=$1 last;
            }

to:

try_files $uri $uri/ /index.php?$args;

If your version of nginx supports it (pretty certain if your nginx version is > 0.7.51 then it supports it)

you should also look at inserting the w3tc nginx rules direct into your vhost file to enabled proper disk enhanced caching of pages (which is faster than APC caching with nginx)

Take a look at the following varnish vcl which I use for sites - you will need to read through and edit a few things for your website - it also assumes that its only WP sites on the server and only 1 site on the server, it can easily be modified for more sites (take a look at the cookie section)

generic vcl: https://gist.github.com/b7332971a848bcb7ecef

With this config I would argue to remove fastcgi_cache to prevent any possible issues with a cache-chain occurring whereby trying locate any stray stale cache entries is more difficult

also tell w3tc that varnish is at 127.0.0.1 and it will purge it for you ;)

I deployed this to a server on Wednesday evening (with a few domain specific modifications) that was handling 2500 active site visitors it reduced load to less than 1 and the approx number of running php children was around 10-20 (this number does depend on number of logged in users and other factors like cookies) this server did have much more ram but the principle is the same, you should be able to easily handle the number of visitors you get at peaks

Best Answer

Related Solutions

Custom 503 Error Page With Varnish

Nginx – Configure php5-fpm for many concurrent users

Related Topic