How do you log the requests that are not getting a cache hit on varnish

A good approach is to inspect the traffic with varnishlog in order to see what makes which rules apply.

Since you're not enforcing a TTL, my guess is that the wordpress site delivers Cache-Control headers like "max-age: 0", or "no-cache" "private" or something, and varnish honors such headers unless you specifically tell it not to, i.e:

 sub vcl_fetch {
     if (!(req.url ~ "wp-(login|admin)")) {
         unset beresp.http.set-cookie;
         set beresp.ttl = 3600s;
     }

First rule in vcl_fetch does not make sense, as second rule will unset all set-cookies for all urls but wp-login/wp-admin.


Add some config to vcl_deliver to output headers that states if it's a hit or not:

sub vcl_deliver {
        if (obj.hits > 0) {
                set resp.http.X-Cache = "HIT";
        } else {
                set resp.http.X-Cache = "MISS";
        }
     return (deliver);
}

Other feedback to you config:

In vcl_recv, you do "unset req.http.cookie" if url is not wp-login or wp-admin, which means that if a user logs on /wp-login then goes to "/", you will unset his cookie. You only need to unset cookies in vcl_recv for urls that shouldn't ever need one, ie .(js|css|png|jpg) etc, and instead remove the set-cookie in vcl_fetch.

req.hash_always_miss should do the trick.

Don't do a full cache flush at the start of the spider run. Instead, just set the spider to work - and in your vcl_recv, set the spider's requests to always miss the cache lookup; they'll fetch a new copy from the backend.

acl spider {
  "127.0.0.1";
  /* or whereever the spider comes from */
}

sub vcl_recv {
  if (client.ip ~ spider) {
    set req.hash_always_miss = true;
  }
  /* ... and continue as normal with the rest of the config */
}

While that's happening and until the new response is in the cache, clients will continue to seamlessly get the older cache served to them (as long as it's still within its TTL).