I am using Nginx as an SSL terminator in front of varnish. Backends are litespeed. I have two wordpress sites in which both of them uses HTTPS protocol. I have a Haproxy in front of my infrastructure which only passes SSL traffics to Nginx.
One of my websites is loading fast and when I check the status of cached items through firebug, everything get a 'HIT' like .js .css and the domain name: domain1.com.
When I check these items for the second wordpress site, .js and HTML file get 'MISS' the same as the domain name domain2.com.
The configuration of Nginx and Varnish and the backend is the same for both web sites. I also disabled all of its plugins and the ones I think affect the caching :WP Fastest Cache, 404 to 301,Broken Link Checker and some others.
The only difference is that the second web site has a robot which gathers news from other sites and put them on the sites(some kind of web crawler).
What causes the varnish not to cache cache everything on the second site?
Edit:
I added the varnish log for a requests which is not cached:
varnishlog -g request -q "ReqUrl ~ '/wp-content/themes/farda/js/jquery.min.js'"
* << Request >> 149
- Begin req 148 rxreq
- Timestamp Start: 1499626236.025652 0.000000 0.000000
- Timestamp Req: 1499626236.025652 0.000000 0.000000
- ReqStart 192.168.1.108 57470
- ReqMethod GET
- ReqURL /wp-content/themes/farda/js/jquery.min.js
- ReqProtocol HTTP/1.0
- ReqHeader X-Real-IP: 192.168.1.105
- ReqHeader X-Forwarded-For: 192.168.1.105
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Nginx: on
- ReqHeader Host: mytourguide.ir
- ReqHeader Connection: close
- ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0
- ReqHeader Accept: */*
- ReqHeader Accept-Language: en-US,en;q=0.5
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Referer: https://mytourguide.ir/
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqUnset X-Forwarded-For: 192.168.1.105
- ReqHeader X-Forwarded-For: 192.168.1.105, 192.168.1.108
- VCL_call RECV
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- ReqUnset X-Forwarded-For: 192.168.1.105, 192.168.1.108
- ReqHeader X-Forwarded-For: 192.168.1.108
- ReqUnset Accept-Language: en-US,en;q=0.5
- ReqUnset User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0
- ReqHeader cookie:
- ReqUnset cookie:
- ReqHeader cookie:
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Debug "XXXX MISS"
- VCL_call MISS
- VCL_return fetch
- Link bereq 150 fetch
- Timestamp Fetch: 1499626236.029726 0.004075 0.004075
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Cache-Control: max-age=2592000, private
- RespHeader Expires: Mon, 09 Jul 2018 18:50:18 GMT
- RespHeader Last-Modified: Tue, 17 Jan 2017 14:46:21 GMT
- RespHeader Content-Type: text/javascript; charset=UTF-8
- RespHeader Content-Length: 33342
- RespHeader Content-Encoding: gzip
- RespHeader Vary: Accept-Encoding,User-Agent
- RespHeader Date: Sun, 09 Jul 2017 18:50:18 GMT
- RespHeader Accept-Ranges: bytes
- RespHeader Server: LiteSpeed
- RespHeader X-Varnish: 149
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: MISS
- RespUnset X-Varnish: 149
- RespUnset Via: 1.1 varnish-v4
- RespHeader X-Configured-By: ServerSetup.ir
- VCL_return deliver
- Timestamp Process: 1499626236.029739 0.004087 0.000012
- Debug "RES_MODE 2"
- RespHeader Connection: close
- RespHeader Accept-Ranges: bytes
- Timestamp Resp: 1499626236.030691 0.005039 0.000952
- Debug "XXX REF 2"
- ReqAcct 445 0 445 448 33342 33790
- End
** << BeReq >> 150
-- Begin bereq 149 fetch
-- Timestamp Start: 1499626236.025722 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /wp-content/themes/farda/js/jquery.min.js
-- BereqProtocol HTTP/1.0
-- BereqHeader X-Real-IP: 192.168.1.105
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Nginx: on
-- BereqHeader Host: mytourguide.ir
-- BereqHeader Accept: */*
-- BereqHeader Referer: https://mytourguide.ir/
-- BereqHeader Pragma: no-cache
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader X-Forwarded-For: 192.168.1.108
-- BereqHeader cookie:
-- BereqProtocol HTTP/1.1
-- BereqHeader X-Varnish: 150
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- Backend 17 apache web1(192.168.1.11,,5050)
-- Timestamp Bereq: 1499626236.025798 0.000076 0.000076
-- Timestamp Beresp: 1499626236.027971 0.002249 0.002173
-- BerespProtocol HTTP/1.1
-- BerespStatus 200
-- BerespReason OK
-- BerespHeader Cache-Control: max-age=2592000, private
-- BerespHeader Expires: Mon, 09 Jul 2018 18:50:18 GMT
-- BerespHeader Last-Modified: Tue, 17 Jan 2017 14:46:21 GMT
-- BerespHeader Content-Type: text/javascript; charset=UTF-8
-- BerespHeader Content-Length: 33342
-- BerespHeader Content-Encoding: gzip
-- BerespHeader Vary: Accept-Encoding,User-Agent
-- BerespHeader Date: Sun, 09 Jul 2017 18:50:18 GMT
-- BerespHeader Accept-Ranges: bytes
-- BerespHeader Server: LiteSpeed
-- BerespHeader Connection: Keep-Alive
-- TTL RFC 2592000 -1 -1 1499626236 1499626236 1499626218 1531162218 2592000
-- VCL_call BACKEND_RESPONSE
-- TTL VCL 120 10 0 1499626236
-- VCL_return deliver
-- Storage malloc Transient
-- ObjProtocol HTTP/1.1
-- ObjStatus 200
-- ObjReason OK
-- ObjHeader Cache-Control: max-age=2592000, private
-- ObjHeader Expires: Mon, 09 Jul 2018 18:50:18 GMT
-- ObjHeader Last-Modified: Tue, 17 Jan 2017 14:46:21 GMT
-- ObjHeader Content-Type: text/javascript; charset=UTF-8
-- ObjHeader Content-Length: 33342
-- ObjHeader Content-Encoding: gzip
-- ObjHeader Vary: Accept-Encoding,User-Agent
-- ObjHeader Date: Sun, 09 Jul 2017 18:50:18 GMT
-- ObjHeader Accept-Ranges: bytes
-- ObjHeader Server: LiteSpeed
-- Fetch_Body 3 length stream
-- Gzip u F - 33342 95992 80 206366 266672
-- BackendReuse 17 web1(192.168.1.11,,5050)
-- Timestamp BerespBody: 1499626236.030637 0.004915 0.002666
-- Length 33342
-- BereqAcct 291 0 291 375 33342 33717
-- End
Best Answer
The problem is with backend (LiteSpeed).
Note this line in your varnishlog:
Varnish built-in VCL will not cache URLs which send
Cache-Control: private. Thus it's always being not cached, andMISS-ed later on.Proper solution: configure LiteSpeed to enable public cache as opposed to private cache.