I have 2 varnish 3.0.4 servers on edge of an Apache cluster (fallback director).
On one of my sites, users log in to download static content such as pdf, images (jpg,jpeg,png, gif), xls/xlsx etc. All but the exel files are told to be cached.
When downloading an Excel file everything works as expected. When trying to download a cached object the behavior is different. Even if the site is correctly displayed (all images appears) the user is disconnected and the file can't be fetched (404 from varnishlog).
No issues browsing the site when logged in.
Using host file to point directly on Apache and the files download perfectly.
As a temporary solution told varnish not to cache for this site and banned all content for it.
Thank you.
EDIT : full vcl and varnishlog.
Thanks for your help. Sadly it's still not working. Here is my full vlc
backend srv1 {
.host = "srv1";
.port = "80";
.first_byte_timeout = 120s;
.probe = {
.url = "/";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend srv2 {
.host = "srv2";
.port = "80";
.first_byte_timeout = 120s;
.probe = {
.url = "/";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend srv3 {
.host = "srv3";
.port = "80";
.first_byte_timeout = 120s;
.probe = {
.url = "/";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
director cluster fallback {
{ .backend = srv1; }
{ .backend = srv2; }
{ .backend = srv3; }
}
sub vcl_recv {
set req.backend = cluster;
if (req.url ~ "\.(htm|html|xml|jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|pdf|flv|swf)$") {
return (lookup);
}
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
# Normalize encoding/compression
if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; }
elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; }
else { remove req.http.Accept-Encoding; }
}
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
}
acl purge {
"localhost";
}
#
sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged.";
}
}
#
sub vcl_miss {
if (req.request == "PURGE") {
error 404 "Not in cache.";
}
}
sub vcl_fetch {
if (req.url ~ "\.(htm|html|xml|jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|pdf|flv|swf)$") {
return (deliver);
}
}
sub vcl_deliver {
if (obj.hits > 0)
{ set resp.http.X-Cache = "HIT from Varnish"; }
else
{ set resp.http.X-Cache = "MISS from Varnish"; }
}
And here is the full varnishlog trace when trying to download a file :
50 BackendOpen b srv1 10.24.1.11 40049 10.24.1.95 80
50 TxRequest b GET
50 TxURL b /de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf
50 TxProtocol b HTTP/1.1
50 TxHeader b Via: 1.1 ClientSiteProxy:3128 (squid/2.7.STABLE4), 1.0 proxy-1_5 (squid/3.1.19), 1.1 cache-3:80, 1.0 SPRX0008
50 TxHeader b Host: mysite.domain.com
50 TxHeader b Referer: http://mysite.domain.com/de-ge/website-material/vet_basic/produktdatenblaetter
50 TxHeader b User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
50 TxHeader b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
50 TxHeader b Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
50 TxHeader b X-Teacup: eFDRDF7Vx0755xk=
50 TxHeader b X-Forwarded-For: CLIENT_IP, PUBLIC_IP
50 TxHeader b X-Varnish: 1630421623
50 TxHeader b Accept-Encoding: gzip
50 RxProtocol b HTTP/1.1
50 RxStatus b 404
50 RxResponse b Not Found
50 RxHeader b Date: Wed, 08 Jan 2014 10:02:24 GMT
50 RxHeader b Server: Apache
50 RxHeader b Expires: Thu, 19 Nov 1981 08:52:00 GMT
50 RxHeader b Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, public, no-transform
50 RxHeader b Pragma: no-cache
50 RxHeader b X-Powered-By: eZ Publish
50 RxHeader b Served-by: mysite.domain.com
50 RxHeader b Content-language: de-DE
50 RxHeader b Set-Cookie: PHPSESSID=ob5a4jaic2ubpvala2qacksl17; path=/
50 RxHeader b Content-Length: 7030
50 RxHeader b Connection: close
50 RxHeader b Content-Type: text/html; charset=utf-8
50 Fetch_Body b 4(length) cls 0 mklen 1
50 Length b 7030
50 BackendClose b srv1
24 ReqStart c REVERSE_PROXY_IP(NOT VARNISH) 57364 1630421623
24 RxRequest c GET
24 RxURL c /de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf
24 RxProtocol c HTTP/1.1
24 RxHeader c Via: 1.1 ClientSiteProxy:3128 (squid/2.7.STABLE4), 1.0 proxy-1_5 (squid/3.1.19), 1.1 cache-3:80, 1.0 SPRX0008
24 RxHeader c Host: mysite.domain.com
24 RxHeader c Cookie: ISAWPLB{515B9AC4-0CEE-4B2B-B32F-DB0AD870EC2F}={DF5E7FFB-0F55-466C-9DDC-2983CFDF75AE}; PHPSESSID=nfcqe0tcq7gqiu253ef3o9svu4; is_logged_in=true
24 RxHeader c Referer: http://mysite.domain.com/de-ge/website-material/vet_basic/produktdatenblaetter
24 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
24 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
24 RxHeader c Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
24 RxHeader c X-Teacup: eFDRDF7Vx0755xk=
24 RxHeader c X-Forwarded-For: CLIENT_IP, PUBLIC_IP
24 RxHeader c Cache-Control: max-age=259200
24 RxHeader c Connection: Keep-Alive
24 VCL_call c recv lookup
24 VCL_call c hash
24 Hash c /de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf
24 Hash c mysite.domain.com
24 VCL_return c hash
24 VCL_call c miss fetch
24 Backend c 50 cluster srv1
24 TTL c 1630421623 RFC 0 -1 -1 1389175345 0 1389175344 375007920 0
24 VCL_call c fetch deliver
24 ObjProtocol c HTTP/1.1
24 ObjResponse c Not Found
24 ObjHeader c Date: Wed, 08 Jan 2014 10:02:24 GMT
24 ObjHeader c Server: Apache
24 ObjHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT
24 ObjHeader c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, public, no-transform
24 ObjHeader c Pragma: no-cache
24 ObjHeader c X-Powered-By: eZ Publish
24 ObjHeader c Served-by: mysite.domain.com
24 ObjHeader c Content-language: de-DE
24 ObjHeader c Set-Cookie: PHPSESSID=ob5a4jaic2ubpvala2qacksl17; path=/
24 ObjHeader c Content-Type: text/html; charset=utf-8
24 VCL_call c deliver deliver
24 TxProtocol c HTTP/1.1
24 TxStatus c 404
24 TxResponse c Not Found
24 TxHeader c Server: Apache
24 TxHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT
24 TxHeader c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, public, no-transform
24 TxHeader c Pragma: no-cache
24 TxHeader c X-Powered-By: eZ Publish
24 TxHeader c Served-by: mysite.domain.com
24 TxHeader c Content-language: de-DE
24 TxHeader c Set-Cookie: PHPSESSID=ob5a4jaic2ubpvala2qacksl17; path=/
24 TxHeader c Content-Type: text/html; charset=utf-8
24 TxHeader c Content-Length: 7030
24 TxHeader c Accept-Ranges: bytes
24 TxHeader c Date: Wed, 08 Jan 2014 10:02:25 GMT
24 TxHeader c X-Varnish: 1630421623
24 TxHeader c Age: 0
24 TxHeader c Via: 1.1 varnish
24 TxHeader c Connection: keep-alive
24 TxHeader c X-Cache: MISS from Varnish
24 Length c 7030
24 ReqEnd c 1630421623 1389175344.435734510 1389175345.230693817 2.376755714 0.794904470 0.000054836
EDIT2 : I can't use curl because I need to be logged in the application. But I'm 200% sure the ressource is available since when I use my host file to hit the web server I can download the files.
Here's the output of Chrome dev tools :
Request URL:http://mysite.domain.com/de-ge/content/download/1004658/18185577/version/2/file/RC_PB_CHN_Junior.pdf
Request Method:GET
Status Code:200 OK
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Cookie:__utma=65825073.183772452.1389198084.1389262355.1389265715.3; __utmc=65825073; __utmz=65825073.1389198084.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=3jr13d7cqmn0kv86iho4n030i0; is_logged_in=true
Host:mysite.domain.com
Referer:http://mysite.domain.com/de-ge/website-material/retail_basic/produktdatenblaetter
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Response Headers
Accept-Ranges:bytes
Cache-Control:public
Connection:close
Content-Disposition:attachment
Content-language:de-DE
Content-Transfer-Encoding:binary
Content-Type:application/pdf
Date:Thu, 09 Jan 2014 14:36:30 GMT
Expires:Thu, 09 Jan 2014 14:46:30 GMT
Pragma:
Served-by:mysite.domain.com
Server:Apache
Transfer-Encoding:chunked
X-Powered-By:eZ Publish
EDIT 3 : I can't get it to work. I and I really don't understand why all images are correctly diplayed in the user zone whereas downloading (litterally, with a download prompt) images or another resource failed.
Thank you.
Best Answer
I suspect that your backend is overwriting the cookie, so I'll try changing
vcl_fetch
to:Anyway it's hard to tell without having a varnishlog trace and your full vcl config.
After seeing your full VCL and the varnishlog trace, its clear that the backend is answering a 404 status.
So please try to issue a curl call to the backend in order to inspect the response, like:
After seeing your EDIT2:
If you need to be logged in to get the resource and logins are cookie based, it's failing because you're stripping the cookies on:
BTW: You can issue curl calls with cookies: