I am trying to setup squid to cache downloading of one of my programs , unfortunately I found out I am not caching anything, my logs looks like:
1414421675.540 1059 127.0.0.1 TCP_MISS/204 288 GET http://serverfault.com/posts/596890/ivc/d85b? - DIRECT/198.252.206.16 -
1414421676.548 769 127.0.0.1 TCP_MISS/200 1539 GET http://careers.stackoverflow.com/gethired/js - DIRECT/198.252.206.17 text/javascript
1414421677.226 674 127.0.0.1 TCP_MISS/204 442 GET http://careers.stackoverflow.com/ad/i/o8dlwjVTTd7-lrgGzmMcoFk-2LAw8 - DIRECT/198.252.206.17 image/gif
1414421685.217 48 127.0.0.1 TCP_MISS/200 4345 GET http://audaupdate.audatex.cz/SEARCH_TREE/ - DIRECT/93.189.233.60 text/html
1414421685.273 44 127.0.0.1 TCP_MISS/200 1407 GET http://audaupdate.audatex.cz/SEARCH_TREE/INDEX.DEF - DIRECT/93.189.233.60 text/plain
1414421689.198 553 127.0.0.1 TCP_MISS/200 16838 GET http://serverfault.com/questions/473940/caching-large-amounts-of-data - DIRECT/198.252.206.16 text/html
1414421689.276 35 127.0.0.1 TCP_MISS/200 1983 GET http://i.stack.imgur.com/F801B.jpg? - DIRECT/103.31.6.34 image/jpeg
1414421689.426 27 127.0.0.1 TCP_MISS/200 614 GET http://www.google-analytics.com/collect? - DIRECT/173.194.40.5 image/gif
1414421689.438 32 127.0.0.1 TCP_MISS/204 395 GET http://b.scorecardresearch.com/b? - DIRECT/193.247.167.139 -
1414421689.517 138 127.0.0.1 TCP_MISS/200 2615 GET http://engine.adzerk.net/ados? - DIRECT/107.20.144.218 application/javascript
1414421689.594 190 127.0.0.1 TCP_MISS/200 439 GET http://pixel.quantserve.com/pixel;r=545276160;a=p-c1rF4kxgLUzNc;fpan=0;fpa=P0-1294698880-1407227396456;ns=0;ce=1;cm=;je=1;sr=1920x1080x24;enc=n;dst=1;et=1414421689399;tzo=-60;ref=http%3A%2F%2Fserverfault.com%2Fquestions%2Fask;url=http%3A%2F%2Fserverfault.com%2Fquestions%2F473940%2Fcaching-large-amounts-of-data;ogl=type.website%2Cimage.http%3A%2F%2Fcdn%252Esstatic%252Enet%2Fserverfault%2Fimg%2Fapple-touch-icon%402%252Epng%3Fv%3D9b1f48ae296b%2Ctitle.Caching%20large%20amounts%20of%20data%2Cdescription.We%20need%20to%20cache%20all%20psd%2Fzip%20files%20from%20our%20amazone%20S3%20storage%252E%20The%20amount%20of%20th%2Curl.http%3A%2F%2Fserverfault%252Ecom%2Fquestions%2F473940%2Fcaching-large-amounts-of-data - DIRECT/95.172.94.12 image/gif
1414421689.996 214 127.0.0.1 TCP_MISS/200 1036 GET http://stackexchange.com/ads/launched-question/js? - DIRECT/198.252.206.16 text/javascript
1414421690.007 228 127.0.0.1 TCP_MISS/200 671 GET http://engine.adzerk.net/i.gif? - DIRECT/23.23.234.111 image/gif
1414421690.093 651 127.0.0.1 TCP_MISS/204 288 GET http://serverfault.com/posts/473940/ivc/d85b? - DIRECT/198.252.206.16 -
1414421690.128 119 127.0.0.1 TCP_MISS/200 671 GET http://engine.adzerk.net/i.gif? - DIRECT/23.23.234.111 image/gif
1414421690.229 153 127.0.0.1 TCP_MISS/200 3163 GET http://ad.doubleclick.net/adj/N7384.171779.STACKOVERFLOW.COM/B8285612.111984239;sz=728x90;click=http://engine.adzerk.net/r? - DIRECT/173.194.40.27 text/javascript
1414421690.320 42 127.0.0.1 TCP_MISS/200 6968 GET http://cdn.sstatic.net/stackexchange/Img/launched-ads/physics.png - DIRECT/141.101.114.59 image/png
1414421690.369 126 127.0.0.1 TCP_MISS/200 603 GET http://googleads4.g.doubleclick.net/pagead/adview? - DIRECT/173.194.40.13 text/html
1414421690.641 103 127.0.0.1 TCP_MISS/200 10047 GET http://s1.2mdn.net/viewad/3742665/SignUp-BannerAds-CodeBugs-c-728x90.gif - DIRECT/74.125.232.156 image/gif
1414421691.158 379 127.0.0.1 TCP_MISS/204 442 GET http://careers.stackoverflow.com/ad/i/o8dlwjVTTd7-lrgGzmMcoFk-2LAw8 - DIRECT/198.252.206.17 image/gif
1414421692.532 669 127.0.0.1 TCP_MISS/200 529 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421693.656 19965 127.0.0.1 TCP_MISS/200 5076 CONNECT adzerk-www.s3.amazonaws.com:443 - DIRECT/54.231.1.233 -
1414421699.259 23788 127.0.0.1 TCP_MISS/200 1560 CONNECT qa.sockets.stackexchange.com:443 - DIRECT/198.252.206.25 -
1414421700.747 91 127.0.0.1 TCP_MISS/200 677 GET http://pagead2.googlesyndication.com/activeview? - DIRECT/173.194.112.218 image/gif
1414421709.400 141 127.0.0.1 TCP_MISS/200 29779 GET http://audaupdate.audatex.cz/ - DIRECT/93.189.233.60 text/html
1414421710.792 1349 127.0.0.1 TCP_MISS/200 1858526 GET http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0000A5/0000A5.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
1414421711.121 297 127.0.0.1 TCP_MISS/200 439819 GET http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0000R8/0000R8.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
1414421711.518 370 127.0.0.1 TCP_MISS/200 508209 GET http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0JD050/0JD050.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
1414421711.665 52 127.0.0.1 TCP_MISS/200 29779 GET http://audaupdate.audatex.cz/ - DIRECT/93.189.233.60 text/html
1414421712.270 548 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/LOG/347035-20141027155316-STATUS.LOG - DIRECT/93.189.233.60 -
1414421712.911 127 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/LOG/347035-20141027155316-STATUS.LOG - DIRECT/93.189.233.60 -
1414421713.531 83 127.0.0.1 TCP_MISS/200 3382 GET http://audaupdate.audatex.cz/LOG/347035-20141027155316-STATUS.LOG - DIRECT/93.189.233.60 text/plain
1414421715.584 42 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/TRACE/347035-20141027.ZIP - DIRECT/93.189.233.60 -
1414421717.463 1878 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/TRACE/347035-20141027.ZIP - DIRECT/93.189.233.60 -
1414421738.346 588 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421784.008 258 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421830.045 298 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421839.845 51 127.0.0.1 TCP_MISS/200 1624 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421839.849 55 127.0.0.1 TCP_MISS/200 1624 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421845.898 5052 127.0.0.1 TCP_MISS/200 2940 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421845.908 5057 127.0.0.1 TCP_MISS/200 2908 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421855.023 200872 127.0.0.1 TCP_MISS/200 9839 CONNECT www.gravatar.com:443 - DIRECT/68.232.35.121 -
1414421876.367 622 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421889.871 199446 127.0.0.1 TCP_MISS/200 1560 CONNECT qa.sockets.stackexchange.com:443 - DIRECT/198.252.206.25 -
1414421894.153 358 127.0.0.1 TCP_MISS/200 17472 GET http://serverfault.com/search/titles? - DIRECT/198.252.206.16 application/json
1414421894.399 229 127.0.0.1 TCP_MISS/204 306 POST http://serverfault.com/gps/event - DIRECT/198.252.206.16 -
1414421896.276 275 127.0.0.1 TCP_MISS/200 17554 GET http://serverfault.com/search/titles? - DIRECT/198.252.206.16 application/json
1414421896.510 233 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-title - DIRECT/198.252.206.16 application/json
1414421896.851 340 127.0.0.1 TCP_MISS/204 306 POST http://serverfault.com/gps/event - DIRECT/198.252.206.16 -
1414421921.868 499 127.0.0.1 TCP_MISS/200 391 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421929.372 292573 127.0.0.1 TCP_MISS/200 4556 CONNECT clients1.google.com:443 - DIRECT/173.194.112.199 -
1414421941.513 334 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-body - DIRECT/198.252.206.16 application/json
1414421943.762 235 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-body - DIRECT/198.252.206.16 application/json
1414421949.862 476 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-body - DIRECT/198.252.206.16 application/json
So it looks it logging what i did, and also log which files program download
http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0JD050/0JD050.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
Anyway when I try to download same file again it is downloaded again from the original webserver instead of from cache. Same when I try to watch some video or anything, it is always downloading , never served from cache. Adding my config as well. Can anybody tell me where can be a problem?
The sanitised config is below:
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access allow localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow all
icp_access allow localnet
icp_access deny all
http_port 3128
hierarchy_stoplist cgi-bin ?
cache_dir ufs c:/squid/var/cache 9000 16 256
maximum_object_size 4096000000 KB
access_log c:/squid/var/logs/access.log squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
range_offset_limit 409600000 KB
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
visible_hostname localhost
coredump_dir c:/squid/var/cache
The full config can be found here: http://pastebin.com/yRV6VE16
Best Answer
Squid is not able to keep objects coming from CDNs like YouTube or URLs with '?' char. To cache videos, images, etc... from CDN URLs, you should use a plugin that'll do the job for you. You can try the SquidVideoBooster taking care 600+ websites including YouTube, DailyMotion, NetFlix, etc...
Link: https://sourceforge.net/projects/squidvideosbooster/ Compatible Squid 2.7, 3.4 and 3.5.
Download this plugin and test, you'll see if it fits your needs ;o)
Bye Fred