The answer I found is, Yes, a Varnish HIT should be recorded for cached pages in Magento 2 when Varnish is properly enabled. By properly, I mean:
- Varnish installed and running
- Magento 2 configured to use Varnish as FPC (Stores -> Configuration -> System -> "Full Page Cache"
- Download the vcl Magento gives you in the above admin area and load it into Varnish.
- Make sure that the "Page Cache" under System -> Cache Management is also enabled and refreshed.
To see the HIT in the HTTP response headers, you need to edit the vcl file and comment out the line:
unset resp.http.X-Varnish;
For Debug lines in the header response, put Magento into Developer mode, and make sure the resp.http.X-Magento-Debug is set (or unset commented out).
It just feels strange because of Magento and all people that post Tutorials and How To's claim that this should work fine just by setting it up(like FPC used to work).
I will join them :).
Magento 2 and Varnish are compatible out of the box. If something is not working, there was a step missed:
On the teams live projects I see the response cookie of PHPSESSID as well as Magento form key on every single page
If you see it on every page, this likely means that the Magento 2 VCL file was not applied. For reference, you can look at this line.
To recap: Magento 2 does spit out PHPSESSID on every page but Magento 2's VCL unsets it and overrides the built-in VCL in a way that even with cookies, the page can be cached.
Troubleshoot
In proper configuration the Expires and Cache-Control header which is emitted by the Magento 2 app will ensure the TTL to be positive, e.g. 1-day expiration. That is the TTL which Varnish will use to cache the page.
You can verify the headers coming from your backend app via curl. Simply launch the curl against Magento 2 directly. This way you can see headers which Varnish sees.
Sample headers from Magento 2 (before Varnish):
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 02 Mar 2018 14:40:55 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: store=default; expires=Sat, 02-Mar-2019 14:40:55 GMT; Max-Age=31536000; path=/; HttpOnly
Set-Cookie: PHPSESSID=hbtgntuutdaa460hjtfdobbjt3; expires=Fri, 02-Mar-2018 15:40:55 GMT; Max-Age=3600; path=/; domain=www.example.com; secure; HttpOnly
Pragma: cache
Cache-Control: max-age=86400, public, s-maxage=86400
Expires: Sat, 03 Mar 2018 14:40:55 GMT
X-Magento-Tags: store,cms_b,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_header_cms_content,cms_b_footer_cms_content,cms_p_58,cat_p
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
There is Set-Cookie which is fine because M2 VCL will take care of it. The Expiress and Cache-Control specify 1 day of TTL for Varnish.
On my clean Magento 2 I can also see that Pragma and Cache-Control do not allow caching of any page.
This is to be expected. Those headers are meant for browsers and do not affect caching by Varnish. They are set by Magento 2 VCL.
Best Answer
Based on your screenshot, the first thing you should pay attention to is the
Set-Cookie.Every time a page returns it Varnish Cache will not cache the page.
So you need to remove all the cookie manipulation from the pages you want to cache.
One way you could do that is to use a specific page to treat all the cookies and call it as an AJAX in the pages that are cached.
That way, even if the page is cached that piece of dynamic information will be handled by AJAX.
Remember to add this specific page to the cache exceptions so it will not be cached at all.