There are two difficult things in computer science:
- Naming things
- Cache invalidation.
Hole punching falls into category #2 :)
General
The best approach is to start at the lower points of the stack and optimize up to the frontend of Magento.
Database and Filesystem
Should always be the first areas to focus on. Because. I/O.
MyTop is a handy Linux based perl script that will mimic the Linux 'top' command and give you insight on the state of your MySQL instance(s).
Htop is a more robust top, The strace feature can help determine ins/outs of a process to find potential bottlenecks.
Iotop is another tool to consider for monitoring I/O.
Other handy utility scripts like mysqltuner.pl and mysql tunning primer can offer insight into your MySQL runtime variables and offer advice to help. Keep in mind these are meant to be guides as the best approach is always an evaluation of requirements and tuning based on known data gathered. Blindly doing so can cause more damage at times than good. And prematurely running these without at least 24 hours of mysql runtime variables may offer bad advice.
Keep in mind Percona, MariaDB and standard MySQL should work with all of the above. Favoring Percona as a MySQL fork, since Magento is so heavy on InnoDB and XtraDB offers many tools and enhancements to the db engine.
Apache or Nginx
Still using Apache as it has served many others well, myself included. I have used and configured Nginx as well. While it does offer some advantages there is a learning curve. While the two are both popular options, it does offer some advantages over Apache, one would be a smaller memory footprint. However a slim downed Apache running PHP-FPM will have a similar memory footprint.
Case in point:
Since this article was about performance, I should point out that one
of the easiest ways to help apache get out of its own way is to not
use .htaccess files. Put what you'd put there in your Directory
stanzas, set AllowOverride to "None" and you end up not asking apache
to traverse the whole document path to figure out if it needs to pay
attention to .htaccess or not. This is a basic, simple tuning hint
that many people seem to miss.
To help facilitate this check out:
Utilizing a CDN to help take the ease off of either will help obviously but will have added benefit on frontend optimization since most end users browsers will be able to connect to both servers with the same number of connection limits. This also frees up Apache from not having to jump through checks and such just to serve up a simple static image. Lighthttpd is an option if you want to run a static web server just for content besides a CDN.
PHP
PHP-FPM and APC. Use them, strip out any unneeded or unrequired PHP modules not needed for Magento.
Magento codebase
AOE_TemplateHints is great to determine if your blocks are caching properly:
AOE_Profiler is good for profiling, be sure and enable its DB layer profiling (in a local/dev environment obviously). This in conjunction with the mytop tool mentioned previously makes finding bad behaving SQL an easier task.
3rd Party modules & Custom code
Some very good best practices for optimization from Magento themselves is a good read, and to keep in mind when reviewing 3rd party modules before using them. (there are lots of bad behaving ones IMO).
A tool Magniffer from Magento ECG will help easily identify bad behaving code based on the PDF provided above. It is symfony/php-parser based however but installable via composer.
Varnish
As an advocate of Varnish being the author was a FreeBSD kernel dev, it offers some crazy sub second load times. However if you even have some of the slightest differences in your templates that isn't out of box, you will spend time configuring varnish / magento to holepunch the content you need. Most I've seen will simply AJAX'ify the needed items uncached from Varnish.
There are a number of Magento modules to help facilitate this hole punching and caching:
Ultimately this should be at the last end of your optimization journey, and MAY require some customization to get things right.
Magento CE FPC
So far the best CE FPC I have found is: Lesti::FPC
it is a very well put together (all observer based) open-source and free FPC for Community.
At the end of the day use your own testing and judgement.
Some further reading:
It just feels strange because of Magento and all people that post Tutorials and How To's claim that this should work fine just by setting it up(like FPC used to work).
I will join them :).
Magento 2 and Varnish are compatible out of the box. If something is not working, there was a step missed:
On the teams live projects I see the response cookie of PHPSESSID as well as Magento form key on every single page
If you see it on every page, this likely means that the Magento 2 VCL file was not applied. For reference, you can look at this line.
To recap: Magento 2 does spit out PHPSESSID on every page but Magento 2's VCL unsets it and overrides the built-in VCL in a way that even with cookies, the page can be cached.
Troubleshoot
In proper configuration the Expires and Cache-Control header which is emitted by the Magento 2 app will ensure the TTL to be positive, e.g. 1-day expiration. That is the TTL which Varnish will use to cache the page.
You can verify the headers coming from your backend app via curl. Simply launch the curl against Magento 2 directly. This way you can see headers which Varnish sees.
Sample headers from Magento 2 (before Varnish):
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 02 Mar 2018 14:40:55 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: store=default; expires=Sat, 02-Mar-2019 14:40:55 GMT; Max-Age=31536000; path=/; HttpOnly
Set-Cookie: PHPSESSID=hbtgntuutdaa460hjtfdobbjt3; expires=Fri, 02-Mar-2018 15:40:55 GMT; Max-Age=3600; path=/; domain=www.example.com; secure; HttpOnly
Pragma: cache
Cache-Control: max-age=86400, public, s-maxage=86400
Expires: Sat, 03 Mar 2018 14:40:55 GMT
X-Magento-Tags: store,cms_b,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_header_cms_content,cms_b_footer_cms_content,cms_p_58,cat_p
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
There is Set-Cookie which is fine because M2 VCL will take care of it. The Expiress and Cache-Control specify 1 day of TTL for Varnish.
On my clean Magento 2 I can also see that Pragma and Cache-Control do not allow caching of any page.
This is to be expected. Those headers are meant for browsers and do not affect caching by Varnish. They are set by Magento 2 VCL.
Best Answer
For example if varnish is running on 127.0.0.1:8080 you can run
If you set up Magento to clear the Varnish cache above, it will only clear the cache for changed items (unless you flush everything which you should avoid if possible). You can use a tool like Siege to read your sitemap.xml if you want to build a little cache warming script.
Magento 2 uses partial reindexing so it only reindexes changed items - this should also clear your full page cache when it completes. I believe there are some known issues with the price indexing in 2.2.