So I'm having a serious issue with Magento which is ruining performance since 90+% of the time we don't have caches.
I'm running Magento 2.3.2 and Varnish and have had several issues with them.
- Magento sending insanely large headers and crashing Varnish constantly.
- Then Magento sent a fix to send these headers in batch but they're improperly cut so Varnish was still crashing.
We've managed to fix these issues ourselves but we've now found out where the large tags are coming from and it seems that one of our indexes in Magento is clearing the cache of every single product and category in our catalog. Is there any reason for Magento to clear the cache of all products/categories constantly, like every 2 minutes or so? Also, this happens even when we have no updates to products, categories or any other actions in the admin area.
We've had situations where this stopped happening when some indexes got 'stuck' and I believe the Product Categories index might be the origin.
Here's the backtrace caught before sending the tags to Varnish (I'd link the tag it but it's really large, it's literally every tag of every product and every category in our catalog):
[2019-08-21 09:25:13] report.WARNING: backtrace cache #0 /home/<user>/magento2_prod/vendor/magento/module-cache-invalidate/Model/PurgeCache.php(75): Magento\CacheInvalidate\Model\PurgeCache->splitTags('((^|,)cat_p_421...')
#1 /home/<user>/magento2_prod/vendor/magento/module-cache-invalidate/Observer/InvalidateVarnishObserver.php(65): Magento\CacheInvalidate\Model\PurgeCache->sendPurgeRequest('((^|,)cat_p_421...')
#2 /home/<user>/magento2_prod/vendor/magento/framework/Event/Invoker/InvokerDefault.php(88): Magento\CacheInvalidate\Observer\InvalidateVarnishObserver->execute(Object(Magento\Framework\Event\Observer))
#3 /home/<user>/magento2_prod/vendor/magento/framework/Event/Invoker/InvokerDefault.php(74): Magento\Framework\Event\Invoker\InvokerDefault->_callObserverMethod(Object(Magento\CacheInvalidate\Observer\InvalidateVarnishObserver), Object(Magento\Framework\Event\Observer))
#4 /home/<user>/magento2_prod/vendor/magento/module-staging/Model/Event/Manager.php(97): Magento\Framework\Event\Invoker\InvokerDefault->dispatch(Array, Object(Magento\Framework\Event\Observer))
#5 /home/<user>/magento2_prod/generated/code/Magento/Staging/Model/Event/Manager/Proxy.php(95): Magento\Staging\Model\Event\Manager->dispatch('clean_cache_by_...', Array)
#6 /home/<user>/magento2_prod/vendor/magento/module-target-rule/Model/ResourceModel/Rule.php(170): Magento\Staging\Model\Event\Manager\Proxy->dispatch('clean_cache_by_...', Array)
#7 /home/<user>/magento2_prod/vendor/magento/module-target-rule/Model/Indexer/TargetRule/AbstractAction.php(110): Magento\TargetRule\Model\ResourceModel\Rule->cleanCachedDataByProductIds(Array)
#8 /home/<user>/magento2_prod/vendor/magento/module-target-rule/Model/Indexer/TargetRule/Product/Rule/Action/Rows.php(34): Magento\TargetRule\Model\Indexer\TargetRule\AbstractAction->_reindexByProductIds(Array)
#9 /home/<user>/magento2_prod/vendor/magento/module-target-rule/Model/Indexer/TargetRule/Product/Rule.php(100): Magento\TargetRule\Model\Indexer\TargetRule\Product\Rule\Action\Rows->execute(Array)
#10 /home/<user>/magento2_prod/vendor/magento/framework/Mview/View.php(294): Magento\TargetRule\Model\Indexer\TargetRule\Product\Rule->execute(Array)
#11 /home/<user>/magento2_prod/vendor/magento/framework/Mview/Processor.php(44): Magento\Framework\Mview\View->update()
#12 /home/<user>/magento2_prod/vendor/magento/module-indexer/Model/Processor.php(104): Magento\Framework\Mview\Processor->update('indexer')
#13 /home/<user>/magento2_prod/vendor/magento/framework/Interception/Interceptor.php(58): Magento\Indexer\Model\Processor->updateMview()
#14 /home/<user>/magento2_prod/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Indexer\Model\Processor\Interceptor->___callParent('updateMview', Array)
#15 /home/<user>/magento2_prod/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Indexer\Model\Processor\Interceptor->Magento\Framework\Interception\{closure}()
#16 /home/<user>/magento2_prod/generated/code/Magento/Indexer/Model/Processor/Interceptor.php(39): Magento\Indexer\Model\Processor\Interceptor->___callPlugins('updateMview', Array, Array)
#17 /home/<user>/magento2_prod/vendor/magento/module-indexer/Cron/UpdateMview.php(31): Magento\Indexer\Model\Processor\Interceptor->updateMview()
#18 [internal function]: Magento\Indexer\Cron\UpdateMview->execute(Object(Magento\Cron\Model\Schedule))
#19 /home/<user>/magento2_prod/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php(317): call_user_func_array(Array, Array)
#20 /home/<user>/magento2_prod/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php(737): Magento\Cron\Observer\ProcessCronQueueObserver->_runJob(1566379500, 1566379506, Array, Object(Magento\Cron\Model\Schedule), 'index')
#21 /home/<user>/magento2_prod/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php(243): Magento\Cron\Observer\ProcessCronQueueObserver->processPendingJobs('index', Array, 1566379506)
#22 /home/<user>/magento2_prod/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php(272): Magento\Cron\Observer\ProcessCronQueueObserver->Magento\Cron\Observer\{closure}('index')
#23 /home/<user>/magento2_prod/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php(244): Magento\Cron\Observer\ProcessCronQueueObserver->lockGroup('index', Object(Closure))
#24 /home/<user>/magento2_prod/vendor/magento/framework/Event/Invoker/InvokerDefault.php(88): Magento\Cron\Observer\ProcessCronQueueObserver->execute(Object(Magento\Framework\Event\Observer))
#25 /home/<user>/magento2_prod/vendor/magento/framework/Event/Invoker/InvokerDefault.php(74): Magento\Framework\Event\Invoker\InvokerDefault->_callObserverMethod(Object(Magento\Cron\Observer\ProcessCronQueueObserver), Object(Magento\Framework\Event\Observer))
#26 /home/<user>/magento2_prod/vendor/magento/module-staging/Model/Event/Manager.php(97): Magento\Framework\Event\Invoker\InvokerDefault->dispatch(Array, Object(Magento\Framework\Event\Observer))
#27 /home/<user>/magento2_prod/generated/code/Magento/Staging/Model/Event/Manager/Proxy.php(95): Magento\Staging\Model\Event\Manager->dispatch('default', Array)
#28 /home/<user>/magento2_prod/vendor/magento/framework/App/Cron.php(86): Magento\Staging\Model\Event\Manager\Proxy->dispatch('default')
#29 /home/<user>/magento2_prod/vendor/magento/module-cron/Console/Command/CronCommand.php(117): Magento\Framework\App\Cron->launch()
#30 /home/<user>/magento2_prod/vendor/symfony/console/Command/Command.php(255): Magento\Cron\Console\Command\CronCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#31 /home/<user>/magento2_prod/vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#32 /home/<user>/magento2_prod/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Cron\Console\Command\CronCommand\Interceptor->___callParent('run', Array)
#33 /home/<user>/magento2_prod/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Cron\Console\Command\CronCommand\Interceptor->Magento\Framework\Interception\{closure}(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#34 /home/<user>/magento2_prod/generated/code/Magento/Cron/Console/Command/CronCommand/Interceptor.php(26): Magento\Cron\Console\Command\CronCommand\Interceptor->___callPlugins('run', Array, Array)
#35 /home/<user>/magento2_prod/vendor/symfony/console/Application.php(893): Magento\Cron\Console\Command\CronCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#36 /home/<user>/magento2_prod/vendor/symfony/console/Application.php(262): Symfony\Component\Console\Application->doRunCommand(Object(Magento\Cron\Console\Command\CronCommand\Interceptor), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#37 /home/<user>/magento2_prod/vendor/magento/framework/Console/Cli.php(105): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#38 /home/<user>/magento2_prod/vendor/symfony/console/Application.php(145): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#39 /home/<user>/magento2_prod/bin/magento(23): Symfony\Component\Console\Application->run()
#40 {main} [] []
Best Answer
So after a bit of debugging I found out it was a Related Product Rule in Magento by following the backtrace. The target-rule index, even if it was only updating 10 products, the rule matched all products for some reason.
The rule we had was for all products, and it was basically set to show 7 random related products where the matching rule was within the same category.