After going in many different directions, I resorted to directly comparing a working product vs a non-working product in the db. Arguably, I should have done this first.
There were some variations between the two and I have managed to trace this to 'page_layout' (attribute_id = 107 (for me at least)) with a value of 'No layout updates' in the value column of the catalog_product_entity_varchar
table.
Scrubbing these records - There may be a record saved for each of the store views (admin, default etc) - and clearing the cache solved my issue.
Hopefully this helps someone in the future.
I keep bumping in this so I read up a bit more and now feel there is a need for better answers
Solutions that tell you to do cron_run = 'false" will disable other queues you do want, like bulk attribute update or export file generation.
Option 1 disable package
php bin/magento module:disable Magento_WebapiAsync
php bin/magento module:disable Magento_Amqp
They are useful for large site for bulk update and integrations, but for smaller sites totally fine without
Option 2 Tweak your queue cron, dont disable it
List your existing queues
bin/magento queue:consumers:list
Add all of them (actually, those that make sense, read update below) except async to your cron consumer, in env.php. If there is functionality you never use, skip that one (but remember you did...)
'cron_consumers_runner' => [
'cron_run' => true,
'max_messages' => 2,
'single_thread' => true,
'consumers' => [
'product_action_attribute.update',
'product_action_attribute.website.update',
'exportProcessor'
]
],
Most places have a higher max_messages but if you are not having rabbitmq you are possibly on a low budget host too and short queues more often are better. Up the number as you want.
UPDATE: these queues can end up staying up forever and there are a lot of them so if your server is not very busy and not very meaty that can take a permanent slice of memory off for no valuable use. Also if you don't set 'single thread' you end up with multiple queues waiting for the same thing.
It is useful to REMOVE any of the queues that are not relevant to your usage, put them in a slower cron with very low message numbers or run them manually should you need them once. Eg: I don't use coupon generation so removed codegenerator. I also don't do mass inventory updates often, so just run it manually as bin/magento queue:consumers:start --single-thread --max-messages=1 inventory.mass.update
UPDATE: there is a "process whats there don't wait around" option now, though it cannot be set via command line, or only at installation time. It is called queue/consumers_wait_for_messages and technically should mean your queue runner clears the queue then exists, not waiting around. This will use a bit more CPU but free memory
I have tried to add it to my env.php in two places and it seems to have worked.
'queue' => [
'consumers_wait_for_messages' => 0,
],
'cron_consumers_runner' => [
'cron_run' => true,
'max_messages' => 2,
'single_thread' => true,
'consumers-wait-for-messages' => 0,
'consumers' => [
'product_action_attribute.update',
'product_action_attribute.website.update',
'exportProcessor',
'inventory.source.items.cleanup',
'inventory.mass.update',
'inventory.reservations.cleanup',
'inventory.reservations.update'
]
],
ALSO: queue names have changed and there are some new ones in 5.4
product_action_attribute.update
product_action_attribute.website.update
exportProcessor
inventory.source.items.cleanup
inventory.mass.update
inventory.reservations.cleanup
inventory.reservations.update
media.storage.catalog.image.resize
codegeneratorProcessor (coupon codes)
inventory.reservations.updateSalabilityStatus
inventory.indexer.sourceItem
inventory.indexer.stock
media.content.synchronization
media.gallery.synchronization
async.operations.all (meant for a real queue system with third party integrations)
** option 3 - out of internal cron and manually run or individually in cron**
The command to run them individually in cron on on command line is
bin/magento queue:consumers:start --single-thread --max-messages=1 inventory.source.items.cleanup
Best Answer
One common use we have for Magento message queues is related to ERP integrations. For example: When a user places an order we want to transmit that data to a remote system. This transmission process over the network takes time that we don't want the user to wait.
Using a message queue allows us to decouple the ERP integration process from the customer order process. When Magento fires the sales_order_place_after event we just pass the order id (message) into the order export message queue (publisher) and let the order complete normally.
Now the message broker (RabbitMQ) will queue the order ids (messages) waiting for export until another process (consumer) to take an order id (message) and export that order. With the consumer being a background process there is more flexibility in running time and error handling.
It really took me playing with RabbitMQ to connect all the dots but hopefully, this is helpful.