Magento – Magento 2.3.2 CE and Cron jobs, flock and RabbitMQ

cronmagento-cronmagento2.3.2message-queue

Last few days I was chasing an interesting bug in production system running Magento 2.3.2 CE
Where as per community recommendation I run cron jobs like this

flock --timeout=0 /run/lock/magento_system_cron.lock  /usr/bin/php /var/www/html/bin/magento cron:run

After few executions of cron task it is suddenly gets locked and flock never releases this lock. The only way to release it is to either remove the lock file (which is incorrect, don't do it) or figure out related process IDs and kill them. One can figure out process IDs like this:

fuser -u /run/lock/magento_system_cron.lock

Once you know process IDs – do something like ps aux | grep <process-id-from-fuser-command>

Well, when you see processes that are involved there you would probably understand why it gets locked forever. Those processes are message consumers and they are supposed to sit and wait (they are actually blocked and effectively sleeping) till they receive a message via the message-queue mechanism and then process them.

Magento documentation is somewhat vague on this:

At the same time I noticed that cron jobs get blocked once I see this in logs:

main.INFO: Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp []

And googling for that was always leading me to the tip to disable the cron configuration for those message consumers, which didn't sound right to me:

// app/etc/env.php
// ...
'cron_consumers_runner' => array(
    'cron_run' => false,
    'max_messages' => 1000,
    'consumers' => array(
        'async.operations.all',
    )
)

See this for instance:
https://community.magento.com/t5/Magento-2-x-Admin-Configuration/Consumer-async-operations-all-skipped-as-required-connection/td-p/129621

Related questions:

Somehow it became clear to me that something is not right with that message-queue setup as well with cron configuration. So I was looking through Magento docs multiple times and were not finding there what is supposed to consume messages that magento pushes into rabbitMQ, see this for example: https://www.rabbitmq.com/tutorials/tutorial-one-php.html

The question that I was asking myself all the time: "What am I missing here?" and "What is a recommended setup here?"

Best Answer

Flock not needed since Magento 2.3.2, because Magento already has own lock mechanism.
See https://devdocs.magento.com/guides/v2.3/install-gde/install/cli/install-cli-subcommands-lock.html#instgde-cli-lockconfig

Related Topic