Magento – Message queue configuration changes in Magento 2 EE 2.1

magento-enterprisemagento2message-queue

I've been looking at the EE message queues functionality, but it seems like it is incomplete.

Prior to 2.1 there was a reasonably functional implementation that allowed you to create a etc/queue.xml to define publishers, topics, consumers and queues as described in the official documentation: http://devdocs.magento.com/guides/v2.0/config-guide/mq/config-mq.html.

There were limitations such as not been able to create bindings unless they matched a topic defined in configuration, which limited the flexibility as you had to define all possible topic possibilities up front. The lack of a recurring install script also meant you had to do a bit of hacking to get the installer script to run again.

As of 2.1 the publisher, topic, consumer and bind elements in etc/queue.xml have been deprecated and the configuration is split between etc/queue.xml and etc/communication.xml, as can be seen here: https://github.com/magento/magento2-samples/blob/master/sample-module-sample-message-queue/etc/. The deprecated publisher/topic/consumer/bind schema can still be used in isolation, but not in conjunction with the revised broker/queue schema.

However, none of this has appeared in the official documentation and its not immediately clear why the configuration has been split and requires duplication in some cases. More importantly, there doesn't seem to be a facility to define binding now, with the topic name used as the routing key instead. This in turn also makes it impossible to use special characters for binding queues. So it appears to have been refactored but lost functionality.

On a positive note, the magento/module-amqp module now uses a recurring installer script so the queue config changes are installed when you run magento setup:upgrade. This change has not been applied to the magento/module-mysql-mq module yet though.

So I'd like to know:
a) Have I got this all wrong and there is actually way to create bindings and it is more flexible than it seems?
b) Why has the configuration been split?

As a side note, as I've been experimenting with this I've been using one of the topology examples from the RabbitMQ tutorials at https://www.rabbitmq.com/tutorials/tutorial-four-php.html:

enter image description here

This deprecated configuration achieved the topology for the most part:

queue.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue.xsd">
    <topic name="quick.orange.rabbit" schema="Example\MessageQueue\Api\MessageInterface" publisher="default" />
    <topic name="quick.orange.fox" schema="Example\MessageQueue\Api\MessageInterface" publisher="default" />
    <topic name="lazy.pink.rabbit" schema="Example\MessageQueue\Api\MessageInterface" publisher="default" />
    <topic name="lazy.orange.elephant" schema="Example\MessageQueue\Api\MessageInterface" publisher="default" />
    <topic name="lazy.brown.fox" schema="Example\MessageQueue\Api\MessageInterface" publisher="default" />
    <consumer name="consumerOne" queue="queueOne" connection="amqp" class="Example\MessageQueue\Model\Subscriber" method="processMessage" executor="Magento\Framework\MessageQueue\BatchConsumer" />
    <consumer name="consumerTwo" queue="queueTwo" connection="amqp" class="Example\MessageQueue\Model\Subscriber" method="processMessage" executor="Magento\Framework\MessageQueue\BatchConsumer" />
    <bind queue="queueOne" exchange="magento" topic="*.orange.*" />
    <bind queue="queueTwo" exchange="magento" topic="*.*.rabbit"  />
    <bind queue="queueTwo" exchange="magento" topic="lazy.#" />
</config>

UPDATE:
The documentation has now been updated. Wildcards are now not supported, so the flexibility of a topic exchange is rendered null. So I tried to recreate the following direct exchange:

enter image description here

communication.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="orange" request="Example\MessageQueueExample\Api\MessageInterface" />
    <topic name="black" request="Example\MessageQueueExample\Api\MessageInterface" />
    <topic name="green" request="Example\MessageQueueExample\Api\MessageInterface" />
</config>

queue.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue.xsd">
    <broker topic="orange" type="amqp" exchange="magento">
        <queue consumer="consumerOne" name="queueOne" handler="Example\MessageQueueExample\Model\Subscriber::processMessage" consumerInstance="Magento\Framework\MessageQueue\Consumer"/>
    </broker>
    <broker topic="black" type="amqp" exchange="magento">
        <queue consumer="consumerTwo" name="queueTwo" handler="Example\MessageQueueExample\Model\Subscriber::processMessage" consumerInstance="Magento\Framework\MessageQueue\Consumer"/>
    </broker>
    <broker topic="green" type="amqp" exchange="magento">
        <queue consumer="consumerTwo" name="queueThree" handler="Example\MessageQueueExample\Model\Subscriber::processMessage" consumerInstance="Magento\Framework\MessageQueue\Consumer"/>
    </broker>
</config>

However, when you run the consumers, only the "green" topic is routed to consumerTwo, it ignores the "black" topic. So it seems that the best that can be accomplished is a direct exchange with only one binding per queue and consumer.

Related Topic