RabbitMQ <--> Magento 2 EE integration

[Talesh]

Hi Alan,

Thank you for sharing all the information on the development of Magento 2 so far, we completely appreciate the insight into progress as we led up to the launch of Magento 2. I am curious as to the capabilities of the RabbitMQ that currently exist in M2EE. This documentation page (http://devdocs.magento.com/guides/v2.0/install-gde/prereq/install-rabbitmq.html) is a little light on details from the Magento side of things.

Basically it doesn't help me gauge how much developer work will be required for Rabbit integrations out of the box in EE vs building it from scratch in CE. If the forum isn't the right place to discuss the details of EE implementations, lets get in touch or direct me to the correct person to discuss the existing functionality, I'm easy to find on twitter or email. 

@_Talesh

[akent99]

Hi. Back at work from a bit of time off - sorry for delay in responding. We have some RabbitMQ support in 2.0, but more is coming in 2.1. I might have some of the details not quite right here, but here goes regardless! ;-)

In EE 2.0 we have a magento CLI command loop that will loop receiving messages and processing them. So its a way of doing async processing of requests. We have a MySQL based implementation and a RabbitMQ implementation. MySQL is basically something polling a database table - nothing flash, but it works if you don't want to have RabbitMQ running.  (I cannot remember if its cron driven or not, so might be up to one minute delay in processing requests.) RabbitMQ is more efficient, more responsive, etc.  But the focus was on background job processing.

In EE 2.1, we are investigating adding support for in bound and out bound messaging via service contracts. That is, we can receive a JSON object over RabbitMQ, deserialize it, then hand over to a module to process. Just like REST and SOAP, we do all the deserialization etc. You just write the PHP interface according to the service contract rules, and write a bit of XML config to turn it on. We are also looking at out-bound messages in the same way - that is, you call a service contract method and behind the scenes we serialize it into JSON and send over RabbitMQ (and wait for a response if appropriate). This leverages the existing JSON serialization/deserialization code we have for REST, but makes it easy to send/receive messages between processes as well. This is for communications, not background processing. The same module code should be able to receive SOAP, REST, or RabbitMQ messages - with only config file changes to turn it on.

So probably my first question back to you would be to understand if you are trying to do some background processing, or do you want to do more sophisticated messaging between disjoint systems?

[Talesh]

The delay over the holidays was fine, I didn't really expect this to move along until the new year anyway. I am definitely looking at more sophisticated communication than just background job processing. However I am curious about the MySQL based implementation. From what I understand they guys on my new team are a little more familiar with the concept of formatting integration data in a data warehouse for asynchronous pickup by the processing application. 

We're still struggling with just getting an associate account for a start so I can take a look at the EE Rabbit integration. I'd be curious to see how the MySQL integration works also. 

[akent99 · Accepted solution]

Hi. In EE 2.0 we provide:

• PublisherInterface :: publish($topicName, $data) to publish a message to a queue (it serializes $data using the same rules as arguments for REST and service contracts)
• A CLI command to pull messages from a queue and pass to a module to process (it deserializes using the same rules as serialization).

We then provide I think three queue implementations controllable via di.xml

• Do it now - don't bother with adding to a queue, just go do the work immediately
• Queue via MySQL - publisher creates a record in a database; subscriber takes a record out of a database for processing
• Queue via RabbitMQ - send to RabbitMQ instead of a MySQL database table

The CLI can be run in a few different ways, such as invoke from cron periodically to see if any new jobs came in. RabbitMQ would be running continuously and so see new work to do more rapidly.

With both MySQL and RabbitMQ, due to concerns around resource leakage in PHP, you may process say 100 requests then restart the code. That way you avoid the overhead of restarting PHP so frequently, but reclaim any resources if PHP has a resource leak of any kind.

[Talesh]

This helps greately, but I still need to eventually get partnership status so we can load test these integrations. One step at a time. Thanks Alan.

[Inna Bezchastna]

Do we need to create exchanges, queues manually to RabbitMQ to work with this new functionality?

I was trying to test new 

ScalableInventory

module.

But no queues and messages appears after publish. 

[brobie]

If you don't initially tell Magento that you want to use message queues upon installation, it will not create the queues that you need:

http://magento.stackexchange.com/questions/94062/add-rabbitmq-to-existing-magent-2-ee-installation

I ended up creating an InstallData.php class in a module that called the install() function on the Topology class.  This created the queues.