cancel
Showing results for 
Search instead for 
Did you mean: 

ZooKeeper support to lock Message queue messages

0 Kudos

ZooKeeper support to lock Message queue messages

We used Magento Queue support extensively in our project, we even created a connector to use Amazon SQS instead of RabbitMQ (I intend to share it with the community soon).

 

During a process of improving the performance of our consumer jobs, I found that the native  message queue module uses MySQL to manage the lock of messages, which ends up competing with the IO of the other calls to the database. As we have a very high volume of messages processed per minute (+ 15k messages), the database ends up being unnecessarily overloaded with this native lock approach.

 

As Magento already provides native support for using ZooKeeper to control the locking of cronjobs processes, it would be very interesting if this feature were also made available to Message Queue, reducing the unnecessary usage of MySQL database.