Since we added message queues and Asynchronous / Bulk API became in Magento Open Source, our great community has provided many new improvements and bugfixes. This article provides details about one of the improvements for Magento scope-related APIs and a patch fix for an issue with specific stores.
The following examples outline an issue with scope APIs that do not provide the information needed to update or create data for specific stores. Patches are available to resolve this issue for current and new Magento release, with Git and Composer based installations.
Affected Magento versions
The issue affects the following Magento versions (on prem and cloud):
- Magento Open Source v2.3.2, 2.3.1
- Magento Commerce v2.3.2, 2.3.1
- Magento Commerce v2.3.2, 2.3.1
Patch information
The patch name for all versions and editions is “Scope parameter for Async/Bulk API patch”. For complete information on the solution, continue reading.
How to apply the patch
A patch is available from the Magento Download page for Magento Open Source, or through your Magento account for Magento Commerce. Locate the patch by the name. We provide Git and Composer based patches. See Applying patches on DevDocs for additional instructions.
For Magento Commerce Cloud, download the appropriate Magento Commerce version patch and see Apply custom patches.
Current workflow
The current Magento message queue implementation does not take into account the store that executes queue operations. The Bulk APIs are currently the largest users of message queues. The following information details how they work together.
Queue and Stores
Standard Magento REST APIs work as follows: Before executing the required Service Contract, Magento uses the request URL to detect which store has to be initialized. Then the system initializes the correct store and executes the required Service Contract. In this workflow, all data for the requested operation will be updated accordingly for the requested store.
But asynchronous implementation executes in another way:
- The user sends an API request to Async/Bulk endpoints. For example:
POST {{URL}}/rest/async/bulk/V1/products [{ "product": { "sku": "sku", "name": "Simple Product", "attribute_set_id": 4, "price": "1.1", "status": 1, "visibility": 4, "type_id": "simple", }]
- Magento detects the Bulk request based on the endpoint URL, async/bulk and packs the message that will be sent to the message queue (RabbitMQ)
Magento creates a message in the
app/code/Magento/AsynchronousOperations/Model/MassPublisher.php:method. The message looks like this:
ublish($topicName, $data)$envelopes = {array} [1] 0 = {Magento\Framework\MessageQueue\Envelope} [2] properties = {array} [2] delivery_mode = 2 message_id = "c015358f897e54f6cd01a396904e6575f5b100e752c7fd55fa6e5ec075d9b081" body = "{"id":3,"bulk_uuid":"62225484-b31a-421f-aa9c-20ef3522253d", "topic_name":"async.magento.catalog.api.productrepositoryinterface.save.post", "serialized_data":"{ … }","result_serialized_data":null, "status":4,"result_message":null,"error_code":null}"
- After the body is generated, Magento sends a message to RabbitMQ. At this step, the operation to create a new message has completed. The API returns the UUID of the operation so the user can track the status later on.
- Finally, the consumer listens to the async queue and processes messages from it.
Issue
Based on this example, the message to the queuing system does not contain information about the store to apply the request to. This causes an issue: the end-user cannot use the Async/Bulk API to update or create data for a specific store. Service contracts will always behave like there is no store code transferred, and they will always update values for the default store code as a fallback.
Solution
To solve this problem, we needed a method to transfer the store code to the message queue and process it through the consumer. This solution must be backward-compatible, as a solution will be released in a Magento Patch release.
comwrap provided the fix for this issue in PR102 while working on an Asynchronous Import community project. This PR became the patch “Scope parameter for Async/Bulk API”. To install the fix, see the How to apply the patch section.
How we created this patch
The following information provides detailed information regarding this fix.
Added to a new Magento module
Using this solution, we created a new Magento Module named magento/module-amqp-store.
Why did we create a new module? Magento follows a Service Isolation approach. In the current state of the message queue implementation, this module has no dependency on Magento Stores at all. To avoid adding new dependencies to the core modules, we need to find another way, and a separate extension is a great approach for it.
An extension provides a set of plugins to add stores support for message queue, and at the same time it does not add new dependencies. As a result, if you do not need stores support, you can remove this extension from your installation.
Transfered the store to the message queue
At first glance, the easier way to add store information to the queue is to extend the message body with a new parameter, but that approach is not backward-compatible.
Also, each message in the queue has a properties list, containing “application_headers”, which is an instance of the AMQPTable class. We add one more application header named “store_id”, which holds the store id for which this message has to be processed.
Plugin implementation:
app/code/Magento/AmqpStore/Plugin/Framework/Amqp/Bulk/Exchange.php
Extended the queue consumer
Finally, we have to teach our consumer what the store_id param is doing and how to use it. The new plugin does this, and it executes together with the consumer. When we have store_id in the message headers, we set the correct scope for the current consumer process.
Plugin implementation:
app/code/Magento/AmqpStore/Plugin/AsynchronousOperations/MassConsumerEnvelopeCallback.php
Future Releases
We provide patches for listed affected Magento versions. The solution will be available in the Magento 2.3.3 release.
Scopes are one of the key features of the Magento infrastructure, and the lack of such support in the Magento Message Queue implementation is a flaw in the framework implementation.
The patch adds support for Magento scopes for the whole Magento Message Queue implementation, which operates with RabbitMQ. Magento also has the DB queue system as a fallback with the same issue. In the context of the Magento Community project, we are working to move forward and provide support for DB queues for Magento Bulk API. When we achieve this, customers will be able to use scopes in the DB queue system.
Join us and Contribute
Interested in contribution and the Asynchronous project and the Magento API? Join us and learn more:
- GitHub: https://github.com/magento/async-import/
- Tickets: ZenHub & Backlog
- Chat: Slack (self signup)
- DevDocs: Bulk API & Async API
1 Comment