Magento very slow on batch updates

[xleybagmaic583]

Hi

We are using Magento 2.4 and are having a big issue on performance.

Our products are extracted from an ERP and sent to Magento as a kind of batch process where another application extracts the products from the ERP and send them to Magento by calling Magento´s end points.

Magento is then receiving a lot of data (more than 1K products) and then the performance become very slow and the CPU usage is 90%.

We have Magento deployed in AWS using a RDS MySQL.
We thought the problem could be slow writes due to the general purpose SSD and some MYSQL log settings but then realised that the write times are not bad but the CPU usage is very high.

We are assuming that the root cause of the issue is the Magento reindex process on each product update (it would explain the high CPU usage) but it is just a guessing as we are not sure.

Then question is: 

- could the reindexing process be the source of our issue. If so, how could we fix it?
- also: is there a recommended way to update a lot of data in Magento?

Thanks in advance.

[theMageComp]

Hello @xleybagmaic583 

1. By default, Magento triggers automatic indexing after each product update. This can cause performance degradation when dealing with a high volume of updates. Instead, you can disable automatic indexing during the data import process and manually trigger the reindexing once the data import is complete. You can disable automatic indexing in the Magento Admin under Stores > Configuration > Catalog > Catalog > Catalog Search.

2. Magento provides two indexing modes, "Update on Save" and "Update by Schedule." The "Update on Save" mode triggers indexing immediately after each product update, while "Update by Schedule" batches the updates and performs indexing at scheduled intervals. Consider switching to the "Update by Schedule" mode to reduce the frequency of indexing and lessen the impact on performance. You can configure the indexing mode in the Magento Admin under Stores > Configuration > Advanced > System > Full Page Cache.

3. Magento 2.4 introduced the option for asynchronous reindexing, which offloads the reindexing process to separate background workers, reducing the impact on the main server. Asynchronous indexing can significantly improve performance when dealing with large amounts of data. You can enable asynchronous indexing in the Magento Admin under Stores > Configuration > Advanced > System > Asynchronous indexing.

4. Ensure that your AWS server resources are properly allocated and optimized for Magento. Pay attention to CPU, memory, and disk performance. Consider using instance types suitable for high-performance applications and optimizing MySQL database configurations. Reviewing AWS documentation and consulting with AWS experts can help you optimize your server setup.

5. Depending on the nature of your data import process, you may consider different strategies to improve performance. For example, you could batch the product updates into smaller chunks instead of sending all the data in a single request. This can help distribute the load and reduce the strain on the server. Additionally, optimizing the data import process itself can have a significant impact. Review your data import scripts or custom integrations to ensure they are efficient and minimize unnecessary database queries or operations.

[xleybagmaic583]

Thanks for your answer.
I will try to apply your recommendations.