cancel
Showing results for 
Search instead for 
Did you mean: 

Indexers parallelization and optimization

lkrell
New Contributor

Starting with the 2.2.6 release, Magento indexers are scoped and multi-threaded and support reindexing in parallel mode. To see the benefits, this post provides our collected results of index operation optimization and best practices.

 

Performance Tracking

 

For this data, we tracked performance against a Magento 2.2.6 implementation with a large profile in the Magento Commerce Cloud environment (Pro 48) with ElasticSearch 5.2 as the full-text search engine with Advanced Search enabled.

 

Index parallelization can affect only the indexers which are scoped, which means Magento split the data into multiple tables by indexer as its scope instead of keeping all data in one table.

 

  • Catalog Search Fulltext is paralleled by store views.
  • Category Product is paralleled by store views.
  • Product Price can be paralleled by websites, customer groups, websites and customer groups by command:
    bin/magento indexer:set-dimensions-mode [<indexer>] [<mode>]
    Where [<indexer>] is “catalog_product_price” and [<mode>] can be one of the following:
    • website
    • website_and_customer_group
    • customer_group
    • none (default)

 

The main reason that the Product Price indexer scope is optional is that the total count of scopes (or dimensions counts) can be huge. When this is the case, adding a scope may not yield any difference, and may even degrade the Product Price index process.

 

Best Practices

 

We recommend the following best practices for Product Price indexer dimensions. 

 

  • Do not set any dimension mode for Product Price indexer if you don't use parallelization.
  • If the Product Price indexer set-dimensions-mode is set to website_and_customer_group, the total count of dimensions is (websites * customer groups). For example, if you have 5 websites and 3 customer groups, a total count of dimensions will be 15 (or 3*5=15).
  • We recommend fitting your threads count in one per core, and not less than one per dimension.
  • Do not change the set-dimensions-mode for Product Price indexer with a huge amount of websites and customer groups if you cannot give the index process an equal amount of threads according to the total count of Product Price index dimensions. This can affect performance.

 

Indexers Performance Graph

 

This graph provides an index comparison according to Product Price indexer mode and threads count.

 indexers.png

 

Consider the following when reviewing the comparison:

 

  • The values represent only indexers that can be affected by parallelization.
  • The environment could efficiently receive no more than 10 threads.
  • Dimension info only affects the Product Price indexer.
  • The Magento instance had 5 websites, 4 customer groups, and 5 store views using the large profile.

Thanks to Vitalii Zabaznov (@vzabaznov) for this blog post.

2 Comments
marcin_szterling
M2 Certified

Can someone test it with the real data? Sample data is not representative enough to prove the point.

Dava Gordon
New Contributor

After testing this on a production website found that it did not give the required result, did not work with elastic search which we found was struggling with the price index ended up switching elastic search off temporarily and the index run correctly.