Indexers parallelization and optimization

[lkrell]

Starting with the 2.2.6 release, Magento indexers are scoped and multi-threaded and support reindexing in parallel mode. To see the benefits, this post provides our collected results of index operation optimization and best practices.

Performance Tracking

For this data, we tracked performance against a Magento 2.2.6 implementation with a large profile in the Magento Commerce Cloud environment (Pro 48) with ElasticSearch 5.2 as the full-text search engine with Advanced Search enabled.

Index parallelization can affect only the indexers which are scoped, which means Magento split the data into multiple tables by indexer as its scope instead of keeping all data in one table.

• Catalog Search Fulltext is paralleled by store views.
• Category Product is paralleled by store views.
• Product Price can be paralleled by websites, customer groups, websites and customer groups by command:
  bin/magento indexer:set-dimensions-mode [<indexer>] [<mode>]
  Where [<indexer>] is “catalog_product_price” and [<mode>] can be one of the following:
  • website
  • website_and_customer_group
  • customer_group
  • none (default)

The main reason that the Product Price indexer scope is optional is that the total count of scopes (or dimensions counts) can be huge. When this is the case, adding a scope may not yield any difference, and may even degrade the Product Price index process.

Best Practices

We recommend the following best practices for Product Price indexer dimensions. 

• Do not set any dimension mode for Product Price indexer if you don't use parallelization.
• If the Product Price indexer set-dimensions-mode is set to website_and_customer_group, the total count of dimensions is (websites * customer groups). For example, if you have 5 websites and 3 customer groups, a total count of dimensions will be 15 (or 3*5=15).
• We recommend fitting your threads count in one per core, and not less than one per dimension.
• Do not change the set-dimensions-mode for Product Price indexer with a huge amount of websites and customer groups if you cannot give the index process an equal amount of threads according to the total count of Product Price index dimensions. This can affect performance.

Indexers Performance Graph

This graph provides an index comparison according to Product Price indexer mode and threads count.

 

Consider the following when reviewing the comparison:

• The values represent only indexers that can be affected by parallelization.
• The environment could efficiently receive no more than 10 threads.
• Dimension info only affects the Product Price indexer.
• The Magento instance had 5 websites, 4 customer groups, and 5 store views using the large profile.

---

Thanks to Vitalii Zabaznov (@vzabaznov) for this blog post.