Search time is too long across hundreds millions products

[sergheitopf5c5]

We imported ~300M configurable digital products (~900M products totally) & noticed that search time is more than 20 seconds & sometimes we get "Too long request" error. I separately did a search queries to ElasticSearch & MySQL. Everything works fast. There is absolutely no stress. Probably there are some problems with default search settings? Would be very glad if somebody suggest how to fix that. Thanks in advance

[toniagency82ed]

900M SKUs is a lot. What are you using specifically for your site search engines? Defualt implementation of Elastic? Which version of Magento? What kind of infrastructure are you running this installation on?

[sergheitopf5c5]

Default settings on ElasticSearch

Magento 2.4.6
DB Container: 126 GB DDR4 Memory, 32 CPU Cores

ES Container: 200 GB DDR4 Memory, 32 CPU Cores

Web Server Container: 64 Gb DDR4 Memory, 8 CPU Cores

By Idea when we do search first ElasticSearch returns IDs & then DB selection will be made by provided IDs

Query curl localhost:9200/<INDEX>/_search?q=New+York+skyline+evening on production goes 0.5 secs across 300M documents

[Milople]

Hello @sergheitopf5c5,

Given the scale of your product catalog, Magento's default search and indexing settings may not be optimized for handling such large volumes.

Optimize Elasticsearch Settings in Magento

• Increase Elasticsearch Query Timeout: Go to Stores > Configuration > Catalog > Catalog Search, and increase the Elasticsearch Server Timeout to prevent timeouts.
• Adjust Elasticsearch Search Configuration:
• Enable Quick Search Suggestions and Recommendations: This can limit the load by suggesting common queries instead of processing all search results.
• Use Faster Sorting Options: By limiting sorting options to simple fields, such as relevance, you can improve response times.

Leverage Elasticsearch Indexing Optimization

• Use Nested Fields for Attributes: For large catalogs, mapping complex attributes as nested fields can help avoid performance bottlenecks.
• Optimize Index Refresh and Merge Policies:
• Set the index refresh interval to a larger value (e.g., 30s or 60s) to reduce the load on Elasticsearch during indexing operations.
• Adjust merge policies to minimize the number of segments, improving search performance at scale.

 Use Magento's Flat Catalog for Products (if compatible)

• For this volume of products, Magento’s Flat Catalog option (if applicable to your version) can speed up catalog queries by reducing joins.

Scale Up Elasticsearch Hardware or Cluster

• Elasticsearch often benefits from more nodes or more powerful nodes, particularly with high product volumes. Consider adding nodes or increasing CPU if your current hardware is not optimized for the scale.

Consider Custom Search Modules

• Explore custom search modules or search configurations that can simplify queries or improve response times, particularly for very high-volume catalogs.

Testing with these adjustments can help identify if configuration changes alone can reduce your search times. If problems persist, consider segmenting the catalog or indexing strategy to distribute load across different instances, you can also check out our Magento 2 store, we have a handful of search extensions that may benefit you. Just simply search Milople Magento 2 Extension.

If the issue is resolved, click Kudos and accept it as a solution.