Elasticsearch error in cron: 429 Too Many Requests

[danielunif1eef]

I'm setting up our environment on AWS and have noticed that there are error messages coming from Elastic Search in the indexer_reindex_all_invalid tasks.

The error looks like this:

[2022-10-28T06:01:09.250871+00:00] report.ERROR: Cron Job indexer_reindex_all_invalid has an error: 429 Too Many Requests /b2b_product_1_v4/document/_bulk. Statistics: {"sum":0,"count":1,"realmem":0,"emalloc":0,"realmem_start":260571136,"emalloc_start":241446712} [] []
[2022-10-28T06:01:09.251907+00:00] report.CRITICAL: Elasticsearch\Common\Exceptions\BadRequest400Exception: 429 Too Many Requests /b2b_product_1_v4/document/_bulk in /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:693
Stack trace:
#0 /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(333): Elasticsearch\Connections\Connection->process4xxError(Array, Array, Array)
#1 /var/app/current/vendor/react/promise/src/FulfilledPromise.php(28): Elasticsearch\Connections\Connection->Elasticsearch\Connections\{closure}(Array)
#2 /var/app/current/vendor/ezimuel/ringphp/src/Future/CompletedFutureValue.php(55): React\Promise\FulfilledPromise->then(Object(Closure), NULL, NULL)
#3 /var/app/current/vendor/ezimuel/ringphp/src/Core.php(341): GuzzleHttp\Ring\Future\CompletedFutureValue->then(Object(Closure), NULL, NULL)
#4 /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(345): GuzzleHttp\Ring\Core::proxy(Object(GuzzleHttp\Ring\Future\CompletedFutureArray), Object(Closure))
#5 /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(241): Elasticsearch\Connections\Connection->Elasticsearch\Connections\{closure}(Array, Object(Elasticsearch\Connections\Connection), Object(Elasticsearch\Transport), Array)
#6 /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php(110): Elasticsearch\Connections\Connection->performRequest('POST', '/b2b_product_1_...', Array, '{"index":{"_id"...', Array, Object(Elasticsearch\Transport))
#7 /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php(1929): Elasticsearch\Transport->performRequest('POST', '/b2b_product_1_...', Array, '{"index":{"_id"...', Array)
#8 /var/app/current/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php(347): Elasticsearch\Client->performRequest(Object(Elasticsearch\Endpoints\Bulk))
#9 /var/app/current/vendor/magento/module-elasticsearch-7/Model/Client/Elasticsearch.php(173): Elasticsearch\Client->bulk(Array)
...

Is this something to be concerned about and why is Magento code overloading the Elastic Search service ?

[danielunif1eef]

We are getting the same error. Please let us know if you found any solution - there is definitely a problem when it's sending too many requests to Elastic Search.

[eduardojem0ba9]

Did you figure it out the problem? We have the same issue.

[ibtisamdev8aef]

The occurrence of a 429 error message indicates a bulk queue error, often referred to as es_rejected_execution_exception. AWS experts suggest that each node's bulk queue capacity can range from 50 to 200 requests, depending on the Elasticsearch version being used. Several variables can contribute to this error, including high values in instance metrics, limitations in data node instance types for search or write operations, and the status of active and queued threads.

To address this issue, AWS specialists recommend monitoring the ThreadpoolSearchQueue and ThreadpoolWriteQueue metrics in Amazon CloudWatch. These metrics can help track the status of queue threads. Additionally, one can check for search rejections using the Active and Queue thread information through a specific command, as well as examine write rejections in a similar manner.

Furthermore, it's important to note that there is a limit on the number of searches or writes that can be performed, depending on the thread pool definition and Elasticsearch version in use.