On behalf of an integration with a client using Magento 2, we are fetching products from their Magento system using the /v1/products GET REST API call for version 2.4.4.
They have a large number of products (10,000+), so we are fetching them in page sizes of 250.
The first pages we get go through fine, they take a second or two. But as we go through the page numbers, starting from 1, the requests take longer and longer to complete, until we hit the default timeout we've set.
We can get all the products if we increase the timeout to stupid levels (10 minutes+) and the page size to e.g. 5000+. But isn't this defeating the purpose of the paging? What is the expected behaviour here?
It's as if, for every request, it's hitting a SQL DB and taking more and more records, without skipping ones not needed first?
Here's an example request, for page number 49, where it starts taking 2 minutes+:
URL decoded
https://********/V1/products?fields=items[id,sku,type_id]&searchCriteria[pageSize]=250&searchCriteria[currentPage]=49&searchCriteria[filter_groups][0][filters][0][field]=status&searchCriteria[filter_groups][0][filters][0][value]=1&searchCriteria[filter_groups][1][filters][0][field]=website_id&searchCriteria[filter_groups][1][filters][0][value]=3
