REST API page size & current page search criteria options return item(s) past end of data set

[apiuser]

Feature request from careys7, posted on GitHub Jan 10, 2017

Preconditions

1. Magento 2.0.0 - 2.1.3 (Community or Enterprise) with Sample Data installed
2. Environment using magento2-docker-compose

Steps to reproduce

This issue appears to affect at least /products/ and /categories/ requests. I have added below an example for products.

1. Make API request for products specifying page size as 1, current page as 99999, eg:

GET /rest/default/V1/products/?searchCriteria[pageSize]=1&searchCriteria[currentPage]=9999 
HTTP/1.1
Host: m2.localhost:8000
Authorization: OAuth oauth_consumer_key="sitoq7tu4b1aj7kikj4irkog8tggh1ch",oauth_token="kr3bly2kbbnrr9flyws9r4mxdaagxngo",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1484095699",oauth_nonce="TivkWr",oauth_version="1.0",oauth_signature="Q02C2wk4AgkDPkYAbACUoLMoXa4%3D"
Cache-Control: no-cache

Note: Page size of '1' above is a nominal value to make examples given below more concise.

Expected result

1. No items returned in response if (current page * page size) > total products, eg:

{
  "items": [],
  "search_criteria": {
    "filter_groups": [],
    "page_size": 1,
    "current_page": 99999
  },
  "total_count": 2048
}

Actual result

1. The product with highest id is returned, eg:

{
  "items": [
    {
      "id": 2048,
      "sku": "24-WG085_Group",
      "name": "Set of Sprite Yoga Straps",
      "attribute_set_id": 12,
      "status": 1,
      "visibility": 4,
      "type_id": "grouped",
      "created_at": "2016-11-13 21:24:31",
      "updated_at": "2016-11-13 21:24:31",
      "extension_attributes": [],
      "product_links": [
        {
          "sku": "24-WG085_Group",
          "link_type": "associated",
          "linked_product_sku": "24-WG085",
          "linked_product_type": "simple",
          "position": 0,
          "extension_attributes": {
            "qty": 0
          }
        },
        {
          "sku": "24-WG085_Group",
          "link_type": "associated",
          "linked_product_sku": "24-WG086",
          "linked_product_type": "simple",
          "position": 1,
          "extension_attributes": {
            "qty": 0
          }
        },
        {
          "sku": "24-WG085_Group",
          "link_type": "associated",
          "linked_product_sku": "24-WG087",
          "linked_product_type": "simple",
          "position": 2,
          "extension_attributes": {
            "qty": 0
          }
        }
      ],
      "tier_prices": [],
      "custom_attributes": [
        {
          "attribute_code": "description",
          "value": "<p>Great set of Sprite Yoga Straps for every stretch and hold you need. There are three straps in this set: 6', 8' and 10'.</p>\n<ul>\n<li> 100% soft and durable cotton.\n<li> Plastic cinch buckle is easy to use.\n<li> Choice of three natural colors made from phthalate and heavy metal free dyes.\n</ul>"
        },
        {
          "attribute_code": "image",
          "value": "/l/u/luma-yoga-strap-set.jpg"
        },
        {
          "attribute_code": "small_image",
          "value": "/l/u/luma-yoga-strap-set.jpg"
        },
        {
          "attribute_code": "thumbnail",
          "value": "/l/u/luma-yoga-strap-set.jpg"
        },
        {
          "attribute_code": "options_container",
          "value": "container2"
        },
        {
          "attribute_code": "required_options",
          "value": "0"
        },
        {
          "attribute_code": "has_options",
          "value": "0"
        },
        {
          "attribute_code": "url_key",
          "value": "set-of-sprite-yoga-straps"
        },
        {
          "attribute_code": "is_returnable",
          "value": "2"
        },
        {
          "attribute_code": "activity",
          "value": "17"
        },
        {
          "attribute_code": "material",
          "value": "41,53"
        },
        {
          "attribute_code": "gender",
          "value": "89,90,93"
        },
        {
          "attribute_code": "category_gear",
          "value": "96"
        },
        {
          "attribute_code": "size",
          "value": "100"
        }
      ]
    }
  ],
  "search_criteria": {
    "filter_groups": [],
    "page_size": 1,
    "current_page": 99999
  },
  "total_count": 2048
}

[apiuser]

Comment from veloraven, posted on GitHub Jan 11, 2017

@careysizer thank you for your report. EE issues should be reported via the Support portal of your account or Partner portal if you are a partner reporting on behalf of a merchant. Github is intended for Community edition reports given no account management for CE users. This will allow for proper tracking of issues at the account level. Please let us know if you already have reported this issue via some portal in order to avoid duplicates in our internal tracking system.

[apiuser]

Comment from careys7, posted on GitHub Jan 11, 2017

@veloraven - thanks, I've updated the issue to specify that CE is also affected (appears to be all versions of both editions that I've checked).

I haven't reported this issue through the partner portal or other Magento channels.

[apiuser]

Comment from mbrinton01, posted on GitHub Jan 12, 2017

@misha-kotov please review MAGETWO-58841 before adding comment, this seems to be the same issue.

[apiuser]

Comment from fvschie, posted on GitHub Jan 25, 2017

The workaround we currently use is to check whether the last item on the previous page is identical to the last item on the current page.

[apiuser]

Comment from misha-kotov, posted on GitHub Jan 27, 2017

Verified and the internal issue is MAGETWO-58841

[apiuser]

Comment from nekobul, posted on GitHub Jun 21, 2017

This appears a very substantial bug and there is still no resolution in the latest 2.1.7 version. Do you have estimate when you will fix this issue?

[apiuser]

Comment from magento-engcom-team, posted on GitHub Oct 10, 2017

@careys7, thank you for your report. We've created internal ticket(s) MAGETWO-58841 to track progress on the issue.

[apiuser]

Comment from stevethedev, posted on GitHub Nov 28, 2017

The root of this problem seems to live in the file vendor/magento/framework/Data/Collection.php:

    /**
     * Get current collection page
     *
     * @param  int  $displacement
     * @return int
     */
    public function getCurPage($displacement = 0)
    {
        if ($this->_curPage + $displacement < 1) {
            return 1;
        } elseif ($this->_curPage + $displacement > $this->getLastPageNumber()) {
            return $this->getLastPageNumber();
        } else {
            return $this->_curPage + $displacement;
        }
    }

The upper-limit overflow can be addressed here, but the lower-limit overflow (page <= 0) is handled by vendor/magento/zendframework1/library/Zend/Db/Select.php:

    /**
     * Sets the limit and count by page number.
     *
     * @param int $page Limit results to this page number.
     * @param int $rowCount Use this many rows per page.
     * @return Zend_Db_Select This Zend_Db_Select object.
     */
    public function limitPage($page, $rowCount)
    {
        $page     = ($page > 0)     ? $page     : 1;  //<---- This line right here
        $rowCount = ($rowCount > 0) ? $rowCount : 1;
        $this->_parts[self::LIMIT_COUNT]  = (int) $rowCount;
        $this->_parts[self::LIMIT_OFFSET] = (int) $rowCount * ($page - 1);
        return $this;
    }

Lower-limit overflow is easy to detect (page <= 0), but upper-limit overflow is not.

I have submitted a Pull Request at: https://github.com/magento/magento2/pull/12475

[apiuser]

Comment from orlangur, posted on GitHub Nov 29, 2017

The workaround we currently use is to check whether the last item on the previous page is identical to the last item on the current page.

@fvschie @nekobul

From the issue description I see that API returns total amount of records: "total_count": 2048.

Could you please add more details why you need to access invalid page at all? It seems like there is some bug with last page determination in your code but why not fix invalid pagination instead of hacking core?