After upgrading my site from Magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as https://www.example.com/cms/index/noCookies/ and https://www.example.com/wishlist/index/add/product/101/form_key/QRh31BjtGTfy2Ur9/
Google and Bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.
I have added the following command to robots.txt a long time ago.
Disallow: /cms/
Disallow: /wishlist/
But this seems to be useless.
When my site running on Magento 1.938, I never found out that Google and bing spiders would crawl these URLs in “Online Customers" tab.
Why is this happening, how to prohibit the crawl of these URLs?
I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt
robots.txt Tester showed these two URLs had been blocked.