I have something strange going on with my site, if you visit it with the same user-agent strong as googlebot, it 302 redirects to itself. You can see this in the screen shot you provided and also using a bit simulator like this one:
https://redbot.org/?uri=https%3A%2F%2Fknitandpearls.com%2F&req_hdr=User-Agent%3AMozilla%2F5.0+%28com...
How to fix it this issue...
Hi @sudhir_singh2,
I'm not sure but I guess I will follow this: https://varvy.com/robottxt.html
If you want to allow everything you should use:
User-agent: * Disallow:
Or maybe even better, you can use a Magento's robots.txt file: https://github.com/Creare/magento-robots/blob/master/robots.txt
Hi Damian,
Thanks for give it some time with this.
We already checked (https://knitandpearls.com/robots.txt) , but it's fine.
I checked htaccess file, magento configuration but still can't able to find the bug.
