Robots.txt

[mark_williams1]

I've noticed that /catalog/product/view/ has been disallowed.

Doesn't this mean that search engines won't index my products?

[Manthan Dave · Accepted solution]

Hi @mark_williams1

Disallow: /catalog/product/view/

It doesn't mean search engine not index your product , But this is for not allowing the direct url !

Meaning  your url of the product should be a user friendly and SEO friendly , not /catalog/product/view/12 (product_Id) , instead of this it should be like test.html etc

So answer here is no - if this line is there your product gets indexed with search engine !

Hope it helps 

[mark_williams1]

Thanks for your super fast response, ManthanDave.

[Manthan Dave]

Hi @mark_williams1

Glad to know that you have resolved your issue !

Happy to help and keep helping

[josh_bachynski]

No they will index it, they just can't show it properly for searches because they cannot read what is on the page. If you have content there you want to rank, then you want to allow google to see it!

[mark_williams1]

Hi all,

Thanks for the help so far.  Just looking at this again.

According to Google Search Console, 138 pages are being indexed but 10,998 are being blocked by robots!

Help!

[mark_williams1]

Anything?

[Damian Culotta]

Hi @mark_williams1,

Can you share your robots.txt file/url?

[mark_williams1]

Hi Damian,

Thank you for your reply.  Robots.txt below: