Hi Guys
I used the Magento default Robot.txt generator in Magent 2
But when I look at the end result it listing the file like this:
User-agent: * Disallow: /index.php/ Disallow: /*? Disallow: /checkout/ Disallow: /app/ Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ Disallow: /catalog/ Disallow: /customer/ Disallow: /sendfriend/ Disallow: /review/ Disallow: /*SID=
and not
User-agent: * Disallow: /index.php/ Disallow: /*? Disallow: /checkout/ Disallow: /app/ Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ Disallow: /catalog/ Disallow: /customer/ Disallow: /sendfriend/ Disallow: /review/ Disallow: /*SID=
Does it matter which way it is appearing? I believe it does as in my search console I'm getting soft 403 errors as the robots are trying to index /catalog etc etc
when you at your own robot.txt file how is it looking?
Also another question where is the robot.txt file kept?
I can't see it in my root director
The correct layout for the robots.txt file is as follows
User-agent: [User-Agent Name] Disallow: [URL Path]