cancel
Showing results for 
Search instead for 
Did you mean: 

What is the Correct Layout for Robot.txt

What is the Correct Layout for Robot.txt

Hi Guys

 

I used the Magento default Robot.txt generator in Magent 2

 

But when I look at the end result it listing the file like this:

 

User-agent: * Disallow: /index.php/ Disallow: /*? Disallow: /checkout/ Disallow: /app/ Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ Disallow: /catalog/ Disallow: /customer/ Disallow: /sendfriend/ Disallow: /review/ Disallow: /*SID=


 

and not

User-agent: *
Disallow: /index.php/
Disallow: /*?
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=


 Does it matter which way it is appearing? I believe it does as in my search console I'm getting soft 403 errors as the robots are trying to index /catalog etc etc 

 

when you at your own robot.txt file how is it looking?

2 REPLIES 2

Re: What is the Correct Layout for Robot.txt

Also another question where is the robot.txt file kept?

 

I can't see it in my root director

Re: What is the Correct Layout for Robot.txt

The correct layout for the robots.txt file is as follows

User-agent: [User-Agent Name]
Disallow: [URL Path]