cancel
Showing results for 
Search instead for 
Did you mean: 

robots.txt is empty

robots.txt is empty

I believe I have properly edited the settings under Content->Design->Configuration for my sites robots.txt but when I try to pull the url I'm getting a blank file.

 

Search Engine Robots:

 
Default Robots :

INDEX, FOLLOW

 

Edit custom instruction of robots.txt File
User-agent: *
Disallow: /index.php/
Disallow: /*?
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Sitemap: https://www.mysite.com/sitemap.xml

 

 

1 REPLY

Re: robots.txt is empty

Hi @treii28,


I don't know if could be related but in previous versions of Magento 2 there was an issue the the robots.txt file and the location (I don't remember all the issues).

Can you share a public URL of your store?

Is the robots.txt file placed at the magento root directory or into path/to/magento/pub ?

--
If you've found one of my answers useful, please give "Kudos" or "Accept as Solution"