Magento Forums
Help
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

robots.txt is empty

‎04-24-2018 11:58 AM
‎04-24-2018 11:58 AM

robots.txt is empty

I believe I have properly edited the settings under Content->Design->Configuration for my sites robots.txt but when I try to pull the url I'm getting a blank file.

 

Search Engine Robots:

 
Default Robots :

INDEX, FOLLOW

 

Edit custom instruction of robots.txt File
User-agent: *
Disallow: /index.php/
Disallow: /*?
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Sitemap: https://www.mysite.com/sitemap.xml

 

 

Labels:
0 Kudos
Reply
1 REPLY 1
‎05-02-2018 10:26 PM
‎05-02-2018 10:26 PM

Re: robots.txt is empty

Hi @treii28,


I don't know if could be related but in previous versions of Magento 2 there was an issue the the robots.txt file and the location (I don't remember all the issues).

Can you share a public URL of your store?

Is the robots.txt file placed at the magento root directory or into path/to/magento/pub ?

Reply