cancel
Showing results for 
Search instead for 
Did you mean: 

robots.txt generates outside the 'pub' web root

robots.txt generates outside the 'pub' web root

Hi,

 

We are seeing an issue in Magento 2.0.0 - 2.0.4 where a robots.txt file is generated from Stores > Configuration > General Design > Search Engine Robots > Reset to default > Save Config.

 

Apart from the very strange UI of 'reset to default & Save' instead of 'Generate default robots.txt'...

 

This file is generated in <base path on disk>/robots.txt INSTEAD OF <base path on disk>/pub/robots.txt

 

Which means hitting <SITE URL>/robots.txt doesn't work out of the box.

 

Easy to workaround with a symlink from pub up a level, or a webserver location block, but doesn't work out of the box.

3 REPLIES

Re: robots.txt generates outside the 'pub' web root

It is normal, the robots.txt file should be located in the Magento 2 root folder: https://mage2.pro/t/1221

Re: robots.txt generates outside the 'pub' web root

Dmitry,

 

It's possible this works as expected under the Apache webserver, but as Magento2 advertises full NginX support, there is no provision within the sample included NginX configs to correctly serve the robots.txt from the $MAGE_ROOT where the robots.txt gets generated, as the actual NginX root is defined as 'root $MAGE_ROOT/pub;'.

 

So if I symlink from /pub up one level it will work, but not otherwise - unless I add an additional location block to match it.

Re: robots.txt generates outside the 'pub' web root

Maybe... I do not use the standard Nginx out-of-box configuration and have written my own:

https://mage2.pro/t/20

https://mage2.pro/t/24

https://mage2.pro/t/23