cancel
Showing results for 
Search instead for 
Did you mean: 

Robots.txt question

SOLVED

Robots.txt question

Hi,

could you tell me if my robots.txt is ok? Is there anything missing?

 

Sitemap: http://www.xxx.pl/sitemap/sitemap.xml

# Crawlers Setup
User-agent: *

# Allowable Index
Allow: /*?p=
Allow: /media/

# Cats
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /stats/
Disallow: /var/

#  (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

## checkout & user
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

# files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# URLs ( URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=

## No sorting.
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Robots.txt question

@jakbau

 

That looks fine to me but I'd also like to disallow the following: 

 

Disallow: /*.CVS

Disallow: /*.Zip$

Disallow: /composer.json

Disallow: /composer.lock
Magento Certified Solution Specialist | Lead Magento developer
If this response was helpful to you, consider giving kudos to this post

View solution in original post

2 REPLIES 2

Re: Robots.txt question

@jakbau

 

That looks fine to me but I'd also like to disallow the following: 

 

Disallow: /*.CVS

Disallow: /*.Zip$

Disallow: /composer.json

Disallow: /composer.lock
Magento Certified Solution Specialist | Lead Magento developer
If this response was helpful to you, consider giving kudos to this post

Re: Robots.txt question

Hi @jakbau

 

Actually your robots.txt needs some improvement.

 

1) Don't disallow js and css files as you will get an error from Google stating that it can't reach the files. It may also negatively affect your rankings (according to Google).

 

2) (recommended) It's better to use canonical tags for sorting/filtering options rather that block such URLs in robots.txt.

Was my answer helpful? You can accept it as a solution.
230+ professional extensions for M1 & M2 with free lifetime updates!