Robots.txt question

[jakbau]

Hi,

could you tell me if my robots.txt is ok? Is there anything missing?

Sitemap: http://www.xxx.pl/sitemap/sitemap.xml

# Crawlers Setup
User-agent: *

# Allowable Index
Allow: /*?p=
Allow: /media/

# Cats
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /stats/
Disallow: /var/

#  (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

## checkout & user
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

# files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# URLs ( URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=

## No sorting.
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*

[ShoaibRehman89 · Accepted solution]

@jakbau

That looks fine to me but I'd also like to disallow the following: 

Disallow: /*.CVS

Disallow: /*.Zip$

Disallow: /composer.json

Disallow: /composer.lock

[Amasty]

Hi @jakbau

Actually your robots.txt needs some improvement.

1) Don't disallow js and css files as you will get an error from Google stating that it can't reach the files. It may also negatively affect your rankings (according to Google).

2) (recommended) It's better to use canonical tags for sorting/filtering options rather that block such URLs in robots.txt.