Is this robots.txt good?

[Espamoto]

Hello, I eanto to have the “better” robots.txt but I dont know if the robots We use its fine or not:

# # Robots.txt for Magento Community and Enterprise

# # GENERAL SETTINGS

# Enables robots.txt rules for all crawlers

User-agent: *

# # Crawl-delay parameter: the number of seconds you want to wait between successful requests to the same server.
# # Set a crawl rate, if your server's traffic problems. Please note that Google ignore crawl-delay setting in Robots.txt. You can set up this in Google Webmaster tool
# Crawl-delay: 30

# # Magento sitemap: URL to your sitemap file in Magento
# Sitemap: https://www.myweb.com/sitemap.xml

# # Settings that relate to the UNDER CONSTRUCTION

# # Do not allow indexing files and folders that are required during development: CVS, SVN directory and dump files
Disallow: / CVS
Disallow: / *. Svn $
Disallow: / *. Idea $
Disallow: / *. Sql $
Disallow: / *. Tgz $

# # GENERAL SETTINGS For MAGENTO

# # Do not index the page Magento admin
Disallow: / admin /

# # Do not index the general technical Magento directory
Disallow: / app /
Disallow: / downloader /
Disallow: / errors /
Disallow: / includes /
Disallow: / lib /
Disallow: / pkginfo /
Disallow: / shell /
Disallow: / var /

# # Do not index the shared files Magento
Disallow: / api.php
Disallow: / cron.php
Disallow: / cron.sh
Disallow: / error_log
Disallow: / get.php
Disallow: / install.php
Disallow: / LICENSE.html
Disallow: / LICENSE.txt
Disallow: / LICENSE_AFL.txt
Disallow: / README.txt
Disallow: / RELEASE_NOTES.txt

# # MAGENTO SEA IMPROVEMENT

# # Do not index the page subcategories that are sorted or filtered.
Disallow: / *? Dir *
Disallow: / *? Dir = desc
Disallow: / *? Dir = asc
Disallow: / *? Limit = all
Disallow: / *? Mode *

# # Do not index the second copy of the home page (example.com / index.php /). Un-comment only if you have activated Magento SEO URLs.
# # Disallow: / index.php /

# # Do not index the link from the session ID
Disallow: / *? SID =

# # Do not index the page checkout and user account
Disallow: / checkout /
Disallow: / onestepcheckout /
Disallow: / customer /
Disallow: / customer / account /
Disallow: / customer / account / login /

# # Do not index the search page and CEO, non-optimized link categories
Disallow: / catalogsearch /
Disallow: / catalog / product_compare /
Disallow: / catalog / category / view /
Disallow: / catalog / product / view /

# # Server Settings

# # Do not index the general technical directories and files on a server
Disallow: / cgi-bin /
Disallow: / cleanup.php
Disallow: / apc.php
Disallow: / memcache.php
Disallow: / phpinfo.php

# # SETTINGS Image indexing

# # Optional: If you do not want to Google and Bing to index your images
# User-agent: Googlebot-Image
# Disallow: /
# User-agent: msnbot-media
# Disallow: /

[Damian Culotta]

Hi @Espamoto,

You can check a few examples here:

• https://gist.github.com/gwillem/feeada60fb6b4ebc5c3f <-- Check the links at the commetns section
• https://gist.github.com/petskratt/016c9dbf159a81b9d6aa
• http://inchoo.net/ecommerce/ultimate-magento-robots-txt-file-examples/
• https://github.com/Creare/magento-robots/blob/master/robots.txt

Why are you asking? I mean, which problem did you find with your store?

[Espamoto]

Thanks, I will take a look now.

because I think that its not very good indexed.

[Espamoto]

I have put this: 

# Sitemaps
Sitemap: https://www.myweb.com/sitemap.xml

User-agent: *
Allow: /

# Magento directories
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /cgi-bin/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
Disallow: /maintenance/
Disallow: /ajax/

# Paths (clean URLs)
Disallow: /admin/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product_compare/
Disallow: /catalogsearch/
Disallow: /catalogsearch/result/
Disallow: /checkout/*
Disallow: /contacts/
Disallow: /control/
Disallow: /customer/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /*/admin/
Disallow: /*/catalog/category/view/
Disallow: /*/catalog/product/view/
Disallow: /*/catalog/product_compare/
Disallow: /*/catalogsearch/
Disallow: /*/catalogsearch/result/
Disallow: /*/checkout/
Disallow: /*/contacts/
Disallow: /*/control/
Disallow: /*/customer/
Disallow: /*/newsletter/
Disallow: /*/review/
Disallow: /*/tag/
Disallow: /*/wishlist/
Disallow: /guest/*

# Magento files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt

# Paths (no clean URLs)
Disallow: /*.php$

[JLHC]

The robots.txt file seem fine to me. 

There can be a lot of reasons as to why your website isn't properly indexed. 

Did you generate and submit your sitemap to Google Search Console? 

You should add your website to Google Search Console if you haven't done so:-

https://www.google.com/webmasters/tools/home

[Espamoto]

Yes yes, and we have a SEO module. But I think that it´s not very good positioned in Google

[kalyanchakri]

yes, robots.txt seems to be fine.