cancel
Showing results for 
Search instead for 
Did you mean: 

the site is not indexed

the site is not indexed

robots.txt is taken from one of the forums, before adding an exception / ajax_cart items indexed, but not all, but only what was extracted through ajax_cart those variations. No description of the goods, etc. for a month by the index is not gone. Only indexed categories of goods, what is the problem?
The following information from the master Yandex and Robots.
Thank you in advance.

 Снимок экрана 2016-06-14 в 22.15.03.png

User-agent: * 
   
## Crawl-delay parameter: number of seconds to wait between successive requests to the same server. 
## Set a custoskim crawl rate if you\'re experiencing traffic problems with your server. 
# Crawl-delay: 30 
   
## Magento sitemap: uncomment and replace the URL to your Magento sitemap file 
Sitemap: http://urbanprojects.ru/sitemap.xml 
Host: urbanprojects.ru
   
## DEVELOPMENT RELATED SETTINGS 
   
## Do not crawl development files and folders: CVS, svn directories and dump files 
Disallow: /CVS 
Disallow: /*.svn$ 
Disallow: /*.idea$ 
Disallow: /*.sql$ 
Disallow: /*.tgz$ 
   
## GENERAL MAGENTO SETTINGS 
   
## Do not crawl Magento admin page 
Disallow: /admin/ 
Disallow: /myadmin/ 
   
## Do not crawl common Magento technical folders 
Disallow: /app/ 
Disallow: /downloader/ 
Disallow: /errors/ 
Disallow: /includes/ 
Disallow: /lib/ 
Disallow: /pkginfo/ 
Disallow: /shell/ 
Disallow: /var/ 
Disallow: /ajaxcart/
Disallow: /quickview/

   
## Do not crawl common Magento files 
Disallow: /api.php 
Disallow: /cron.php 
Disallow: /cron.sh 
Disallow: /error_log 
Disallow: /get.php 
Disallow: /install.php 
Disallow: /LICENSE.html 
Disallow: /LICENSE.txt 
Disallow: /LICENSE_AFL.txt 
Disallow: /README.txt 
Disallow: /RELEASE_NOTES.txt 
   
## MAGENTO SEO IMPROVEMENTS 
   
## Do not crawl sub category pages that are sorted or filtered. 
Disallow: /*?dir* 
Disallow: /*?dir=desc 
Disallow: /*?dir=asc 
Disallow: /*?limit=all 
Disallow: /*?mode* 
   
## Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs. 
## Disallow: /index.php/ 
   
## Do not crawl links with session IDs 
Disallow: /*?SID= 
   
## Do not crawl checkout and user account pages 
Disallow: /checkout/ 
Disallow: /onestepcheckout/ 
Disallow: /customer/ 
Disallow: /customer/account/ 
Disallow: /customer/account/login/ 
Disallow: /firecheckout/
   
## Do not crawl seach pages and not-SEO optimized catalog links 
Disallow: /catalogsearch/ 
Disallow: /catalog/product_compare/ 
Disallow: /catalog/category/view/ 
Disallow: /catalog/product/view/ 
   
## SERVER SETTINGS 
   
## Do not crawl common server technical folders and files 
Disallow: /cgi-bin/ 
Disallow: /cleanup.php 
Disallow: /apc.php 
Disallow: /memcache.php 
Disallow: /phpinfo.php 
   
## IMAGE CRAWLERS SETTINGS 
   
## Extra: Uncomment if you do not wish Google and Bing to index your images 
# User-agent: Googlebot-Image 
# Allow: / 
# User-agent: msnbot-media 
# Allow: / 

Disallow: /checkout/cart/  
1 REPLY 1

Re: the site is not indexed

Please add below mentioned code in robots.txt file

 

# Group 1
User-agent: Googlebot
Disallow: /subfolder/
Disallow: /nogooglebot/
Disallow: /subfolder/page-url.html


# Group 2
User-agent: *
Allow: /

# Directories & files
Disallow: /wishlist/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /customer/account/login/
Disallow: /review/product/
Disallow: /catalogsearch/
Disallow: /catalog/
Disallow: /catalogsearch/
##Disallow: /checkout/
Disallow: /catalogsearch/
Disallow: /downloader/
Disallow: /razorpay/
Disallow: /order/
Disallow: /rma/
Disallow: /sendfriend/
Disallow: /returns/
Disallow: /sales/
Disallow: /review/
Disallow: /invitation/
Disallow: /newsletter/
Disallow: /install/
Disallow: /enable-cookies/
Disallow: /skin/
Disallow: /js/
Disallow: /directory/
Disallow: /var/
Disallow: /private/
Disallow: /admin/
Disallow: /api/
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/
Disallow: /tag/
Disallow: /review/

Disallow: /lib/
Disallow: /includes/
Disallow: /composer.json
Disallow: /composer.lock
Disallow: /CONTRIBUTING.md
Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow: /COPYING.txt
Disallow: /Gruntfile.js
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /nginx.conf.sample
Disallow: /package.json
Disallow: /php.ini.sample
Disallow: /RELEASE_NOTES.txt
Disallow: /*.Zip$
Disallow: /*.php$


# Parameters
Disallow: /*?cat=
Disallow: /*&cat=
Disallow: /*?carrier=
Disallow: /*&carrier=
Disallow: /*?color_filter=
Disallow: /*&color_filter=
Disallow: /*?display_size=
Disallow: /*&display_size=
Disallow: /*?network=
Disallow: /*&network=
Disallow: /*?os=
Disallow: /*&os=
Disallow: /*?price=
Disallow: /*&price=
Disallow: /*?timestamp=
Disallow: /*&timestamp=
Disallow: /*?___store=
Disallow: /*?SID=
Disallow: /*?___SID=
Disallow: /*?name=
Disallow: /*&name=
Disallow: /*?top_seller=
Disallow: /*&top_seller=
Disallow: /pkginfo/
Disallow: /report/