cancel
Showing results for 
Search instead for 
Did you mean: 

Google crawl error increase, it shows me weird mix in all the url's in magento 1.9.0.1

Google crawl error increase, it shows me weird mix in all the url's in magento 1.9.0.1

Magento 1.9.0.1

 

Hi,

I have since 10 days an increase in 404 errors on webmaster tools, and in the crawl error section i found 150 weird url that make no sense :
http://sitename.com/ (cms page)/index.php/(another cms page or product).

 

In my memory i haven't change anything, but this has to come from somewhere. I've checked robot.txt (i only deleted prior to that the line User-agent: Googlebot-Image Disallow: ), htaccess , local.xml and friendly url in system/config/web. Looks all good.

 

Here is a screenshot : http://s15.postimg.org/k82bu8317/image.jpg

 

In my robot.txt i've got :

Disallow: /index.php/
Disallow: /.css$
Disallow: /
.php$
Disallow: /?p=&
Disallow: /?SID=
Disallow: /?___from_store=
Disallow: /
___from_store=

 

Thanks in advance for your help.

 

Regards,

Marc

2 REPLIES 2

Re: Google crawl error increase, it shows me weird mix in all the url's in magento 1.9.0.1

I just found out in my access.log that i had an attack to my /downloader folder :

"GET /downloader/index.php?A= HTTP/1.1" 200 2518 "-" "-"65.98.60.146 - - [07/Mar/2016:12:37:45 +0100]

"POST /downloader/ HTTP/1.1" 302 - "-" "-".

 

And after every 100 attacks i get the URL error that appear :

"GET /bache-tendue/index.php/tendeur-elastique/tendeur-caoutchouc/tendeur-caoutchouc-25cm HTTP/1.1" 404 111180 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"

 

Hope this helps.

 

Regards,

Marc

Re: Google crawl error increase, it shows me weird mix in all the url's in magento 1.9.0.1

I managed to block the brut force attack, and protect my downloader/ folder with the htaccess file.

 

But when i look at the access.log, it keeps getting the weird urls and they make absolute no sense. I'm searching for a way to redirect these wrong urls to my homepage via the htaccess but have no idear how. Can anyone help me ? 

 

- [12/Mar/2016:09:10:40 +0100] "GET /bache-tendue/cadre-mural/index.php/revendeur HTTP/1.1" 404 103472 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

- [12/Mar/2016:09:22:59 +0100] "GET /bache-tendue/index.php/revendeur/kit-revendeur HTTP/1.1" 404 103472 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

- [12/Mar/2016:06:43:58 +0100] "GET /info-creation-graphique/index.php/maquette/ HTTP/1.1" 404 103472 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

 

Regards, 

Marc