cancel
Showing results for 
Search instead for 
Did you mean: 

Website being Crawled for checkout/cart/add/uenc/ This is an issue?

Website being Crawled for checkout/cart/add/uenc/ This is an issue?

Hi all

My site has been getting crawled allot by this IP range: 46.229.168 specifically 46.229.168.72

 

And it always seems to be looking/crawling:

 

https://www.sparetoolparts.co.uk/checkout/cart/add/uenc/aHR0cHM6Ly93d3cuc3BhcmV0b29scGFydHMuY28udWsv...

 

or similar

 

Is this an issue? is it an attempt to hack my site? I would have thought If I had told my .htaccess file to ignore all checkout pages that spider would have simply moved on when it saw a checkout page, this may be a coincidence but I get the feeling my sales slow right down when these pages are being looked it/crawled. I have blocked the IP address in the past, but it obviously is not working

 

Any thoughts/ideas?

1 REPLY 1

Re: Website being Crawled for checkout/cart/add/uenc/ This is an issue?

It less likely than this is a hack but rather a bot spidering your site. 

 

Your product pages, e.g. https://www.sparetoolparts.co.uk/milwaukee-ag-1000-spare-parts-list-type-4000458156.html have add to cart buttons which are "setLocation" and therefore suggesting to bots that they can be accessed with GET requests rather than form POSTs. It could simply be a bot finding these buttons and following the location set. 

 

You could change each button so that it was in it's own form with a post with the correct URL which would make this less likely to happen. 

 

From running whois with those IP addresses, it's not obvious which bot is doing that, perhaps you'll be able to see from the user agents that are visiting with those IP addresses. If you've tried to block but it's not working, could this be because you added something to robots.txt and the bot is ignoring these requests? Or perhaps the IP has just changed to a similar looking one. 

 

----
If you've found one of my answers useful, please give "Kudos" or "Accept as Solution" as appropriate. Thanks!