2.6 Million Rewrites

[ApplianceDepot]

Hi,

Having just taken over a popular Magento 1.9 website I've noticed that url re-indexing takes a very very long time... Checking the rewrite management in Magento I see 2,699,505 rewrites and 2,599,299 of these are custom rewrites! Now we only have 2622 active products and looking at how this site works all their products are individually different so when the product is sold it's then been disabled, and then duplicated for the next model... so there is also 14227 disabled products in the catalogue.

What would you think is the best way for us to approach this issue?

[Damian Culotta]

Hi @ApplianceDepot,

Maybe the first step is to understand why they've created all those custom redirects. I would like to think that there is a valid reason.

[ApplianceDepot]

Hi Damien, there is no reason they would need so many redirects, I think the issue has come from the staff not removing the tick from create permanent redirect from old URL when duplicating products but also a sloppy admin who has let it build up so big, now i know I can just truncate the core_url_rewrite table and then reindex to restore the system rewrites but I'm concerned about the SEO impact and the amount of custom rewrites that would be hard to extract just the required from 2.6 million rewrites.

[Damian Culotta]

I can understand your problem (maybe you'll need to get asistance from some SEO specialist... I'm just a developer) but at some point you'll need to forget about some (most of) of the rewrites if that the quantity of products in your catalog.

Or can can maintain that insane amount of rewrites and forget about performance. (Yes, I know, I didn't help.... ) :-)

[Finestshops · Accepted solution]

Hi there,

We had a similar problem with a client we migrated to us from another host.

That store was overloading a server without much traffic because rewrites table had several million records. We discussed it with SEO guys and decided that slow store does much more damage to SEO compare to not having those rewrites. And, as long as there is no link to old product URL anywhere in the store (product descriptions or old sitemap for example) so Google does not see thousands of 404 pages, there is no much reason to keep it. We cleaned rewrite table, setup a few dozens 301 redirects to the most popular products (client provided the list) and let other old URLs go to a custom 404 page with the list of main categories and prominently placed search form so customers will have an easy way out if they ended up on that 404 page. That client did not report any issues with dropped traffic or Google rank so looks like that worked but you still want to check with your SEO guys because every store is different.

Hope this helps.

[ApplianceDepot]

I think your approach is the best way forwards for us too.

Many thanks

[ApplianceDepot]

To follow up on this we have dropped the 2.6 million rewrites now and reindexing takes minutes over hours and everything SEO wise is fine.

Great result,

Thanks again