occasionally getting 404s for valid urls, refresh usually solves it, but why?

[DWJames]

Hi all,

on our magento v1.9.2.2. store we get occasional 404 error pages for valid urls. Typically when clicking a link from a google search result or similar. The user gets presented with a 404 page, but if they refresh, usually this clears it and they get delivered the content.

It's as though the magento url rewrite process just craps out sometimes and doesn't answer a request, but is that even possible?   

Has anyone seen this before or can anyone recommend a way to diagnose and troubleshoot?

thanks,

James

[Damian Culotta]

Hi @DWJames,

I didn't see this behavior before but I guess I should start by:

• Checking Magento and Webserver logs. Also the access.log.
• Compare the urls on the access.log with the Magento urls to be sure those are the same.
• Checking how big is your core_url_table data. Maybe you have too many and there is an issue there? How much entries have that table?

Is there an example you can share?

[DWJames]

Hi @Damian Culotta,

thanks for the reply,

I can see these ocasional 404s on our nginx access log and can also see 200 response codes for the same URI within the same day.

These urls are the same in magento. We're using friendly names, so there is a rewrite taking place in magento. 

The core_url_rewrite table is quite large, but not huge: 90,000 rows and 60mb in size approx.

The only thing which is non-standard with our url handling is that we use this flat category extension  . So I guess there is an extra layer of rewrite in place. If I can construct a test where we can provoke the failure, then I can try disabling things and see if there's any difference.    

I wonder if there's a simple way I could ask magento to look up a url 1,000 times and record the responses to see if I can witness the failures.

[Damian Culotta]

Hi @DWJames,

I think you're in the right track.

If I were you probably I'll do the same. Normally I start removing/disabling all third party modules to be sure if the problem is Magento core or not.

Then I start to add features (sometimes is a pain in the... ) to try to catch the issue.

It would be nice if you can come back with the problem description (if you can find it, of course).

[sasoriza]

We are having the same problem but we are not using the extension mentioned. I too believe it is related to too many rewrites happening although our table is even smaller (~70.000 entries, ~45mb). I know that this problem started at some point and never existed before so I guess there is some threshold that we exceeded. The problem is ocurring randomly and (luckily) not that often, however this makes it so hard to debug. 

[Damian Culotta]

Hi @sasoriza,

Which version of Magento are you using?

[sasoriza]

Hi Damian,

we were on 1.9.3.7, but the problem might have started with an earlier version. We have now updated to 1.9.3.8 and will have to see if that makes any difference.

[DWJames]

I've been investigating this further and have removed the flat category extension, but it hasn't affected this issue, so we can rule that out.

I also made a powershell script that would repeatedly request the page headers of 1 of the pages we identified as failing, but this ran for hours and never got anything other than a perfect 200 response code.   

I can see an example of this error in our server logs from this week where a particular page was requested approx 30 times in the last 7 days. In these 30 times, there were 1 x 404 code, 1 x 403, 1 x 301 and the rest were code 200.    

If anyone can suggest where to test next, that would be much appreciated.

[ArchbaldCortez]

We're having the same issue...although our core_url_rewrite is quite big....131MB

We're starting to get customer complaints so I need to solve this ASAP.  

[Giftio]

Hi,

I am too trying to solve this issue for years now, no help either but there is simple solution I just found out.

Just put auto refresh on 404 page. If there is real issue it stays 404, if there is this little nasty bug, it will refresh and customer gets served. Not great but you won't loose customer at 404.

<script type='text/javascript'>
(function()
{
  if( window.localStorage )
  {
    if( !localStorage.getItem('firstLoad') )
    {
      localStorage['firstLoad'] = true;
      window.location.reload();
    }  
    else
      localStorage.removeItem('firstLoad');
  }
})();
</script>