It's a quick and dirty bodge, but it seems to get the end results for the customers and work nicely
Hopefully someone can find a solution.
A customer is experiencing this issue in a big way, when using the search feature on Magento 2.2.4, we have been though every possible scenario of testing from reviewing the crons ( quite a few running from Brightpearl inventory management tool), and also now after upgrading the search tool to agolia, this seems to have made the issue worse (more crons and indexing). It is a server 404 not a Magento 404 so the site is not loading at all but once refreshed the page loads no problem. we are running a beefy AWS set up to server resources not an issue.
We have other sites for same customer with same set-up and no problems, only key difference would be database size (qty products) and the sites running without the error had products manually added, the site in question had product data migrated via lite extension.
Any ideas would be fantastic, we may have to apply this auto refresh workaround also as a stop gap ..
give me strength!
Holy moly. Even with Magento 2?!
Well, I'm using Magento 184.108.40.206, and my core_url_rewrite table has only 81 rows, 11 virtual products in the store, running at nexcess with redis and no catalog or category flat enabled.
I faced the 404 issue 3 times myself. And I can see 404 being logged on valid URLs on awstats.
Still have no idea why.
It would be nice if others share more details about the environment and configuration (redis, flat category, etc).
Did anyone ever find a solution to this?
Im on 220.127.116.11 and getting this problem, its killing the crawl rate of Google bot and annoying a few customers too