After upgrading from Magento 2.4.1 to 2.4.2 on Feb 25th our install has been experiencing very poor performance. The 2.4.1 install ran flawlessly with an uptime of in excess of 80 days, now symptoms include;
I'm guessing that there are some changes between 2.4.1 and 2.4.2 that are causing elasticsearch to overload now possibly with reindexing or a cron job change? Sooo...
Environment (off the top of my head)
Things I have tried so far...
Anyone else having similar issues after upgrading from 2.4.1 to 2.4.2? any thoughts? I' sure this is impacting on my sales.
After spending time observing the process manager during these events it seems to be the cron jobs causing the high server load. When it happens i am seeing multiple instances of:
opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run
jailshell (andy) [init] ilshell -c /opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /home/andy/public_html/var/log/magento.cron.log
When the load is really high (40+) ther are upwards of 10 of each of these processes.
I tried deleting the crontab temporarily and truncating the cron_schedule table and the site ran fine with minimal load for about 12hrs, but the high load issues returned as soon as i reinstalled the crontab.
I have compared cron.php in 2.4.1 and 2.4.2 and there are no changes.The release note for 2.4.2 dont point to any obvious changes to the crons either.
Any ideas or pointers on how i can work out why the crons are now causing such high load?
Hello. I don't think the performance hit is caused by the version update from 2.4.1 to 2.4.2.
The first thing I would do is reach out to the hosting company and see if they can look deeper into possible root cause.
The next thing I would check is PHP memory. My site currently has a 2GB limit (2048 MB). And also check cron. Did you reinstall cron after upgrading to 2.4? There were known issues with cron specifically locking up. Backup cron and remove the old Magento entry. Then run "php bin/magento cron:install".
Lastly, I would check all 3rd party extensions installed. Is it possible to clone your site / copy to a staging environment. It's a pain and very time consuming, but try to disable 3rd party extensions one-by-one and check your store's performance after removal.
Thanks for your help. I did indeed check the PHP memory limits, they were set to 2g in php.ini I double checked because i went through this when i migrated from PHP7.3 to PHP7.4, furthermore i checked that the limit was also 2g in .htaccess and user.ini.
The cron was also reinstalled by removing the cron tab and executing "php bin/magento cron:install -f". I did this as soon as i upgraded because the cron scriipts were still using the ea-php73 directory after switlching to PHP7.4 and had to reinstall to get them to run from ea-php74.
I tried tinkering with the elasticsearch heap size too and made sure it wasn't using any of the swap file using instructions from elastic.co although i'm not convinced i was successful.
None of the above worked however, so in the end i moved the site to a 6CPU/8g VPS and all the issues are resolved, still not 100% whether it's the increased CPU or the increased memory and not relying on swap anymore that cured it though.
Is there any log message in /var/log/message ?
I think it will give some hints to fix server hanging issue like this
Jun 13 19:00:26 localhost kernel: [85431961.167454] Fatal resource shortage: privvmpages, UB 2412. Jun 13 20:45:19 localhost kernel: [85438254.049458] CPU0: Core temperature above threshold, cpu clock throttled (total events = 9751576984) Jun 13 20:45:19 localhost kernel: [85438254.049463] CPU12: Core temperature above threshold, cpu clock throttled (total events = 9751578755)
Initially, I'd contact the hosting company and ask if they can explore the root cause more deeply. Afterward, I would check the PHP memory. Currently, 2GB (2048 MB) is the limit on my site. Also, it is a good idea to check cron. During the upgrade to 2.4, did you install cron anew? Specifically, cron was known to lock up. The old Magento entry should be deleted after backing up the cron. You will then need to run "php bin/magento:install". To conclude, I recommend checking the extensions installed by third parties. Do you have the capability of copying your site to a staging environment? It's a long and tedious process, but try disabling third-party extensions one by one and seeing how the store performs afterward. << Snipped >>