cancel
Showing results for 
Search instead for 
Did you mean: 

Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

After upgrading from Magento 2.4.1 to 2.4.2 on Feb 25th our install has been experiencing very poor performance. The 2.4.1 install ran flawlessly with an uptime of in excess of 80 days, now symptoms include;

  • Random (every hour or so) high server loads and very high frontend response times, these sometimes settle back down
  • Random server hangs (no response and 500s every 24-48hrs) requiring an external reboot to get it back online
  • Magento double processing orders (ie customers are charged twice and two orders appear backend)
  • Magento takes customer payment (via stripe or Paypal) but does not complete order and no order appears in the backend (this usually coincides with a high server load event) - a while later a failed payment email will come through saying products are out of stock that should have a saleable quantity
  • when server hangs and recovers sometimes Elasticsearch will not recover in this instance the site and server will run fine but no products will be available on the site obviously!
  • During these high load events if i execute top -c the CPUs seem to be tied up with a high %wa

I'm guessing that there are some changes between 2.4.1 and 2.4.2 that are causing elasticsearch to overload now possibly with reindexing or a cron job change? Sooo...

Environment (off the top of my head)

  • Centos 7.9 VPS
  • WHM/cPanel
  • 2CPUs
  • 2gb Ram
  • 3gb Swap
  • 60gb disk sapce (69% used)
  • Litespeed websserver 5.4.12 and Litemage cache plugin
  • PHP 7.4.16
  • MySql 8.0.23

Things I have tried so far...

  • switched from PHP7.3 to PHP7.4.16
  • Upgraded the MFTF from 2.5.4 to 3.4
  • Upgraded from MySQL 7 to MySQL 8
  • Set elasticsearch Xms1g Xmx1g in jvm.options.d - this was done before upgrade and worked fine with 2.4.1
  • added LimitMEMLOCK=infinity in etc/systemd/system/elasticsearch.service.d/override.conf - this was done before upgarde and worked fine with 2.4.1
  • added Restart=always in etc/systemd/system/elasticsearch.service.d/override.conf - this hasnt stopped the overloads though.
  • Installed sodium PHP module - Had to be done otherwise users could not log in after upgrade

Anyone else having similar issues after upgrading from 2.4.1 to 2.4.2? any thoughts? I' sure this is impacting on my sales.

5 REPLIES 5

Re: Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

Update 27/3/21

After spending time observing the process manager during these events it seems to be the cron jobs causing the high server load. When it happens i am seeing multiple instances of:

 

opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run

and

 

jailshell (andy) [init] ilshell -c /opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /home/andy/public_html/var/log/magento.cron.log

When the load is really high (40+) ther are upwards of 10 of each of these processes.

I tried deleting the crontab temporarily and truncating the cron_schedule table and the site ran fine with minimal load for about 12hrs, but the high load issues returned as soon as i reinstalled the crontab.

 

I have compared cron.php in 2.4.1 and 2.4.2 and there are no changes.The release note for 2.4.2 dont point to any obvious changes to the crons either.

 

Any ideas or pointers on how i can work out why the crons are now causing such high load?

Re: Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

Hello. I don't think the performance hit is caused by the version update from 2.4.1 to 2.4.2.

 

The first thing I would do is reach out to the hosting company and see if they can look deeper into possible root cause. 

 

The next thing I would check is PHP memory. My site currently has a 2GB limit (2048 MB). And also check cron. Did you reinstall cron after upgrading to 2.4? There were known issues with cron specifically locking up. Backup cron and remove the old Magento entry. Then run "php bin/magento cron:install".

 

Lastly, I would check all 3rd party extensions installed. Is it possible to clone your site / copy to a staging environment. It's a pain and very time consuming, but try to disable 3rd party extensions one-by-one and check your store's performance after removal.

Re: Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

Hi There,

 

Thanks for your help. I did indeed check the PHP memory limits, they were set to 2g in php.ini I double checked because i went through this when i migrated from PHP7.3 to PHP7.4, furthermore i checked that the limit was also 2g in .htaccess and user.ini. 

 

The cron was also reinstalled by removing the cron tab and executing "php bin/magento cron:install -f". I did this as soon as i upgraded because the cron scriipts were still using the ea-php73 directory after switlching to PHP7.4 and had to reinstall to get them to run from ea-php74.

 

I tried tinkering with the elasticsearch heap size too and made sure it wasn't using any of the swap file using instructions from elastic.co although i'm not convinced i was successful.

 

None of the above worked however, so in the end i moved the site to a 6CPU/8g VPS and all the issues are resolved, still not 100% whether it's the increased CPU or the increased memory and not relying on swap anymore that cured it though.

Re: Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

Is there any log message in /var/log/message ?

I think it will give some hints to fix server hanging issue like this

 

Jun 13 19:00:26 localhost kernel: [85431961.167454] Fatal resource shortage: privvmpages, UB 2412.
Jun 13 20:45:19 localhost kernel: [85438254.049458] CPU0: Core temperature above threshold, cpu clock throttled (total events = 9751576984)
Jun 13 20:45:19 localhost kernel: [85438254.049463] CPU12: Core temperature above threshold, cpu clock throttled (total events = 9751578755)

Re: Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

Initially, I'd contact the hosting company and ask if they can explore the root cause more deeply. Afterward, I would check the PHP memory. Currently, 2GB (2048 MB) is the limit on my site. Also, it is a good idea to check cron. During the upgrade to 2.4, did you install cron anew? Specifically, cron was known to lock up. The old Magento entry should be deleted after backing up the cron. You will then need to run "php bin/magento:install". To conclude, I recommend checking the extensions installed by third parties. Do you have the capability of copying your site to a staging environment? It's a long and tedious process, but try disabling third-party extensions one by one and seeing how the store performs afterward. << Snipped >>