Poor Magento Performance after upgrade from 2.4.1 to 2.4.2

[andrew_dean]

After upgrading from Magento 2.4.1 to 2.4.2 on Feb 25th our install has been experiencing very poor performance. The 2.4.1 install ran flawlessly with an uptime of in excess of 80 days, now symptoms include;

• Random (every hour or so) high server loads and very high frontend response times, these sometimes settle back down
• Random server hangs (no response and 500s every 24-48hrs) requiring an external reboot to get it back online
• Magento double processing orders (ie customers are charged twice and two orders appear backend)
• Magento takes customer payment (via stripe or Paypal) but does not complete order and no order appears in the backend (this usually coincides with a high server load event) - a while later a failed payment email will come through saying products are out of stock that should have a saleable quantity
• when server hangs and recovers sometimes Elasticsearch will not recover in this instance the site and server will run fine but no products will be available on the site obviously!
• During these high load events if i execute top -c the CPUs seem to be tied up with a high %wa

I'm guessing that there are some changes between 2.4.1 and 2.4.2 that are causing elasticsearch to overload now possibly with reindexing or a cron job change? Sooo...

Environment (off the top of my head)

• Centos 7.9 VPS
• WHM/cPanel
• 2CPUs
• 2gb Ram
• 3gb Swap
• 60gb disk sapce (69% used)
• Litespeed websserver 5.4.12 and Litemage cache plugin
• PHP 7.4.16
• MySql 8.0.23

---

Things I have tried so far...

• switched from PHP7.3 to PHP7.4.16
• Upgraded the MFTF from 2.5.4 to 3.4
• Upgraded from MySQL 7 to MySQL 8
• Set elasticsearch Xms1g Xmx1g in jvm.options.d - this was done before upgrade and worked fine with 2.4.1
• added LimitMEMLOCK=infinity in etc/systemd/system/elasticsearch.service.d/override.conf - this was done before upgarde and worked fine with 2.4.1
• added Restart=always in etc/systemd/system/elasticsearch.service.d/override.conf - this hasnt stopped the overloads though.
• Installed sodium PHP module - Had to be done otherwise users could not log in after upgrade

Anyone else having similar issues after upgrading from 2.4.1 to 2.4.2? any thoughts? I' sure this is impacting on my sales.

[andrew_dean]

Update 27/3/21

After spending time observing the process manager during these events it seems to be the cron jobs causing the high server load. When it happens i am seeing multiple instances of:

opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run

and

jailshell (andy) [init] ilshell -c /opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /home/andy/public_html/var/log/magento.cron.log

When the load is really high (40+) ther are upwards of 10 of each of these processes.

I tried deleting the crontab temporarily and truncating the cron_schedule table and the site ran fine with minimal load for about 12hrs, but the high load issues returned as soon as i reinstalled the crontab.

I have compared cron.php in 2.4.1 and 2.4.2 and there are no changes.The release note for 2.4.2 dont point to any obvious changes to the crons either.

Any ideas or pointers on how i can work out why the crons are now causing such high load?

[seattletechie]

Hello. I don't think the performance hit is caused by the version update from 2.4.1 to 2.4.2.

The first thing I would do is reach out to the hosting company and see if they can look deeper into possible root cause. 

The next thing I would check is PHP memory. My site currently has a 2GB limit (2048 MB). And also check cron. Did you reinstall cron after upgrading to 2.4? There were known issues with cron specifically locking up. Backup cron and remove the old Magento entry. Then run "php bin/magento cron:install".

Lastly, I would check all 3rd party extensions installed. Is it possible to clone your site / copy to a staging environment. It's a pain and very time consuming, but try to disable 3rd party extensions one-by-one and check your store's performance after removal.

[andrew_dean]

Hi There,

Thanks for your help. I did indeed check the PHP memory limits, they were set to 2g in php.ini I double checked because i went through this when i migrated from PHP7.3 to PHP7.4, furthermore i checked that the limit was also 2g in .htaccess and user.ini. 

The cron was also reinstalled by removing the cron tab and executing "php bin/magento cron:install -f". I did this as soon as i upgraded because the cron scriipts were still using the ea-php73 directory after switlching to PHP7.4 and had to reinstall to get them to run from ea-php74.

I tried tinkering with the elasticsearch heap size too and made sure it wasn't using any of the swap file using instructions from elastic.co although i'm not convinced i was successful.

None of the above worked however, so in the end i moved the site to a 6CPU/8g VPS and all the issues are resolved, still not 100% whether it's the increased CPU or the increased memory and not relying on swap anymore that cured it though.

[TutorialMagento]

Is there any log message in /var/log/message ?

I think it will give some hints to fix server hanging issue like this

Jun 13 19:00:26 localhost kernel: [85431961.167454] Fatal resource shortage: privvmpages, UB 2412.
Jun 13 20:45:19 localhost kernel: [85438254.049458] CPU0: Core temperature above threshold, cpu clock throttled (total events = 9751576984)
Jun 13 20:45:19 localhost kernel: [85438254.049463] CPU12: Core temperature above threshold, cpu clock throttled (total events = 9751578755)

[calculerun1406]

Initially, I'd contact the hosting company and ask if they can explore the root cause more deeply. Afterward, I would check the PHP memory. Currently, 2GB (2048 MB) is the limit on my site. Also, it is a good idea to check cron. During the upgrade to 2.4, did you install cron anew? Specifically, cron was known to lock up. The old Magento entry should be deleted after backing up the cron. You will then need to run "php bin/magento:install". To conclude, I recommend checking the extensions installed by third parties. Do you have the capability of copying your site to a staging environment? It's a long and tedious process, but try disabling third-party extensions one by one and seeing how the store performs afterward. << Snipped >>

[lepkmagene3be2]

It sounds like you have experienced some performance issues and bugs after upgrading from Magento 2.4.1 to 2.4.2. Here are some suggestions to try and address the issues:

- Check for Magento 2.4.2 known issues: Before investigating further, check the official Magento 2.4.2 Release Notes and known issues. This will help you to understand if the issues you are experiencing have been previously reported and how to address them.

- Monitor server resources: Monitor the CPU, memory, disk usage, and network traffic of your server to identify if any resources are being exhausted. High server loads and CPU usage could indicate that something is causing excessive processing, such as a poorly optimized extension or too many concurrent tasks running at the same time.

- Check Magento logs: Check the Magento logs for any errors, exceptions, or warnings related to the issues you are experiencing. This can help you identify the root cause of the problem and provide valuable information to Magento support or other experts.

- Check third-party extensions: Review any third-party extensions installed on your site to ensure they are compatible with Magento 2.4.2. Disable any extensions that are not essential and see if this helps improve performance.

- Check cron jobs: Review the cron jobs running on your server to ensure they are set up correctly and not overlapping. A poorly optimized or incorrectly configured cron job can cause performance issues.

- Review Elasticsearch configuration: Check the Elasticsearch configuration and ensure it is optimized for your server. Elasticsearch can be resource-intensive, so you may need to adjust its settings to improve performance.

- Check payment gateway integration: Review your payment gateway integration and ensure it is compatible with Magento 2.4.2. Check if the issues you are experiencing are related to payment processing and try to reproduce the issues using a test payment gateway.

- Upgrade to the latest version: Consider upgrading to the latest version of Magento, which may include bug fixes and performance improvements.

If you are unable to resolve the issues, consider reaching out to Magento support or hiring a Magento expert to assist you with troubleshooting and resolving the issues.

[ibtisamdev8aef]

Based on the symptoms you described after upgrading from Magento 2.4.1 to 2.4.2, it does sound like the issues are stemming from Elasticsearch getting overloaded. Here are some things I would recommend troubleshooting:

• Check cron jobs and look for any new or modified jobs added in 2.4.2 that could be hitting Elasticsearch heavily. Disable any unnecessary crons.
• Review 2.4.2 release notes for mentions of changes to indexing or search behavior. Major version upgrades often include reindexing that can spike load.
• Look at Elasticsearch index size and growth - a large bloated index can cause performance issues. You may need to reindex in batches.
• Increase Elasticsearch heap size in config to allocate more memory for indexing operations.
• Enable Elasticsearch throttling to cap load during peak times.
• Check for queue backlogs - heavy queue usage from jobs/indexing can overload the DB and cache.
• Review server logs for Elasticsearch errors and warnings.
• Consider adding a dedicated Elasticsearch server or cluster to handle search load separately.
• Try tweaking storefront caching and indexers for less frequent full reindexing.

With the right Elasticsearch optimizations and getting cron jobs/indexing under control, the upgrade issues should stabilize. It's not about updating from 2.4.1 to 2.4.2, there can be various reasons. Also, checking up with your hosting provider is a considerable factor. Issues like these can arouse from the end of your hosting provider, that's why opting a good hosting for your Magento site is very important. Going for Managed Magento Hosting is an ideal option for the stability of your site, as it helps optimizing the performance and reliability. And if any issue of this kind occurs, their customer support team is available 24/7.