Diagnosing slow site with high server load

[LaXiS]

Hello,

I am diagnosing a CE 2.3 instance on a quad core 4GB RAM VPS that is running extremely slow and putting a high load on the VPS. This site was migrated from Magento 1 and rebuilt. I can't deny that there are quite a lot of installed extensions and products/prices, so this does certainly have an effect. The site is in production mode with all static content deployed.

All page loads take a long time to "connect", but once the first element loads, the rest of the page loads quickly. I found that PHP is constantly using 100% of all CPUs, thanks to Magento index.php (so the problem is Magento frontend and not some other process or cron schedule).

I enabled the profiler and the result is quite strange, look at the profiler html table here (https://laxis.it/magento.html). The "magento" timer says 41 seconds, but if you look below there is no obvious cause for this (only some 2 or 3 seconds records).

I have been searching and trying things for days (like tuning MySQL, PHP, disabling extensions) with no difference.

How can I precisely diagnose what part of Magento or extension is causing this nonsense?

Thank you

[LaXiS · Accepted solution]

I succeeded in fixing the issue, thanks to the XHProf PHP extension.

The main problem was that the WHM/cPanel installation for PHP 7.3 did not include the mcrypt extension (that even the official Magento installation guide says is "optional but recommended" for PHP < 7.2). This most probably caused phpseclib to use a PHP-only implementation of mcrypt instead of the native library through the extension, causing long delays at every encryption.

This and the fact that Magento cache was disabled for some reason... Now the site loads in under 1 second (instead of 15) and server load is under 1 (instead of 8-15, on a 4 core VPS).

Hope this helps anyone stumbling upon this thread.