I have Magento 2.3.4
Site Ground Cloud Server: 5 CPUs, 4 GB RAM
Every time run the cron, the server kills it after several seconds.
I made a script that kills all the pending crons from the table before the cron runs
but i still face the same issue while running the cron and this is why only several cron jobs complete every time and not all of them (server kills the process).
I have attache the answer from the server support.
I will be very happy if someone can help me here to solve this issue.
The statistics in your User Area are not in real time. The RAM is being exhausted only for a few seconds which is why you cannot see the spikes in the statistics inside the User Area.
Here is what happens:
1) The following cron job runs at the beginning of every hour.
0 * * * * /usr/local/php73/bin/php-cli -f /home/customer/www/arabinstruments.com/web/bin/magento cron:run
2) The queries spawned by the Magento cron are causing a very high number of read requests to the actual hard disk.
2020-08-03 01:03:24.638695-05 | IOPs-Read: 12411 | IOPs-Write: 2248
2020-08-03 01:02:22.869841-05 | IOPs-Read: 24143 | IOPs-Write: 3033
2020-08-03 01:01:21.723683-05 | IOPs-Read: 35169 | IOPs-Write: 1687
2020-08-02 03:01:23.304137-05 | IOPs-Read: 39000 | IOPs-Write: 2086
2020-08-02 03:00:21.864408-05 | IOPs-Read: 13095 | IOPs-Write: 1458
3) Because of that the server's RAM depletes for a few seconds and this causes a high load on the server.
4) During the high server load our monitoring system starts killing the PHP processes in order to stabilize back the server.
Aug 03 01:00:28 c16608: Critical load (101.80) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
Aug 03 02:00:23 c16608: Critical load (43.42) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
A well-optimized application should not generate more than 4000 read and write request in total. As you can see from the above logs, the queries spawned by the Magento cron are generating between 10000-35000 requests per second.
If the number of the input/output operations is too high, the corresponding processes become very slow and RAM consumable. Adding more RAM to your server may solve the issue, but this is not the correct approach in this case. The correct approach would be to check why the queries spawned by the Magento cron are generating that much read requests and to optimize them.
Did you manage to find out the cause of this issue?
Any help would be much appreciated - I believe I have the same problem.
Siteground Cloud Server: 4 CPU cores, 16 GB RAM
Sample provided by Siteground
2020-11-19 20:36 | CPU: 16.18 | IOPs-Read: 53067 | IOPs-Write: 1643
2020-11-19 20:35 | CPU: 15.48 | IOPs-Read: 41958 | IOPs-Write: 784
2020-11-19 20:34 | CPU: 30.42 | IOPs-Read: 50391 | IOPs-Write: 2909
2020-11-19 20:33 | CPU: 32.89 | IOPs-Read: 69533 | IOPs-Write: 2171
2020-11-19 20:32 | CPU: 15.20 | IOPs-Read: 31875 | IOPs-Write: 745
2020-11-19 20:31 | CPU: 23.27 | IOPs-Read: 35467 | IOPs-Write: 1673
2020-11-19 20:30 | CPU: 19.73 | IOPs-Read: 19918 | IOPs-Write: 1421
2020-11-19 20:29 | CPU: 27.90 | IOPs-Read: 28271 | IOPs-Write: 2088
2020-11-19 20:28 | CPU: 424.27 | IOPs-Read: 825340 | IOPs-Write: 24498
What i found out is that all the issues i had for years in my website with magento 1 and also magento 2 has been solved once i have moved to a different server company.
My conclusion is Never work with siteground server company if you have a magento shop.
Not only that i paid a lot (around 120$ per month) to siteground. They will always gave the same answer that something wrong with you SQL or Cron or anything...
and the website worked so slow...
Once i moved to cloudways, all the problems have been solved.
Website is fast, no issues with crons, no more issues with website stop working or server erros and elastic search is FREE and i pay 80$ per month
Hope it will help
Sorry I only just saw your reply.
Thank you very much for your advice - I will check out Cloudways and give them a try.
Great ,please update me
I took the digital ocean server.
You can start with the 42$ plan. I chose the 82$ plan
and i am so so happy that i run away from siteground.
Looking forward to hear if it worked for you too.