Cron jobs pile up and never stop running, thus crashing the server

[DK_dev]

Hi everyone!

For some unknown reason Magento 2 never stops running cron jobs. Before one cron job finishes, the next one starts causing a lot of cron jobs to pile up and crash the server. 

I've got the following Magento configuration: 2.2.7 CE.

Has anyone had similar behavior and how can I solve this?

[Mukesh Tiwari]

Hi @DK_dev 

Can you share the cron configuration which you made on server? It can help to identify the issue.

[Tarandeep Singh]

@DK_devNormally it does not piles up. After certain amount of minutes you may see the status as missed and an error message. There can be bunch of following questions which you may have already looked into..

• Do you know which job is causing the error? Many time indexing takes time due to big catalog and cause that error.
• While you are running the cron are you running out of memory or CPU etc?
• Is that appeared suddenly or after an upgrade/patch/code update etc...

You can slow down the cron from the Magento admin. In case running and generating jobs every minute generate for every five minute. This will slow down the processing a bit. Down side is you have to wait 5 minutes for your updates.

Problem solved? Please give 'Kudos' and accept 'Answer as Solution'.

[DK_dev]

I've got the following config. As you can see, everything is set to default values (other groups have the default settings as well).

Since restarting MySQL server yesterday I haven't had any issues (not sure yet if DB server config may be an issue), will check once more today if the issue still exists. Though there are missed cron jobs in cron_schedule database table, which is definitely not a good sign.

For some reason I have the following processes which have been running for 20 minutes now:

[DK_dev]

And that's the pile of php processes currently running, which makes server basically unresponsive. New cron jobs start while unfinished previous ones haven't finished yet.

I don't have a huge catalog - there are about 4500 products. While these cron jobs are running I have server load average growing infinitely high and server is slowly running out of memory. I can't recall making any code changes other that started using Magento 2 connector for Odoo ERP. Will try to disable these extensions to see if this helps as well as reschedule cron jobs for every five minutes.

[Mukesh Tiwari]

@DK_dev Your cron jobs are running every minute.

Try to run the job every 5 minutes instead  */5 * * * *

*/5 * * * * /usr/bin/php -c /etc/php5/apache2/php.ini /var/www/magento2/bin/magento cron:run > /var/www/magento2/var/log/magento.cron.log&

*/5 * * * * /usr/bin/php -c /etc/php5/apache2/php.ini /var/www/magento2/update/cron.php > /var/www/magento2/var/log/update.cron.log&

*/5 * * * * /usr/bin/php -c /etc/php5/apache2/php.ini /var/www/magento2/bin/magento setup:cron:run > /var/www/magento2/var/log/setup.cron.log&

[DK_dev]

Thank you for the reply!

I will try applying your solution and let you know how it works!

[DK_dev]

After updating cron schedule I still get some unknown processes like the following:

Is this normal behavior since I don't user New Relic Reporting?

[DK_dev]

These processes work for up to 5 minutes, often in parallel with cron jobs, causing server load average to rise to 15-20 on a 5-core virtual server.

[Tarandeep Singh]

@DK_dev  Can you make sure that New Relic integration is disabled and cron for the same is disabled too. If you have multiple site or store then make sure to switch the store and confirm.

Settings for the same can be found in Admin -> Store -> General -> New Relic Reporting.

Problem solved? Please give 'Kudos' and accept 'Answer as Solution'.