Cron jobs pile up and never stop running, thus crashing the server

[DK_dev]

@Tarandeep Singh for every Magento store I have this extension disabled and it's cron disabled as well, as on the screenshot (default config is used). I've also disabled the New Relic Reporting module via Magento CLI.

[DK_dev]

Unfortunately due to unknown "newrelic" cron jobs, I still get a bad server performance issue. Has anybody else had similar problem?

[DK_dev]

After restarting MySQL server the issue disappeared. Will let the system to work for some time and see if the issue re-appears.

[Tarandeep Singh]

@DK_devThose pending task may be causing the issue. Now just keep an eye on the job. If you see any similar job which you are not using and then kill them.

You should be good. Next step I Can recommend it so optimize the Magento so that things can work more smoothly.

Problem solved? Please give 'Kudos' and accept 'Answer as Solution'.

[DK_dev]

I was able to figure out that the issue is related to cron jobs execution being delayed in cron_schedule DB table after some time (some cron jobs aren't executed on schedule and stay in "pending" status), which causes too many pending jobs execute at once during Magento cron:run command.

The solution I was able to figure out is to truncate "cron_schedule" table, which normalizes the cron schedule and lets cron jobs to execute normally on schedule without piling up.

[mcaashu133]

I was also facing the same issues with cron on one of my website  and it fills up my db cron table to 60,000 records thus  making system heavier, I tried this technique on two of my website and it now keeps the record to 12,000 max. 

First of all you need to truncate the cron table to empty it.
Go to admin > Stores>Configuration >Cron (Scheduled Tasks)> Cron configuration options for group: default and make Failure History Lifetime = 600

https://prnt.sc/oc0n5e

This solved my problem. Hope it helps you.

[DK_dev]

Somehow I've had the Magento cron working fine for about 4 days, no problems at all. Today suddenly the issue re-appeared. Not quite sure where will it be better to start digging, but try looking through logs to find the cause

[mcaashu133]

@DK_dev  Try the steps I have shared. I was benefited with those steps.

[DK_dev]

Unfortunately, after applying all the recommended settings, I'm still facing the issue. May it be related that cron jobs don't execute in time and stay in "pending" status due to inappropriate MySQL config (I have only a couple of settings set to non-default values) and how to find it out?

[DK_dev]

I noticed that if I delete the delayed cron jobs which stay in "pending" status, cron jobs execute normally according to schedule, but at some point cron jobs won't execute at crontab schedule and will be left for execution during next crontab heartbeat (set to execute every minute), thus delaying next cron jobs.