Cron 'catalogrule_apply_all' locks cron 'indexer_all_invalid' and catalogsearch_fulltext stuck
Took me quite some time to debug this. Every night (default) at 0:00 catalogrule_apply_all is run by cron, and resets all indexes for all store. As a consequence catalogserach_fulltext is launched (takes about 8 minutes in my store).
At first five items, but after a day, it's about 60 of these, every minute. And catalogsearch_fulltext is then processing constantly. If it finished, another reindex is started again. Probably, because the all invalid index doesn't work anymore. indexer_reindex_all_invalid does not come back in action again.
The system does not unlock indexer_reindex_all_invalid once this error occurs.
If I do a grep I then see that the cronjob cron:run seems to hang and for about 20 minutes I have cron.php called at least twice.
[USER]+113210.00.04340724?Ss10:090:00/bin/sh -c /usr/bin/php /[PATH]/public_html/bin/magento cron:run 2>&1| grep -v "Ran jobs by schedule">>/[PATH]/public_html/var/log/magento.cron.log
[USER]+1132560.24.8590908197924? R 10:0910:14/usr/bin/php /[PATH]/public_html/bin/magento cron:run
[USER]+125420.00.04340772?Ss10:260:00/bin/sh -c /usr/bin/php /[PATH]/public_html/bin/magento setup:cron:run >>/[PATH]/public_html/var/log/setup.cron.log
[USER]+1254457.02.148427288712? R 10:260:01/usr/bin/php /[PATH]/public_html/bin/magento setup:cron:run
All crons work perfect, normally - no single cron missed in cron_schedule. I can reset the indexing by truncating all _cl tables, resetting mview_state version_id, truncating cron_schedule. Manual reindex and flush cache.
Everything works perfectly (also indexer_reindex_all_invalid) untill the next time 'catalogrule_apply_all is launched. I tested this by adding this manually to cron_schedule.
But now? I don't know what to do anymore, hopefully you have an answer.
The error is thrown by /vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php I wouldn't know if this needs updating or adjustment.
I do not have any catalog rules. Tried making one and deleting again, no difference.
I'm using Elasticsearch plugin by Mirasvit, Magento 2.3.5 on a cloudways server. Using Redis for sessions. (don't think it is Mirasvit related).
In env.php I have 'lock' set to 'db', I think just default settings.
I've set all cronjobs in config->system for seperate process to 'no'.
Migrated site from 1.9.x to 2.3.5
It is just the catalogrule_apply all that is messing up things. Does anyone have a suggestion?