Took me quite some time to debug this. Every night (default) at 0:00 catalogrule_apply_all is run by cron, and resets all indexes for all store. As a consequence catalogserach_fulltext is launched (takes about 8 minutes in my store).
In the meantime, new crons run, and logs begin:
main.WARNING: Could not acquire lock for cron job: indexer_reindex_all_invalid [] []
At first five items, but after a day, it's about 60 of these, every minute. And catalogsearch_fulltext is then processing constantly. If it finished, another reindex is started again. Probably, because the all invalid index doesn't work anymore. indexer_reindex_all_invalid does not come back in action again.
The system does not unlock indexer_reindex_all_invalid once this error occurs.
If I do a grep I then see that the cronjob cron:run seems to hang and for about 20 minutes I have cron.php called at least twice.
[USER]+ 11321 0.0 0.0 4340 724 ? Ss 10:09 0:00 /bin/sh -c /usr/bin/php /[PATH]/public_html/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /[PATH]/public_html/var/log/magento.cron.log
[USER]+ 11325 60.2 4.8 590908 197924 ? R 10:09 10:14 /usr/bin/php /[PATH]/public_html/bin/magento cron:run
[USER]+ 12542 0.0 0.0 4340 772 ? Ss 10:26 0:00 /bin/sh -c /usr/bin/php /[PATH]/public_html/bin/magento setup:cron:run >> /[PATH]/public_html/var/log/setup.cron.log
[USER]+ 12544 57.0 2.1 484272 88712 ? R 10:26 0:01 /usr/bin/php /[PATH]/public_html/bin/magento setup:cron:run
All crons work perfect, normally - no single cron missed in cron_schedule. I can reset the indexing by truncating all _cl tables, resetting mview_state version_id, truncating cron_schedule. Manual reindex and flush cache.
Everything works perfectly (also indexer_reindex_all_invalid) untill the next time 'catalogrule_apply_all is launched. I tested this by adding this manually to cron_schedule.
But now? I don't know what to do anymore, hopefully you have an answer.
The error is thrown by /vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php I wouldn't know if this needs updating or adjustment.
Additional info
- I do not have any catalog rules. Tried making one and deleting again, no difference.
- I'm using Elasticsearch plugin by Mirasvit, Magento 2.3.5 on a cloudways server. Using Redis for sessions. (don't think it is Mirasvit related).
- In env.php I have 'lock' set to 'db', I think just default settings.
- I've set all cronjobs in config->system for seperate process to 'no'.
- Migrated site from 1.9.x to 2.3.5
It is just the catalogrule_apply all that is messing up things. Does anyone have a suggestion?
