M1: When frequently adding products programatically, what are best practices around reindexing?

[tmillsmonin]

Here is my setup. I have reindex on save enabled as well as EE Smart Categories. I'm calling an API every 15 minutes to pull in new products - two our three new products may be added in a single "sync" (but not every sync, of course). When the cron begins the module turns off index on save, adds the products and then reindexes - it does not check to see if the indexer is already executing.

One thing I am noticing is a lot of "Integrity Constraint Violations" when saving products after enabling this module. Also, we ran into an issue yesterday where only 3 of 10 total products were being displayed in a category as if the indexer had failed. After I executed php -f shell/indexer.php reindexall the products were restored to the category.

So, I'm clearly doing something wrong but am not quite sure what it is.

[tj_gamble · Accepted solution]

If you are running the latest version of EE, indexing is more of an automated process than in the past.   Try just not re-indexing after the update in your module and let Magento handle it.   Within a short timeframe (if cron, etc is running as expected) it should automatically re-index.

[tmillsmonin]

I turned off the index "killer" in the module as well as updated my index settings to Update on Schedule. My understanding is that this runs each time the cron Magento cron is run - which would be every minute based on magento's recommended settings (which I adhere to). Is that correct?

[tj_gamble]

Cron runs according to your schedule, but sometimes things can back up a bit by tasks running long.  So, it's not exactly every minute that things will execute, but it should happen very frequently as needed.