Hello All,
We are a marketplace (say MP.com) and aggregate products from 100s of ecommerce companies and sell on our website. We have built our marketplace using Magento CE 1.9.X. Currently we integrate our marketplace with ecommerce retailers (say R1.com, R2.com.. R100.com) which are only built on Magento. All the products available on various Retailer Magento stores (R1.com, R2.com… R100.com) are imported into our system using a custom extension we have created and which is installed by us on each of these retailer websites (R1.com, R2.com etc.). The products which are imported into our system can be bought from our site (MP.com). It’s also possible for a customer to go to R1.com and using a link on R1.com; come to our website (MP.com) and do the checkout. We provide value added services which most retailers don’t provide and thus retailers don’t mind sending these customer to us.
Each retailer (R1.com etc.) has one store view setup on our marketplace site (R1.MP.com) and it shows only the products offered by a given retailer. Currently we have on-boarded about 100 retailers and thus we are integrating our marketplace with 100 different magento stores. We carry 200,000 products in our marketplace website.
If any product is changed in any retailer’s website (say R1.com), then the system prompts the retailer to sync this product with our Magento system (MP.com). If the retailer doesn’t sync the product, we have a cron which runs in the night at 1AM and all the changes for a product get synched.
We are rapidly expanding and are facing a specific issue as described below:
Problem:
We were facing problem with reindexing in our Magento system (MP.com) . When requests are coming in parallel from multiple retailers for sync; product import and reindexing are going on in parallel. When import and reindexing are running together, we are facing Deadlock issues. We are using Fast Asynchronous Reindex as we can not run full reindex for 200,000+ products everytime a product is being synced
Temporary Solution we are using:
To avoid this problem, we have come up with a Queuing system. Once import, reindex and sending sync status is completed for one retailer; then only the import, reindex and sending sync status will start for second retailer and so on. Our system is working but most of the times other retailers have to wait for a long time (sometime upto 24 hours) for products to be synched. For example 50 retailers has requested for syncing 500 products each; this means we need to sync 500*50 i.e. 25000 products. For importing 500 products, reindexing and sending sync status to each retailer, it takes close to 45 minutes (Reindex itself takes 40 minutes. So 50th vendor has to wait for 45*50 i.e. 2250 minutes which is 37.5 hours. This is becoming a bottleneck and we need to get this process completed in maximum of 2-3 hours.
Solution Required:
How can our Magento system (MP.com) handle multiple imports & reindex in parallel and running requests for multiple retailers in parallel?
What solution will you propose, so that system can handle requests without deadlock and it should take much lesser time to sync products?
How will we ensure that the new proposed system will be able to handle more retailers as we grow and upto how many retailers it will work fine? (We may have 500 retailers as well)
Thanks
