cancel
Showing results for 
Search instead for 
Did you mean: 

how to import 40,000 products hourly

how to import 40,000 products hourly

Hello,

 

We have 40,000 products that we need to import Magento. Initially we just need to get the products in the system, this will be one massive import of all the products data including 30 custom attributes.

 

After our initial import we will need to update the inventory of each product hourly, most likely with a cron job

 

What is the best way to do this..

 

So far I imported 1000 product categories and this took 12 hours and kept timing out, Magento is slower than molasses, so I need a 100% guaranteed way to import this volume of products

 

9 REPLIES

Re: how to import 40,000 products hourly

You product updates will go faster than product imports so you can update 40,000 products in an hour easily.

 

What are the specs of your server, it sounds a little under powered?

 

If you are looking for something really fast try this 

 

https://secure.unirgy.com/products/urapidflow/

Magento Moderator since 2009
Keep Calm and Clear Cache!

Re: how to import 40,000 products hourly

Hello,

 

Thank you for getting back to me.

 

It is a shared server, do I need a dedicated server? If I get what what are the minimum specs it needs to run Magento quickly and smoothly?

 

 

Do you have an opinion on what is URapidFlow or Magmi?

 

Re: how to import 40,000 products hourly

Server specs

 

24 processors, all of them

 

Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz

Speed 2600.024 MHz

Cache 15360 KB

 

 

Memory: 32822208k/35651584k available (5329k kernel code, 2132792k absent, 696584k reserved, 7010k data, 1280k init)

 

Re: how to import 40,000 products hourly

With such a huge amount of products, shared hosting wouldn't cut it. 

 

You may need to look for a more powerful server with guaranteed resource, which can either be a Cloud Server, Dedicated Server or VPS. Be sure to also look for a Managed solution as it will save you a lot of headache on server management, security, optimizations, etc. 

James Lee | Moderator • Magento Master
See My Recommended Magento Hosting & Security Tips

Re: how to import 40,000 products hourly

Realistically would all 40000 records change hourly? Your best option would be to only update those records that have changed.

 

Not sure a single dedicated server would handle such a large amount of data, whilst serving your website as well. Could you run an additional server just for updating?

Triple Certified Magento Specialist.
Magento Solution Specialists

Re: how to import 40,000 products hourly

They wouldn't all change completely, and only the inventory and possbly some prices would change, i'll check if old database has a timestamp on updates

Re: how to import 40,000 products hourly

you should consider using MAGMI for this task.

 

Magmi is a script that runs direct SQL queries updating your product information thus it is very fast.

 

It can be scripted to run with cron - check it out - https://sourceforge.net/projects/magmi/

Re: how to import 40,000 products hourly

thank you, i will check it out. I keep hearing about it

Re: how to import 40,000 products hourly

Magmi is fine for initial loads but with large datasets you have the indexing issue.

 

You can normally mitigate it with the correct hosting and a multi-server environment.

 

Otheriwse you need delta loading with an external app that checks the contents for changes before load.