how to import 40,000 products hourly

[justinblayney]

Hello,

We have 40,000 products that we need to import Magento. Initially we just need to get the products in the system, this will be one massive import of all the products data including 30 custom attributes.

After our initial import we will need to update the inventory of each product hourly, most likely with a cron job

What is the best way to do this..

So far I imported 1000 product categories and this took 12 hours and kept timing out, Magento is slower than molasses, so I need a 100% guaranteed way to import this volume of products

[Brent W Peterson]

You product updates will go faster than product imports so you can update 40,000 products in an hour easily.

What are the specs of your server, it sounds a little under powered?

If you are looking for something really fast try this 

https://secure.unirgy.com/products/urapidflow/

[justinblayney]

Hello,

Thank you for getting back to me.

It is a shared server, do I need a dedicated server? If I get what what are the minimum specs it needs to run Magento quickly and smoothly?

Do you have an opinion on what is URapidFlow or Magmi?

[justinblayney]

Server specs

24 processors, all of them

Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz

Speed 2600.024 MHz

Cache 15360 KB

Memory: 32822208k/35651584k available (5329k kernel code, 2132792k absent, 696584k reserved, 7010k data, 1280k init)

[JLHC]

With such a huge amount of products, shared hosting wouldn't cut it. 

You may need to look for a more powerful server with guaranteed resource, which can either be a Cloud Server, Dedicated Server or VPS. Be sure to also look for a Managed solution as it will save you a lot of headache on server management, security, optimizations, etc. 

[pinkdigital]

Realistically would all 40000 records change hourly? Your best option would be to only update those records that have changed.

Not sure a single dedicated server would handle such a large amount of data, whilst serving your website as well. Could you run an additional server just for updating?

[justinblayney]

They wouldn't all change completely, and only the inventory and possbly some prices would change, i'll check if old database has a timestamp on updates

[kgerasimov]

you should consider using MAGMI for this task.

Magmi is a script that runs direct SQL queries updating your product information thus it is very fast.

It can be scripted to run with cron - check it out - https://sourceforge.net/projects/magmi/

[justinblayney]

thank you, i will check it out. I keep hearing about it

[CommerceStudio]

Magmi is fine for initial loads but with large datasets you have the indexing issue.

You can normally mitigate it with the correct hosting and a multi-server environment.

Otheriwse you need delta loading with an external app that checks the contents for changes before load.