Import from Dataflow Advanced Products too slow

[ivi_d]

Hi to everybody in the community of magento.

Recently I have moved my webstore to a new dedicated server. I have done all the configurations on the php and mysql but I have problem with import.. In the previous server the import speed was about 50000 imports per hour. Now I can only import about 1500 items per hour.

As I noticed in phpMyAdmin, magento_db_user after inserting the data in the dataflow_bach_import goes in sleep mode and executes one query every 30 seconds and then goes again in sleep mode.

What is the problem and how can I fix it.

Thank you,

[magesolutions]

It belong to your hardware. How 's your server RAM, CPU?

[ivi_d]

The server runs in a Virtual Machine and below you can see its hardware.

system                  vServer

bus                        Motherboard

memory                96KiB BIOS
processor             ntel(R) Xeon(R) CPU E5-26xx (Sandy Bridge)
memory               16GiB System Memory
memory               16GiB DIMM RAM
bridge                  440FX - 82441FX PMC [Natoma]
bridge                  82371SB PIIX3 ISA [Natoma/Triton II]
storage                82371SB PIIX3 IDE [Natoma/Triton II]
bus                       82371SB PIIX3 USB [Natoma/Triton II]
bridge                  82371AB/EB/MB PIIX4 ACPI
display                 VGA compatible controller
network               82540EM Gigabit Ethernet Controller
storage               Virtio SCSI
generic               Virtio memory balloon

/dev/sda    disk       409GB SCSI Disk
/dev/sda1   volume     381GiB EXT4 volume

[hgpit]

We are seeing similar results running it on a VM.

We have given the VM 8GB RAM and 3 CPUS (CentOS6.7/PHP5.6/MariaDB10.0.23).

Takes over 12 hours (still waiting...) to import 77,000 products from a CSV. Our company's master products database is in excess of a million products...

Looking at the actual SQL statements sent from M2 to the DB, around the 32000 product mark, they are in excess of 1.3 million characters! So with every product insertion, it sends a 1.3MB+ data packet. These packets grow with *every* single product it imports (another 'OR' clause gets added). Seems very wrong to us.

We can't really justify spending £1,000's on a dedicated server, on the suggestion that these speed issues are just down to it being on a VM. Even on a VM, managing 77,000 records in a database should never take this long.

Very confused and disappointed with Magento 2 at this stage. I really hope we are overlooking something silly, like we haven't enabled 'FAST MODE'!

[ivi_d]

Finally I managed to deal with that problem by installing Magin. With this (free) extension now i van import 50000-70000 items per hour and i can update 50000 items in 10 minutes. All you need is to configure it correctly.

Thank you.

[hgpit]

Hi ivi_d, 

Forgive me, but I am having trouble finding Magin. Can you possibly fire a link over?

Thanks!

[ivi_d]

The MagMi documentation is here, you will find 2 download links there.

Another way to download it is from GitHub.

[hgpit]

Thank you

[Jeeva Rathinam]

Help me @ https://community.magento.com/t5/Technical-Issues/How-can-i-import-2-images-in-same-items-using-magm...