Hi to everybody in the community of magento.
Recently I have moved my webstore to a new dedicated server. I have done all the configurations on the php and mysql but I have problem with import.. In the previous server the import speed was about 50000 imports per hour. Now I can only import about 1500 items per hour.
As I noticed in phpMyAdmin, magento_db_user after inserting the data in the dataflow_bach_import goes in sleep mode and executes one query every 30 seconds and then goes again in sleep mode.
What is the problem and how can I fix it.
The server runs in a Virtual Machine and below you can see its hardware.
memory 96KiB BIOS
processor ntel(R) Xeon(R) CPU E5-26xx (Sandy Bridge)
memory 16GiB System Memory
memory 16GiB DIMM RAM
bridge 440FX - 82441FX PMC [Natoma]
bridge 82371SB PIIX3 ISA [Natoma/Triton II]
storage 82371SB PIIX3 IDE [Natoma/Triton II]
bus 82371SB PIIX3 USB [Natoma/Triton II]
bridge 82371AB/EB/MB PIIX4 ACPI
display VGA compatible controller
network 82540EM Gigabit Ethernet Controller
storage Virtio SCSI
generic Virtio memory balloon
/dev/sda disk 409GB SCSI Disk
/dev/sda1 volume 381GiB EXT4 volume
We are seeing similar results running it on a VM.
We have given the VM 8GB RAM and 3 CPUS (CentOS6.7/PHP5.6/MariaDB10.0.23).
Takes over 12 hours (still waiting...) to import 77,000 products from a CSV. Our company's master products database is in excess of a million products...
Looking at the actual SQL statements sent from M2 to the DB, around the 32000 product mark, they are in excess of 1.3 million characters! So with every product insertion, it sends a 1.3MB+ data packet. These packets grow with *every* single product it imports (another 'OR' clause gets added). Seems very wrong to us.
We can't really justify spending £1,000's on a dedicated server, on the suggestion that these speed issues are just down to it being on a VM. Even on a VM, managing 77,000 records in a database should never take this long.
Very confused and disappointed with Magento 2 at this stage. I really hope we are overlooking something silly, like we haven't enabled 'FAST MODE'!
Finally I managed to deal with that problem by installing Magin. With this (free) extension now i van import 50000-70000 items per hour and i can update 50000 items in 10 minutes. All you need is to configure it correctly.
Forgive me, but I am having trouble finding Magin. Can you possibly fire a link over?