Not able to add/update 16k product programitally by text file

[webkeonsanjeev]

Hi Guys,

In my magento 2.3.5-p1 new website I am trying to add/update 16k product by txt file, My one magento instance have 2 websites and 6 stores in each website means 12 total stores.

One website (6 stores) 16 k products I am uploading in one time by txt file.

In my text file have first column is sku & then  store wise product title, short description, descriptioin, price, image and status.

By programitically I am checking If sku is available then update those product otherwise add new product. Below is my code.

Issue is that In script does not work if I use all 16 k products in one time, If I use only 300-400 product then it work but some time show duplicate url key error. 

My requirement is, I need to add/update all 16k product in one time 

<?php
include 'common.php'; if(!isset($_SESSION['DIID'])){$f->redirect(APP_URL);exit();}
use Magento\Framework\App\Bootstrap;
require __DIR__ . '../../app/bootstrap.php';
$bootstrap = Bootstrap::create(BP, $_SERVER);
$obj = $bootstrap->getObjectManager();
$state = $obj->get('Magento\Framework\App\State');
$state->setAreaCode('frontend');
set_time_limit(0);
$txtfile="upload/{Keyboards4Laptops}-Product-Update.txt";
$myfile = fopen($txtfile, "r") or die("Unable to open file!"); 
$array = fread($myfile,5600000);
$signlelinearray = explode("\n",trim($array));
//print_r($signlelinearray);
for($i=1;$i<count($signlelinearray);$i++){
$single=explode("\t",trim($signlelinearray[$i]));
try {
	$sku = $single[0];	
	
	// $url = preg_replace('#[^0-9a-z]+#i', '-', $single[1]);
	 //  $url = strtolower($url."-".$sku."-".1);
	 
	$product = $obj->get('Magento\Catalog\Api\ProductRepositoryInterface')->get($sku);		
	$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(1)
	//->setName($single[1])
	//->setUrlKey($url)
	 ->setPrice($single[16])
	->setShortDescription($single[11])
    ->setDescription($single[6])
    ->save();

 //$url = preg_replace('#[^0-9a-z]+#i', '-', $single[2]);
 //$url = strtolower($url."-".$sku."-".2);

$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(2)
	//->setName($single[2])
	//->setUrlKey($url)
	 ->setPrice($single[17])
	->setShortDescription($single[12])
    ->setDescription($single[7])
    ->save();
	
 //$url = preg_replace('#[^0-9a-z]+#i', '-', $single[4]);
  //$url = strtolower($url."-".$sku."-".3);
	
$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(3)
	//->setName($single[4])
	//->setUrlKey($url)
	 ->setPrice($single[19])
	->setShortDescription($single[14])
    ->setDescription($single[9])
    ->save();
	
 //$url = preg_replace('#[^0-9a-z]+#i', '-', $single[3]);
// $url = strtolower($url."-".$sku."-".4);
$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(4)
	//->setName($single[3])
	//->setUrlKey($url)
	 ->setPrice($single[18])
	->setShortDescription($single[13])
    ->setDescription($single[8])
    ->save();	
 //$url = preg_replace('#[^0-9a-z]+#i', '-', $single[5]);
 //$url = strtolower($url."-".$sku."-".5);
$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(5)
	//->setName($single[5])
	//->setUrlKey($url)
	 ->setPrice($single[20])
	->setShortDescription($single[15])
    ->setDescription($single[10])
    ->save();
 //$url = preg_replace('#[^0-9a-z]+#i', '-', $single[1]);
 //$url = strtolower($url."-".$sku."-".6);
$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(6)
	//->setName($single[1])
	//->setUrlKey($url)
	 ->setPrice($single[21])
	->setShortDescription($single[11])
    ->setDescription($single[6])
    ->save();
	
	//echo "UPDATED";
	
	} catch (\Magento\Framework\Exception\NoSuchEntityException $e){


$product = $obj->get('Magento\Catalog\Model\Product');

$status=$single[22]=='Enabled'?1:0;

$img="upload/image/".$single[0]."jpg";

$sku=$single[0];

$product
    ->setSku($single[0])   
	->setName($single[1])	
    ->setStatus($status)// Enable
    ->setWebsiteIds([2])
    ->setVisibility(4)
    ->setAttributeSetId($product->getDefaultAttributeSetId())
	->setImage($img)
    ->setTypeId('simple')
    ->save();

//echo "Saved product ID " . $product->getId() . "\n";

//$url = preg_replace('#[^0-9a-z]+#i', '-', $single[1]);
//$url = strtolower($url."-".$sku."-".1);

$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(1)
	->setName($single[1])
	//->setUrlKey($url)
	 ->setPrice($single[16])
	->setShortDescription($single[11])
    ->setDescription($single[6])
    ->save();

//$url = preg_replace('#[^0-9a-z]+#i', '-', $single[2]);
//$url = strtolower($url."-".$sku."-".2);

$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(2)
	->setName($single[2])
	//->setUrlKey($url)
	 ->setPrice($single[17])
	->setShortDescription($single[12])
    ->setDescription($single[7])
    ->save();

//$url = preg_replace('#[^0-9a-z]+#i', '-', $single[4]);
//$url = strtolower($url."-".$sku."-".3);

$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(3)
	->setName($single[4])
	//->setUrlKey($url)
	 ->setPrice($single[19])
	->setShortDescription($single[14])
    ->setDescription($single[9])
    ->save();

//$url = preg_replace('#[^0-9a-z]+#i', '-', $single[3]);
//$url = strtolower($url."-".$sku."-".4);

$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(4)
	->setName($single[3])
	//->setUrlKey($url)
	 ->setPrice($single[18])
	->setShortDescription($single[13])
    ->setDescription($single[8])
    ->save();	
	
//$url = preg_replace('#[^0-9a-z]+#i', '-', $single[5]);
//$url = strtolower($url."-".$sku."-".5);
	
$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(5)
	->setName($single[5])
	//->setUrlKey($url)
	 ->setPrice($single[20])
	->setShortDescription($single[15])
    ->setDescription($single[10])
    ->save();

//$url = preg_replace('#[^0-9a-z]+#i', '-', $single[1]);
//$url = strtolower($url."-".$sku."-".6);

$obj->create('Magento\Catalog\Model\Product')
    ->load($product->getId())
    ->setStoreId(6)
	->setName($single[1])
	//->setUrlKey($url)
	 ->setPrice($single[21])
	->setShortDescription($single[11])
    ->setDescription($single[6])
    ->save();
	
	//echo "CREATED";
	}
	
	//break;
}

fclose($myfile);

$table = array(
'activity' => 'Create product keyword'
);
$add_query = $db->insert('dimportlog', $table );
//exit;
?>

Thanks

Sanjeev

[gaurav_harsh1]

Hello @webkeonsanjeev 

The duplicate URL key error comes when multiple products are having same name, so make sure first if you are not having same product name.

Hope it works !

[webkeonsanjeev]

Thank you @gaurav_harsh1 thank you for your reply but my text file have can have same product name multiple time.

My main query is how can we add/update 16k product in one time

Thanks

Sanjeev

[gaurav_harsh1]

@webkeonsanjeev 

Are you referring textfile with CSV ?

if not than Magento provides a way to import products which you can find in admin at :

System > Data transfer > Import

there you can select your CSV for products and general details, you can download a sample CSV of coulmns as well.

So let me tell you how import works, CSV contains all details of product including URL key, if your CSV doesn't have URL for product then it will create url_key for it.

so here two things you must know :

1. if you have a huge data then it will create URL key one by one so it will take a huge time consumption for creating url-key for every product.

2. the URL key which Magento will generate will be based on product name so if your product has same name then definitely  it will create same url-key for both products, so when it will create second time, it will throw error of same URL-key and execution will stop.

Solution : You can pass URL-key from your end so Magento doesn't need to create themselves so the time will decrease and also even if you have products name, you can pass a unique key from your end.

for creating URL key, you can create a very basic script that what your url key should be.

for ex: if you have two products with same name "Men Shoes" then Magento will create URL rewrites as :

url_key(product 1) => men-shoes

url_key(product 2) => men-shoes

for both of the products which will collapse later.

so you can do like, you can handle the similar products like

url_key(product 1) => men-shoes

url_key(product 2) => men-shoes-1

url_key(product 3) => men-shoes-2

url_key(product n) => men-shoes-[n-1]

Hope you are clear now, good luck !