custom product sync

[mayhem1]

I'm trying to sync an offline product database with Magento 2.3.1.  The builtin sync does not work how we need it to.

I have originally been trying to upload a csv and then process it with a php script that uses the PHP API (not rest/soap). This isn't working because the php script is triggered via the browser and just times out or hits memory limits. 

Even if i used the rest api from a php script, it would have the same problem i'm assuming. 

I'm looking for recommendations on the best way to do this. 

[Sunil Patel]

Hello @mayhem1 

if it is PHP script then you can run using the command line as well, for this, you need ssh access.

e.g. your script name is test.php

then run using php test.php ( no time out occurs)

Hope it will help you if works then mark as a solution.

[mayhem1]

Hi, thanks for the suggestion. 

I have already tried the command line and it gets around the timeouts but memory limits still get hit eventually. Also, I was hoping to use the magento gui, so i can take advantage of the magento user access control instead of using shell accounts. I have a custom gui module set up for this. If i could get around the cmdline memory issues, i was thinking i could use magento gui module to allow someone to upload the csv and then have a cron job to look for and process newly uploaded csv's. Maybe the REST api is the way to go. I think most of the memory usage is from object instantiation and i guess if each rest request is processed on it's own, it would limit the instantiation within my php script process. Can anyone tell me if this makes sense ? haha

[mayhem1]

This explains my issue. I haven't done enough php to realise these limitations until now. https://wpshout.com/beyond-avoiding-php-timeout-memory-limit-errors-ajax/ 

[mayhem1]

I think i found a potential solution. Working on implementing it. Instead of doing the full product sync at once, i'm doing it in stages and after each stage, i'm creating a new php request so the timeout / memory refreshes. Tracking the stage via POST variables and auto-submitting a form after each iteration.

echo '<form id="LtsRedirect" action="'.$block->getUrl('ltsproductsync/main/index').'" method="post">';
echo '<input type="hidden" name="form_key" value="'.$block->getFormKey().'"/>';
echo '<input type="hidden" name="start_sync_confirmed" value="true" />';
echo '<input type="hidden" name="sync_stage" value="'.$iSyncStage.'" />';
if($iSyncStage == 4){
echo '<input type="hidden" name="chunk_index" value="'.$iChunkIndex.'" />';
echo '<input type="hidden" name="total_products" value="'.$iTotalProducts.'" />';
}
echo '<input type="submit" value="Not Used" style="display:none;"/>';
echo '</form>';
echo "<script type=\"text/javascript\"> document.forms['LtsRedirect'].submit();</script>";