Solution for hosting magento (Millions of product images)

[kneeraj]

Dear All
While i have hosted many simple websites on opencart, woocommerce and magento having less than 100 products which worked well even on shared hosting plans with some hosting leaders like bluehost, inmotion etc.

But now i am stuck with a big ecommerce website to be made which could comprise of more than 1-2 million of products, each product having 5-6 images and each image could be upto 100KB of size.

Initially i thought of opting for some VPS, but VPS has limitations on SSD only upto 50-80GB SSD storgae, which will easily end up in 20-25 days:-

**Reg. project on magento**

 - We will be importing products from tmart, etsy to our store with
   images and description with a extension we have developed for
   importing the products.
 - Each day 9000-10000 products will be imported with their images.
 - This would range upto 1 Million products to 10 Million products in
   span of 3-6 months.

**Requested solution for problem i am into:**

1. Is there any other way to deal with hosting images with our ecommerce website on magento?
2. Like i have heard about AWS and google cloud (i have no experience though)
3. Our main problem for hosting will be hosting of so many images which could take upto 1TB of images if we continue to import products and scaleup.
4. How does companies like ebay,amazon manages so much images?
5. Is there any other solution available like, when we first import images onto our server, Media folder of magento will sync with other server (which is setup just for images) as magento provides subdomain feature for images media.

I have no idea whatsoever to deal with this problem and tried every post at stackexchange forum, webhostingtalk and other forums but still looking for the solution after 30 days.

[AHLeslie]

Hi Kneeraj,

For the most part, the summarized answer is you will simply need the funds for that much space, especially from a reputable host.

What you could do is use multiple small VPS servers as image gataways. The setup for that can be tricky, and it would require a good developer, but it's likely the cheapest option. 

Your next option to consider is collocation itself, in which you could either rent the hardware from the data center or send in your own hardware. In this case you should be able to easily get multiple terabytes of usable space. However, it will be costly.

Using Amazon or any other pay-by-the-usage major cloud provider will be the most expensive route, but also the easiest.

Sorry I couldn't be of more help, it would take awhile to delve into the aspects of each option.

[Finestshops]

Hi Kneeraj,

Hosting those images will be expensive, especially if you want to use a faster solid state drive space. One of our clients had the same problem - his store has almost 1 mil products with 3 mil to be added in the future and hosting all the product images would cost much more than they are willing to spend at this time. 

We customized that Magento store to hold product thumbnails and product images complete URLs in the database (instead of only image file name) and those URLs were pointed to the original image location used by the product supplier, so the products are loading from the Magento store we host but images are loading from the outside URL.
Just an idea to consider but you may want to check with your supplier first if you can use those images like that because they may block you at any time for excessive traffic.

[davkan]

@AHLeslie- Thanks for your kind words

As you have mentioned below:-

Using Amazon or any other pay-by-the-usage major cloud provider will be the most expensive route, but also the easiest.

But my developer is saying it is very cost effective. As below chart

https://aws.amazon.com/s3/pricing for Asia-Pacific Mumbai

Price is --> First 50 TB / month  -->  $0.025 per GB

Am i missing something? How does amazon will charge me?

[Finestshops]

Davkan,

Amazon AWS can be a good way to do this but make sure you calculate the expected storage and bandwidth:

- $0.023 per GB is $23/mo per TB or $1150/mo for 50TB of storage.

and this is for the storage only, do not forget, you need to pay for the bandwidth as well.

- $0.140 per GB which is $140 per TB or $7000/mo for 50TB of bandwidth.

Maybe you will not use 50TB or storage and bandwidth but make sure you calculate your approximate storage and bandwidth.

Another option is like Leslie suggested - get a custom dedicated server and stick multiple hard drives into it or better yet, set up a separate file server to get the amount of storage you need but make sure you pay attention to the cost of bandwidth because it may add up in your case.

Hope this helps.

[davkan]

@Finestshops

Thanks for the expert views.

I have also dig down and now came across amazon S3 bucket.

I saw that prices for S3 bucket is less as compared to amazon cloudfront CDN.

Also i think as our customer base is restricted to only one country, will S3 bucket give us some relief in above scenario of budget and image hosting?

What i basically expect from S3 bucket is:-

1. Host my images with my subdomain (maintaining magento media directory)

2. With https:// (not http) i.e SSL

3. Only for one country

Certainly i will explore other options as told by @AHLeslie once i will have funds to scaleup

[Finestshops]

you can check some articles below on how to use S3 for hosting images:

https://blog.codinghorror.com/using-amazon-s3-as-an-image-hosting-service/

https://learnetto.com/blog/tutorial-how-to-use-amazon-s3-and-cloudfront-cdn-to-serve-images-fast-and...

but i'm not sure if S3 will be less expensive. Note: the price there is per GB, not TB and you sill have to pay for the bandwidth.

[cloudcontrol]

Hi Kneeraj,

Sorry this search for the right solution has been difficult for you. See may be able to help as we've successfully managed sites with millions of SKUs/items and moved them into the cloud in a very cost effective way. We support customers in India in AWS Singapore and Mumbai as well as Google Cloud in Asia. Happy to connect you to a more technical resource here if this problem still exists. 

Sonal

[chathuramk]

Hi @kneeraj,

Can you please connect me to more technical resource?