Plan for media storage API?

[apiuser]

Feature request from owenhaynes, posted on GitHub May 31, 2015

Is there any plan for a media stroage api, was looking to see if it was possible to use S3/Cloud Stroage for media assets. So that Elastic Beanstalk can be used effectively.

I don't think storing the images in the database is the correct option when S3 storage is far cheaper and faster to run then a database.

Have had a look at the code and it seems very tightly coupled with File and Database, so creating a plugin seems a lot of work.

[apiuser]

Comment from jameshalsall, posted on GitHub Feb 03, 2016

I'm working on a module for Magento2 that will offer S3, Google Cloud Storage for assets. Will post it here when it's done.

Sent from my iPhone

On 3 Feb 2016, at 10:23, Khaleel ibn Anwar Mughal notifications@github.com wrote:

Sounds good. I too am using Rackspace UK as LON region. Also international API. It would have been nice to have official support from the Magento platform for CDN integration rather than modules. Varnish (personally not a fan) and things like caching come out of box. CDN should be default now across Enterprise.

— Reply to this email directly or view it on GitHub.

[apiuser]

Comment from ostark, posted on GitHub Feb 03, 2016

@jameshalsall +1

[apiuser]

Comment from craigcarnell, posted on GitHub Feb 04, 2016

@jameshalsall Sounds good, maybe I can extend it for Rackspace

[apiuser]

Comment from amansrivastava, posted on GitHub Feb 04, 2016

@jameshalsall Looking forward for the implementation. This will help me alot in using S3. Thanks in advance.

[apiuser]

Comment from jameshalsall, posted on GitHub Feb 14, 2016

@alankent @mazhalai

Whilst I think this is "achievable" with a 3rd party module through the combination of plugin code and completely extending core Magento classes and replacing them through DI, I don't think this is a good solution. Currently the Magento\MediaStorage\Helper\File\Storage\Database class does not implement any interface meaning that it's difficult to know exactly what needs to be implemented in a custom storage layer.

I'm not even sure why it's referred to as database storage when all of the files reside on disk, not in the database. The only information stored in the database are the asset paths relative to the media path, and this would continue to be the case if we swapped it out for cloud storage alternatives. So "Database" is a poor name for this class.

To allow a bespoke storage layer we will need interfaces creating that will allow us to define logic for

• creation of assets
• deletion of assets
• modification of assets
• listing of assets

Everything in Magento\MediaStorage\Helper\File\Storage\Database only handles the management of assets, not the fetching of them. To customise behaviour of fetching assets, I can see a couple of places that would need to be refactored for images (although I could be wrong so please correct me) but haven't found anything on generic file assets on my travels through M2 yet:

• \Magento\Catalog\Block\Product\ImageBuilder::create()
• \Magento\Catalog\Helper\Image::getUrl()

Again there needs to be a clear set of interfaces defined that handle the fetching of assets which would allow us to customise this behaviour. The workflow I was hoping to easily implement in a module would be:

1. Check if asset exists locally
2. If not, fetch it from storage bucket e.g. S3
3. Cache asset locally
4. Return local URL for asset

@alankent Your suggestion of just using a CDN doesn't really make sense, how is a Magento2 store currently meant to scale horizontally if all of the assets reside on phyiscal disk? The only easy solutions I can see right now are:

• s3:// stream wrappers which makes for an extremely slow store
• shared disk services like gluster or EFS (which still are not ideal)

Please let me know if I'm missing something obvious here.

I look forward to hearing your thoughts

[apiuser]

Comment from thaiphan, posted on GitHub Feb 15, 2016

Hey everyone,

Here is an extension that I've built that allows you to sync all your Magento 2 media files, e.g. WYSIWYG images, product images, category images, to Amazon S3.

https://github.com/arkadedigital/magento2-s3

Regards,

Thai

[apiuser]

Comment from craigcarnell, posted on GitHub Feb 15, 2016

@thaiphan Thanks for that. If a file is not on the CDN does it still fall back to media?

[apiuser]

Comment from thaiphan, posted on GitHub Feb 15, 2016

@craigcarnell

There are actually different setups that you can use that allows fallback to media.

CloudFront

If you're using CloudFront then you can set different origin servers for your images. One origin server can be S3 (which Magento should automatically sync newly uploaded images to) and then you can set up Magento to be your second origin server for things like thumbnails (which needs to first be generated via Magento before being automatically synced to S3).

S3

If you're not using CloudFront then you can use S3's routing rules to allow you to fallback to Magento if a object doesn't exist in the bucket. Here's a quick article explaining how the routing rule engine works:

http://hipsterdevblog.com/blog/2014/06/22/lazy-processing-images-using-s3-and-redirection-rules/

Magento

Another method is if you're routing your media files through get.php then Magento will download an image from S3 if it doesn't already exist on the server.

I hope that was helpful!

[apiuser]

Comment from mrkhoa99, posted on GitHub Feb 15, 2016

@thaiphan +1

[apiuser]

Comment from peec, posted on GitHub Feb 22, 2016

@thaipan thanks, tested it and it works. Helpful hint is also just to set media url under Store -> General -> Media URL unsecure to S3 bucket or CDN url.

Core Team, are you going to officially support S3 and other storages ? Magento2 can't scale using regular file system storage.