Magento Marketplace, Composer Repository, and Entitlements

[akent99]

You can buy extensions from the online Magento Marketplace (marketplace.magento.com) and they appear in your Composer repository for download (repo.magento.com), but how does this work behind the scenes? Why do I need to go through Marketplace to get free extensions, but not to get Magento Community Edition (CE)? And why does Magento Enterprise Edition (EE) appear only after I pay? This post describes the relationship between the Marketplace online store, the Composer repository, and the entitlements that link the two.

Composer Repository Structure

Composer is now the de facto way to download 3rd party PHP code. There is a public packagist.org repository that provides links to a wide range of public domain libraries. Each separately downloadable library of code is shipped in a separate “package”.

Magento 2 has adopted Composer as the standard way to download code and patches for the CE and EE products. It is also used to download extensions. Having all the code in our own repository makes it easier to do things like quality and security scans across the code – something we will continually monitor and improve upon.

(Quick side note, we still make ZIP files available for CE and EE for convenience, but these ZIP files are built from Composer. We just did a few steps for you. So the ZIP files you can consider the same as using Composer – they behave in the same way and have the same structure.)

Metapackages

Magento has the concept of “products” and “extensions”. Put simply, products refer to CE and EE and extensions are everything you add to a product. For example, each project has a “composer.json” file that will always contain a reference to CE or EE plus zero or more additional extensions. (For simplicity in this post I am going to include themes and language packs in with extensions – they behave exactly the same way.)

Composer however does not have the concepts of products and extensions – it only knows about “packages”. If you explore the Composer repository (e.g. use ‘curl’ to fetch https://repo.magento.com/packages.json using your repository keys for authentication) you will see a flat list of packages. Magento has taken the approach to create a Composer “metapackage” per product and per extension which references all the other packages to include. (Extensions with a single package can skip the metapackage.) So a project’s composer.json file points at a single metapackge for each product and extension to include, and that metapackage draws in all the individual packages that form the entire product or extension.

Entitlements

So how does the Magento Composer repository know which packages to make available to each user? It does so by authenticating users, a database of “entitlement” records, and product/extension metdata. There are several steps in this process.

• Authentication is required because different users have access to different products and extensions. Without knowing who the user is, the Composer repository cannot determine what to provide access to.
• Entitlement records identify per login what products and extensions that user has access to.
• The product/extension definitions then identify what packages make up that product/extension.

So it is the combination of authentication, entitlement records, and product/extension metadata (listing all packages in the product/extension) that drive what the Composer repository makes available for download.

Creating Entitlements

There are different ways in which entitlements are created that align with different user experiences.

• All users by default get a CE entitlement. That is why all users get access to CE by default without any additional effort.
• Users who contact our sales team and purchase EE get an EE entitlement record. (We also can create entitlements for partners in the same way.)
• Users who go to our Marketplace online store and purchase an extension get an entitlement record for that extension. Note that even free extensions on the Marketplace must go through the checkout flow in order to create the entitlement record. This was done on purpose to allow users to control what appears in their Composer repository.

So there are several different ways in which entitlements are created behind the scenes. This is the linkage between the online store and the Composer repository.

Magento Extensions

So what about Magento extensions? They will follow the same rules as above.

• If we decide that an extension should be in every user’s Composer repository by default, it will be aligned with the CE entitlement.
• If we decide that an extension should be free for every EE customer, it will be aligned with the EE entitlement.
• If we decide that an extension does not map on to one of the above cases, we will list it in the Marketplace (whether paid or free). It is the Marketplace checkout flow that will create the entitlement.

We may introduce other options in the future, but the internal mechanics of entitlement records will be the same for Magento extensions as community extensions.

Future Ideas

Internally we are discussing ideas about possible changes related to the Composer repository. These are just ideas – they have not been committed to (there is certainly no planned release date).

Currently everyone shares the same Composer repository URL. The repository authentication credentials are used to determine what the user can download. These credentials are typically shared by all developers on a project. There is support for creating multiple keys if the project owner wants to, allowing them to invalidate individual keys.

One possible change would be to create a different URL per project – like GitHub does for GitHub projects. Users would be then granted access to projects and entitlements would similarly be changed to be associated with projects. What this would allow is developers at a solution partner or agency to move more easily between projects without changing their identity. (In practical terms, the "~/.composer/auth.json" file for a user would not have to change when they switch between projects.)

Further, that in turn may allow users to user their normal Magento account name to authenticate with and control their own download password, eliminating the need for the current long hex sequences we have today (the “public and private keys”). It would still be a separate password from their normal Magento account as Composer stores this password clear text on disk, but it would potentially take some of the pain around key management away.

There are other similar options being considered. They may be incremental steps where keys are kept but the project structure is introduced. We are currently investing more into extension quality than improving the download experience for priority reasons, but community feedback on whether such changes to the Composer repository would help (or other ideas) are always welcome.