Magento Packages Repository update

taras.pogrebniak · ‎04-27-2020

Current state

Every time a Composer client requests packages from https://repo.magento.com, the first call is to get packages.json - a file that contains information about all packages that the current customer can download. Here is an example of the current structure of packages.json:

{
    "packages": {
        "magento/composer": {
            "1.0.2": {
                "name": "magento/composer",
                "description": "Magento composer library helps to instantiate Composer application and run composer commands.",
                "keywords": [],
                "homepage": "",
                "version": "1.0.2",
                "version_normalized": "1.0.2.0",
                "license": [
                    "OSL-3.0",
                    "AFL-3.0"
                ],
                "authors": [],
                "dist": {
                    "type": "zip",
                    "url": "https://repo.magento.com/archives/magento/composer/magento-composer-1.0.2.0.zip",
                    "reference": null,
                    "shasum": "6bfdbff4c23aace1e6d14ab598c81c790375aba0"
                },
                "type": "library",
                "autoload": {
                    "psr-4": {
                        "Magento\\Composer\\": "src"
                    }
                },
                "require": {
                    "php": "~5.5.0|~5.6.0|~7.0.0",
                    "composer/composer": "1.0.0-alpha10",
                    "symfony/console": "~2.3 <2.7"
                },
                "require-dev": {
                    "phpunit/phpunit": "4.1.0"
                }
            },
            "1.2.0": {
                "name": "magento/composer",
                "description": "Magento composer library helps to instantiate Composer application and run composer commands.",
                "keywords": [],
                "homepage": "",
                "version": "1.2.0",
                "version_normalized": "1.2.0.0",
                "license": [
                    "OSL-3.0",
                    "AFL-3.0"
                ],
                "authors": [],
                "dist": {
                    "type": "zip",
                    "url": "https://repo.magento.com/archives/magento/composer/magento-composer-1.2.0.0.zip",
                    "reference": null,
                    "shasum": "303513c83d4f6562ff1480a954bb298b37261fb3"
                },
                "type": "library",
                "autoload": {
                    "psr-4": {
                        "Magento\\Composer\\": "src"
                    }
                },
                "require": {
                    "php": "~5.5.0|~5.6.0|~7.0.0|~7.1.0",
                    "composer/composer": "1.4.1",
                    "symfony/console": "~2.3, !=2.7.0"
                },
                "require-dev": {
                    "phpunit/phpunit": "4.1.0"
                }
            }
        }
    }
}

The issue here is that this file grows continuously. With every new Magento release, the file becomes bigger and bigger. Every change in the list of available packages causes changes in the packages.json file and the Composer client must re-download it every time.

Solution

To solve this issue, we will use providers: https://getcomposer.org/doc/05-repositories.md#provider-includes-and-providers-url. Instead of one huge list of packages, the Composer client will get a list of 'provider-includes'. Packages will be grouped into providers based on editions: 'CE', 'EE' and 'B2B': 'CE provider-include' will contain the list of Core CE packages, 'EE provider-include' will contain the list of Core Enterprise packages, 'B2B provider-include' will contain the list of Core B2B packages. There will also be a 'Marketplace provider-include' for packages bought on https://marketplace.magento.com. Each provider-include will contain a list of providers of available packages. Each package will have its own provider.

Benefits

The Composer client will cache all providers locally and will download only the updated providers when a new version of a package is available. All other providers will be taken from the local cache instead of pulling the huge packages.json file every time a new version is available.

Example of packages.json structure with providers

{
    "packages": [],
    "providers-url": "/p/%package%$%hash%.json",
    "provider-includes": {
        "p/provider-ce$%hash%.json": {
            "sha256": "f80acc93935ca21dfaa91ed18d4e27a63d48612daf9ba3a609d14ae82c4a8162"
        }
    }
}

Caution

The Composer client is able to work with both approaches, so there is no need to change anything if you only get packages from repo.magento.com via Composer. You may have noticed that the 'packages' array is empty. If you parse 'packages' from the packages.json file directly for some automation scripts, you must change your application logic, as the 'packages' array will be empty. Instead, information about versions will be contained in the individual package provider file:

{
    "providers": {
        "magento/magento-semver": {
            "sha256": "8b59a0ec802fcedd30d718b74e675e1c14cdf75a901125a0a624e77a8e7eccc0"
        },
        "magento/magento-cloud-patches": {
            "sha256": "7fe70630ff4546653f12abac9fcb08b77637508fb1abb8a59e2937c2a0806530"
        },
        "magento/magento-cloud-components": {
            "sha256": "65aa4c251bf71d64d75515b5b4a08d2200adf1406e6ea2e537845c397a617435"
        },
        "magento/module-graph-ql-cache": {
            "sha256": "2dad357574b1c84acf7806aea1565329aeaa47b5911b7728daded1666ac5f910"
        },
        "magento/module-graph-ql": {
            "sha256": "f6ac1b0e3aa1a3eb4bd41753b76cfd6164cac24b980289f5494fe3a35322060f"
        }
    }
}