This blog post is a summary of changes coming in 2.2 related to deployment. This is based on my talk at MagentoLive UK 2017. Improvements include:
- The removal of static asset generation dependence on the database (so it can be run on a Jenkins or similar server),
- Performance improvements for compilation and static asset generation (much of which has already been backported to 2.1.x patches), and
- Better per environment vs. shared configuration support.
Acknowledgements go to the great team behind the scenes that designed and built these new features.
12 Factor Apps
12 factor apps talks about good patterns for web applications to follow to make deployment easier. These factors are generally accepted best practices, and the new features directly align with these factors. If you are interested in deployment strategies, the 12 factor app site is a good read.
Better Support for a Build Phase
Currently, a typical Magento deployment procedure will put the server in maintenance mode, get the new code, update generated code and web asset files, upgrade the database schema if needed, then go live again. This however takes the site down for that duration. A key goal of the Magento 2.2 improvements is to shorten the length of time a site is down for.
As can be seen in the diagram above, compilation (di-compile), web asset generation (static-deploy), and the database schema upgrade can all add undesirable delays to site deployment downtime. Good deployment strategies improve upon this by at least moving the compilation and web asset generation out of the maintenance window by running them in a separate directory, but this still requires these commands to be run on the production server where you really want to keep the capacity available for customer requests.
To improve upon this, Magento 2.2 optimizes the compilation and web asset generation steps and removes the database dependency so that these steps can be run on a different server. Many of the optimizations have also been backported to Magento 2.1.x patches
So how will the above look with a new build phase?
Notice that the database dependence of the web asset generation step has been removed. The code now can get the settings from config.php, env.php, or environment variables. Note also that an automated test step has been added to the above. Running tests per build (this would include setting up a database if functional or integration tests are to be run) may automatically detect an issue with the optimized production code before it hits your production server, providing an extra level of safety. Advanced sites might also include performance measurements during testing to stop unexpected code slow-downs from getting deployed.
No improvements to the database schema upgrade step have been done in 2.2, but we are considering an additional command to test if any data formats have changed (including Redis). If no such formats have changed, then it is likely safe to run the old and new code in parallel. Basically, if the only code changes are minor, then you can look at techniques such as spinning up new servers with the new code and flipping a load balancer from the old web servers to the new web servers atomically. That can result in a zero-downtime deployment. Even if that is not possible, you can still minimize the downtime for the site by avoiding the database schema upgrade step. The new flow may look something like the following.
The Compact Web Asset Generation Strategy
One additional speed improvement in Magento 2.2 is a new “compact” deployment strategy. It works by identifying files that do not change per locale, such as the jQuery.js library. By writing such files to disk once instead of once per locale, the web asset generation step is faster and the time taken to transfer the files to the production server and extract them is also reduced.
In one test, a site with 3 locales completed web asset generation 2x faster using the “compact” deployment strategy and another site with 15 locales completed 10x faster. This is due to most of the JavaScript libraries never needing to be localized. Localization is more common for images and CSS files.
To use the new mode, a new “--strategy” command line option is available.
$ magento setup:static-content:deploy --strategy compact ...
The command writes out a map.php and requirejs-map.js file for the URL generation functions in PHP and JavaScript files to use. Common files are written into a base directory for per locale references to share.
Configuration as Code
Another 12 Factor App consideration is to treat configuration as code. It should be under source code version management and propagated as a part of your standard deployment process. For this reason, Magento has formalized that config.php is intended to be committed to git and shared between all environments. env.php overrides settings in config.php and is expected to be local per environment. (Environment variables can also be used to override the env.php file.)
In addition, the list of web sites, stores, and store views can now be put into the config.php file. This allows the web asset generation commands to run without database access, and allows the deployment of new scopes to be pushed between sites, along with themes during deployment of the code.
Normally, developers would define new websites and store views in the Admin, then dump the settings to a configuration file using the magento app:config:dump command. During deployment, other sites would run either magento app:config:import to synchronize the database with the dumped settings, or use magento setup:upgrade which will perform the import and perform any other database upgrade commands required.
Locked Settings
If you put any store configuration settings into config.php or env.php, those settings in the Admin will become “locked”. The Admin will show the current value, but not let you edit the value. This gives the developer better control over what a merchant can change on the live site vs. what the developer controls and deploys with the code.
The magento config:set command has a new --lock command line argument for store configuration settings that, if specified, will write the setting to the env.php file instead of the database. This means that the setting will now be locked for that local environment. (You can manually move that setting to config.php if you want to share it across all environments.)
Sensitive Settings
Magento 2.2 also has the concept of “sensitive” settings for passwords or other personally identifiable information. When dumped, these values are written to the env.php file and not the config.php file. (We had experimented with not writing them to a file at all but this confused some users.) There is an interactive mode that can be used in other environments to interactively prompt the user for each sensitive setting, to make it easier to set up a new environment.
Source Code Management
12 Factor Apps always put the source code for your site under source code control and only deploys code that has been committed to the repository (typically by pulling code from the repository during the build process).
In terms of what files to put under source code management, config.php should be included and env.php should not. The composer.json and composer.lock files should both be included, but vendor generally should not. (Some like to commit the vendor directory to avoid needing a “composer install” command during deployment. It can also help with patched core code. Magento is still reviewing several Composer extensions for applying patches in order to standardize such practices.)
But what about modules you develop locally? The app/code directory is provided for locally developed modules that can be committed with the project. But Composer supports another approach that is particularly interesting if you want to share locally developed modules between projects. Composer supports the ability to download packages either in a distribution format (such as a ZIP file) or as source code (such as checking the code out of git). If you check the code out as source code, you can edit it and commit the changes directly back to the git repository. A negative of downloading as source code is Composer is not able to cache downloads, unlike distribution formats, making Composer slower to run.
When using Composer, you can run composer install --prefer-source to say you prefer getting modules as source code when possible, but it is also worth having a look at the “preferred-install” section of the composer.json file.
"config": {
"preferred-install": {
"alankent/*": "source",
"*": "dist"
}
}
In this example, all modules with a vendor name of “alankent” will be automatically checked out in source code mode (useful on my laptop), and all others will be in installed using the “distribution” format, which Composer will cache between runs. Caching greatly improves the speed of running composer install and composer upgrade, so this approach can be useful to use the faster distribution formats for all modules except ones you develop inside your company.
Security Improvements via Read-only File System
In addition, the var/generated directory has now been moved out of the var directory. The purpose for this change is to allow the whole directory tree (except var and pub/media) to be made read-only, improving site security. No code or static assets will be generated at run time.
One approach for doing this is using two distinct Linux user accounts combined with Linux file permissions. In this approach, the web server runs as one Linux user (e.g. www-user is a common default for Apache) and is given read-only access to PHP, JavaScript, etc. files on disk. To deploy code, a different Linux user account is used that has write access to the files on disk. Set up correctly, the Linux OS will block the web server from making any PHP or JavaScript file changes. (The web server still needs to be able to write files to var and pub/media.) In 2.1 and earlier releases, it was possible to use a read-only file system, but there were some corner cases such as changing cache settings would try to update the env.php file, resulting in strange Admin error messages. Improvements have been made in 2.2 to clean up such inconsistencies, such as the Admin UI no longer letting you edit store configuration settings in config.php or env.php.
What is Next
The new 2.2 features are useful low-level improvements being shipped with 2.2. Please try out the new features and let us know what you think. For example, sensitive settings is a new concept that might need some fine tuning.
With these commands in place, the next goal is to better standardize deployment flows. The general goal is to provide a more consistent experience from development (using say Magento DevBox) through to production across different hosting options. There will always be some variation due to deployment differences (e.g. a cluster of web nodes with load balancers will be different than a single node deployment). The goal is to minimize the differences for developers to allow them to more easily switch between environments, as well as promote best practices such as minimizing downtime during deployment and improving site security.
Finally, please note that this blog post has not attempted to describe all 2.2 changes in the area of deployment. It has highlighted particular areas of note. Check out the release notes for other improvements.
9 Comments