cancel
Showing results for 
Search instead for 
Did you mean: 

Magento 2.2 Deployment Improvements

akent99
Regular Contributor

This blog post is a summary of changes coming in 2.2 related to deployment. This is based on my talk at MagentoLive UK 2017. Improvements include:

 

  • The removal of static asset generation dependence on the database (so it can be run on a Jenkins or similar server),
  • Performance improvements for compilation and static asset generation (much of which has already been backported to 2.1.x patches), and
  • Better per environment vs. shared configuration support.

 

Acknowledgements go to the great team behind the scenes that designed and built these new features.

 

12 Factor Apps

 

12 factor apps talks about good patterns for web applications to follow to make deployment easier. These factors are generally accepted best practices, and the new features directly align with these factors. If you are interested in deployment strategies, the 12 factor app site is a good read.

 

Better Support for a Build Phase

 

Currently, a typical Magento deployment procedure will put the server in maintenance mode, get the new code, update generated code and web asset files, upgrade the database schema if needed, then go live again. This however takes the site down for that duration. A key goal of the Magento 2.2 improvements is to shorten the length of time a site is down for.

  

image.png

 

 

As can be seen in the diagram above, compilation (di-compile), web asset generation (static-deploy), and the database schema upgrade can all add undesirable delays to site deployment downtime. Good deployment strategies improve upon this by at least moving the compilation and web asset generation out of the maintenance window by running them in a separate directory, but this still requires these commands to be run on the production server where you really want to keep the capacity available for customer requests.

 

To improve upon this, Magento 2.2 optimizes the compilation and web asset generation steps and removes the database dependency so that these steps can be run on a different server. Many of the optimizations have also been backported to Magento 2.1.x patches

 

So how will the above look with a new build phase?

 

image.png

 

 

Notice that the database dependence of the web asset generation step has been removed. The code now can get the settings from config.php, env.php, or environment variables. Note also that an automated test step has been added to the above. Running tests per build (this would include setting up a database if functional or integration tests are to be run) may automatically detect an issue with the optimized production code before it hits your production server, providing an extra level of safety. Advanced sites might also include performance measurements during testing to stop unexpected code slow-downs from getting deployed.

 

No improvements to the database schema upgrade step have been done in 2.2, but we are considering an additional command to test if any data formats have changed (including Redis). If no such formats have changed, then it is likely safe to run the old and new code in parallel. Basically, if the only code changes are minor, then you can look at techniques such as spinning up new servers with the new code and flipping a load balancer from the old web servers to the new web servers atomically. That can result in a zero-downtime deployment. Even if that is not possible, you can still minimize the downtime for the site by avoiding the database schema upgrade step. The new flow may look something like the following.

 

image.png

 

  

The Compact Web Asset Generation Strategy

 

One additional speed improvement in Magento 2.2 is a new “compact” deployment strategy. It works by identifying files that do not change per locale, such as the jQuery.js library. By writing such files to disk once instead of once per locale, the web asset generation step is faster and the time taken to transfer the files to the production server and extract them is also reduced.

 

In one test, a site with 3 locales completed web asset generation 2x faster using the “compact” deployment strategy and another site with 15 locales completed 10x faster. This is due to most of the JavaScript libraries never needing to be localized. Localization is more common for images and CSS files.

 

To use the new mode, a new “--strategy” command line option is available.

 

$ magento setup:static-content:deploy --strategy compact ...

 

The command writes out a map.php and requirejs-map.js file for the URL generation functions in PHP and JavaScript files to use. Common files are written into a base directory for per locale references to share.

 

Configuration as Code

 

Another 12 Factor App consideration is to treat configuration as code. It should be under source code version management and propagated as a part of your standard deployment process. For this reason, Magento has formalized that config.php is intended to be committed to git and shared between all environments. env.php overrides settings in config.php and is expected to be local per environment. (Environment variables can also be used to override the env.php file.)

 

In addition, the list of web sites, stores, and store views can now be put into the config.php file. This allows the web asset generation commands to run without database access, and allows the deployment of new scopes to be pushed between sites, along with themes during deployment of the code.

 

Normally, developers would define new websites and store views in the Admin, then dump the settings to a configuration file using the magento app:config:dump command. During deployment, other sites would run either magento app:config:import to synchronize the database with the dumped settings, or use magento setup:upgrade which will perform the import and perform any other database upgrade commands required.

 

Locked Settings

 

If you put any store configuration settings into config.php or env.php, those settings in the Admin will become “locked”. The Admin will show the current value, but not let you edit the value. This gives the developer better control over what a merchant can change on the live site vs. what the developer controls and deploys with the code.

 

The magento config:set command has a new --lock command line argument for store configuration settings that, if specified, will write the setting to the env.php file instead of the database. This means that the setting will now be locked for that local environment. (You can manually move that setting to config.php if you want to share it across all environments.)

 

Sensitive Settings

 

Magento 2.2 also has the concept of “sensitive” settings for passwords or other personally identifiable information. When dumped, these values are written to the env.php file and not the config.php file. (We had experimented with not writing them to a file at all but this confused some users.) There is an interactive mode that can be used in other environments to interactively prompt the user for each sensitive setting, to make it easier to set up a new environment.

 

Source Code Management

 

12 Factor Apps always put the source code for your site under source code control and only deploys code that has been committed to the repository (typically by pulling code from the repository during the build process).

 

In terms of what files to put under source code management, config.php should be included and env.php should not. The composer.json and composer.lock files should both be included, but vendor generally should not. (Some like to commit the vendor directory to avoid needing a “composer install” command during deployment. It can also help with patched core code. Magento is still reviewing several Composer extensions for applying patches in order to standardize such practices.)

 

But what about modules you develop locally? The app/code directory is provided for locally developed modules that can be committed with the project. But Composer supports another approach that is particularly interesting if you want to share locally developed modules between projects. Composer supports the ability to download packages either in a distribution format (such as a ZIP file) or as source code (such as checking the code out of git). If you check the code out as source code, you can edit it and commit the changes directly back to the git repository. A negative of downloading as source code is Composer is not able to cache downloads, unlike distribution formats, making Composer slower to run.

 

When using Composer, you can run composer install --prefer-source to say you prefer getting modules as source code when possible, but it is also worth having a look at the “preferred-install” section of the composer.json file.

 

 

"config": {
    "preferred-install": {
        "alankent/*": "source",
        "*": "dist"
    }
}

 

In this example, all modules with a vendor name of “alankent” will be automatically checked out in source code mode (useful on my laptop), and all others will be in installed using the “distribution” format, which Composer will cache between runs. Caching greatly improves the speed of running composer install and composer upgrade, so this approach can be useful to use the faster distribution formats for all modules except ones you develop inside your company.

 

Security Improvements via Read-only File System

 

In addition, the var/generated directory has now been moved out of the var directory. The purpose for this change is to allow the whole directory tree (except var and pub/media) to be made read-only, improving site security. No code or static assets will be generated at run time.

 

One approach for doing this is using two distinct Linux user accounts combined with Linux file permissions. In this approach, the web server runs as one Linux user (e.g. www-user is a common default for Apache) and is given read-only access to PHP, JavaScript, etc. files on disk. To deploy code, a different Linux user account is used that has write access to the files on disk. Set up correctly, the Linux OS will block the web server from making any PHP or JavaScript file changes. (The web server still needs to be able to write files to var and pub/media.) In 2.1 and earlier releases, it was possible to use a read-only file system, but there were some corner cases such as changing cache settings would try to update the env.php file, resulting in strange Admin error messages. Improvements have been made in 2.2 to clean up such inconsistencies, such as the Admin UI no longer letting you edit store configuration settings in config.php or env.php.

 

What is Next

 

The new 2.2 features are useful low-level improvements being shipped with 2.2. Please try out the new features and let us know what you think. For example, sensitive settings is a new concept that might need some fine tuning.

 

With these commands in place, the next goal is to better standardize deployment flows. The general goal is to provide a more consistent experience from development (using say Magento DevBox) through to production across different hosting options. There will always be some variation due to deployment differences (e.g. a cluster of web nodes with load balancers will be different than a single node deployment). The goal is to minimize the differences for developers to allow them to more easily switch between environments, as well as promote best practices such as minimizing downtime during deployment and improving site security.

 

Finally, please note that this blog post has not attempted to describe all 2.2 changes in the area of deployment. It has highlighted particular areas of note. Check out the release notes for other improvements.

9 Comments
baldwin_pieter
Core Contributor

Hi Alan

 

Thanks for the write up, very usefull information!

 

Regarding the recommendation for including the config.php in source control: we've been explicitly not including this file in Magento 2.1, because it's a very frustrating experience whenever a new module gets added and the whole sort order of modules defined in that file gets shifted around. This then has as a consequence that when multiple devs working on multiple branches on different modules and then trying to merge their branches back in the main branch, get constant conflicts due to the changes being made in the config.php file where we then end up with the config.php file not being in an consistent state unless you run bin/magento setup:upgrade after you merged your branch in the main branch.

We then decided after seeing multiple issues popping up with this workflow to no longer include the config.php file in source control and haven't noticed any issues since.

 

Is there something in version 2.2 which will also address this? To no longer keep the sort order of installed modules in the config.php file?

There are some issues on Github talking about that: https://github.com/magento/magento2/issues/8479 & https://github.com/magento/magento2/issues/10331

 

Thanks!

akent99
Regular Contributor

Thanks for the feedback. That is interesting. A part of the reason is the config.php list of module names is it worked out all the dependencies to make sure modules get loaded in the correct order. So adding a new module could add different dependencies. But it is probably randomizing (at the mercy of the algorithm) items where there is no difference in dependencies. This could make a good GitHub request - ask that modules get ordered alphabetically where dependencies are the same to minimize config.php changes when things are added (or even better do a pull request!).

baldwin_pieter
Core Contributor

In my opinion, it would be better to save the sort order of modules in some other generated file which we don't have to include in version control, and the list of the modules in config.php should be completely alphabetically to avoid the problem. This way we can include the config.php file in source control and keep track of enabled/disabled modules (which we currently can't in our particular workflow, we assume all installed modules are enabled all the time).

But it's probably too late now to change something like this and have it included in Magento 2.2.0? So we'll have to deal again with all these merge conflicts for now, until this is further improved unfortunately.

Thanks for the feedback though!

akent99
Regular Contributor

Its definitely too late for 2.2, and we would have to think about it. The file must exist, so it would have to be generated at some time and kept up to date. The current approach is its recomputed when the data it depends on changes. If not in git, you would have to recompute when... ummm... get new code from git, change the list, etc. I suspect it would make the overall workflow more complex. It would also need to be taken into account into deployment workflows etc too. Changing the ordering of the current file could be snuck into a patch easily as its backwards compatible. Nothing else changes. But worth sleeping on a little.

 

If you want to experiment and consider submitting a pull request, the relevant function is here: https://github.com/magento/magento2/blob/develop/lib/internal/Magento/Framework/Module/ModuleList/Lo...

 

akent99
Regular Contributor

Sorry, for some reason I did not notice your links to the GitHub issues already talking about it! I will add https://github.com/magento/magento2/pull/10649 to this thread if anyone wants to check it out and support it.

EnnoStuurman
Senior Member

@akent99, in "The Compact Web Asset Generation Strategy" you explain that "Common files are written into a base directory for per locale references to share." . Does that refer to the Asset Generation itself (the process under the hood) or to the actual static files output?

To explain: I am running a 2.2.0-rc22 now with two locales, I see that for example jquery.js is deployed to both locales. I was under the assumption, after reading this post, that general libraries like jquery.js would be referenced from a single location (pub/static/base or something) and only, if different, would be written to the (pub/static/theme/locale) ...

 

Edit

No need to answer anymore @akent99, using the "compact strategy" creates a base directory e.g. static/base/Magento/base/default/jquery.min.js , see also  => http://devdocs.magento.com/guides/v2.2/config-guide/cli/config-cli-subcommands-static-view.html

 

akent99
Regular Contributor

Another happy customer (without me having to lift a finger). Yes, you found the right answer.

EnnoStuurman
Senior Member

@akent99happy with the answer indeed ;-) Not yet sure about the new deployment strategies. I just filed an issue, I do really hope it's on my end somewhere... https://github.com/magento/magento2/issues/10766

dgallegos
Senior Member

@akent99 How do you "compile" without a database connection? I get errors trying to connect to redis.