Deployment and Front End Development

[apiuser]

Feature request from Khaleel, posted on GitHub Aug 12, 2016

Hi

Where do I begin to vent my frustrations with this platform? Firstly I will bring to your attention this Stackoverflow bounty which no one from the Magento Community gave an answer for that I a hoping will garner here: http://stackoverflow.com/questions/38589283/a-faster-route-to-deploying-static-content-in-magento-2-...

1. Why is it that the Magento Team thought it was acceptable to turn a live website into Maintenance mode to carry out a deployment? Our company makes serious money and it is just - absurd - to consider ever placing our site in downtime (at any time or day).

We literally ship our code to git and then pull on the live server and then deploy static content and this takes an incredible amount of time. Just this week, our developers made a few mistakes and we had to re-deploy twice. 45 mins of no sales. Clearly this is not a solution. There must be something were not doing correctly; but your own documentation, recommends it.

1. @alankent who says 'I am not a frontend dev' Clearly not. Otherwise you would have chosen Sass and dropped the insane idea of Magento UI, which no one really knows what it is. And then further you would have scrapped RequireJS and all the insane (not SEO friendly) KnockoutJS bloat even on the cart. Please can you answer regarding this. It is been raised across the web on many threads on Stackoverflow which my bounty failed and even on this post: https://github.com/magento/magento2/issues/2104 other devs see the insane feature of this.

Why is it sooooo slow to make changes to Less and JS even in a developer mode node. You have to recompile files (no Livereload; which I get people not a fan of) but seriously? Make a padding or margin change; delete all var folders, re compile static files for all languages and then run grunt to generate less etc - seriously? ALL our Front End Engineers are strong with JavaScript and CSS and most are (including me) previously Back End Developers. It is insane.

Please can this be addressed? Gulp and Grunt is one argument but the process of static deploy on development mode is one thing - a live environment? Seriously?

[apiuser]

Comment from boldhedgehog, posted on GitHub Aug 12, 2016

Hi @Khaleel,

We've ended up with a 'build' server which does composer install, compile and deploy steps, then force commits vendor and static directories to git. This commit is then tagged and we switch the production environment to this tag. This way you don't have to compile or deploy the static files on the production server with maintenance. Git workflow described here is inaccurate, of course, but I hope you get the idea.

[apiuser]

Comment from Khaleel, posted on GitHub Aug 12, 2016

Hey @boldhedgehog

Can you explain a little further in detail?

What about the LESS and JS - some of my coders hate the speed of making things in M2

[apiuser]

Comment from boldhedgehog, posted on GitHub Aug 12, 2016

@Khaleel

I'm not sure what git workflow you have for deployments, still the main idea is to create a special build branch, install all composer packages, perform static content deploy and compilation, then force commit and push all generated files into this branch, then tag it. The tag is then fetched on the production and the file base is switched to this tag, thus we update all generated files (/vendor/, static, di, compiled files) in 1 'atomic' git operation. With this scheme it does not really matter how much time is spent to prepare the generated files, as git check to the tag usually takes seconds.

Our FEDs use grunt to compile less files: grunt watch, less, exec tasks, as the documentation describes; and this is not that terribly slow.

I hope this helps.

[apiuser]

Comment from choukalos, posted on GitHub Aug 12, 2016

CC'ing @xcomSteveJohnson into this thread to add to our documentation.

Hi @boldhedgehog - any suggestions on how to improve the build server flow? We've looked at it internally and are exploring some improvements but would love to get your insights. CC'ing @Tanya-soroka and @piotrekkaminski.

[apiuser]

Comment from piotrekkaminski, posted on GitHub Aug 12, 2016

also cc @bwaters

[apiuser]

Comment from hostep, posted on GitHub Aug 13, 2016

Hi all

I kind of agree with @Khaleel but kind of also don't, because I think it all boils down to the fact that for newcomers in the Magento 2 ecosystem it's really hard to understand what is happening all over the place, and this is (I think) because some of the documentation is lacking a bit in certain areas.

So for the deployments. @Khaleel: it is possible to deploy on your production server without using a build server, but you'll need to reorganize your filestructure a bit to execute a bunch of commands in a new release directory (composer install, setup:di:compile, setup:static-content:deploy) and when that is done, create a symlink from that new releases directory to your docroot, put the shop in maintenance mode, run setup:upgrade --keep-generated and disable maintenance mode again. You also need a 'shared' directory for all the media files that get uploaded (and some other stuff) which needs to stay put in between deploys. We have this kind of deployment flow and it works (most of the time, not always), but takes a long time, about 8 minutes every deploy with a downtime of about 6 to 8 seconds, which are all not very ideal. If you don't want to write your own deploy scripts, you can take a look at the magento2 capistrano scripts from @DavidAlger over here: https://github.com/davidalger/capistrano-magento2 We use Fabric ourselves and have written all these scripts by hand.

Now, it would be great if Magento could write documentation about this to see what they recommend we do for deployments. Because they have more inside information about how everything works, they might have a better knowledge of how we need to do this. There is already a pending issue over at the devdocs for this. For the part where the above deployment flow doesn't always work in our case: if you create a new theme, you'll need to run setup:upgrade before setup:static-content:deploy otherwise the latter one crashes. So in this case I think the Magento guys hopefully have more insights into when and in what order you'll need to run all these deployment steps, because we just have to keep guessing...

@choukalos: for an improvement in the deployment flow: I would really really really like you to get rid of locale specific css/less compilation. It makes no sense in my opinion. I understand you need it for javascript translations and images in different locales with text written on them. But I don't understand why we need it for css/less files. If you could remove this (using a configuration option, to maintain backwards compatible) it would improve deployment speeds a lot I think. It would also help frontend development, if they switch to a different locale not everything has to be recreated. Also this PR will help a lot: https://github.com/magento/magento2/pull/4294 And I've written a small module myself which switches the less.php compiler into using less.js which also improves deployment speeds (but you'll need to be able to install nodejs on your server).

Then for the frontend development part.

I think it makes sense if somebody wrote some documentation about what happens with all the frontend assets when you refresh your browser when you are working in developer mode. Symlinks, var/view_processed, pub/static, ... Also what happens if a file already exists, how does Magento determine it should be recreated or not, stuff like that.

And if you could somehow improve the speed of compiling less into css (which now takes about 8 seconds if I remember correctly) that would also be very helpful. 8 seconds is a lot, it completely breaks your concentration if you have to sit and wait for so long. In Magento 1 we also did less precompiling, and it only took 2 or 3 seconds, which is a lot better.

And what I think will also help out a lot, is if you let javascript bundling work in developer mode. For example: if you now open the cart or checkout page in your browser, it downloads (and generates if they don't exist yet) like 200 or 300 javascript files, which takes a looooooooooong time (more then a minute). If we could get javascript bundling work in developer mode, it would only download/generate 3 javascript files (not exactly sure about the number) and the page refresh would be much faster. I do understand this won't be an easy feature to implement.

And we also need documentation of all the caches, what every cache is responsible for, what caches should we disable for certain scenario's. For example, for frontend development (css/less, changing phtml's, changing xml's, translations), I think it makes sense to disable these caches: layout, block_html, full_page, translate and leave the others enabled. This has some impact on the performance when refreshing the page every couple of minutes. And then you'll probably have another list of caches for people working on modules who don't have to work a lot on css/less, translations for example. So an overview of what each cache type actually caches and what caches you recommend we disable in certain scenario's would be helpful. I see some of my frontend colleagues disabling all caches so they don't have to manually refresh every time they change an xml file for example, and then they refresh, and sit and wait more time than actually needed, because just disabling some specific caches would be enough.

@xcomSteveJohnson: And then one last request: maybe create a weekly changelog of the devdocs somewhere. A lot of new content is added on a daily basis, but we can't just open devdocs every week and take a look at all the pages to see if something new has popped up somewhere. I think a weekly changelog with a short overview of what was done on the devdocs would help out a lot. I'm talking about mentioning big new topics being added. Important changes being made. But please don't add 'fixing typo's' to the changelog, that's not really needed

I hope this helps somewhat.

[apiuser]

Comment from Khaleel, posted on GitHub Aug 13, 2016

@hostep

Thank you so much for your messages and to all the above. What we are doing at the moment.

Development:

All developers have Magento 2 installed on their local machines - database releases are done weekly and images and media are already on Cloudinary as a CDN. We considered Rackspace/Akamai and may change to Ampliance in future but for now that is how it works. The developers have pretty powerful machines so they develop there code locally. Once done, they push and commit there changes to the staging repo. The developers have ALL got M2 configured in developer mode and still have to delete files and re build the cache libraries. This is what they do. None of the developers have less compilation working. Even in developer mode the LESS files do not compile automatically even with Magento 2 admin setup - is this is a bug. Even the JS. So they have to delete all files and re-compile. I am really happy that many other people here on Github are so open and share these same concerns. The FE code, KnockoutJS, requireJS, less is sooooo bloated. @alankent is pretty ignorant that he does not consider Front End Development the core component system to the platform. WordPress and many others are shipping to FE and JavaScript only servers with only API and JSON talking between. @alankent and many other Magento engineers (European and American based) who built the Front End system I do not think give much thought nor have the years of experience in ecommerce. Most of the developers are locked in the "Magento mindset"

bin/magento maintenance:enable
rm -rf var/cache var/page_cache var/view_preprocessed pub/static
mkdir var/cache var/page_cache var/view_preprocessed pub/static
bin/magento setup:static-content:deploy
bin/magento setup:static-content:deploy ar_SA
grunt less
bin/magento maintenance:disable

This is every time they want to see a change to less or JS. So most of the senior developers work directly in static files and then ship it back to the structure files in app for modules and themes. They hate and loathe this process and are already voting to our board to leave Magento. This takes 15mins max every single time.

Staging:

The staging environment gets a few "deploys" as we call them daily. A release manager goes onto the staging box (we use Digital Ocean) and have wildcard SSL, nginx, varnish and fpm and a few features like redis etc. Release manager executes the command below. Only difference is the upgrade on the CLI.

bin/magento maintenance:enable
rm -rf var/cache var/page_cache var/view_preprocessed pub/static
mkdir var/cache var/page_cache var/view_preprocessed pub/static
bin/magento setup:upgrade
bin/magento setup:static-content:deploy
bin/magento setup:static-content:deploy ar_SA
grunt less
bin/magento maintenance:disable

This takes around 15-20mins. Then we have QA and testing done on the staging server. We have around 14 stores in English and Arabic so various testing takes place; some automated some manual. Once it is good to go we go to the live nodes.

Node 2 and Node 1 are r-sync servers and load balanced. (We may switch off Node 1 and do a deployment on the Node 1 and once done enable it and allow Node 2 to download the new files) At the moment we have not done that so we have to do:

bin/magento maintenance:enable
rm -rf var/cache var/page_cache var/view_preprocessed pub/static
mkdir var/cache var/page_cache var/view_preprocessed pub/static
bin/magento setup:upgrade
bin/magento setup:static-content:deploy
bin/magento setup:static-content:deploy ar_SA
grunt less
bin/magento maintenance:disable

This is on our live machine. Keep in mind that any errors, changes even if the FE or BE makes errors we have to re-deploy from the staging. 25mins this process takes to execute one staging to live deployment.

CTOs from some big Magento 1.9 and EE versions asking reference:

Take your existing system offline by enabling maintenance mode using magento maintenance:enable. (DOWNTIME STARTS) Install the Magento software in a clean database (optionally using the --cleanup-database option). Import database data from your development system. Move your existing file system to an archive location. Move the file system you copied from your production system to the Magento docroot. Change to production mode and set file system permissions and ownership for production. Test everything thoroughly and carefully. Switch DNS to the new server. (DOWNTIME ENDS) Source: http://devdocs.magento.com/guides/v2.1/config-guide/prod/prod_deploy.html

We have to turn our live website offline and switch in maintenance mode and run a deployment and then switch live. Absurd. Who thought of this? Even in your documentation you mention this? What is the rational that this is considered even acceptable? Imagine turning eBay offline and telling customers we will be right back once we deploy some three new lines of CSS on our website.

[apiuser]

Comment from hostep, posted on GitHub Aug 13, 2016

Hi @Khaleel: I'm not going to try to help your specific case (because that's not what github issues are for), but let me give you this link, it more or less explains what you need to do to keep the downtime during deploys low enough (max 10 seconds mostly): http://www.classyllama.com/blog/deploying-magento-2-using-capistrano Don't copy/paste everything without thinking about it, try to understand what is happening and then make your own version of this. This might take you a couple of days to fully figure out, so be prepared for that. Also be aware that the article is tailored for Magento 2.0, not 2.1, so you might need to change some things here and there (like bin/magento setup:di:compile instead of bin/magento setup:di:compile-multi-tenant).

Btw: the article you refer to is only for the initial setup, not for continuous deployments.

Hope this helps.

Looking forward to see any kind of activity from Magento about these issues (I know it's weekend, don't worry )

[apiuser]

Comment from Khaleel, posted on GitHub Aug 15, 2016

Hi @hostep

This seems the best way for us:

Node1 Node2 Node3 (all load balanced and sync files)

1. All Developers have their own nodes locally.
2. All Developers push to Git and The Release Manager merges and controls a staging site.
3. Testing on staging.
4. Release manager turns off Node 3. He pulls repo and runs static files and deployments. Done.
5. Release manager turns Node 1 and Node 2 off and turns Node 3 off. (11 seconds possible downtime)
6. Release manager runs sync for Node 1 and Node 2 which get the latest from Node 3. Once done switches these boxes back online.

Weekly task. Still awaiting feed back from Magento.

[apiuser]

Comment from stamster, posted on GitHub Aug 15, 2016

Have you guys tried to disable certain cache types in M2, at least during development? Our FE devs were happy after we disabled the following cache types:

• Layouts
• Blocks HTML output
• Collections Data
• Page Cache

Apart from that, I agree with @Khaleel regarding levels of insanity M2 introduces. :-1: