Current thoughts on preprocessors and M2 directions

[akent99]

There is a twitter thread discussion, which is pretty painful with 144 character limits. So putting my comments here instead.

History: M2 included a Less preprocessor built in. The general feedback is it would be better to not include any preprocessor in M2 in order to allow existing great tools like Gulp and Grunt to be used instead, and allow frontend devs to come up with new tools without being held back by M2 catching up. The Sass port work by Bartek for example does this. You can also do this today even using Less if you develop in a particular way, but I wanted to share here some background and current thoughts of what might be a good way to overcome the problems - feedback welcome!

Let me talk about some rough personas and their needs:

• Magento - we product the core code base and a platform for others to build upon.
• Extension developers - create extensions for sale / reuse across multiple projects.
• Theme developers - create themes for sale to make it quick to build a Magento site with a non-default look-and-feel.
• Solution partners - want to take the core product and a set of extensions (and locally developed code) and produce a great site for a merchant. Some projects will have big budgets with advanced needs, some projects just want to get something up quickly so may buy and tweak an existing theme.
• DIY merchants - grab Magento, buy a few extensions, limited dev resources available (or limited budget), want a functional and decent looking site.

The DIY merchant might be a bad name, but the point is they don't have much access (or budget) to technical resources. There are many of them, so extension developers like them (more sales). But it means Magento has to make it easy for a merchant to buy an extension and a theme and have them all work together well.

So why is Less CSS preprocessing built in? I will simplify here. The original idea was to use Less variables, set by theme developers, to adjust the look-and-feel of a site. (Layout files etc would be involved as well, but I will stick to CSS here.) Using variables allows say button properties to be changed consistently with out adding additional CSS rules. (E.g. you could just keep piling on more CSS rules to override previous CSS rules to change colors and style.)

• A proposal here is to break down the CSS files into more modular files, then use file replacement rather than overriding individual Less variables.
• Maybe the assumption of adding new CSS rules is not correct - adding a few extras might be acceptable.

The Less preprocessor in Magento has another ability to include a bit of Less files from every loaded module (they are merged).

• A proposal here is that Magento maintains core CSS, with extensions adding new CSS files for any new rules they need. Then use CSS file merging techniques to reduce the number of CSS files for a site (for performance).
• Expert frontend developers can create their own files that import selected collections of files, to control CSS merging by hand.
• Magento could look at a more sophisticated CSS merging capability to help, so its not merge everything but rather merge selected sets of CSS files.

There was a question of whether its acceptable to say "NodeJS must be available in the production environment for DIY sites". Then you can use Grunt/Gulp etc in production, to merge themes and extensions Less/Sass/etc at run time. Default position is no, we don't want to require NodeJS on production sites for DIY merchants if we can avoid it - it would not keep up with new tools and innovations.

• One question here was if instead of CSS preprocessor variables, could theme and extension merging be done purely at the CSS file level. Then developers can use Gulp/Grunt/whatever, and ship themes and extensions with just CSS files. Then the DIY merchant site has no need for a CSS preprocessor. It is fine to add NodeJS as a requirement for developers - only production sites are really the issue.

There are not definite plans here - the above is a quick brain dump of ideas that resonated with me from other discussions. I thought this might be a good starting point for any future discussions. If there was a way to keep all personas above happy, that would be brilliant.

And yes, this is only dealing with CSS - not JavaScript or other frontend areas. This discussion is also not relevant to BEM or other CSS class naming standards. The above feels tantilizingly close - like there may be a way to achieve the above without change to Magento - the change would only be in conventions around using what is already in the product. E.g. if extension developers only ship CSS, then we can keep the Less preprocessor in place for legacy sites, but new sites would simply not use it. We just tell developers they should ship CSS (following certain standards).

[bassplayer7]

So let me throw this in - what do you think? I like the idea of having CSS more of the centralized end-point. However, the core SCSS is important as well. So what about Magento shipping with the core SCSS (or LESS for now, I suppose) and an output version of that CSS just like Magento 1.9?

The CSS would be nested not compressed. Then, Magento would take all the CSS files at runtime (or deploy) and concatenate and minify them. Sort order is one area of consideration with that. I know Plugins have sort order, so what about having addCss nodes in XML also have that? This would allow DIY Merchants to add their own custom CSS and that would then be added to the end (or wherever) of the distribution CSS. 

With this, though, Magento would still ship with the NodeJS build tools necessary to compile the SCSS. This way, theme developers like me could take and override core SCSS files and run the Gulp (not Grunt, please ) necessary to build the SCSS into CSS. At the end of the day, node would be required if a developer wanted to re-build the SCSS, but not a requirement for developing on Magento.

One other thing would be if a DIY Merchant started by appending plain CSS to add some custom styles. Then, perhaps they hire an agency that sets up a deployment process and has the ability to compile stylesheets and take that approach. The thing is that it is very easy to port CSS to SCSS. So if that happened, they could port the CSS to SCSS partials and move on.

If CSS files, instead of the SCSS files were overridden on a module basis (instead of just an custom stylesheet), this could be a more substantial impact. A case like that, though, seems a little beyond the scope of what needs to be accounted for?

I haven't given this a huge amount of thought, but seems like it would facilitate the different "personas". Also, I think that having some way to add tasks to the task-runner would be important. With the current Gulp/SASS setup, there is no way to add tasks that would do other, non-core things.  

[akent99]

Yes, that is inline with what I was trying to say. Work out how to make merging in themes and extensions only need CSS files. And yes, still ship SCSS/Less/Sass/whatever files as well so solution providers (experts) can do "clever stuff". If its possible for DIY merchants to not have to run a CSS preprocessor then the world is simple for the low end (with less power), and more serious developers can use NodeJS etc tools. The question is can theme developers do everything they need acceptably, by adding extra CSS rules, or replacing CSS files.

I was hoping if we deferred CSS file merging to the last step then replacing CSS files (one for buttons? One for forms?) would be good enough for theme developers. No Less variables to change button colors - just replace that CSS file. Plus introduce more consistent CSS class names (BEM, SUITCSS, whatever).

[bassplayer7]

---

The question is can theme developers do everything they need acceptably, by adding extra CSS rules, or replacing CSS files.

I never like only adding styles because it means overriding - that translates to wasted bytes going down the pipe to the end user. (That's one thing that's made me grimace with the large mixins in the Magento UI library - there are many "inherit" values. I know Gzip helps and all that, but it's still being shipped to the user and not actually used.) But that is me as a theme dev and I think there's a place for only adding styles. I think that often it is better that a "right way" is provided, but not forced. 

The goal of the variables and massive mixins in the Magento UI library was very noble. However, in my working with it, I have found it to feel quite brittle. It seems like if variables were to work well for a store, there should be a much smaller amount of them that are standardized on. When there are ~50 shades of gray, it's hard to know which ones are the real ones being used. I suppose that this is then an instance where it comes back to ground zero: what if someone just wants to change 15 variables (colors, etc.) and then use near-core styles? They would still have to deploy and run NodeJS, etc.

Then, if the SCSS was written well (BEM, or really any set, structured approach), all of the files would be small partials, and not mixins, that would be trivial to override. Obviously, if all one was doing was changing theme variables, it would still then require a NodeJS installation. That's an instance where CSS variables would work nicely.

Ultimately, everything has to end up as CSS. SCSS is just syntactic sugar. Since it's still important, it seems best to make it as easy to "turn on" (use with development) as possible, but not a requirement. (That's me thinking out loud.)

[Bartek Igielski]

I don't want to bloat this topic with Frontools related things, but changing styles processing pipeline and adding another dependecies, isn't just "adding new task" and really can't be handled without forking codebase and maintaing them manually in future. This is the nature of Gulp (or even any node.js) tools, b/c AFAIK there is no way to extend package.json or overwite files with same name like in M.

[bassplayer7]

I concur that Frontools is exactly the focus of this thread, but it could be related in a sense.

@Bartek Igielski keep in mind that neither me or @svossen suggested changing styles processing pipeline. In my experience, there is often a way to do something if necessary. Consider this: perhaps Magento builds several Gulp plugins, and hosts them on NPM, that take care of most of the more complicated parts of a build system - like getting all of the web/ folders from modules. And then perhaps, there is a gulpfile.js.sample that is small but includes the plugins and is an entry point for the tasks. Then, we can take and add tasks as desired. 

I suspect that are other approaches that would work on this as well. Perhaps it wouldn't be a problem to have a core Gulp setup and a secondary one that can be customized?

I think SVG vs. font icons are a great example of this. From what I see, SVGs are clearly better than font icons. As such, it is reasonable that front end developers would want to include SVGs in the theme and need the build tools for that. Yes, there would be additional dependencies, and yes, it may take a little different approach, but it would seem to clearly fit into a new task category.

As such, it seems to me that there are things that we can work together on to come to solutions that are as flexible as possible and facilitate the needs of developers across many different backgrounds and with different goals.

Also, is the Magento 2 front end project going to be open source? I would be interested in contributing some to that.

[Bartek Igielski]

package.json works at directory scope and there is no way to extend them with external files. Without ability to add new dependencies, customizations doesn't have any point.

For my point of view, forking tools and customizing them for project needs is super easy and it's not necessary to change it.

Adding another tasks is as simple as adding new file to `task` folder, thanks to gulp-task-loader. Same about new dependencies, thanks to gulp-load-plugins everything from package.json is automatically available in every task.

[akent99]

"I never like only adding styles because it means overloading - that translates to wasted bytes going down the wire"...

Two questions

1. Does anyone have strong feelings of replacing smaller CSS files vs adding to existing CSS files (via overriding)?
2. Are there CSS minification tools that can merge/minimize such overriding and eliminate it?

[Bartek Igielski]

1. Replacing small CSS files will be good, as long as they will be really small and single purpose (b/c currently LESS partials, which are equivalent of this files are way to long).

2. There is PostCSS plugin cssnano, but it's mostly designed for safe minification (except "custom identifier reduction, z-index rebasing, unused at-rule removal"), not actions like removing overrides, b/c it's developers job.

[andypimlet]

Couldn't agree more about BEM structured Sass. I inherited a theme a few months back in which someone had basically BEM-ified the core templates and then built custom partials for all these components, completely dispensing with the base theme and starting from scratch.

So nothing like starting from ground zero but in terms of styling, scaling and maintaining it's been a dream. Everything else I work with that has some sort of limping connection to base theme styling is just a dog of a thing.