Advanced JavaScript Bundling: My setup using gulp

[klaas_werkt]

Based on Magento's documentation on Advanced JavaScript Bundling for Magento 2.3 and Anton Kril's (deprecated) gist about JavaScript optimization, I've build my setup using gulp.

I've created an "optimize" gulp-task that needs to run after:

bin/magento setup:static-content:deploy

.. that uses RequireJS's optimizer (alsoo know as r.js) to bundle part of the static assets.

General steps of the "optimize" gulp-task:

1. Duplicate the pub/static/frontend/MyTheme/default/en_US to a *-tmp directory next to it
2. Generate the optimization configuration
3. Run the optimizer (process the static assets from the *-tmp direcroy to pub/static/frontend/MyTheme/default/en_US)
4. Remove the *-tmp directory

Step 1 & 4 are trivial, step 3 is simple:

const requireJsModule = require("requirejs");

requireJsModule.optimize(rjsConfig, callbackRemoveTempDirectory);

Step 2 is the tricky part and contains the magic (and ducttape) and consist of the following sub-steps:

1. Load and prepare a base configuration (which contains global configuration, stubs & the module definitions)
2. Extend the base configuration with (the aggregated configuration for RequireJS that is located in) requirejs-config.js (warning: this step contains magic)
3. Save bundle definitions for the loading of bundles
4. Remove mixin-ed components from the bundle definition (this is pure ducttape)

Load and prepare a base configuration

In the theme directory (app/design/frontend/MyTheme/default) I've added a JS module base-rjs-config.js file that exports the base configuration, stubs and module definitions. and it looks a little like this:

const baseRjsConfig = {
    // configuration
    optimize: 'none',
    inlineText: true,

    // stubs
    deps: [
        'jquery/jquery.cookie',
    ],
    shim: {},
    paths: {
        mixins: 'mage/requirejs/mixins',
    },
    map: {},
    config: {
        mixins: {},
    },

    // bundle definitions
    modules: [
        {
            name: 'bundles/require',
            create: true,
            include: [
                'requirejs/require',
            ],
            exclude: [
                'jquery',
            ],
        },
        {
            name: 'bundles/default',
            create: true,
            include: [
                'loaderAjax',
                'mage/common',
                'mage/cookies',
                'mage/dataPost',
                ...
            ],
            exclude: [
                'requirejs/require',
            ],
        },
        {
            // parts such as the minicart items of recently viewed products
            name: 'bundles/dynamic-product-list',
            create: true,
            include: [
                'Magento_Catalog/js/product/list/column-status-validator',
                'Magento_Catalog/js/product/list/columns/final-price',
                ...
            ],
            exclude: [
                'requirejs/require',
                'bundles/default',
                'bundles/category-product-list', // more bundle definitions
                'bundles/product-view', // more bundle definitions
                'bundles/cart', // more bundle definitions
                'bundles/checkout', // more bundle definitions
            ],
        },
    ],
};

module.exports = baseRjsConfig;

Not all bundles are included and complete in the example above, but I've made the configuration for all bundles manually using Chrome's dev-tools to see which files are loaded on which URL's.

Note: I've included stubs for the RequireJS configuration (deps, shim, paths, map & config). These stubs will be fully extended in the next step.

Note: The stubs already contains the deps 'jquery/jquery.cookie' to fix a broken/missing global deps. Also already present in the stub is the mixin loader to load mixins - sadly I haven't got this working.

Note: No optimization is configured (optimize: 'none'). This is because the default optimizer ('uglify2') can't handle ES6 code and although I run all my JS through Babel to rewrite ES6 to IE11 code, some third party library contains ES6 .

Extend the base configuration with requirejs-config.js

Important: Before being able to execute the next step a little rewriting is required because of the clash between RequireJS's "require" method and the "require" method from ES6 (as used in gulp). RequireJS adds the "require" method to the global namespace (in non-ES6) and the alias "requirejs" for that method. Magento however uses "require" by default in the generated aggregation of the RequireJS configuration in pub/static/frontend/MyTheme/default/en_US/requirejs-config.js. To use the alias "requirejs" I've added a plugin to Magento\Framework\RequireJs\Config, that does:

    public function afterGetConfig(Config $config, $result)
    {
        return str_replace('})(require);', '})(requirejs);', $result);
    }

As for the magic, I use the following code to extend the stubs in my rjsConfig:

global.requirejs = () => {};
global.requirejs.config = (c) => {
    if (c.deps) {
        rjsConfig.deps = [...rjsConfig.deps, ...c.deps];
    }
    if (c.shim) {
        rjsConfig.shim = {...rjsConfig.shim, ...c.shim};
    }
    if (c.paths) {
        rjsConfig.paths = {...rjsConfig.paths, ...c.paths};
    }
    if (c.map) {
        for (let [key, maps] of Object.entries(c.map)) {
            rjsConfig.map[key] = {...rjsConfig.map[key], ...maps}
        }
    }
    if (c.config && c.config.mixins) {
        rjsConfig.config.mixins = {...rjsConfig.config.mixins, ...c.config.mixins};
    }
};
require('pub/static/frontend/MyTheme/default/en_US/requirejs-config.js');

And I top it of with some ducttape:

// remove unusable keys
delete rjsConfig.shim['paypalInContextExpressCheckout'];
delete rjsConfig.paths['paypalInContextExpressCheckout'];

// replace magento text loader by default require-js text loader
rjsConfig.paths.text = 'requirejs/text';

Now my configuration (rjsConfig) contains all information to generated the bundles, thanks to spoofing the requirejs method/object on the global namespace. Yes, this is indeed something you should never do. I've yet to find a cleaner solution.

Note: The paypal keys in the ducttape are removed because they reference externally hosted assets that can't be accessed and bundled on the deployment server.

Note: Magento's text-loader can't handle loading of "text!..." directives in the bundle definitions during this optimization step. So this is replaced by (reverted to) the default RequireJS text loader.

Save bundle definitions for the loading of bundles

To eventually load the bundles in stead of single files and being able to fallback on loading single files, an extra configuration is loaded in the head after loading "requirejs-config.js". A layout directove is used to include this javascript file in the head. To write this file during the optimization process the following method is added to optimization configuration:

const optimizedrequirejsConfigFile = 'pub/static/frontend/MyTheme/default/en_US/requirejs-config-optimized.js';
fs.closeSync(fs.openSync(optimizedrequirejsConfigFile, 'w'));

rjsConfig.onModuleBundleComplete = (data) => {
    const bundleConfig = {};
    bundleConfig[data.name] = data.included;
    bundleConfig[data.name] = bundleConfig[data.name].map(bundle => bundle.replace(/\.js$/, ''));
    const contents = `
requirejs.config({
    bundles: ${JSON.stringify(bundleConfig)},
});
`;
    fs.appendFile(optimizedrequirejsConfigFile, contents);
}

After running the optimization, take a look into this file. It simply contains a mapping from single assets to the bundle that instructs the RequireJS loader to load a bundle in stead of a single asset on request.

Remove mixin-ed components from the bundle definition

During development and testing I noticed that components that are mixin-ed (i.e.: components on which a mixin is defined) fail to load when bundled. So I remove these from the bundle definition using the following ducttape code:

const mixinedComponents = Object.keys(rjsConfig.config.mixins);

rjsConfig.modules.map(module => {
    module.include = module.include.filter(componentName => {
        if (mixinedComponents.indexOf(componentName) >= 0) {
            module.exclude = [...module.exclude, componentName];
            return false;
        }
        return true;
    });
});

The end result

Bundles are generated based on my manually generated definitions and loaded accordingly, supporting fallback on loading single assets. And a slight headache.

Obstacles still to overcome

1. Bundled assets from the Magento_Ui module are still loaded as single assets
2. Bundled assets from the lib/web/js/ directory are still loaded as single assets
3. Bundled mixin-ed components are not loaded at all
4. The default 'uglify2' library doesn't support ES6 (and it uglifies which always scares me a little, I prefer just a comments-stripping minifier)

The sad thing is that 1 & 2 contain most of the assets

Any help on these points is more than welcome!

[joshuadickerson]

This is awesome. Do you have a repo you can share? I'm having troubles with mixins but I don't understand how you solved them.

Thanks,

Josh

[klaas_werkt]

Hi Joshua,

I have no repo that I can share, because I've applied this method commercially.

About the mixins, as you can read, I haven't solved them  Because I couldn't solve it (mixin-ed components weren't loaded/bundled correctly) I've remove Magento's mixin loader and added a script that removes mixin-ed components from the configuration that's sent to r.js. This is described here above.

When I find the time to further optimize the bundler and I make progress on either one of my obstacles, I'll post it here.

[Janek11]

@klaas_werkt thanks for the guide. I also had to remove mixins from my requirejs configuration config before running requirejs optimizer. It solved issues with js errors in checkout for me.

I guess you load requirejs synchronously and rest is loaded asynchronously via require ? (I use this approach). It's little different than what imply usage of configuration builder m2-devtools (really helpful to create build config and later just customize it). 

M2-devtools generates build config in a way that (with connection of Magento_ConfigBundle (orBundleConfig module) load all assets synchronously (shared.js and pageSpecific bundle). 

If you have any knowledge about usage of one or other approach in real projects you could share some info here: https://github.com/magento/m2-devtools/issues/42

[electroid]

---

@Janek11 wrote:

@klaas_werkt thanks for the guide. I also had to remove mixins from my requirejs configuration config before running requirejs optimizer. It solved issues with js errors in checkout for me.

I guess you load requirejs synchronously and rest is loaded asynchronously via require ? (I use this approach). It's little different than what imply usage of configuration builder m2-devtools (really helpful to create build config and later just customize it). 

---

@Janek11 can you give more information about how do you exclude mixins and then load others script. I still have errors on checkout page.

Thanks!

[Janek11]

What errors do you have?

I did the same way like author of the post. I added removeMixins array like

"removeMixins": [
        "jquery/jstree/jquery.jstree",
        "Magento_Ui/js/view/messages",
         ...
]

to my build config file.

Then in my script that prepare assets and run requirejs optimizer I modify my build config looping trough all modules and filtering each "include" in every module

"modules": [
       ...
        {
            // Modules used on > 1 page(s) of the store
            "name": "bundles/shared",
            "include": [
                "mage/common",
                "mage/dataPost"
                ...
            ]
         },
]

so in the end I will have config without files that were "mixined". So I don't put in removeMixins actually mixin files but original files that are "mixined".

THe only difference I import my build config using require() in node.
I also append requirejs bundles to requirejs-config.js 

In my config I also use this settings

    "uglify": {
        mangle: false,
        compress: {
          "collapse_vars": false
        },
        output: {
            "max_line_len": false
        }
    },

to avoid issues with JS code when uglifying.

I also had to add deps in my build config this to resolve some JS issues.

"deps": [ "jquery/jquery.mobile.custom", "js/responsive", "js/theme" ],

[electroid]

---

@Janek11 wrote:

What errors do you have?

I did the same way like author of the post. I added removeMixins array like

"removeMixins": [
        "jquery/jstree/jquery.jstree",
        "Magento_Ui/js/view/messages",
         ...
]

to my build config file.

Then in my script that prepare assets and run requirejs optimizer I modify my build config looping trough all modules and filtering each "include" in every module

"modules": [
       ...
        {
            // Modules used on > 1 page(s) of the store
            "name": "bundles/shared",
            "include": [
                "mage/common",
                "mage/dataPost"
                ...
            ]
         },
]

so in the end I will have config without files that were "mixined". So I don't put in removeMixins actually mixin files but original files that are "mixined".

---

@Janek11 thank you very much for provided information. Most important part is about mixins and that we need to remove script on what "mixins" was applied. However in magento2 original manual they told about remove mixin but that way it doesn't work.

I was using "removeMixins" key like you did and then I wrote few lines of code in build.js onModuleBundleComplete function to remove that scripts from bundles

onModuleBundleComplete: function (data) {
function onBundleComplete (config, data) {
const fileName = `${config.dir}/requirejs-config.js`;
const bundleConfig = {};

const mixinedComponents = config.removeMixins;



bundleConfig[data.name] = data.included;
bundleConfig[data.name] = bundleConfig[data.name].map(bundle => bundle.replace(/\.js$/, ''));

bundleConfig[data.name] = bundleConfig[data.name].filter( ( el ) => !mixinedComponents.includes( el ) );

const contents = `require.config({
bundles: ${JSON.stringify(bundleConfig)},
});`;

fs.appendFile(fileName, contents, data, function(err, result) {
if(err) console.log('error', err);
});

}
onBundleComplete(config, data);

}

After that I still have to check modules includes and comment some more scripts in order for onestepcheckout to work (it wasn't worked with cached scripts in browser, only then I fully clear browser cache with Ctrl + Shift + R). Finally it give good results in frontend.

[Janek11]

Which scripts had you to remove? 

[electroid]

---

@Janek11 wrote:

Which scripts had you to remove? 

---

                "mage/bootstrap",
                "mage/loader",
                "mage/dropdown",

However I want to mention I was working with custom purchased module for onepage checkout and not default magento2 checkout module.