Introduction

As a software engineer in my opinion, I think that we should have the capability to know how the tools we are using in our daily life works under the hood, to better understand and improve our skills. In this journey we will see step by step how to create a javascript module bundler.

Many of us has used to work with webpack, parcel,... but they don't know the magic behind, by understanding we will be able to make a better design decision about our code .

Before starting the source code is available on Github

Let's start

• The first thing that came in mind is to create a function that reads a file and resolve its dependencies like the following :

const fs = require('fs');


function describe(absolutePath) {
    const sourceCode = fs.readFileSync(absolutePath, "utf-8");
    return {
        sourceCode:sourceCode
    }

}

We didn't do much here just reading a file and return a description object, but to resolve its dependencies we need to capture import declaration there is two options:

• Regex
• AST

Definitely we will not bother ourselves with regex, we choose AST, but stop What is it ?

It is a hierarchical program representation that presents source code structure according to the grammar of a programming language,
each AST node corresponds to an item of a source code.

In other way transform your code into a data structure like below :

But how to transform a code into this data structure, Hmm let me see well compilers does those stuffs:

but in this tutorial we will not go through all its phases we will be limited to lexical and syntax analyzer, as usual we will not reinvent the wheels we'll choose babel to handle this part.

We will continue our implementation as follows

const fs = require("fs");
const path = require("path");
const {parse} = require('@babel/parser');
const {default: traverse} = require("@babel/traverse");


function describe(absolutePath) {
    const sourceCode = fs.readFileSync(absolutePath, "utf-8");
    const ast = parse(sourceCode, {
        sourceType: 'module',
    });
    let dependencies = [];
    traverse(ast, {
        ImportDeclaration({node}) {
            dependencies.push(node.source.value)
        }
    });
    return {
        dependencies: dependencies,
        sourceCode: transformSourceCode(sourceCode),

    }
}

What we did so far we generated the AST with the parse method from the @babel/parser and we traversed the tree to capture import declaration.

Why did you use traverse? Well, you can create your own implementation to traverse the tree, but babel offers @babel/traverse that implements the visitor pattern and we are interested to import nodes.

Let's now test our describe function :

• main.js

import a from "./a.js";
console.log("Hello !!");

• bundler.js

const fs = require('fs');
const path = path.join(__dirname,'main.js')
console.log(describe(path))

• output :

{
  dependencies: [ './a.js' ],
  sourceCode: '"use strict";\n' +
    '\n' +
    'var _a = _interopRequireDefault(require("./a.js"));\n' +
    '\n' +
    'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
    '\n' +
    'console.log("Hello !!");'
}

Wait wait the sourceCode is changed, we can't find the import keyword instead there is require .
Yes we transpiled the code source with transformSourceCode to make sure that will work in all browsers

const {transformFromAst} = require('babel-core');

function transformSourceCode(rawSource) {
    let {code} = transformSync(rawSource, {
        presets: ['@babel/preset-env']
    })
    return code;
}

So far so good our describe function works as we desired, now we need to traverse the root file in order to capture its dependencies and for each dependency we capture its dependencies and so want so forth.

const path = require('path');

function resolveModules(identifier,rootAbsolutePath) {
    let {dependencies, sourceCode} = describe(rootAbsolutePath);
    let modules = `['${identifier}'] : function(require,exports){ ${sourceCode} },`;
    dependencies.forEach((dependency) => {
        modules += resolveModules(dependency,path.resolve(path.dirname(rootAbsolutePath),dependency))
    });
    return modules;
}

What we did a simple function that traverse recursively a tree of dependencies, but the tricky part is:

let {dependencies, sourceCode} = describe(rootAbsolutePath);
let modules = `['${identifier}'] : function(require,exports){ ${sourceCode} },`;

We wrapped each dependency source code into a function that takes require and exports as params .
Why you did that ? Humm for the simple reason each module should be encapsulated and we will inject a require function and exports object that holds exported elements from the module

The final step is to generate the bundle . wait wait we are done ? Yes will just define our require function
to load the proper modules .

function bundle(entryPoint) {
    let modules = `{${resolveModules(entryPoint,path.resolve(__dirname,entryPoint))}}`;
    return `
        let modules = ${modules}
        function require(module){
            let exports = {};
            modules[module](require,exports)
            return exports;
        }
        (modules['${entryPoint}'])(require,{})
    `
}

Let's now test our bundle function :

• a.js

export default "module a"
console.log("module a invoked")

• main.js

import a from "./a.js";
import b from "./a.js";
console.log("----- main start ------")
console.table({a,b});
console.log("----- main end  -------")

• bundler.js

let entryFile = "../webapp/main.js";
fs.writeFileSync(path.join(__dirname, "./output.js"), bundle(entryFile));

if executed output.js will see in the console

module a invoked
module a invoked
----- main start ------
┌─────────┬────────────┐
│ (index) │   Values   │
├─────────┼────────────┤
│    a    │ 'module a' │
└─────────┴────────────┘
-----  main end  ------

If you notice there something wrong with our require implementation, first one is each time we import the module it get executed and the modules object is available on the window object.

The fix should be like the following :

function bundle(entryPoint) {
    let modules = `{${resolveModules(entryPoint, path.resolve(__dirname, entryPoint))}}`;
    return `
      (function(){
    let modules = ${modules};
    let cache = {};
    function require(module){
        let exports = {};
        if(!cache[module]){
            modules[module](require,exports)
            cache[module]=exports;
            return exports;
        }else{
            return cache[module];
        }
    }

    (modules['${entryPoint}'])(require,{})
})()     
    `
}

We reached the end of this tutorial i really enjoyed writing it and i hope it will be useful, If you have any questions or remarque's please reach me out on twitter or github

References :

• babel
• AST for js developers