Muffin Series 4: Module Loader and Package Management
In this post I would like to discuss in more detail how Muffin handles module loading and package management. These are complex issues, and a number of solutions have been devised for them over the years. However, I would like to argue that Muffin handles these issues in a more elegant manner.
Muffin handles module loading, package management, build process and dependency management all by itself. This is in stark contrast to many other tools. For example, Yeoman enlists four different tools to handle these tasks: Yo for scaffolding, Grunt for the build process, Bower for package management, and RequireJS for module loading.
Muffin makes use of this synergy and deals with all these issues in an integrated manner. For example, Muffin’s build process automatically wraps the CommonJS modules into an AMD-compatible format, while at the same time setting up module paths and dependencies. This significantly reduces the complexity of the module loader, thus Muffin’s built-in module loader is only 1675 bytes long when minified and gzipped, and can be easily included inline with the HTML file.
CommonJS adds a simple module loader named
require, which imports a module. This design is very similar to what many other languages have built-in, for example, Python’s
import function. The syntax is clean and simple.
It wraps the module definition inside a function, so that it can be loaded later. The syntax looks okay, until you have a lot of dependencies:
This starts to get unwieldy. Things get really bad when a library developer tries to support both CommonJS and AMD formats:
That is just painful to look at.
In my opinion, the library developer should choose the simplest module format and let the module loader worry about how to load the module. This is where Muffin can help.
Modules in Muffin Apps
In Muffin apps, you write modules in the CommonJS format. During the build process, Muffin automatically wraps the CommonJS module into an AMD-compatible format, so that it can be loaded asynchronously in the browser.
For example, this is a simple module for the
UserList, can easily import the
During the build process, Muffin compiles and wraps the
User module into this:
This is a variation of the AMD format and is compatible with the AMD spec.
Note that Muffin has automatically set the module path and traced module dependencies. A module with an explicit module path is called a “named module”, and is much easier to work with. This becomes clear when you need to load dependent modules with relative paths. For example, Muffin compiles and wraps the
UserList module into this:
If this module’s path is not known, it will be quite difficult to resolve the full path for
./User. A hack is to do some bookkeeping when the browser evaluates a module, but that adds a lot of complexity to the module loader. Since Muffin always sets the module path and traces the module dependencies during the build process, Muffin’s built-in module loader can be simple and small.
Wrapping a CommonJS module into the AMD format has a few benefits: the module can be loaded asynchronously, the module can delay its evaluation until it’s required, and most important of all, the module can load its dependencies before evaluating itself.
You have to be very careful with the order of these scripts, otherwise the app will break. For example, jQuery must be loaded before jQuery plugins, and application scripts using these plugins must be loaded even later. When you have a lot of scripts that depend on each other, manually sorting out the dependencies quickly becomes tedious and error-prone, if not infeasible.
Fortunately there is a much better solution. When we define a module in the AMD format, we can specify the module dependencies, thus the module loader can load all the dependencies before evaluating the module itself. Take an earlier example:
When the module loader loads this
User module, it sees that the
User module depends on the
Backbone module, thus the loader will fetch and load the
Backbone module before evaluating the
User module. The wrapper function in the AMD format makes this kind of dynamic loading possible.
However, if we have to manually specify the module dependencies in the wrapper function, this approach can be just as tedious and error-prone as managing the order of scripts. This is where the standard AMD format fails. It comes with too much boilerplate and requires you to specify all the dependencies up front:
With Muffin you don’t have to specify these module dependencies in the wrapper function at all, because Muffin automatically traces the module dependencies and wraps the module for you. It does a static analysis of the module content, extracts all the
require calls, and puts the dependencies in the wrapper. So you can write your code in the clean CommonJS format:
And Muffin wraps the above into this:
Note that this approach is different from what Browserify does. Browserify also traces the module dependencies using the static analysis, but bundles all the dependencies into one big file. It does not provide any runtime support for dynamic module loading. Muffin does automatic module dependency tracing, but still lets you use dynamic module loading in the runtime, if you choose to.
For example, if we add a dynamic
require to the above module:
ArticleList module will only be loaded when
loadArticles is called. This gives Muffin apps great flexibility: you can separate your application code into a few parts, load all the essential code at once, and use dynamic loading for the non-essential code.
Another benefit of wrapping a CommonJS module into the AMD format is that module evaluation can be separated from downloading the script. This separation has profound implications:
- Modules don’t have to evaluated all at once on the initial page load, thus reducing the initial loading time.
- Modules only need to be evaluated when they are required.
- Modules can be concatenated into one or a few files and downloaded at once.
With Muffin you can specify which module should have all the dependencies concatenated to it. For example, you may specify in
When you create a production build with
muffin minify, Muffin will analyze the file
Templates as Modules
For client-side webapps, how to dynamically load template files is another challenge. Template files are HTML files with template functions in it. For example, this is a template file using the Underscore.js template engine:
Since template files are text files, they can’t be dynamically loaded like scripts. So how can we deliver them to the client app?
Muffin has a built-in package manager that is compatible with TJ’s components. Muffin can install TJ’s components in Muffin apps, and Muffin packages follow TJ’s component spec. Just like Component, Muffin’s package manager can install packages from any GitHub repository, and only copies essential files instead of cloning the full git repository.
There are some differences, too. Muffin installs the packages in a slightly different directory structure. And Muffin prefers to keep code in its original format.
If a script is not set up as a component or a Muffin package, you can set up a shim for it. The shim follows the component spec as well, so that it behaves just like a component after installation.
For example, this is a shim for
You can also specify dependencies in the shim, just like in real components. For example,
bry4n/backbone-shortcuts depends on
madrobby/keymaster, so we can set up shims like this:
Specifying the script type is optional. Muffin wraps both scripts and modules, but the wrapper function is slightly different. The wrapper function for scripts doesn’t pass in variables like
exports. This only matters for scripts that support multiple formats. If you don’t specify the script type, Muffin loads the script as a module.
Muffin also adds package dependencies to the wrapper function, thus it can load the dependencies of traditional scripts just as easily.
By now you should have a good idea how Muffin handles module loading, module dependencies and package management. Check out muffin.io website if you want to learn more about Muffin.