Yaogang Lian

Muffin Series 4: Module Loader and Package Management

In this post I would like to discuss in more detail how Muffin handles module loading and package management. These are complex issues, and a number of solutions have been devised for them over the years. However, I would like to argue that Muffin handles these issues in a more elegant manner.

Muffin handles module loading, package management, build process and dependency management all by itself. This is in stark contrast to many other tools. For example, Yeoman enlists four different tools to handle these tasks: Yo for scaffolding, Grunt for the build process, Bower for package management, and RequireJS for module loading.

Why doesn’t Muffin build upon the existing tools? The short answer is synergy. If you think long and hard about these issues, you will realize that they all originated from the same root — the lack of module support in the JavaScript language. Because of the deep connections among these issues, dealing with them separately adds a lot of complexity to each tool, and also requires writing lengthy configuration files to bind the tools together.

Muffin makes use of this synergy and deals with all these issues in an integrated manner. For example, Muffin’s build process automatically wraps the CommonJS modules into an AMD-compatible format, while at the same time setting up module paths and dependencies. This significantly reduces the complexity of the module loader, thus Muffin’s built-in module loader is only 1675 bytes long when minified and gzipped, and can be easily included inline with the HTML file.

Module Formats

The lack of module support in JavaScript has resulted in a lot of bad programming practices over the years. We have witnessed the hideous pollution of the global namespace, the monolithic script that is impossible to maintain, and the dependency hell. To solve these issues, several JavaScript module formats have been proposed. The most clean and usable of all is still the CommonJS format. It’s the module format used in Node.js and looks like this:

1
2
3
4
5
6
7
8
9
fs = require 'fs'
class MyModule
showPassword: ->
fs.readFile '/etc/passwd', (err, data) ->
console.log data
module.exports = MyModule

CommonJS adds a simple module loader named require, which imports a module. This design is very similar to what many other languages have built-in, for example, Python’s import function. The syntax is clean and simple.

However, the CommonJS format doesn’t work well in the browser, because it doesn’t support asynchronous module loading. Unlike other languages which typically run on a server, JavaScript needs to run in the browser and may have to download scripts on demand. This is where things get messy. Another module format, the Asynchronous Module Definition (AMD), was proposed and it looks like this:

1
2
3
define(['jquery'] , function ($) {
return function () {};
});

It wraps the module definition inside a function, so that it can be loaded later. The syntax looks okay, until you have a lot of dependencies:

1
2
3
4
5
6
7
8
9
10
11
define([ "require", "jquery", "blade/object", "blade/fn", "rdapi",
"oauth", "blade/jig", "blade/url", "dispatch", "accounts",
"storage", "services", "widgets/AccountPanel", "widgets/TabButton",
"widgets/AddAccount", "less", "osTheme", "jquery-ui-1.8.7.min",
"jquery.textOverflow"],
function (require, $, object, fn, rdapi,
oauth, jig, url, dispatch, accounts,
storage, services, AccountPanel, TabButton,
AddAccount, less, osTheme) {
});

This starts to get unwieldy. Things get really bad when a library developer tries to support both CommonJS and AMD formats:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// AMD support
if (typeof define === 'function' && define.amd) {
define(function () { return Cookies; });
// CommonJS and Node.js module support.
} else if (typeof exports !== 'undefined') {
// Support Node.js specific `module.exports` (which can be a function)
if (typeof module != 'undefined' && module.exports) {
exports = module.exports = Cookies;
}
// But always support CommonJS module 1.1.1 spec (`exports` cannot be a function)
exports.Cookies = Cookies;
} else {
window.Cookies = Cookies;
}

That is just painful to look at.

In my opinion, the library developer should choose the simplest module format and let the module loader worry about how to load the module. This is where Muffin can help.

Modules in Muffin Apps

In Muffin apps, you write modules in the CommonJS format. During the build process, Muffin automatically wraps the CommonJS module into an AMD-compatible format, so that it can be loaded asynchronously in the browser.

For example, this is a simple module for the User model:

1
2
3
4
5
6
Backbone = require 'Backbone'
class User extends Backbone.Model
initialize: -> {}
module.exports = User

Another module, UserList, can easily import the User module:

1
2
3
4
5
6
7
8
Backbone = require 'Backbone'
User = require './User'
class UserList extends Backbone.Collection
model: User
initialize: -> {}
module.exports = UserList

During the build process, Muffin compiles and wraps the User module into this:

1
2
3
4
define('javascripts/apps/main/models/User', ["Backbone"], function(require, exports, module) {
...
module.exports = User;
});

This is a variation of the AMD format and is compatible with the AMD spec.

Note that Muffin has automatically set the module path and traced module dependencies. A module with an explicit module path is called a “named module”, and is much easier to work with. This becomes clear when you need to load dependent modules with relative paths. For example, Muffin compiles and wraps the UserList module into this:

1
2
3
4
define('javascripts/apps/UserList', ["Backbone","./User"], function(require, exports, module) {
...
module.exports = UserList;
});

If this module’s path is not known, it will be quite difficult to resolve the full path for ./User. A hack is to do some bookkeeping when the browser evaluates a module, but that adds a lot of complexity to the module loader. Since Muffin always sets the module path and traces the module dependencies during the build process, Muffin’s built-in module loader can be simple and small.

Module Dependencies

Wrapping a CommonJS module into the AMD format has a few benefits: the module can be loaded asynchronously, the module can delay its evaluation until it’s required, and most important of all, the module can load its dependencies before evaluating itself.

Managing JavaScript dependencies has always been a headache, way before module formats were created. For example, this is how CNN.com includes traditional scripts:

1
2
3
4
5
<script src="/.e/js/libs/jquery-1.7.2.min.js"></script>
<script src="/.e/js/libs/jquery.timeago.min.js"></script>
<script src="/.e/js/libs/protoaculous.1.8.2.min.js"></script>
<script src="/tmpl_asset/static/www_homepage/2705/js/hplib-min.js"></script>
<script src="http://cdn.optimizely.com/js/131788053.js"></script>

You have to be very careful with the order of these scripts, otherwise the app will break. For example, jQuery must be loaded before jQuery plugins, and application scripts using these plugins must be loaded even later. When you have a lot of scripts that depend on each other, manually sorting out the dependencies quickly becomes tedious and error-prone, if not infeasible.

Fortunately there is a much better solution. When we define a module in the AMD format, we can specify the module dependencies, thus the module loader can load all the dependencies before evaluating the module itself. Take an earlier example:

1
2
3
4
define('javascripts/apps/main/models/User', ["Backbone"], function(require, exports, module) {
...
module.exports = User;
});

When the module loader loads this User module, it sees that the User module depends on the Backbone module, thus the loader will fetch and load the Backbone module before evaluating the User module. The wrapper function in the AMD format makes this kind of dynamic loading possible.

However, if we have to manually specify the module dependencies in the wrapper function, this approach can be just as tedious and error-prone as managing the order of scripts. This is where the standard AMD format fails. It comes with too much boilerplate and requires you to specify all the dependencies up front:

1
2
3
4
5
6
7
8
9
10
11
define([ "require", "jquery", "blade/object", "blade/fn", "rdapi",
"oauth", "blade/jig", "blade/url", "dispatch", "accounts",
"storage", "services", "widgets/AccountPanel", "widgets/TabButton",
"widgets/AddAccount", "less", "osTheme", "jquery-ui-1.8.7.min",
"jquery.textOverflow"],
function (require, $, object, fn, rdapi,
oauth, jig, url, dispatch, accounts,
storage, services, AccountPanel, TabButton,
AddAccount, less, osTheme) {
});

With Muffin you don’t have to specify these module dependencies in the wrapper function at all, because Muffin automatically traces the module dependencies and wraps the module for you. It does a static analysis of the module content, extracts all the require calls, and puts the dependencies in the wrapper. So you can write your code in the clean CommonJS format:

1
2
3
4
5
6
7
8
9
Backbone = require 'Backbone'
ArticleList = require '../models/ArticleList'
ReaderView = require './ReaderView'
class ArticleListView extends Backbone.View
articleCellTemplate: _.tpl(require '../templates/_article_cell.html')
# ...
module.exports = ArticleListView

And Muffin wraps the above into this:

1
2
3
4
define('javascripts/apps/main/views/ArticleListView', ["Backbone","../models/ArticleList","./ReaderView","../templates/_article_cell.html"], function(require, exports, module) {
//...
module.exports = ArticleListView;
});

Note that this approach is different from what Browserify does. Browserify also traces the module dependencies using the static analysis, but bundles all the dependencies into one big file. It does not provide any runtime support for dynamic module loading. Muffin does automatic module dependency tracing, but still lets you use dynamic module loading in the runtime, if you choose to.

For example, if we add a dynamic require to the above module:

1
2
3
4
5
6
7
8
9
10
11
Backbone = require 'Backbone'
ReaderView = require './ReaderView'
class ArticleListView extends Backbone.View
loadArticles: ->
require ['../models/ArticleList'], (ArticleList) ->
collection = new ArticleList()
collection.fetch()
module.exports = ArticleListView

The ArticleList module will only be loaded when loadArticles is called. This gives Muffin apps great flexibility: you can separate your application code into a few parts, load all the essential code at once, and use dynamic loading for the non-essential code.

Module Concatenation

Another benefit of wrapping a CommonJS module into the AMD format is that module evaluation can be separated from downloading the script. This separation has profound implications:

  1. Modules don’t have to evaluated all at once on the initial page load, thus reducing the initial loading time.
  2. Modules only need to be evaluated when they are required.
  3. Modules can be concatenated into one or a few files and downloaded at once.

With Muffin you can specify which module should have all the dependencies concatenated to it. For example, you may specify in client/config.json:

1
2
3
"concat": [
"javascripts/apps/main/start"
]

When you create a production build with muffin minify, Muffin will analyze the file javascripts/apps/main/start, extract its dependencies, and recursively concatenate all the dependent modules and their dependencies. Thus the client app only needs to download this one file to get all the necessary modules. If you want to separate your application code into a couple large files, you can specify multiple files in the above configuration, and use dynamic loading to load some of them at a later time.

Templates as Modules

For client-side webapps, how to dynamically load template files is another challenge. Template files are HTML files with template functions in it. For example, this is a template file using the Underscore.js template engine:

1
2
3
4
5
6
7
8
9
10
11
12
<li class="article-cell">
<% if (article.imageUrl != null) { %>
<a href="#">
<img src="<%- article.imageUrl %>" class="article-image"/>
</a>
<% } %>
<div class="content">
<div class="title"><a href="#"><%- article.title %></a></div>
<div class="summary"><%= article.summary %></div>
<div class="dateline"><%- article.updated %><a href="<%- article.link %>" target="_blank" class="hidden-xs">Link</a></div>
</div>
</li>

Since template files are text files, they can’t be dynamically loaded like scripts. So how can we deliver them to the client app?

There are two viable solutions and Muffin implements both. The first solution is to use AJAX to dynamically download the template files as text files. This is useful when we need to dynamically load a single template. The second solution is to wrap the template file into a JavaScript module, such as:

1
2
3
define('javascripts/apps/main/templates/_article_cell.html', [], function() {
return '\n<li class="article-cell"><% if (article.imageUrl != null) { %><a href="#"><img src="<%- article.imageUrl %>" class="article-image"/></a><% } %>\n <div class="content">\n <div class="title"><a href="#"><%- article.title %></a></div>\n <div class="summary"><%= article.summary %></div>\n <div class="dateline"><%- article.updated %><a href="<%- article.link %>" target="_blank" class="hidden-xs">Link</a></div>\n </div>\n</li>';
});

This is useful when we need to pack template files together with JavaScript modules.

As for the developer, templates can be treated exactly the same as JavaScript modules, and can be easily imported with the require function:

1
cellTemplate = require '../templates/_article_cell.html'

Package Management

Now that we have a great way to write clean JavaScript modules, let’s think about how to share and reuse them.

TJ Holowaychuk wrote a decent article about creating reusable components for webapps. Later he implemented his ideas in a package manager named Component. TJ’s components are not just JavaScript modules, but can also include markups and styles.

I think TJ’s components are a step in the right direction. The only reason we are using massive libraries like jQuery today, is that we didn’t have a good way to write modular JavaScript. Now that is changing, so I think web developement practices will change too. The future of webapps is not based on a few monolithic libraries, but many smaller, more modular components.

Muffin has a built-in package manager that is compatible with TJ’s components. Muffin can install TJ’s components in Muffin apps, and Muffin packages follow TJ’s component spec. Just like Component, Muffin’s package manager can install packages from any GitHub repository, and only copies essential files instead of cloning the full git repository.

There are some differences, too. Muffin installs the packages in a slightly different directory structure. And Muffin prefers to keep code in its original format.

Shim

If a script is not set up as a component or a Muffin package, you can set up a shim for it. The shim follows the component spec as well, so that it behaves just like a component after installation.

For example, this is a shim for EightMedia/hammer.js:

1
2
3
4
5
"EightMedia/hammer.js": {
"version": "*",
"type": "script",
"main": "dist/hammer.js"
}

You can also specify dependencies in the shim, just like in real components. For example, bry4n/backbone-shortcuts depends on madrobby/keymaster, so we can set up shims like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
"madrobby/keymaster": {
"version": "*",
"type": "script",
"main": "keymaster.js"
},
"bry4n/backbone-shortcuts": {
"version": "*",
"type": "script",
"main": "backbone.shortcuts.js",
"dependencies": {
"jashkenas/backbone": "*",
"madrobby/keymaster": "*"
}
}

Specifying the script type is optional. Muffin wraps both scripts and modules, but the wrapper function is slightly different. The wrapper function for scripts doesn’t pass in variables like require, module and exports. This only matters for scripts that support multiple formats. If you don’t specify the script type, Muffin loads the script as a module.

Muffin also adds package dependencies to the wrapper function, thus it can load the dependencies of traditional scripts just as easily.

Summary

By now you should have a good idea how Muffin handles module loading, module dependencies and package management. Check out muffin.io website if you want to learn more about Muffin.

Yaogang Lian

An iOS, Mac and web developer. Focusing on building productivity and educational apps.