JavaScript and Maps (in that order)

A Thousand Days of Kublai

We picked you because you licked our iced coffee. I said ‘she likes what we like’ and that’s how we picked between you and your sister. Somehow we missed the (physical) thing that truly set you apart from your sister (who is named Zelda and eventually ended up in the care of Rachel’s family), which is your thumbs thumbs.

I remember being so worried on the drive home with your carrier in my lap. I couldn’t help thinking that we would get into a car accident and I would blow it, this responsibility. We made it home without incident but that drive remains one of if not the most nerve wracking drives of my life.

Rachel and I fell into roles that may or may not be an echo of things to come, where I am the one who spoiled you which Rachel was the one who gave you tough love.

I remember when you brought in your first bird, there were feathers everywhere, I didn’t know whether to scold you or high five you.

The days when I got home before Rachel were always my favorite because you were always very affectionate to the first one home. You would jump into my lap and rub your head against my face. Sometimes you then curl up in my lap and take a nap. On weeks where Rachel worked late coming home to you was the highlight of my day. Some mornings I’d wake up and find that you had gotten between Rachel and I you would be cuddled up in my arms.

We got you a couple weeks after Rachel and I moved in together. You were always there for me when I needed someone to hug, and those first couple of months after we moved in together I needed a lot of hugs. I’m not sure we would have been able to make living together work if you hadn’t been there, and no that isn’t me just talking you up, I’ve been saying that for years. You can also take some of the credit of helping me quit smoking.

Rachel and I loved you so much, On your first birthday we threw a party for you and served crab cakes, you didn’t like them, but had a fun time anyway.

You always liked to go outside and then come right back inside an hour or so later when it rained that could be a problem as you’d want to get in bed with us sopping wet. I’d often let you out about an hour before we got up and then let you back in when we got up. The joke was that you are our weather cat, because if you were wet when you came back in, it was raining, if you was cold, it was cold.

One time when it was snowing you were out for 14 hours, that was by far the longest you’d ever been out and when you came back you were warm and dry. We’ve always speculated you were cheating on us with another family that day.

On the night of the 5th I let her you at 3am, it was raining so I expected you to be back fairly soon. But when we woke up you hadn’t come home. As the day progressed and you didn’t come home things got very nerve wracking. When I finally went to bed that evening it suddenly hit me and I burst into tears for the first time in 18 years.

The next day we put up signs and that’s when a neighbor told us that the morning of the 5th a tri-color cat was killed by a coyote very close to our house. Rachel went home and I talked to the neighbor a little longer. When I got home I could hear Rachel crying on the back steps from the street out front.

It wasn’t definitely you, but it might be, the only way we’ll ever know for sure one way or the other, is if we learn it isn’t you by you coming home. We keep hope alive but everyday it becomes a little less likely that you are alive. It’s been a month and you haven’t come back, while for a part of me you will always be my Schrodinger’s cat, I’ve had to accept that it is more likely then not that a coyote took you from me.

Accepting this has been one of the hardest things I have ever done in my life. I just have so much trouble accepting that you are no more, but even if it’s true, you will always have a place in my heart until the day I die.

You were only on this world for a little over 1000 days. But I wouldn’t trade that time for anything because while there is a piece missing from my heart right now, for 1000 days our apartment was a home because you were in it.

Goodbye Kublai Linso Metcalf 8/9/11-7/5/14 I miss you so much.

Module: The Nitty Gritty

My last post covered some of the general import syntax of modules and today I’m going to go deeper into some of the features that tend not to be covered when people talk about modules. Similar to my last post I’m going to try to stick to facts in the first part and super biased opinions in the second part.

Bindings and what not

In JavaScript numbers and strings are references to specific values and are passed around by reference, in other words if you do:

var a = 3;
var b = a;
a++;

a is now equal to 4 and b is equal to 3, because a and b were both just references to the number 3, when you incremented a all that did was set a equal to 4, it didn’t effect b as that was a separate reference to the number 3.

On the other hand objects (and thus arrays and functions) do not work that way so:

var a = {num:3}
var b = a;
a.num++;

now both a.num and b.num are both equal to 4. That is because a and b do not both refer to objects that have the value {num:3} but instead they refer to the same object that originally had a value of {num:3}.

In the ES6 module syntax if you want to export an object you can do it via

// lib.js
export default {
    num: 3,
    increment: function () {
        this.num++;
    }
}

If you then import it into multiple files they are just references to the same object so

// a.js
import a from './lib.js'
console.log(a.num); // 3
setTimeout(function (){
    a.increment();
}, 1000);

and

// b.js
import b from './lib.js'
console.log(b.num); // 3
setTimeout(function (){
    console.log(b.num); // 4
}, 2000);

The other thing to note is that if there was no increment method on the exported object, you would only get an error when you tried to use the method, a second into the script being run.

In ES6 you can also write the script as follows

// lib.js
export let num = 3;
export function increment () {
    num++;
}

then

// a.js
import {num, increment} from './lib.js'
console.log(num); // 3
setTimeout(function (){
    increment();
}, 1000);

and then

// b.js
import {num as b} from './lib.js'
console.log(b); // 3
setTimeout(function (){
    console.log(b); // <-- what does this print
}, 2000);

It prints 4, this is because what modules export are bindings. Put another way, you could think of ‘num’ and ‘b’ as both being references to an object, but the object has a value of 3 (and latter 4) instead of it being a set of key value pairs.

Static Analysis

The other thing to note is that if there was no increment function exported from lib.js then instead of getting an error 2 seconds into running module a.js you would get an error before a.js was run due to modules being statically analyzed before they are evaluated and then imports fetched before hand, in other words somewhat how browserify works (browserify actually works more like the first example as it would only throw a static error if you had an import for a module that didn’t exist, but this is contrasted to node.js and unbundled AMD modules which only bother to look up whether a module exists when the code that imports it is run).

Circular dependencies

A tricky thing in any module system is how to deal with situations where 2 modules require each other. After some discussion with Guy Bedford I somewhat understand how ES6 deals with them. Basically function declerations hoist accross modules so this example (from Guy’s es6-module-loader) would work:

even.js

import { odd } from './odd'

export var counter = 0;

export function even(n) {
  counter++;
  return n == 0 || odd(n - 1);
}

odd.js

import { even } from './even';

export function odd(n) {
  return n != 0 && even(n - 1);
}

but on the other hand

a.js;

export var obj = {};
import {something} from './b';
something();

b.js

import {obj} from './a';
obj.val = 'asdf';
export function something() {}

would throw an error because ‘obj’ wouldn’t be a object yet when b runs because … I’m just going to quote Guy here

The point here is that obj is bound by the time b executes, but the assignment has not been made as we still have to execute sources in some underlying order.

In other words you can think of them as being like regular JavaScript variables where the variable to be used is initialized at the top of the scope, but doesn’t have a value until later on.

Commentary

Static Analysis

Lets look at the static analysis first, this has the benefit of preventing a whole class of errors from ever happening in the first place, namely importing something that doesn’t exist but that you don’t use until later. For example in node you could have a file that has

//lib.js
exports.foo = function () {/*someting*/};

and then another with a typo

//app.js
var lib = require('./lib');
function rarelyRun(){
    lib.fooo();
}

You would have to actually run the rarelyRun function to spot that error in node.js, but in ES6 modules you this would cause an error the first time you tried to run it.

The downside of static analysis isn’t actually anything to do with the static analysis but what is done in the name of static analysis, namely you can’t put import (or export) statements anywhere except top level code, i.e. no

var bar, baz;
if (foo) {
    import {_bar, _baz} from 'bat';
    bar = _bar;
    baz = _baz;
} else {
    bar = 'bar';
    baz = 'baz';
}

Putting statically analyzed imports inside conditionals is less stupid that it seems at first glance as it makes writing polyglot programs that can run in different environments possible. For instance, it’s common to see something like the following in the wild:

var myThing = something;
if (typeof define === 'function' && define.amd) {
    define(function () {
        return myThing; 
    });
} else if (typeof exports === 'object') {
    module.exports = myThing;
} else {
    window.myThing = myThing;
}

this exports your module as an AMD one if that exists, a Common JS one if that exists and otherwise assumes you are in a regular browser and attaches it to the window. Creating things like this is impossible in ES6 modules.

Circular References

This is a step back compared to Node.js, remember this

a.js;

export var obj = {};
import {something} from './b';
something();

b.js

import {obj} from './a';
obj.val = 'asdf';
export function something() {}

well the node.js version of this

a.js

exports.obj = {};
var b = require(./'b');
b.something();

b.js

var a = require('a');
a.obj.val = 'asdf';
exports.something = function() {};

would work fine. Why this works is best stated in the require.js documentations

[…] use exports to create an empty object for the module that is available immediately for reference by other modules. By doing this on both sides of a circular dependency, you can then safely hold on to the the other module.

The ES6 version seems like a step back compared to CJS/AMD as it doesn’t cover as many of the edge cases.

Mutable Binding

The ability to import a name, from a module, and have it look like a global variable while at the same time changing like an object property is the killer ES6 module feature nobody asked for, nobody needs, and nobody wants.

Module Drama Recap

A recent proposal to change ES6 modules slightly has unleashed a torrent of drama. What follows is a recap of the ES6 module system, followed by some background, followed by a parable, followed by my opinions.

ES6 Modules

The original module system for ES6 involved modules being able to export names via either putting export in front of declarations

export var/let/const foo = bar;
export function foo(){}

or with export statments

function foo(){}
export {foo}
export {foo as bar}

these names could then be imported into another module

import {foo, bar as baz} from 'path'

if you needed to import an object with all of the names you could use

module name from 'path'

much work was put into making sure that circular dependencies and other edge cases were worked out leading to the restriction that import, module, and export statements must be at the top level so that they will always run, so no

if (something) {
  export let foo = bar
}

There was also originally an ability to create inline modules via

Module name {
   // stuff
}

but this was removed on the grounds that it was too complex for now and can be added later.

A major issue with this setup was that there are no anonymous exports, it is very common in node.js and require.js systems to have a module be a single function or an instance of a class, but with the original setup you would have to give a name to that single thing. You see this in python with super ugly stuff like “the foo module which exports a foo member” but just for fun also stuff like “the bar module which exports a Bar member” and leads to usage where people just use foo.foo and bar.Bar.

The last major change to modules before recently was the addition of ‘default exports’ which added the ability to export something without giving it a name via

export default function (){}
//or
function foo() {}
export default foo

which is actually a more flexible syntax then the export from syntax which can only take a name

// good
function foo(){}
export {foo as bar}
// bad
export {function (){} as bar}

on that note the other thing you can’t do is

// bad
function foo(){}
export foo

but I digress these default exports could be imported with

import foo from 'path'

this was done about a year ago and always seemed to be as a bit of a stop gap as it left the module system in a somewhat inconsistent place where different ways of exporting functions meant they had to be imported differently, for instance with underscore which is a collection of methods, to get it’s map method you can either do

 module _ from 'underscore'
_.map();

or

import {map} from 'underscore'
import {map as _map} from 'underscore'

though you can’t do

//BAD
let _ = {};
import {map as _.map} from 'underscore'

now on the other hand for jquery which is also a collection of methods, but are attached to a function you would have to do

import $ from 'jquery'
$.map();

because if you tried to do

module $ from 'jquery'

map would actually be at

$['default'].map()

the solution proposed by TC39 is to remove the module/from syntax so that you either use a default import or you import specific methods and if you need all the full module object do something like

import 'foo'
let foo = this.get('foo')

if you can’t tell the idea seems to be that you would need this very often.

This change has spawned one of the most contentious threads on the es-discuss mailing list since somebody pointed out that Promise.resolve and Promise.cast were functionally identical. To understand why you have to remember that these aren’t the only JavaScript modules.

A lengthy digression on AMD and Common JS modules.

The JavaScript community currently uses 2 module systems, AMD and Common JS (CJS), while some people describe them as ‘competing module systems’ those people are wrong. The CJS and AMD are for all intents and purposes the same module system used by 2 different groups. The major difference between the 2 systems is that CJS modules are individual files while AMD are JavaScript objects, a CJS module wrapped in

define(function (require, exports, module) {
...
});

is an AMD module and while AMD modules do have some ways of importing and exporting code that CJS modules does not have, the trend has been (in my biased experience) away from using them.

These days when people talk about a AMD/CJS split they are usually meaning browserify/webpack/npm vs r.js/cujo.js/bower split.

The important thing to remember is that since inline modules were removed from the spec, the things that the ES6 spec is covering are the things that the AMD/CJS communities agree on, namely syntax and that in code a module is just an object.

Another Tangent

A coder spends his free time writing a module to download cat pictures, he publishes it to NPM. A user opens an issue about the possibility of using it to download dog pictures or maybe even generic mammal pictures. The coder feels that the user does not understand the purpose of this module and instead of adding the feature, or even just dismissively pointing out that pull requests are accepted, the coder argues with the user about whether the user really wants this.

This is a bad habit that many of us occasionally suffer from. The reason for it is that when you put a lot of work into writing something, its easy to to take people trying to offer suggests to your project as people attacking your code baby. If you publish your project then you’re no longer the only user of it and other people have different needs than you and need different features then you, this is natural.

My opinions

With modules I feel like TC39 is suffering from this syndrome as well. Instead of taking criticism as suggesting a need to work towards adapting the system based on the experience learned from actual JavaScript developers using modules in the wild, they seem to be taking criticism as something they need to defend against and this makes the node.js community feel like TC39 doesn’t care about them, because when they point out that they’ve learned a thing or 2 about modules in the past several years, TC39 seems to say ‘we don’t care’.

People put a lot of hard work into making the module system, but having an inflexible take it or leave it approach does nobody any good. Modules have yet to be implemented by any engine so if comprehension, which firefox already implemented, can be taken out of the spec, then there is no reason that anything should be off the table regarding modules.

Making Promises Fast

As alluded to in my last post there were some performance killing bugs I had to fix in my promise library lie, through the majority of the issues were actually with immediate, the scheduling library lie uses.

Lie

The only perf oriented change I made to Lie was to make Promise.all speed up by adding the following line to Promise.resolve

if (value instanceof Promise) {
  return value;
}

basically i forgot about step 2 of the spec which allows Promise.resolve to simply return the promise if it was created by this constructor, with that not there I was treating all promises as untrusted thenables, which meant I was going through the somewhat time consuming unwrapping procedure an extra time. This sped up Promise.all because all values that are passed to Promise.all go through Promise.resolve so I was doubling my work unwrapping them twice.

Immediate

The idea behind immediate is that every time you call immediate(func) if this is the only thing in the queue we schedule a drain to happen async but very soon (sooner then setTimeout which has a min of 4ms). When the drain happens it goes through and calls each of the functions in the queue, the tricky thing is that if immediate is called by function while the queue is draining, it needs to be appended to the current queue not scheduled for the next one.

e.g. if you have

function a() {
  immediate(b);
}
function b() {
  setTimeout(c, 5);
}

then it would run go

  • async wait
  • a
  • b
  • 5ms wait
  • async wait
  • c

So the first reason it was slow was because I implemented a very naive double ended queue via .shift

function drainQueue() {
  var task;
  while ((task = handlerQueue.shift())) {
    task();
  }
}

so this is an awful way to do this because every time you call shift it moves all of the items every single time. This is not efficient, I replaced it with

function drainQueue() {
  var task;
  var i = -1;
  while ((task = handlerQueue[++i])) {
    task();
  }
  handlerQueue = [];
}

making this change meant that Lie could actually finish the parallel tests, doing so in 9548ms this is pretty slow, one of the reasons being that drainQueue will always try to access an item past the end of the array which can trigger various deoptimizations, switching the code to

function drainQueue() {
  var i = -1;
  while (++i < handlerQueue.length); {
    handlerQueue[i]();
  } 
  handlerQueue = [];
}

fixes that getting the speed to somewhere between 2829 and 8465, basically there is a race condition that sometimes super slows down the code. The issue was that v8 optimizes functions that are ‘hot’ by compiling them with type info to native code. If a type changes then the function gets deoptimized and the compiled code is discarded. Since compiling the function is not free we want to avoid having to throw out the result. The problem with the previous queue was that the array could grow so long that the i variable can grow big enough that it is no longer a small integer and deoptimizing the function, changing the code to the following fixes that.

function drainQueue() {
  draining = true;
  var i, newQueue;
  var len = handlerQueue.length;
  while (len) {
    newQueue = handlerQueue;
    handlerQueue = [];
    i = -1;
    while (++i < len) {
      newQueue[i]();
    }
    len = handlerQueue.length;
  }
  draining = false;
}

this brings the time down to 2490ms, making lie the 3rd fastest of the 6 we looked at yesterday. Likely at some point in the future I’m going to want to do what asap, bluebird, and when do and implement a circular buffer, but some tests with the implementation that when uses suggest that that isn’t a chock point at the moment.

Back to Lie

Right before I was going to publish this I wondered, what if we double the amount of parallel things we do?

By default the tests do an array of 25 promises resolved by all, but what happens if we double it to 50. At worst the amount of time should double right?

No, for lie the amount of time increased 10 fold ~20,000ms, through it could have been worse, RSVP runs out of memory and crashes.

the problem turned out to be the function I used to make sure the handler’s were only called once

function once() {
  var called = 0;
  return function wrapper(wrappedFunction) {
    return function () {
      if (called++) {
        return;
      }
      wrappedFunction.apply(this, arguments);
    };
  };
}

true to it’s name once was only used in one place and I was able to replace

var onceWrapper = once();
var onError =  onceWrapper(function (value) {
    handlers.reject(self, value);
 });
var onSuccess =  onceWrapper(function (value) {
    handlers.resolve(self, value);
 });

with

   var called = false;
    function onError(value) {
      if (called) {
        return;
      }
      called = true;
      handlers.reject(self, value);
    }
    function onSuccess(value) {
      if (called) {
        return;
      }
      called = true;
      handlers.resolve(self, value);
  }

this causes lie to go from taking 20,000 ms to taking 3,000ms and bringing the original version with 25 parallel things down to 1825ms. This is faster then when.js.

Conclusion

Use version 2.7.4 of lie, it’s super fast.

Comparing promise libraries

Today we’re going to compare promise libraries, this isn’t about finding the best library as such a thing does not exist, all of the libraries make different trade offs to try to be good for certain things.

So disclosure: I’m going to try to be as unbiased as possible but I did write one of them (lie) so if nothing else, that library is going to be optimized for the same trade offs I find important.

There are a lot of spec compliant promise implementations, my criteria for picking these are

  1. I must have heard of it.
  2. It must be used in the wild.
  3. It must have ~ES6 syntax (take a resolver function to create a promise at the minimum).
  4. Provide microtasks in a reasonable manner (aka avoid setTimeout in modern browsers).
  5. Be aplus spec compliant.

The libraries I chose were

And these broadly fall into 2 categories:

bluebird, Q, and when are tool kits that in addition to providing promises also provide a plethora of helper functions for dealing with promises in the footsteps of the venerable async library.

lie, RSVP and then/promise are basic promise libraries which stick close to what the ES6 promise object provides, namely they have static resolve, reject, and all methods and their prototype has .then and .catch methods.

I had a bunch of comparisons I was going to do but then I found a nasty bug in Lie and spent a bunch of time on that so I’ll just cut the my findings.

My benchmark results are here If you want to do the benchmarks yourself just download the bluebird repo and run them there, the size comparisons are here.

Bluebird

Bluebird is the fastest by far, through the speed should be taken with a grain of salt as the bench marking tests were written by the guy who wrote bluebird written by somebody else originally but the live in the bluebird repo, that being said they are the best promise speed tests (as opposed to latency) that I can find so that’s what we got.

Bluebird is approximately equal in size to the other 5 libraries combined (15k minified and gzipped) , so my recommendation would be to without a doubt use it in Node and if you don’t have to be hyper concerned about download size use it on your client size App. Client size libraries are where I tend to be most concerned about size and I would avoid using bluebird there, if your library works in node and the browser you can use browserify to set up an automatic substitution that replaces bluebird with something smaller on the client side while using it in node, it’s what I do in a couple libraries.

When

When was pretty consistently the second fastest of the libraries but at less then a third of the size of bluebird when is by far the best choice for promise tool belts in size contentious situations (i.e. client side libraries). Plus a couple of the guys who write it are Boston based like me so extra credit.

Q

Q is an order of magnitude slower then all the other libraries. I really can’t recommend using it in production, which is sad because it is the most popular of all of these (I’ve seen articles using Q and promises interchangeably). Q is the only one of these libraries to that does not implement version 1.1 of the A+ promise spec, implementing the 1.0 version. The 1.1 test suite is deviously thorough and the lack of conformity at the moment is another reason to avoid Q.

RSVP

RSVP is a bit of a strange case, unlike the previous 3 libraries the only extra method it has is a ‘hash’ method which is like all but for objects, (edit: as pointed out on twitter, it also has finally) but at the same time it’s more similar in size to Q and When (~4k minified and gzipped) and has some other features like uncaught promise catching and long stack traces that tend to be in the tool kit libraries, its performance is a a bit strange in that it is very bad in version 0.10.28 of node but in 0.11.13 it (and then-promise) are both significantly faster making me suspect some new optimization in v8 that is helping it. Though even in 0.11.13 it’s performance isn’t that good and if I was picking a 4k library I’d go for when which is faster and has more features. That being said it’s bundled with ember so if you’re in an ember app or something else that already includes it it’s a fine choice (and now works in web workers)

Then-Promise

This library is the first of smaller ones aiming to be drop in replacements for ES6 spec promises in order to make it easy for libraries to return promises. That being said it does offer a few nonstandard methods. Size and performance wise it’s pretty similar to Lie so while I’d recommend my own library this is an equally good choice.

Lie

My promise library which like then-promise is a minimal one, it’s performance is now the same as then-promise but when doing this article I had to fix a performance killing bug (which will probably be the subject of another blog post). Overall it’s size is pretty similar to then-promise (200 bytes larger then then-promise when gzipped and minified), its performance is similar (faster in certain situations slower in others) and while it doesn’t offer any helper functions it does use low latency async techniques on a wider set of platforms via immediate.

Honestly I use this one for my stuff and we use it on pouchdb for the browser but you’d do well with either this or then-promise.

Edit: see my next post, I ended up making lie considerably faster and can now say that it is faster then then-promise and you should use it.

Wrap up

I’m somewhat concerned about how badly Q did, if there is an issue with the benchmarks I would be more then happy to rerun them after a fix goes through. Also in the interest of fairness as I did wait until after I fixed a bug in my own library if any of the other libraries put through speed improvements I can rerun these tests.

NPM as a build system

EDIT: In case I wasn’t clear enough, when you use npm to run a script with npm run foo or npm test it makes sure all of the local modules are in your path, so if you only have uglify and mocha installed locally, you can still write "test": "uglifyjs ./lib && mocha test/*.js"

The reasons that you often don’t need grunt have been done to death, so I don’t feel any need to lend my voice except to agree. I’ve also seen Make thrown around as another alternative. Make is a fine program on *nix systems but for node.js it’s missing a copule things,

  1. it is global, you have to specify ./node_modules/uglify-js/bin/uglifyjs instead of just uglifyjs.
  2. it is a pain to use on windows.

Now before we turn this into a windows/*nix flame war let me point out, I do not develop on windows, I despise developing on windows, the only time I ever voluntarily touch windows machines is when I’m helping my fiance fix her laptop. But that being said I work with people who use windows and I’d like to continue working with them as I have no interest in doing the heroic work they do with our legacy .NET apps.

Many people starting out with node are also on windows. While we may later be able to seduce them over to the *nix side, for now they are learning on the machine they have.

So back to your build step needing to work on windows, keep in mind not every node project on windows is simply going to get deployed to a linux box so simply using vagrant is not the answer. I’ve used IIS node to replace pieces of old asp apps, and I’ve used node to compile static assets that are then pushed somewhere.

And yes it is theoretically possible to get make installed on windows, but not easy, most of the suggestions I’ve seen either contain sourceforge links or references to Cygwin. These are not things that sane people want to do.

What instead?

For the 99% of projects that are a build step and some tests, you can use NPM as a build system, just put them in your package.json, when you run the script that way, npm makes sure your path has your node_module .bin folder is in the paths so if you use mocha to test you can just add

"test": "mocha -R nyan test/test.js"

to the scripts object in your package.json, and it will run tests on all your files that start with test. and are in the test folder, if you like tape you can do

"test": "node test/test.js | tspec"

jshinting can be done with

"lint": "jshint lib/*",

or you can combine it with your testing setup

"test": "jshint lib/* && node test/test.js | tspec"

both of these will look to .jshintrc for the rules, no need to specify them in a config.

karma can be run this way easily and testling can as well with browserify

"test": "browserify test/tests.js | testling"

For building browserify excels at using a package.json script, but if you must use [require.js] instead of putting the overly verbose and totally unnecessary 30 lines of config options to do something simple in with the build stuff, you can put it in a build.js file and then go

"build": "r.js -o build.js"

but if you don’t like having to specify:

you can just have a package.json with:

(I’ll get to the syntax in a minute) though to be fair since I often have a couple steps here I usually just separate it out into a couple steps:

with browserify you can set specify other things in the package.json, for instance the “browser” key defines a map where you can exclude or substitute things from the build, when you specify the substitutions here instead of in the grunt file then other people browserifing a library that uses your lib as a dep also get the substitution applied, in other words, you need to be doing this anyway, why do it twice?

browserify-shim is a very helpful library in this respect (so helpful, grunt-browserify includes it but fails to mention it causing a confusing blending of the options in the docs).

If things get too busy we can just launch a batch script with

"build": "./build.sh"

over in build.sh same rules apply, it will work in windows if you stick to node commands (and git) you work cross platform, you can also call node scripts aka

"build": "node build.js"

in the some the examples I used | and && to separate commands on a single line and used > to out put a file. In shell commands && means the same thing as in JavaScript, it’s a lazy and operator so command1 && command2 means run command1 and if it doesn’t fail, run command2.

The other commands related to pipes/stdin/stdout. Commands can read from stdin and write to stdout and in node those are both streams, available at process.stdin and process.stdout, though another thing that outputs to stdout in node is console.log. By default unless it is piped anywhere your console will print stdout and thats why programs like browserify will print your bundle on the screen if you forget the -o option.

The | command (pipe) takes the output from one commands stdin and pipes it to the next commands stdin, so when you want to browserify a file and give it to testling, you can do

browserify . | testling

which takes the output of browserifing the current directory (the dot) and pipes it to testling.

or if you want to take the ugly tap output from tape and make it look nice, use

node test/test.js | tspec

which pipes it to tap-spec, or you could use

node test/test.js | tnyan

to pipe it to tap-nyan, because testing should be fun.

the > command means write stdout to a file, so you’ll sometimes see people write

browserify . > bundle.js

which means write the bundle to bundle.js (yes I know you could also write -o)

then can be combined with pipes as well so to browserify and minify you can write

browserify . | uglifyjs -mc > dist/script.js

which browserifies it and then pipes the output to uglifyjs to minify it with the mangle and compress option.

N.B. uglifyjs is the command line program, but it’s in NPM as uglify-js so you need to do npm install --save-dev uglify-js

The < operator reads the specified file as stdin so the following, while stupid, would work

browserify . > temp.js && uglifyjs -mc < temp.js > dist/script.js

in which we write our browserify output to a tempfile and then read it into stdin for uglify, a non contrived example would involve something like istanbul which always writes code coverage to a file and prints test results to stdout, so if you wanted to run coverage on a mocha script and then pipe it to coveralls then

istanbul cover ./node_modules/mocha/bin/_mocha --report lcovonly -- tests/*.mocha.js -R spec -t 5000 && coveralls < ./coverage/lcov.info

would do it

N.B. ./node_modules/mocha/bin/_mocha is the path to the real mocha binary, the mocha in your path (that we would get if we ran mocha) is a command line parser that will prevent istanbul from working right.

You will sometimes see people writing the above as

istanbul ... spec -t 5000 && cat ./coverage/lcov.info || coveralls

those people are wrong, cat is for concatenating files, not for reading them into stdout for you because cat isn’t going to be available on windows.

You want to watch your bundle and have it rebuild when you change anything, try watchify

"watch": "watchify . -o ./dist/bundle"

note you need to specify the output folder, you can’t use > because … it throws an error for some reason.

WINDOWS!

There is no rm -rf so instead use rimraf which will remove a directory and all the stuff in it recursively so from our previous example

rimraf ./foo

will remove ./foo, and ./foo/bar, and ./foo/bar/baz and anything that might be in them.

N.B. Don’t be an idiot with rimraf, if your careless you can delete everything, especially on windows.

To do batch copies ala grunt-copy you can use copyfiles which I wrote expressly as a replacement for grunt-copy, so to move all the JavaScript files from the vendor and src folder to the dist

copyfiles ./vendor/*.js ./src/*.js dist

but if you want them with out the outer vendor and src folder

copyfiles -f ./vendor/*.js ./src/*.js dist

if if you want to do a globstar

copyfiles -f './vendor/**/*.js' './src/**/*.js' dist

for operating system consistency you need the quotes around globstars, and for once that’s not windows that is the problem, it’s OSX with its out of date bash.

Conclusion

You probably don’t need to use any build system for many node projects because npm can act as a build system and for the love of god don’t use make because that’s practically holding up a sign telling .NET programers to continue writing .NET program. We want them publishing all the things to NPM.

discuss on reddit

Other articles on the subject

JavaScript style mistakes I see a lot

  • Edit: removed recursive reference, it was incorrect (my bad) but you should probably avoid recursive references in general.
  • Edit2: discuss on reddit
  • Edit3: forgot hoisting was such a hot button issue, switching to the hopefully less controversial statement that declarations should be your default

A quick run down of a few very common JavaScript style issues, that while aren’t outright errors you should probably avoid.

Function expressions, not on objects

This is when you write

var name = function () {};

instead of

function name(){}

the bottom version should be your default which you should use unless you have a reason to do it the other way. Some of the benefits include

  • hoisting (though you might passionately disagree)
  • reference to the functions name inside the function can’t be changed.
  • because you have to pick one for consistency and I’ve never seen project choose the top version.

using module.exports, to export an object

That is when you write

module.exports = {foo: function(){}, bar: function(){}}

instead of

exports.foo = function (){};

exports.bar = function (){};

the difference is that the bottom syntax can handle circular dependencies, and the top can’t, they are otherwise identical.

making me guess your exports key.

that is if you only export one thing from your module, doing

exports.someRandomName = thing;

as opposed to

module.exports = thing

unless you have circular dependency you have to deal with, making me write

var yourThing = require('yourModule').someRandomName;

is going to cause me to judge you.

PouchDB 2.0.0

PouchDB 2.0.0 is out and has a lot of great stuff, first off you’ll notice breaking changes they are:

  • allDbs has died in a fire. This optional disabled by default feature complicated the db creation process to no end and was the source of many race conditions in testing. If you were one of the 3 people using it fear not, while it was previously impossible to get this behavior otherwise it now is due to another new feature.
  • PouchDB now uses prototype for inheritance, no more of that copying methods from object to object. This does mean that var get = db.get will blow up, so write var get = db.get.bind(db) instead.
  • plugin now take advantage of prototype, meaning you can add them after you create a database.
  • For indexedDB and webSQL adapters the schema has been updated to give correct doc counts, your db should migrate fine,
  • for the leveldb adapter we switched from being a folder with 4 leveldb and an annoying ‘.uuid’ file it is a single leveldb with 4 sublevels and no ‘.uuid’ file. This also will transfer your data over the first time you open an old style one in the new.
  • these last 2 mean that v1 PouchDBs will open in v2 but not the other way around. Also v1 ones will become v2 ones as soon as they are opened in the new release so don’t switch back and forth.

Non breaking changes:

  • leveldb based PouchDB creation is now totally async with no fs.
  • all the methods return promises (except the replication ones). This is in addition to taking callbacks. If you hate promises then the only change you’ll notice is that we more consistently return errors instead of throwing. We use the superfast bluebird in node and in the browser we default to native promises and fallback to the supersmall lie.
  • it works in web worker now, including in chrome where you have access to indexedDB inside a worker.
  • we prevent overly enthusiastic caches in IE from preventing requests from reaching CouchDB
  • we no longer screw around with your options object so you can re use it for multiple calls.
  • if you pass emit as the second argument to your temp query function, map reduce no longer evals it.
  • a whole host of things to make replication much faster
  • a whole lot of tweaks to the webSQL adapter thanks to Nolan.
  • the PouchDB constructor is now an event emitter and you can listen for create and destroy events.
  • If somebody has shat all over your JavaScript environment by adding enumerable properties to Object.prototype, PouchDB has you covered.
  • Put sugar you can specify _id and _rev as the 2nd and 3rd arguments to put, i..e. instead of db.put({some: 'stuff', _id: 'myId', _rev: '1-rev'}), you can do, db.put({some: 'stuff', 'myId', '1-rev'), when you are creating a document you can omit the rev (though if you omit the _id and do include the rev you’ll create a document with that rev).

Some fixes that make developing much easier include

  • qUnit has been totally replaced with mocha and chai, if you’ve ever had the misfortune of using qUnit in node or with async tests or both you’ll know why this is so awesome.
  • all the tmp test dbs are in their own folder which is deleted afterwards, no more mess after failed tests.

Grab it from npm, component or bower, or download the regular or minified version.

Dear TC-39, can you please take another look at your module syntax

Dear TC-39,

You’ve been doing some fantastic work recently, all the new features slated for ES6 look amazing. Arrow functions and the functional parameters sugar do such a good job at making functions just that much more fun to use. Set, Map, WeakMap great and really going to help for data centric apps (like all the GeoSpatial stuff I do). Generators, well I’m already using them and they are fantastic. Promises are already in browsers, that’s so fast it’s unbelievable. All these great things you’ve done make this just so much harder. But the module syntax you folks are proposing for ES6, I think you should take a fresh look at it.

Most of the discussion we’ve had on the module system has been framed poorly, it has been people like me who it rubs the wrong way attacking it and people like you who spent a lot of time developing it defending it. But really we’re all on the same side, we want the best possible module system that we are able to actually get into browsers in a realistic time frame. More specifically a module system that does static analysis (finds the dependencies without running the code), so that we can avoid the significantly added layer of complexity that comes with dynamically loading modules.

I originally was going to write a while thing about the flaws in the system you guys are proposing, I even made this flow chart to show how complicated the export syntax is and a whole list of links to code that wouldn’t be writable in the new syntax. But then it occurred to me that it is hard to tell if you guys writing the spec really even think this syntax is any better then what is currently out there, because if you do your doing a very bad job of selling it to us in the node community. Static analysis of dependencies and imports with browserify or the CommonJS wrapper in require.js (define(function(require, exports, module){...});) show that in practice there is no need for restrictions on the location of imports to successfully find them in static analysis and people trying to conditionally load modules in a situation where they can’t isn’t a problem in the wild.

There are sources which describe ES6 modules as a compromise between CommonJS and AMD modules, but again that isn’t really the case any more as using the CommonJS wrapper has become more prevalent in AMD code to the point that most debates about CommonJS vs AMD have much more to do with tooling and deployment strategy. In many ways ES6 modules are redefining the one thing about modules that the AMD and CommonJS communities agree on, syntax.

Which brings us to the root issue, TC-39 hasn’t sold the node and AMD community on their new modules system. Coming from node and AMD systems this one is more complicated with significantly less functionality, specifically

  • Lack of import statements (as opposed to declarations).
  • Syntax that is still geared towards exporting a bag of values as opposed to a thing.
  • Generally high level of complexity of import export syntax.

The people in the node community giving you grief about this syntax aren’t just bikeshedding but are people building production apps with a module system that they like and you haven’t convinced them that you have something better. And that’s just sad because the node community is your biggest fans. We love generators, we are using block scoping in the wild, and have some very strong feelings about Promises, much of which is positive.

TC-39 that’s what I’m asking you, take a hard look at your module syntax and ask yourself

  • Are we taking advantage of the experience that the community has gained building production systems with CommonJS and AMD?
  • Are we taking advantage of the examples offered by more modern build tools like browserify and require.js?
  • Are there real and non-hypothetical advantages gained from restricting imports to declarations?
  • Have we taken into account other features of ES6 like destructuring in designing these modules?
  • If you weren’t involved in the spec writing process, would the export syntax make as much sense?

From an outside perspective it feels like TC-39 is suffering from Not Invented Here syndrome ,if committee members don’t think that this is the case then I think a lot of people would appreciate it if you guys tried to sell us on the advantages of this syntax as compared to current tools if members don’t feel that way I don’t think I’m the only one who would prefer modules done right in ES7 to settling in ES6.

Edit:

  • Comments on reddit
  • Also in case it’s not clear, I’m not advocating to just drop in node modules into the JavaScript spec, just it would be nice to have a similar level of features.

Allow me to reintroduce you to Proj4js

tl;dr Proj4js used to be really bad, now it’s better and you should use it and help build it.

Background

A while ago I decided the write a program to parse shapefiles in the browser into geojson. Despite the shapefile’s reputation as as perversely bizarre format (it’s binary and mixed endian) the part that ended up being the trickiest had nothing to do with the file itself. The problem was with the communities that produce shapefiles and consume geojson. Geojson almost exclusively records coordinates in unprojected latitude, longitude notation (WGS84 to be exact, though somebody did give me NAD83 once). Shapefiles tend to store coordinates in a projected coordinate references system.

The problem I faced was how could I get shape files in any of the thousands of projections out there to just work. There was a program called Proj4js, but it was in a sorry state, the math was all there, but the interface was cumbersome to say the least.

I (and quite a few others) have spent quite a bit of time fixing it up and I want to reintroduce you to this very helpful library.

Getting it

You can install it with npm, bower, component, jam or just stick the script in your browser. It’s build in browserify and exports a UMD that should work in any module system (Dojo might give you some trouble still, but that is an upstream problem fixed in browserify which will be fixed next time you build).

Usage

The global it exports if you drop in the script is proj4 (as is the name in npm, bower, and jam package but component is proj4js/proj4js as it goes by repo) and proj4 is a function which will be all you need 90% of the time. You use it like so

proj4(fromProj, toProj, coords);

It’s that simple, a few notes:

  • From and to proj can be either wkt strings or proj strings, note the lack of need to to create a new Proj object or anything.
  • Ihe coordinates can be in 2 formats, [x,y] or {x:x, y:y} if you pass in an array you get one back, if you pass in an object, you get that back instead. No point class needed.
  • If you only give one projection then it’s assumed you want to project from wgs84
  • If you don’t pass in a coordinate you can an object with forward and inverse methods
  • You can’t just pass in an EPGS code, there is no realistic way to store all of them in a easy way. Most of craziness of the old version related to trying to let people use arbitrary epgs codes.

So if you want to convert a latlng to a projected coordinate,

var newCoords = proj4(wkt/projString, latlng);

If you want to avoid having to reparse the string each time

var trans = proj4(wkt/projString);

var newCoords = trans.forward(latlng);

var newLatLng = trans.inverse(xy);

Bottom line

Proj4 works for every projection I’ve been able to test against, find a problem, open an issue, or even better, open a pull (the code is still ugly, but not as bad as it used to be thanks to browserify).