Ian Bishop

Nearly a year ago, I wrote a gem wrapper for the Yellow Pages Yellow API.

I had mostly forgotten about it, since no issues have been filed against it and no changes have happened to the API itself.

But, I was happy to see when I check its RubyGems page that it has had over 1,000 downloads in the last year.

That sort of feedback (1K downloads and 0 issues?) definitely inspires me to keep on giving.

Hurray for open source!

Brief Intro

In 2009, I was working as a student for a small consulting firm, tasked to develop some real-time visualization stuff for the web.

Normally, this would have meant writing some Flash. But it was 2009, so Flash Is Dead and HTML5 Is King.

It was the first project I would work on where Javascript would make up the meat of the work, rather than just “sprinkling some jQuery on it” – another thing people said back then.

Over the next 8 months, I travelled down the path of re-learning Javascript. It’s likely a path many of you are familiar with.

At first, I tried to make everything object-oriented. Then, oh god, so much global state. I sat there, puzzled, googling desperately:

I eventually realized that I was among the many who never became familiar with the slumbering beast in our browsers, soon to be awoken to bring on the next generation of the web.

Despite this enlightenment, something kept coming back to haunt every project I worked on. See, the difficulty was not using the new shiny HTML5 canvas APIs to draw spinning circles, but rather, it was this pesky AJAX thing.

Infact, it was the A in particlar.

A (for Asynchronous)

1
2
3

function getItem(id) {
  return $.get('/api/item/' + id);
}

This is genuinely how I expected things would work.

But it isn’t. Instead, we have callbacks. Mysterious functions which are called when our request completes (if ever) and passes the resulting data as arguments.

That doesn’t sound bad at all.

1
2
3
4
5
6
7
8
9

function displayItem (item) {
  console.log(item);
}

function loadItem (id) {
  $.get('/api/item/' + id, function(data) {
      displayItem(data);
  });
}

Callback Hell

Callback Hell is the affectionate name given to what happens when you want to do a bunch of sequential things using these asynchronous functions.

Let’s use The Echo Nest as our guinea pig. These guys are really cool and let you upload songs to be analyzed. With their analysis, you can create some really effing amazing stuff.

But that doesn’t necessarily look that cool or read that cool.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$.getJSON(url, {id:trackID, api_key:apiKey}, function(data) {
    var analysisURL = data.response.track.audio_summary.analysis_url;
    track = data.response.track;

    // This call is proxied through the yahoo query engine.
    // This is temporary, but works.
    $.getJSON("http://query.yahooapis.com/v1/public/yql",
      { q: "select * from json where url=\"" + analysisURL + "\"", format: "json"},
      function(data) {
        if (data.query.results != null) {
          track.analysis = data.query.results.json;
          remixer.remixTrack(track, trackURL, callback);
        }
        else {
          console.log('error', 'No analysis data returned:  sorry!');
        }
    });
 });

Everything is fun and games until you have callbacks calling callbacks calling callb…backs on backs on backs.

Deferral

Luckily, the Javascript community is filled with smart, hardworking people who make tools to fix these problems. In 2011, a group of people whom I respect dearly were really excited to talk about how the antidote had arrived.

Our Virgil is here, ready to guide us out of this place, and his name is $.Deferred.

Breaking It Down

Promises are by no means a new idea. They were invented back in the 1970s by some really smart guys who were trying to figure out how to write programs that did more than one thing at a time. Even though Javascript is single threaded, promises prove to be extrodinarily useful for managing asynchronous calls.

A promise is like saying that you’re doing something and you don’t know how long it will take but, when you’re done, you promise you’ll do some other stuff. In reality, it’s much like me promising that I will remember to pick up some milk after work, except computers don’t break promises.

Fulfilling a Promise

1
2
3
4
5

function createAPromise() {
  var dfd = $.Deferred();

  return dfd.promise();
}

As you can see, making a promise is pretty easy. Especially one like this, which doesn’t do anything.

1
2
3
4
5
6
7
8
9

function doIt() {
  var dfd = $.Deferred();

  setTimeout(function() {
    dfd.resolve('hello world');
  }, 5000);

  return dfd.promise();
}

In this example, we can see that we now actually doing something asynchronous, namely waiting for 5s.

The key though is that, once we’re done waiting, we’re going to call dfd.resolve('hello world').

By resolving the deferred object, we are fulfilling the promise we made. We can also use reject to say that we failed to fulfill the promise (computers sometimes forget to pick up milk, after all). Finally, we can also use notify to inform everyone how we are progressing.

Everything is Already a Promise

Anything that is done asychronously in jQuery is already re-written to return a promise.

1
2
3
4
5

function createItem(item, callback) {
  $.post("/api/item", item, function(data) {
    callback(data);
  });
}

I used to write stuff like this a lot.

Now you might be seeing stuff that looks a bit more like this:

1
2
3
4
5

function createItem(item, successCallback, errorCallback) {
  var req = $.post("/api/item", item);
  req.success(successCallback);
  req.error(errorCallback);
}

This is better, it definitely makes it clearer what exactly is going on. The only magic here is what are these success and error things? It turns out that they are actually just deferreds.

Chaining Things Together

1
2
3
4
5
6
7
8
9
10
11
12
13

function displayError(message) {
  console.log("ERROR! " + message);
}

function displayItem(item) {
  console.log(item);
}

function getItem(id) {
  return $.get("/api/item/" + id);
}

getItem(5).then(displayItem, displayError);

In this simple example, we can see that we are going to try and an item with id = 5 and then either log the data (on success) or complain about errors (on error).

Working in Parallel

Let’s say that you have a couple things you want to do at the same time and then maybe when they are both done you want to do something else.

1
2
3
4
5

function getItem(id) {
  return $.get("/api/item/" + id);
}

$.when(getItem(3), getItem(4)).then(handleSuccess, handleFailure);

$.when will create a wrapper deferred that tracks the results of each deferred. If any of the promises fail, it will call reject on the master promise, otherwise it will call resolve.

There are many more functions at your disposal, check out the $.Deferred documentation for more information.

Cooking with Deferreds

Wait

I really like this helper function as it allows you to do deferred waits when you’re doing stuff like long polling. I’m not sure who the credit for this belongs to, it might have been from an SO post or maybe I wrote it myself. If you know, please leave a comment with its source.

1
2
3
4
5
6
7

$.wait = function(time) {
  return $.Deferred(function(dfd) {
    setTimeout(dfd.resolve, time);
  });
};

$.wait(5000).then(doSomething);

Polling for Change

Sometimes you will have to do some polling, waiting to see if a delayed job has completed.

The Echo Nest is a good example of this – uploading a file, waiting for it to be analyzed and then getting that analysis when it’s done.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

var checkTrackStatus = function () {
  return $.get('http://developer.echonest.com/api/v4/track/profile?format=json&bucket=audio_summary',
              { 'api_key': r.apiKey, 'id', r.trackId });
};

var checkTrackAnalyzed = function () {
  return checkTrackStatus()
    .then(function(data) {
      var status = data["response"]["track"]["status"];

      if (_.isEqual(status, "complete"))
        return true;
      else if (_.isEqual(status, "pending"))
        return $.wait(2000).then(checkTrackStatus)
      else
        return false;
    });
};

var attemptAnalyze = function () {
  return uploadTrack().then(checkTrackAnalyzed, displayFailedToUpload);
};

Progress

In some situations, you might have the opportunity to provide feedback to the user on how something is doing. .notify makes this easy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

var displayTrackLoading = function (p) {
  if (p == 100)
    console.log("TRACK LOADED, REMIXIN'");
  else
    console.log("TRACK LOADING: " + p);
};

var loadTrack = function () {
  var dfd = $.Deferred();

  dfd.progress(displayTrackLoading);

  r.remixer.remixTrackById(r.trackId, r.trackUrl, function (t, p) {
    dfd.notify(p);

    if (_.isEqual(t.status, "ok"))
      return dfd.resolve(t);
  });

  return dfd.promise();
};

Asynchronous Native Functions

There is some stuff in Javascript that are done asynchronously as well. We can reduce the number of brain cycles spent on serializing a file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var serializeFile = function(file) {
  var dfd = $.Deferred();

  var reader = new FileReader();
  reader.onloadend = function() {
    dfd.resolve({
      name: file.name,
      data: this.result
    });
  }

  reader.readAsDataURL(raw);

  return dfd.promise();
};

That’s pretty cool, but let’s spice it up and grab a couple files at once.

1
2
3
4
5

$.when(serializeFile(file1), serializeFile(file2))
  .then(function (encodedFile1, encodedFile2) {
    console.log("File 1: " + encodedFile1);
    console.log("File 2: " + encodedFile2);
  });