7. Implementing a data source

In this chapter, I will look at implementing a data source.

Defining a REST-based, chainable API for the data source

Let's start off by writing some tests in order to specify what we want from the data source we will build. It's much easier to understand the code once we have an idea of what the end result should look like.

Given the following fixture:

var fixture = [
  { name: 'a', id: 1, role: 2 },
  { name: 'b', id: 2, role: 4, organization: 1 },
  { name: 'c', id: 3, role: 4, organization: 2 }
var db.user = new DataSource();

... here are tests describing how I'd like the data source to work:

Can load a single item by ID

'can load a single item by ID': function(done) {
  db.user(1, function(user) {
    assert.equal(fixture[0], user);

Can load multiple items by ID

'can load multiple items by ID': function(done) {
  db.user([1, 2, 3], function(users) {
    assert.deepEqual(fixture, users);

Can load items by search query

The data source should support retrieving items by conditions other than IDs. Since the details depend on the backend used, we'll just allow the user to add search terms via an object. The parameters are passed to the backend, which can then implement whatever is appropriate (e.g. SQL query by name) to return the result JSON.

'can load items by search query': function(done) {
  db.user({ name: 'c'}, function(user) {
    assert.deepEqual(fixture[2], user);

Can add more search conditions using and()

We'll also support incrementally defining search parameters:

'should allow for adding more conditions using and()': function(done) {
  db.user({ role: 4 })
    .and({ organization: 1 }, function(users) {
    assert.deepEqual(fixture[1], users);

Implementing the chainable data source API

The full implementation for a chainable data source API is below. It almost fits on one screen.

function Search(options) {
this.uri = options.uri;
this.model = options.model;
this.conditions = [];
Search.prototype.and = function(arg, callback) {
if(!arg) return this;
return this.end(callback);

The data source accepts two parameters:

  • uri, which is a function that returns a URL for a particular id
  • model, an optional parameter; if given, the results will be instances of that model instead of plain Javacript objects (e.g. JSON parsed as a JS object).

The idea behind chainable APIs is that the actual action is delayed until a callback is passed to the API.

conditions is a simple array of all the parameters (model ID's and search parameters) passed to the current data source search.

Also note how all the functions return this. That allows function calls to be written one after another.

Search.prototype.end = function(callback) {
if(!callback) return this;
var self = this,
params = {},
urls = [];
function process(arg) {
if(typeof arg == 'number') {
} else if (Array.isArray(arg)) {
urls = urls.concat(arg.map(function(id) {
return self.uri(id);
} else if(arg === Object(arg)) {
Object.keys(arg).forEach(function(key) {
params[key] = arg[key];
(urls.length == 0) && (urls = [ this.uri() ]);
this._execute(urls, params, callback);

The end() function is where the conditions are processed and stored into url and params. We call process() on each condition in order to extract the information.

process(arg) looks at the type of each argument. If the argument is a number, we assume it's a model ID. If it is an array, then it is considered an array of IDs. Objects are assumed to be search parameters (key: value pairs).

For numbers, we map them to a url by calling this.uri() on them. That parameter is part of the resource definition.

Search.prototype._execute = function(urls,
params, callback) {
var self = this, results = [];
urls.forEach(function(url) {
.end(Client.parse(function(err, data) {
if(err) throw err;
results.push((self.model ?
new self.model(data) : data));
if(results.length == urls.length) {
callback((urls.length == 1 ?
results[0] : results));

This is where the magic happens (not really). We call the HTTP client, passing each URL and set of parameters.

Once we get each result, we store it in the results array. When the results array is full, we call the original callback with the results. If there was only one result, then we just take the first item in the array.

Search.prototype.each = function(callback) {
return this.end(function(results) {
module.exports = function(options) {
return function(arg, callback) {
return new Search(options).and(arg, callback);

If .each(function() { ...}) is called, then we take the callback, and wrap it in a function that iterates over the results array and calls the callback for each result. This requires ES5 (e.g. not IE; since we rely on Array.forEach to exist). For IE compatibility, use underscore or some other shim.

Finally, how do we define a datasource?

We return a function that accepts (arg, callback) and itself returns a new instance of Search. This allows us to define a particular data source and store the configuration in another variable. Every search is a new instance of Search.

See the full usage example at the end of the chapter for details.

Making ajax a bit nicer: Client

Since I wanted the same code to work in Node and in the browser, I added a (chainable) HTTP interface that works both with jQuery and Node.js. Here is a usage example:

  .data({q: 'hello world'})
  .end(function(err, data) {

And the full source code: for jQuery (~40 lines; below) and for Node (~70 lines; w/JSON parsing).

var $ = require('jquery');

function Client(opts) {
  this.opts = opts || {};
  this.opts.dataType || (this.opts.dataType = 'json');
  this.opts.cache = false;

Client.prototype.data = function(data) {
  if(!data || Object.keys(data).length == 0) return this;
  if(this.opts.type == 'GET') {
    this.opts.url += '?'+jQuery.param(data);
  } else {
    this.opts.contentType = 'application/json';
    this.opts.data = JSON.stringify(data);
  return this;

Client.prototype.end = function(callback) {
  this.opts.error = function(j, t, err) {
    callback && callback(err);
  this.opts.success = function(data, t, j) {
    callback && callback(undefined, data);

module.exports.parse = Client.parse = function(callback) {
  return function(err, response) {
    callback && callback(undefined, response);

['get', 'post', 'put', 'delete'].forEach(function(method) {
  module.exports[method] = function(urlStr) {
    return new Client({
      type: method.toUpperCase(), url: urlStr

Putting it all together

Now, that's a fairly useful data source implementation; minimal yet useful. You can certainly reuse it with your framework, since there are no framework dependencies; it's all (ES5) standard Javascript.

Defining a data source

Now, let's create a page that allows us to use the datasource to retrieve data. For example, you might want to use the datasource with a model. You may have noticed that I slipped in support for instantiating models from the result (see the this.model parameter in implementation). This means that we can ask the data source to instantiate objects from a given model constructor by passing the model option:

// Find instances of Todo using Todo.find()
Todo.find = new DataSource({
  uri: function(id) {
    return 'http://localhost:8080/api/todo/'
      + (id ? encodeURIComponent(id) : 'search');
  model: Todo

As you can see, the uri function simply returns the right URL depending on whether the search is about a specific ID or just a search.

The code also demostrates composition over inheritance. The inheritance-based way of setting up this same functionality would be to inherit from another object that has the data source functionality. With composition, we can simply assign the DataSource to any plain old JS object to add the ability to retrieve JSON data by calling a function.

Building a backend.

The server-side for the datasource can be fairly simple: there are two cases - reading a model by ID, and searching for a model by property.

var http = require('http'),
    url = require('url');
var todos = [
      { id: 1, title: 'aa', done: false },
      { id: 2, title: 'bb', done: true },
      { id: 3, title: 'cc', done: false }
    server = http.createServer();

var idRe = new RegExp('^/api/todo/([0-9]+)[^0-9]*$'),
    searchRe = new RegExp('^/api/todo/search.*$');

server.on('request', function(req, res) {
  res.setHeader('content-type', 'application/json');
  if(idRe.test(req.url)) {
    var parts = idRe.exec(req.url);
    // return the ID
    if(todos[parts[1]]) {
  } else if (searchRe.test(req.url)) {
    var data = '';
    req.on('data', function(part) {
      data += part;
    req.on('end', function() {
      var search = undefined;
      try {
        search = JSON.parse(data);
      } catch (error) {}
      res.end(typeof(search) === 'undefined' ? undefined : JSON.stringify(
        // search the todos array by key - value pair
        todos.filter(function(item) {
          return Object.keys(search).every(function(key) {
            return item[key] && item[key] == search[key];
  } else {
    console.log('Unknown', req.url);
comments powered by Disqus