7. Implementing a data source

In this chapter, I will look at implementing a data source.

Defining a REST-based, chainable API for the data source

Let's start off by writing some tests in order to specify what we want from the data source we will build. It's much easier to understand the code once we have an idea of what the end result should look like.

Given the following fixture:

var fixture = [
  { name: 'a', id: 1, role: 2 },
  { name: 'b', id: 2, role: 4, organization: 1 },
  { name: 'c', id: 3, role: 4, organization: 2 }
];
var db.user = new DataSource();

... here are tests describing how I'd like the data source to work:

Can load a single item by ID

'can load a single item by ID': function(done) {
  db.user(1, function(user) {
    assert.equal(fixture[0], user);
    done();
  });
},

Can load multiple items by ID

'can load multiple items by ID': function(done) {
  db.user([1, 2, 3], function(users) {
    assert.deepEqual(fixture, users);
    done();
  });
},

Can load items by search query

The data source should support retrieving items by conditions other than IDs. Since the details depend on the backend used, we'll just allow the user to add search terms via an object. The parameters are passed to the backend, which can then implement whatever is appropriate (e.g. SQL query by name) to return the result JSON.

'can load items by search query': function(done) {
  db.user({ name: 'c'}, function(user) {
    assert.deepEqual(fixture[2], user);
    done();
  });
},

Can add more search conditions using and()

We'll also support incrementally defining search parameters:

'should allow for adding more conditions using and()': function(done) {
  db.user({ role: 4 })
    .and({ organization: 1 }, function(users) {
    assert.deepEqual(fixture[1], users);
    done();
  });
},

Implementing the chainable data source API

The full implementation for a chainable data source API is below. It almost fits on one screen.

Making ajax a bit nicer: Client

Since I wanted the same code to work in Node and in the browser, I added a (chainable) HTTP interface that works both with jQuery and Node.js. Here is a usage example:

Client
  .get('http://www.google.com/')
  .data({q: 'hello world'})
  .end(function(err, data) {
    console.log(data);
  });

And the full source code: for jQuery (~40 lines; below) and for Node (~70 lines; w/JSON parsing).

var $ = require('jquery');

function Client(opts) {
  this.opts = opts || {};
  this.opts.dataType || (this.opts.dataType = 'json');
  this.opts.cache = false;
};

Client.prototype.data = function(data) {
  if(!data || Object.keys(data).length == 0) return this;
  if(this.opts.type == 'GET') {
    this.opts.url += '?'+jQuery.param(data);
  } else {
    this.opts.contentType = 'application/json';
    this.opts.data = JSON.stringify(data);
  }
  return this;
};

Client.prototype.end = function(callback) {
  this.opts.error = function(j, t, err) {
    callback && callback(err);
  };
  this.opts.success = function(data, t, j) {
    callback && callback(undefined, data);
  };
  $.ajax(this.opts);
};

module.exports.parse = Client.parse = function(callback) {
  return function(err, response) {
    callback && callback(undefined, response);
  };
};

['get', 'post', 'put', 'delete'].forEach(function(method) {
  module.exports[method] = function(urlStr) {
    return new Client({
      type: method.toUpperCase(), url: urlStr
    });
  };
});

Putting it all together

Now, that's a fairly useful data source implementation; minimal yet useful. You can certainly reuse it with your framework, since there are no framework dependencies; it's all (ES5) standard Javascript.

Defining a data source

Now, let's create a page that allows us to use the datasource to retrieve data. For example, you might want to use the datasource with a model. You may have noticed that I slipped in support for instantiating models from the result (see the this.model parameter in implementation). This means that we can ask the data source to instantiate objects from a given model constructor by passing the model option:

// Find instances of Todo using Todo.find()
Todo.find = new DataSource({
  uri: function(id) {
    return 'http://localhost:8080/api/todo/'
      + (id ? encodeURIComponent(id) : 'search');
  },
  model: Todo
});

As you can see, the uri function simply returns the right URL depending on whether the search is about a specific ID or just a search.

The code also demostrates composition over inheritance. The inheritance-based way of setting up this same functionality would be to inherit from another object that has the data source functionality. With composition, we can simply assign the DataSource to any plain old JS object to add the ability to retrieve JSON data by calling a function.

Building a backend.

The server-side for the datasource can be fairly simple: there are two cases - reading a model by ID, and searching for a model by property.

var http = require('http'),
    url = require('url');
var todos = [
      { id: 1, title: 'aa', done: false },
      { id: 2, title: 'bb', done: true },
      { id: 3, title: 'cc', done: false }
    ],
    server = http.createServer();

var idRe = new RegExp('^/api/todo/([0-9]+)[^0-9]*$'),
    searchRe = new RegExp('^/api/todo/search.*$');

server.on('request', function(req, res) {
  res.setHeader('content-type', 'application/json');
  if(idRe.test(req.url)) {
    var parts = idRe.exec(req.url);
    // return the ID
    if(todos[parts[1]]) {
      res.end(JSON.stringify(todos[parts[1]]));
    }
  } else if (searchRe.test(req.url)) {
    var data = '';
    req.on('data', function(part) {
      data += part;
    });
    req.on('end', function() {
      var search = undefined;
      try {
        search = JSON.parse(data);
      } catch (error) {}
      res.end(typeof(search) === 'undefined' ? undefined : JSON.stringify(
        // search the todos array by key - value pair
        todos.filter(function(item) {
          return Object.keys(search).every(function(key) {
            return item[key] && item[key] == search[key];
          });
        })
      ));
    });
  } else {
    console.log('Unknown', req.url);
    res.end();
  }
});