Visualize ElasticSearch data with Angular and C3.js

C3 is a visualization library that makes it easy to generate D3-based charts by wrapping the code required to construct the entire chart. Its a perfect candidate to display data stored in ElasticSearch.

In this post, we will describe how to implement Angular directives to make easier to display data returned by ElasticSearch queries.

Initializing a Yeoman project

We don’t describe here how to initialize a Yeoman project but how to configure it to use libraries elasticsearch.js and c3.js. Since they are both present in Bower, we simply need to add them in the file bower.json of our project, as described below:

{
  "name": "elasticsearch-ui",
  "version": "0.0.1",
  "dependencies": {
    "angular": "~1.2.0",
    (...)
    "c3": "~0.3.0",
    "elasticsearch": "~2.4.3"
  },
  "devDependencies": {
    "angular-mocks": "~1.2.0",
    "angular-scenario": "~1.2.0"
  },
  "appPath": "app"
}

Simply run then the command bower install and they are usable within our Angular application (controllers, directives and so on).

Structure of directives

Angular provides a powerful frame to build directives. It allows to associate controllers to directives to implement their processing and also link directives with parent / child relationships. We will use this to implement the directive that build the chart and the ones that provide the data to display in it.

First of all, lets define the different components:

app.controller('d3ctrl', [ '$scope', '$q', '$parse', 'esClient',
                                    function($scope, $q, $parse, esClient) {
  (...)
}])
.directive('c3', function () {
  return {
    restrict: 'AE',
    replace: false,
    templateUrl: 'views/c3-view.html',
    controller: 'd3ctrl',
    transclude: true,
    link: {
      post: function (scope, element, attrs, ctrl) {
        (...)
      }
    }
  };
}])
.directive('data', [ '$parse', function($parse) {
  return {
    restrict: 'AE',
    require: '^c3',
    link: {
      post: function (scope, element, attrs, ctrl) {
        (...)
      }
    }
  };
});

In the snippet above, we define first the controller that will be used that the directives. We inject in it all the elements that we will need to implement processing.

We define then the main directive of the c3 chart. It specifies that we use the previous controller as controller of the directive with the attribute controller.

The last directive corresponds to the inner one used to configure the data retrieval. As the directive will be used within the previous one, we use the attribute require to specify the directive c3 as parent. In this case, the controller of this parent directive is provided as fourth parameter of the function link > post.

Build chart using C3

All processing to build the chart must be implemented in the function link > post. For this, we use the function generate with a configuration object as parameter. The latter contains the element to attach the chart on (attribute bindto), an empty object for data (attribute data) and some configuration for axis (attribute axis).

When the chart is initialized, we can trigger the retrieval of data from the controller with a function called loadData. When gotten the data, we set them within the chart with function load.

The building of the chart is described below:

post: function (scope, element, attrs, ctrl) {
  var type = attrs.type;

  var chartDef = {
    bindto: '#'+attrs.id,
    data: {
      columns: []
    },
    axis: {
      y: {
        label: {
          text: 'Y Label',
          position: 'outer-middle'
        },
        tick: {
          format: d3.format("$,")
        }
      }
    }
  };
  if (type!=null) {
    chartDef['data']['type'] = type;
  }

  var chart = c3.generate(chartDef);

  ctrl.loadData(function(allData) {
    chart.load({
      columns: allData
    });
  });
}

The view associated with the chart directive is simply useful to attach the chart on an HTML element. Nothing complicated here but just dont forget to add the attribute ng-transclude to be able to use inner directives. The content of the view is described below:

<div id="{{id}}" ng-transclude></div>

As you can see, the configuration of axis of our chart is willingly hard-coded. We dont want and its obviously an area for improvement.

Lets focus now on the way to retrieve data.

Retrieve data from ES

As said previously, we want to load from ElasticSearch based on specific queries. We also want to be able to specify the queries at the directive level when defining the chart.

We configure the ElasticSearch queries as content of the directive data. This means that they will be evaluated

post: function (scope, element, attrs, ctrl) {
  var esQuery = $parse(element.text());
  ctrl.addQuery(attrs.name, attrs.path, attrs.index, attrs.type, esQuery);
}

When the directive is evaluated, the ElasticSearch query is retrieved and registered within the controller of the parent directive, i.e. the chart directive. This processing is located within the function post > link of the data directive. Since we use directive transclusion (i.e. inner directives), the inner directives are evaluated before the parent one. When the function link > post is called for the chart directive, all the data directives have already been evaluated and corresponding queries registered in the shared controller.

We need now to update the chart directive to trigger the data loading after the chart initialization. For this, we simple need to call the function of the controller to load data, as described below:

post: function (scope, element, attrs, ctrl) {
  var type = attrs.type;

  var chartDef = {
    (...)
  };

  (...)

  var chart = c3.generate(chartDef);

  ctrl.loadData(function(allData) {
    chart.load({
      columns: allData
    });
  });
}

Lets focus now on the controller to implement the query registration and the data loading. First, the query registration is implemented within the function addQuery that accepts as parameters the following elements:

  • The name of the data corresponding to the query. This will be used in the chart to identify the data.
  • The path that defines where to find the data to display within the returned hits.
  • The name of the index to make the query on.
  • The type in the index to make the query on.
  • The ElasticSearch query. This content will be provided as JSON object the function search of elasticsearch.js.

The following code describes the implementation of the function addQuery:

app.controller('d3ctrl', [ '$scope', '$q', '$parse', 'esClient',
                                 function($scope, $q, $parse, esClient) {
  var ctrl = this,
        queries = ctrl.queries = $scope.queries = {};

  ctrl.addQuery = function(name, path, index, type, esQuery) {
    queries[name] = {
      name: name,
      path: path,
      index: index,
      type: type,
      esQuery: esQuery
    };
  };

  (...)
}

Lets describe now how to load data based on this list of queries. The thing to have in mind here is that we can have several queries configured to display several data set in the chart, for example several lines in it. For this, we need to aggregate all the retrieved data in a common JSON structure and to wait for all request against ElastSearch to be retrieve. For this aspect, we will leverage the promise support of Angular with the object $q and its function all.

The first step is to execute the query and keep the corresponding promise in an array. The following code describes how to implement this:

var promises = [];
var keys = [];
var paths = [];

angular.forEach(queries, function(query, key) {
  var esQuery = query['esQuery'];
  var path = query['path'];
  var index = query['index'];
  var type = query['type'];
  promises.push(esClient.search({
    index: index,
    type: type,
    body: esQuery
  }));
  keys.push(key);
  paths.push(path);
});

You can notice that we keep the keys (name of data) and the path to retrieve data from hits in separate arrays.

Now we have all the promises, we can use the $q.all to be notified when all promises return successfully. When its the case, we will simply build the data structure of the chart, as described below:

$q.all(promises).then(function (results) {
  var allData = [];
  angular.forEach(results, function(result, i) {
    var hits = result.hits.hits;
    var data = [];
    var path = $parse(paths[i]);
    data.push(keys[i]);
    for (var i=0; i<hits.length; i++) {
      var hit = hits[i];
      data.push(path({hit: hit}));
    }
    allData.push(data);
  });
  callback(allData);
});

The data structure corresponds to a two-dimension array. For each line, the first element must be the name of the data. You can notice the use of the Angular object $parse to evaluate the corresponding value according to a defined path.

Below you can find the complete code of the function loadData:

app.controller('d3ctrl', [ '$scope', '$q', '$parse', 'esClient', 
                                  function($scope, $q, $parse, esClient) {
  var ctrl = this,
        queries = ctrl.queries = $scope.queries = {};

  (...)

  ctrl.getData = function(callback) {
    var promises = [];
    var keys = [];
    var paths = [];

    angular.forEach(queries, function(query, key) {
      var esQuery = query['esQuery'];
      var path = query['path'];
      var index = query['index'];
      var type = query['type'];
      promises.push(esClient.search({
        index: index,
        type: type,
        body: esQuery
      }));
      keys.push(key);
      paths.push(path);
    });

    $q.all(promises).then(function (results) {
      var allData = [];
      angular.forEach(results, function(result, i) {
        var hits = result.hits.hits;
        var data = [];
        var path = $parse(paths[i]);
        data.push(keys[i]);
        for (var i=0; i<hits.length; i++) {
          var hit = hits[i];
          data.push(path({hit: hit}));
        }
        allData.push(data);
      });
      callback(allData);
    });
  };
}

Using directives

Now we have all the pieces implemented, lets use them to display our charts based on ElasticSearch data.

Since we enable the element mode within directives, we can define then in the HTML of our views as HTML tags. We have two levels:

  • The tag c3 to define the chart globally.
  • The inner tag data to define how to get data to display in the chart. Several tags can be defined at this level.

The following snippet displays the way to configure a line chart with our directives to display a list of bank operations:

<c3 id="id1">
  <data name="data1" index="dev" type="operations" path="hit._source.amount['eur']">
    {
      query: {
        match: {
          paymentMeans: 'cb'
        }
      }
    }
  </data>
  <data name="data2" index="dev" type="operations" path="hit._source.amount['eur']">
    {
      query: {
        match: {
          paymentMeans: 'cb'
        }
      }
    }
  </data>
</c3>

This will display a chart like as below:

chart

The attribute type of the directive c3 allows to switch to another chart type. By specifying the value pie, we can display a pie chart:

<c3 id="id1" type="pie">
  (...)
</c3>

This will display a chart like as below:

pie

Conclusion

This article describes how to integrate ElasticSearch, Angular and c3.js and leverage the power of each tools to display charts for your data in a simple and declarative way. We can go further by extending the chart directive to allow to configure axis and the data directive to handle errors when retrieving data. An interesting improvment would be also to add some dynamics at the data configuration level. This would make possible to configure the chart based on the view model of your Web UI.

This entry was posted in Angular, Dataviz, ElasticSearch, Web2 and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s