Implementing integration testing for ElasticSearch with Java

ElasticSearch is particularly convenient to implement integration tests for application components that are based on it.

The ElasticSearch Java client provides all the necessary support to embed an ElasticSearch Node and use it to implement integration tests.

Configuring ElasticSearch client

The simplest way to configure the ElasticSearch Java client to interact with the server is to use Maven and define the client as a dependency in the file pom.xml, as described below:

<?xml version="1.0" encoding="UTF-8"?>
<project (...)>

Maven can be used then to generate the configuration for your IDE. For example, for Eclipse, simply execute the following command:

mvn eclipse:eclipse

Now we have a configured project, lets have a look at how to embed an ElasticSearch node within integration tests.

Embedding an ElasticSearch node

The Java client of Elasticsearch allows to embed an engine of this kind by creating an in-memory node. This can be done using the class NodeBuilder, as described below:

Node node = NodeBuilder.nodeBuilder().node();

This code uses the default configuration of Elasticsearch. We can override configuration entities by using the method setSettings of the class NodeBuilder. Following code describes how to enable scripting for the node:

Settings settings = ImmutableSettings.settingsBuilder()
                 .put("script.disable_dynamic", false).build();
Node node = NodeBuilder.nodeBuilder().settings(settings).node();

You can notice that a folder data in the current directory to store data. This folder isnt deleted after the node shuts down. If we want to Clearing data, we can remove it.

Getting a client instance to interact with this node is simple thanks to the method client of the class Node.

Client client = node.client();

To shutdown embedded node, we can simply called the method close of the class Node, as described below:


Managing index structures

An important point when implementing integration tests is to create the data structures before test suites and drop them after. All these operations can be executed using the indices admin from the client API.

Creating indices and types

ElasticSearch provides a create index request to create an index on a node, as described below:

Settings indexSettings = ImmutableSettings.settingsBuilder()
    .put("number_of_shards", 1)
    .put("number_of_replicas", 1)
CreateIndexRequest indexRequest = new CreateIndexRequest(
                               schemaName, indexSettings);

Now the index is created, we can create types in it. A best practice consists in specifying the mapping for types. For more details about mapping, we can refer the post . The put mapping request can be used to create a type and configure its mapping:

String mapping = (...)

PutMappingResponse response = client.admin().indices()

The mapping can be provided as string (from a file for example) and looks like something like that:

    "properties": {
        "personId": { "type":"integer" },
        "age": { "type":"integer" },
        "gender": { "type":"boolean" },
        "phone": { "type":"string" },
        "address": {
            "properties": {
                "street": { "type":"string" },
                "city": { "type":"string" },
                "state": { "type":"string" },
                "zipCode": { "type":"string" },
                "country": { "type":"string" }

Dropping indices

To drop an index, we can simply use the request deleteIndex, as described below:

DeleteIndexRequest indexRequest = new DeleteIndexRequest(indexName);

To implement reproductible integration tests, we need to base on data sets that are initialized before tests and cleared after. Lets tackle now this aspect.

Manipulating dataset for tests

For data sets, we must be initialize them before each test and reset after.

Initializing data sets

The initialization of data sets can leverage the bulk API of ElasticSearch. This allows to send several data to add in a single call. We need to distinguish two cases regarding data inserts. Do we want to provide an identifier for the document to insert or do we let ElasticSearch to generate it? We can support these two use cases with a structure in our data files described below:

0;{"name":"Bread","description":"Whole grain bread", ...}
{"id": "0","name":"Bread","description":"Whole grain bread", ...}

With such structure in data files, we simply check if an identifier is provided. If not we use index request with two parameters (index name and type name) and in the other case, we add a third parameter corresponding to the identifier.

Following code describes how to insert a data set into ElasticSearch from a data file:

StringTokenizer st = new StringTokenizer(bulkData, "\n");
while (st.hasMoreTokens()) {
    String data = st.nextToken();
    StringTokenizer dataTokenizer = new StringTokenizer(data, ";");
    if (dataTokenizer.countTokens()==1) {
        bulkRequest.add(client.prepareIndex(indexName, typeName)
    } else if (dataTokenizer.countTokens()==2) {
        String pk = dataTokenizer.nextToken();
        String content = dataTokenizer.nextToken();
        bulkRequest.add(client.prepareIndex(indexName, typeName, pk)

BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) {

Clearing type data

After each test, we need to remove all the data for a specific type. This can be easily done using a delete by query request. The specific query here must be a match all one. The use of this request is described below:


Implementing integration tests for ElasticSearch

Now we have described all we need to manage the embedded ElasticSearch node and its data, we now focus on integration tests themselves.

Test suite

JUnit leverages Java annotations to define before and after processing for the whole test suite. They are the places to handle an embedded ElasticSearch node. This garantees that this node is available for all integration tests contained in the suite.

Following class describes a test suite that initializes and finalizes an embedded ElasticSearch node that will be used by integration tests.

@SuiteClasses({ ElasticSearchMetadataLoaderTest.class })
public class ElasticSearchSuite {
    private static Node node;
    private static Client client;

    public static void setUpClass() throws Exception {
        // Initialize ElasticSearch

        // Create index and types
        createTypeFromClasspathFile("myindex", "mytype",

        // Initialize suite context
        IntegrationTestContext currentContext
                    = IntegrationTestContext.getCurrentContext();

    public static void tearDownClass() throws Exception {
        // Finalize suite context

        // Drop index

        // Finalize ElasticSearch

To make available the client instance, we can implement a context with static fields to store it.

Integration tests

Integration tests can be simply implemented using the client instance managed by the test suite. We can notice that the component to test must be correctly designed to inject this instance in them.

Following code describes a sample integration test for a component that interacts with ElasticSearch to build :

public class ElasticSearchMetadataLoaderTest {

    public void testLoadMetadata() throws Exception {
        IntegrationTestContext currentContext
               = IntegrationTestContext.getCurrentContext();
        Client client = currentContext.getClient();

        ElasticSearchMetadataLoader loader = new ElasticSearchMetadataLoader();
        loader.setIndexNames(Arrays.asList(new String[] { "odata" }));

        List<TargetEntityType> types = loader.getTypes();
        Assert.assertEquals(2, types.size());

This entry was posted in Client, ElasticSearch and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s