Handling OData queries with ElasticSearch

Olingo provides an Java implementation of OData for both client and server sides. Regarding the server side, it provides a frame to handle OData requests, specially the queries described with the OData within the query parameter $filter.

We dont provide here a start guide to implement an OData service with Olingo (it will be the subjet of another post) but focus on the way to handle queries. We first deal with the basic frame in Olingo to implement queries and then how to translate them to ElasticSearch ones. To finish, we also tackle other query parameters to control the entity fields returned ($select) and the data set returned and pagination ($top and $skip).

Handling OData queries in Olingo

Olingo is based on the concept of processor to handle OData requests. The library allows to register a processor class that implements a set of interfaces describing what it can handle. In the following snippet, we create a processor that can handle entity collection, entity collection count and entity requests.

public class ODataProviderEntityProcessor
                       implements EntityCollectionProcessor,
                                  CountEntityCollectionProcessor,
                                  EntityProcessor {
    @Override
    public void readEntityCollection(final ODataRequest request,
                                 ODataResponse response, final UriInfo uriInfo,
                                 final ContentType requestedContentType)
                  throws ODataApplicationException, SerializerException {
        (...)
    }

    @Override
    public void countEntityCollection(ODataRequest request,
                                 ODataResponse response, UriInfo uriInfo)
                    throws ODataApplicationException, SerializerException {
        (...)
    }

    @Override
    public void readEntity(final ODataRequest request, ODataResponse response,
                  final UriInfo uriInfo, final ContentType requestedContentType)
                    throws ODataApplicationException, SerializerException {
        (...)
    }
}

Imagine that we have a entity set called products of type Product. When we access the OData service with the URL http://myservice.org/odata.svc/products, Olingo will route the request to the method readEntityCollection of our processor. The objects provided as parameters will contain of the hints regarding the request and allow to set elements to return within the response.

If we want to use queries, we simply need to leverage the query parameter $filter. So if we want to get all the products with name MyProductName, we can simply use this URL: http://myservice.org/odata.svc/products?$filter=name eq 'MyProductName'. Within the processor the query expression can be reached using the parameter uriInfo, as described below:

@Override
public void readEntityCollection(final ODataRequest request,
                             ODataResponse response, final UriInfo uriInfo,
                             final ContentType requestedContentType)
              throws ODataApplicationException, SerializerException {
    FilterOption filterOption = uriInfo.getFilterOption();
    (...)
}

The query support of Olingo doesnt stop here, since it parses the query string for us and allows to based on the classical pattern Visitor. To implement such processing, we simply need to create a class that implements the interface ExpressionVisitor and uses it on the parsed expression, as described below:

Expression expression = filterOption.getExpression();
QueryBuilder queryBuilder = expression
                .accept(new ElasticSearchExpressionVisitor());

The visitor class contains the methods that can will be called when an element of the parsed expression is encountered. A sample empty implementation is described below with the main methods:

public class ElasticSearchExpressionVisitor implements ExpressionVisitor {
    @Override
    public Object visitBinaryOperator(BinaryOperatorKind operator,
                   Object left, Object right)
                     throws ExpressionVisitException,
                            ODataApplicationException {
        (...)
    }

    @Override
    public Object visitUnaryOperator(UnaryOperatorKind operator, Object operand)
                    throws ExpressionVisitException, ODataApplicationException {
        (...)
    }

    @Override
    public Object visitMethodCall(MethodKind methodCall, List parameters)
                    throws ExpressionVisitException, ODataApplicationException {
        (...)
    }

    @Override
    public Object visitLiteral(String literal)
                    throws ExpressionVisitException, ODataApplicationException {
        (...)
    }

    @Override
    public Object visitMember(UriInfoResource member)
                    throws ExpressionVisitException, ODataApplicationException {
        (...)
    }
}

This approach allows to handle several levels within queries. The returned elements of methods corresponds to the elements that will be passed as parameters to other method calls. Lets take a simple example based on the expression name eq 'MyProductName'. Here are the different method calls:

  • method visitMember. The variable member of type UriInfoResource contains potentially several parts to support something like that field1/subField2. We can here simply extract the string name and returns it.
  • method visitLiteral. The variable literal contains the value 'MyProductName'. Since we are in the case of a string literal, we need to extract the string value MyProductName and returns it. If it was an integer, we could convert it to an integer and return it.
  • method visitBinaryOperator. The variable operator contains the type of operator, BinaryOperatorKind.EQ in our case. The other parameters correspond to the values returned by the previous method.

Here is a sample implementation of methods visitMember and visitLiteral:

@Override
public Object visitLiteral(String literal)
         throws ExpressionVisitException, ODataApplicationException {
    return ODataQueryUtils.getRawValue(literal);
}

@Override
public Object visitMember(UriInfoResource member)
         throws ExpressionVisitException, ODataApplicationException {
    if (member.getUriResourceParts().size() == 1) {
        UriResourcePrimitiveProperty property
                                 = (UriResourcePrimitiveProperty)
                                              member.getUriResourceParts().get(0);
        return property.getProperty().getName();
    } else {
        List<String> propertyNames = new ArrayList<String>();
        for (UriResource property : member.getUriResourceParts()) {
            UriResourceProperty primitiveProperty
                                  = (UriResourceProperty) property;
            propertyNames.add(primitiveProperty.getProperty().getName());
        }
        return propertyNames;
    }
}

Now we have described general principles to handle OData queries within Olingo, we can focus now on how to convert these queries to ElasticSearch ones.

Implementing the interaction with ElasticSearch

Now we have tackle generic concepts and have a look at Olingo classes to implement queries, we will now focus on the ElasticSearch specific stuff. We will use the official Java client to execute such queries from Olingo processors. We leverage the class SearchRequestBuilder and create it using the method prepareSearch of the client. The query can be configured within this request. The corresponding result data will be then convert to OData entities and send back to the client.

The following code shows a sample implementation of such processing within the processor previously described:

@Override
public EntitySet readEntitySet(EdmEntitySet edmEntitySet,
                  FilterOption filterOption, SelectOption selectOption,
                  ExpandOption expandOption, OrderByOption orderByOption,
                  SkipOption skipOption, TopOption topOption) {
    EdmEntityType type = edmEntitySet.getEntityType();
    FullQualifiedName fqName = type.getFullQualifiedName();

    QueryBuilder queryBuilder = createQueryBuilder(
                                  filterOption, expandOption);

    SearchRequestBuilder requestBuilder = client
                          .prepareSearch(fqName.getNamespace())
                          .setTypes(fqName.getName())
                          .setQuery(queryBuilder);
    configureSearchQuery(requestBuilder, selectOption,
                          orderByOption, skipOption, topOption);

    SearchResponse response = requestBuilder.execute().actionGet();

    EntitySet entitySet = new EntitySetImpl();
    SearchHits hits = response.getHits();
    for (SearchHit searchHit : hits) {
        Entity entity = convertHitToEntity(
                            searchHit, type, edmProvider);
        entity.setType(fqName.getName());
        entitySet.getEntities().add(entity);
    }

    return entitySet;
}

We will now describe how to actually create ElasticSearch queries.

Creating ElasticSearch queries from OData requests

With OData, we can get all data for a particular type but also filter them using a query. If we want to get all data, we can use the query . In other case, the ElasticSearch query creation will be a bit more tricky. The latter will be created within a Olingo query expression visitor and can have serveral levels.

The following code describes the entry point method to create the ElasticSearch query:

public QueryBuilder createQueryBuilder(FilterOption filterOption) {
    if (filterOption != null) {
        Expression expression = filterOption.getExpression();
        return expression.accept(
             new ElasticSearchExpressionVisitor());
    } else {
        return QueryBuilders.matchAllQuery();
    }
}

We dont describe here all possible cases but focus on two different operators. The first one is the equality one. Its implementation is pretty straightforward using a match query within the method visitBinaryOperator of our expression visito. We need however be careful to handle the case where the value is null.

@Override
public Object visitBinaryOperator(
                  BinaryOperatorKind operator, Object left, Object right)
                     throws ExpressionVisitException, ODataApplicationException {
    if (BinaryOperatorKind.EQ.equals(operator)) {
        String fieldName = left;
        Object value = right;
        if (value!=null) {
            return QueryBuilders.matchQuery(fieldName, value);
        } else {
            return QueryBuilders.filteredQuery(QueryBuilders
                .matchAllQuery(), FilterBuilders.missingFilter(fieldName));
        }
    }
    (...)
}

We can notice that in the case where the field isnt indexed, a term query would be much relevant.

In the case of an operator, we only one level within the ElasticSearch query. The Olingo approach based on an expression visitor allows to compound more complex queries. We can take the sample of an operator that associates to sub queries, something like with OData query name eq 'MyProductName' and price eq 15. In this case, the following visitor methods will be called successfully:

  • method visitMember with member name.
  • method visitLiteral with value 'MyProductName'.
  • method visitBinaryOperator with operator eq that create the first sub query (query #1).
  • method visitMember with member price.
  • method visitLiteral with value 15.
  • method visitBinaryOperator with operator eq that create the second sub query (query #2).
  • method visitBinaryOperator with operator and. The first parameter corresponds to query #1 and the second to query #2.

Having understand this, we can leverage an ElasticSeach filter to create our composite query within the method visitBinaryOperator, as describe below:

@Override
public Object visitBinaryOperator(
                  BinaryOperatorKind operator, Object left, Object right)
                     throws ExpressionVisitException, ODataApplicationException {
    (...)
    if (BinaryOperatorKind.AND.equals(operator)) {
        return QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),
                                  FilterBuilders.andFilter(
                                      FilterBuilders.queryFilter((QueryBuilder) left),
                                      FilterBuilders.queryFilter((QueryBuilder) right)));
    }
    (...)
}

We describe here how to translate OData queries to ElasticSearch ones by leveraging the expression visitor of Olingo. We took concrete samples of an equals query and of a composite one.

In the next section, we will describe how to take into account nested fields within queries

Handling queries on nested fields

Within our equals operator support, we didnt take into account the fact that OData supports sub fields. As a matter of fact, we can have something like that: details/fullName eq 'My product details'. The field details would be an OData complex field and an ElasticSearch nested field. For such use case, we need to extend our support of the operator to handle both case:

  • normal fields with match or term queries
  • complex fields with nested queries.

The following code describes an adapted version of our method visitBinaryOperator to support this case:

@Override
public Object visitBinaryOperator(BinaryOperatorKind operator,
                  Object left, Object right)
                     throws ExpressionVisitException,
                            ODataApplicationException {
    if (BinaryOperatorKind.EQ.equals(operator)) {
        List<String> fieldNames = getFieldNamesAsList(left);
        if (fieldNames.size() == 1) {
            String fieldName = fieldNames.get(0);
            Object value = right;
            if (value!=null) {
                return QueryBuilders.matchQuery(fieldName, value);
            } else {
                return QueryBuilders.filteredQuery(QueryBuilders
                  .matchAllQuery(), FilterBuilders.missingFilter(fieldName));
            }
        } else if (fieldNames.size() > 1) {
            Object value = right;
            if (value!=null) {
                return QueryBuilders.nestedQuery(getRootFieldName(fieldNames),
                    QueryBuilders.matchQuery(
                              getTargetNestedFieldNames(fieldNames), value));
            } else {
                return QueryBuilders.nestedQuery(getRootFieldName(fieldNames),
                    QueryBuilders.filteredQuery(QueryBuilders
                  .matchAllQuery(), FilterBuilders.missingFilter(
                           getTargetNestedFieldNames(fieldNames))));
            }
        }
        (...)
    }
    (...)
}

The last point will see here consists in the ability to parameterizing a subset of returned data.

Parameterizing the returned data

OData allows to specify a subset of data to return. This obviously applies to queries based on the following query parameters:

  • $select to specify which fields will be included in returned entities
  • $top to specify the maximum number of returned entities
  • $skip to specify the index of the first entity of the returned subset

The two last parameters are particularly convenient to implement data pagination with OData.

Such parameters can be used to parameterized the ElasticSearch search request, as described below:

public void configureSearchQuery(
                       SearchRequestBuilder requestBuilder,
                       SelectOption selectOption, OrderByOption orderByOption,
                       SkipOption skipOption, TopOption topOption) {
    requestBuilder.setSize(1000);

    if (selectOption!=null) {
        for (SelectItem selectItem : selectOption.getSelectItems()) {
            requestBuilder.addField(selectItem.getResourcePath()
                                      .getUriResourceParts().get(0).toString());
        }
    }

    if (topOption!=null) {
        requestBuilder.setSize(topOption.getValue());
    } else {
        requestBuilder.setSize(DEFAULT_QUERY_DATA_SIZE);
    }

    if (skipOption!=null) {
        requestBuilder.setFrom(skipOption.getValue());
    }
}

This entry was posted in ElasticSearch, OData, Olingo, Queries and tagged , , , . Bookmark the permalink.

One Response to Handling OData queries with ElasticSearch

  1. Muthu Kader says:

    Hi… Thanks for your good post.i read your post and it is very useful for me to implement filter in my project But i struck to implement now .I don’t know how to implement method present in above code snippet.You just call convertHitToEntity method in readEntitySet method but you don’t mention implementation of convertHitToEntity method.same thing is for different method you posted here ElasticSearchExpressionVisitor class.Kindly give full source code of the implementation.So that it is helpful for all as.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s