Designing a Web API

A Web API corresponds to an application programming interface that can levarage the Web technologies to execute processing and manipulate data. Modern Web APIs follow the REST principles to be the easiest to use as possible and lightweight. Typically it provides a set of resources that exchange representations.

Far from being an high level guide, we want to provide here a practical and technical guide describing how to design Web APIs and pointing out the pitfalls to avoid when doing this. Its based on our experience and feedbacks when implementing Web APIs.

Before diving into the way to design a Web API, lets recall some common concepts and principles of REST the Web APIs are based on.

Concepts

REST (REpresentational State Transfer) is a style of architecture targetted distributed systems in the context of the Web technologies. Whereas it doesnt depend on the HTTP protocol, the latter is particularly suitable to implement these concepts. We will focus here only on the concepts that will be useful to design a Web API.

The central element of REST is the resource. We can see a resource as a programming element that manages a kind of data and a state and provide processing on this kind. Several resources can manage a same data kind (for example, one for the list of data and one for a particular element).

A resource is accessible on the Web from an URL and we can interact with it using a predefined set of methods to get its state, update it or execute processing. When used with HTTP, we can use all the methods (also called verbs) the protocol provided. We will describe later how to use them correctly following the REST approach.

The formats of exchanged data correspond to representations. The latters provide for the managed data kind. We can notice that a representation defines a structure of data and isnt linked to a particular format such as JSON, XML or YAML. This concept is called variant.

The following figure describes these concepts and the relationships between them:

REST concepts

Defining resources and methods

Now we have seen all these high-level concepts. Great but you probably wonder how to use them to actually build your Web API. Youre right! Its not so simple and you need more details. We will focus on this all along this section.

When designing an API, we first need to define all the entities we want to implement then the resources to manage them. We used to define several resources to handle each entity kind:

  • A resource to get the list of elements of this kind and to add ones
  • One to manage a particular element of this kind
  • Resources that manage no state and provide specific processing.

A resource is attached to an URL in order to make it accessible from the Web. We commonly call this URL resource path. Like all URLs, they are hierarchical and this aspect can help us to define relationships between managed entities.

These are several ways to send parameters to resources:

  • Within the resource path itself. Such parameters are called path variables. They correspond to parameters that identify the resource. We commonly define them when attaching a resource to a path. For example, the identifier of an element: /myentity/{entityid}. Most of REST frameworks know how to extract the parameter based on previous patterns.
  • Within the query string (what is specified after the ? within the URL). Such parameters are called query parameters and provide additional parameters. Be sure not to use such parameters for hints that should take place in the path variables.

Lets take a sample with an entity Product. We define the following resources to manage entities of this kind.

A resource ProductListResource attached on a path /products or /products/. TODO: Note about the trailer /. It provides two methods:

  • Method GET to return a list of products, either the complete list or a filtered list based on criteria provided as query parameters.
  • Method POST to add a new product.

A resource ProductResource attached on a path /products/{productId}. It provides three or four methods:

  • Method GET to return a particular product
  • Method PUT to update the full content of a particular product
  • Method PATCH to update a sub content (set of fields) of a particular product
  • Method DELETE to delete a particular product

We could also add a resource CreateProductFromResource attached on a path /products/{productId}/createfrom with one method:

  • Method POST to create a new product from the current one.

We can see that the resources ProductResource and CreateProductFromResource are under the resource ProductListResource if we see at corresponding path level. This means that processing of the first resource tackles the list of products and the two other ones manage a particular product within the list.

The following figure summarizes these resources and what they do:

Resources for entity management

Regarding identifiers present within resource paths, we need to be careful to distinguish them with the internal identifiers. As a matter of fact, if we leverage the resource path hierarchies, identifiers allow to identify elements in the parent elements. For the sample of products, the value of the path variable productId correspond to the internal product identifier itself since we are located at the root of path. Lets take the sample of a two level hierarchy for a versioned element: /elements/{elementId}/versions/{versionId}. In this case, the value of the path variable versionId identifies the version for a value elementId.

When implementing Web APIs with HTTP, its very important to levergage status codes. They allow to specify if processing works well or if something wrong occurs. It really an anti-pattern to always return a status code 200 and use the response payload to tell this. However in the case of an error and in addition to an error status code, the payload can be used to provide more details and eventually a structured error message.

The (well-known) status codes for successful requests are:

  • 200 (OK): everything works well and the response contains a payload with the result.
  • 201 (Created): everything works well and an element was created
  • 204 (No content): everything works well but the payload of the response is empty

We will describe un more details the status codes in the case of errors later in the section .

Lets focus now on the exchanged data structures.

Defining exchanged data structures

Within REST, data structures are called representations. They correspond to the structure of exchanged data indepently of the used format. The latter is called variant and is closely linked to a media type. The following figure describes the relationships between these three concepts.

REST concepts / representations

We can distinguish two kinds of representations:

  • Structured representations. This corresponds to content that is structured to be understand by both client and server applications. On the Web, they classically use the following formats: JSON, XML and YAML.
  • Raw representations. This corresponds to both text or binary content that arent structured for application communication. They can be zip content, images, and so on.

In the following, we use the JSON format for our samples since its leigthweight, self descriptive and easy to read. Moreover we will base our samples on the GitHub API (see http://developer.github.com).

Regarding representations, there are some aspects we need to take care of:

  • how to define and choose the data contained in representations
  • how to handle collections
  • how to data graph
  • how to handle relationshipts between data within representations

Lets start with the first bullet. Representations correspond to data we exchange with the API consumers. Thats why were free to choose the data they contain for each method of resources. Be careful not to directly exposed your underlying business objects. A good approach consists in defining different entities and populate them according to your needs.

Moreover input and ouput representations dont need to be necessarily the sames. It seems obvious but its not. Take the sample of a resource that manages a particularly element, for example an issue element from the GitHub API. Do we need to use the same representations for the GET method (retrieve issue hints) and the POST method (add issue hints)?. The answer is no. Hints like a field lastUpdated can be deduced on the server-side and not directly updatable by the end user. Following content describes the input representation:

{
  "title": "Found a bug",
  "body": "I'm having a problem with this.",
  "assignee": "octocat",
  "milestone": 1,
  "labels": [
    "Label1",
    "Label2"
  ]
}

and this one the output representation:

{
  "id": 1,
  "url": "https://api.github.com/(...)/issues/1347",
  "html_url": "https://github.com/(...)/issues/1347",
  "number": 1347,
  "state": "open",
  "title": "Found a bug",
  "body": "I'm having a problem with this.",
  "user": {
    "login": "octocat",
    "id": 1,
    (...)
  },
  "assignee": {
    "login": "octocat",
    "id": 1,
    (...)
  },
  "milestone": {
    "url": "https://api.github.com/(...)/milestones/1",
    "number": 1,
    "state": "open",
    (...)
  },
  "created_at": "2011-04-22T13:33:48Z",
  "updated_at": "2011-04-22T13:33:48Z",
  (...)
}

The last bullet is the way to handle collections. You should simply say: use arrays. Yes, youre of course right, but We could also have a need to have additionnal hints about the collections: number of elements, current page of data, number of elements within the current page, and so on. As you can see, its tight to pagination and its up to you to choose. Following sample describes a representation containing a collection of issues using the array approach:

[
  {
    "id": 1,
    "url": "https://api.github.com/(...)/issues/1347",
    "html_url": "https://github.com/(...)/issues/1347",
    "number": 1347,
    "state": "open",
    "title": "Found a bug",
    "body": "I'm having a problem with this.",
    "user": {
      "login": "octocat",
      "id": 1,
      (...)
    },
    (...)
  },
  (...)
  {
    "id": 5,
    "url": "https://api.github.com/(...)/issues/1349",
    "html_url": "https://github.com/(...)/issues/1349",
    "number": 1349,
    "state": "open",
    "title": "Found another bug",
    "body": "I'm having another problem.",
    "user": {
      "login": "octocat",
      "id": 1,
      (...)
    },
    (...)
  }
]

We could adapt this representation to contain the pagination hints, as described below:

{
  "number": 23,
  "page_number": 2
  "number_per_page": 10,
  "list": [
    {
      "id": 1,
      "url": "https://api.github.com/(...)/issues/1347",
      "html_url": "https://github.com/(...)/issues/1347",
      "number": 1347,
      "state": "open",
      "title": "Found a bug",
      "body": "I'm having a problem with this.",
      "user": {
        "login": "octocat",
        "id": 1,
        (...)
      },
      (...)
    },
    (...)
    {
      "id": 5,
      "url": "https://api.github.com/(...)/issues/1349",
      "html_url": "https://github.com/(...)/issues/1349",
      "number": 1349,
      "state": "open",
      "title": "Found another bug",
      "body": "I'm having another problem.",
      "user": {
        "login": "octocat",
        "id": 1,
        (...)
      },
      (...)
    }
  ]
}

As you can see in the samples above, the data can contain several kinds of data and really correspond to data graphs with specific depths. The following lists these different kinds:

  • Fields with primitive types. The value of the field corresponds to a primitive type.
  • Fields with complex types. This corresponds to fields with inner fields
  • References to other elements. This corresponds to relationships between elements. They can be displayed as a reference or embedded to simulate a complext type but the element has its own lifecycle.

For each kinds, we can have single or multiple cardinality with null values allowed or not. Such rules are defined by the back-end itself but need to be check when adding or updating data.

Following code describes this aspect:

{
  "id": 1,
  "number": 1347,
  "state": "open",
  "title": "Found a bug",
  "body": "I'm having a problem with this.",
  "user": {
    "login": "octocat",
    "id": 1,
    (...)
  },
  "assignee": {
    "login": "octocat",
    "id": 1,
    (...)
  },
  "labels-ref": [
    "https://api.github.com/(...)/labels/bug",
    "https://api.github.com/(...)/labels/enhancement"
  ],
  (or)
  "labels": [
    {
      "url": "https://api.github.com/(...)/labels/bug",
      "name": "bug",
      "color": "f29513"
    }
  ],
  "milestone-ref": "url": "https://api.github.com/(...)/milestones/1",
  (or)
  "milestone-ref": {
    "url": "https://api.github.com/(...)/milestones/1",
    "number": 1,
    "state": "open",
    (...)
  }
}

Most of time the structure of the returned data hard-coded for Web APIs. Moreover some Web APIs allow to specify which data we want in the returned data graph. We can distinguish two approaches:

  • Specify the fields we want to get in the request response. This can be done using a query parameter containing the list of field names. The OData specification uses the query parameter $select for this.
  • Specify the sub element to load and integrate in the request response. This can be done either using headers (like with Parse – https://parse.com/) or query parameters. The OData specification uses the query parameter $expand for this.

The following URL describes how to use this feature with OData:

http://services.odata.org/V4/Northwind/Northwind.svc/Customers?$select=CustomerID,Orders&expand=Orders

To end this section, we will give some best practices and errors to avoid regarding representations:

  • Be careful not to duplicate data. As matter of fact, some data can be deduced from the path variables at the resource paths or from the security context. In general, we dont need to put it again within the representations. For example, its not necessary to send the identifier of an element if its already present as path variable.
  • Its not also necessary to put the parent element (in the case of parent child relationships) if we use resource paths provided hierarchical levels. For example for resource path like this: /elements/{elementid}/subelements/{subelementid}.
  • We should always refer to hints defined in the resource paths. It mainly applies to element identifiers to prevent from identifier usurpation and also to parent element references to prevent from re-attaching element to another one if not allowed

The two last bullets can correspond to potential data security issues of your API and should be handled with care.

Filtering data

Its not a good idea to return all data of a collection in a one shot. It can correspond to a lot of data and its not necessary (explopitable) by the end user. Paging corresponds to a good approach in such use cases. Paging can be controlled by a set of query parameters:

  • The size of elements in a page. The API should returned the first page when not specified.
  • The number of the current page. The API should support a default value when not specified.

Following request describes the way to provide these parameters to the API using URL:

https://api.github.com/(...)/issues?page=2&page_size=10

Regarding pagination, Github provides an interesting approach based on the header Link that provides the previous and next pages regarding pagination. The representations are in this case simple arrays of data. Following code describes the content of this header:

(...)
Link: <https://api.github.com/(...)/issues?page=2>; rel="next",
<https://api.github.com/(...)/issues?page=5>; rel="last"
(...)

Querying (also called filtering) is an important part of Web APIs since we can only be interested in a subset of data. Two approaches can be implemented for this aspect:

  • Provide query parameters to filter the list of data to be returned. This typically applies to an HTTP method GET.
  • Provide a query parameter to identify the subset of data to be returned. This typically applies to an HTTP method GET.
  • Provide a query as request payload to identify the subset of data to be returned. This typically applies to an HTTP method POST.
  • Leverage the resource paths.
  • Leverage the navigation links within data.

The first one simply gives parameters to filter the returned data, as described below:

http://(...)/issues?state=open&label=bug

The second one simply use a query parameter that contains an expression correspond to the query string itself. The expression syntax is depend on the Web API but commonly applies to the element targetted by the resource path. For example, we can call it query and use the following expression:

http://(...)/issues?query=name%20eq%20'test'

The third differs from the fist one in the way that the payload of the request is used to specify the query content. The REST API of ElasticSearch works like this, as described below:

{
  "match" : {
    "message" : {
      "query" : "this is a test",
      "operator" : "and"
    }
  }
}

The two last ones are more limited but can be very powerful to provide a simple to apply criterias directly within the resource paths. The Github API uses this approach to get list of issues:

  • Resource path /issues lists all issues across all the authenticated user’s visible repositories including owned repositories, member repositories, and organization repositories
  • Resource path /user/issues lists all issues across owned and member repositories for the authenticated user
  • Resource path /orgs/:org/issues lists all issues for a given organization for the authenticated user
  • Resource path /repos/:owner/:repo/issues lists all issues for a repository

We can use the resource paths to navigate through the returned data graph. This corresponds to implicite queries since the inclusion within a parent element automatically handle the filtering. With the issue sample, we could have something like that:

http://(...)/user/issues/1/user
http://(...)/user/issues/1/labels
http://(...)/user/issues/1/labels
http://(...)/user/issues/1/milestone/creator

For a reminder, here is the content of a Github issue:

{
  "id": 1,
  "url": "https://api.github.com/(...)/issues/1347",
  "html_url": "https://github.com/(...)/issues/1347",
  "number": 1347,
  "state": "open",
  "title": "Found a bug",
  "body": "I'm having a problem with this.",
  "user": {
    "login": "octocat",
    "id": 1,
    (...)
  },
  "labels": [
    {
      "url": "https://api.github.com/(...)/labels/bug",
      "name": "bug",
      "color": "f29513"
    }
  ],
  "assignee": {
    "login": "octocat",
    "id": 1,
    (...)
  },
  "milestone": {
    "url": "https://api.github.com/(...)/milestones/1",
    "number": 1,
    "state": "open",
    "title": "v1.0",
    "description": "",
    "creator": {
      "login": "octocat",
      "id": 1,
      (...)
    },
    "open_issues": 4,
    (...)
  },
  "comments": 0,
  "pull_request": {
    (...)
  },
  "closed_at": null,
  "created_at": "2011-04-22T13:33:48Z",
  "updated_at": "2011-04-22T13:33:48Z",
  "closed_by": {
    "login": "octocat",
    "id": 1,
    (...)
  }
}

We can notice that we can combine approaches based the resource paths with the one using a query parameter. Here is a sample from the GitHub API:

http://(...)/user/issues?state=open&label=bug

As we saw, some APIs always return the same data structure (its the case of the Github one) but other allow to select the exact data to get. This can be efficient when amount of received data can be huge (list of elements with a lot of fields). Such Web APIs leverage headers or query parameters to select the fields to be present in the response representations.

Handling list

We need to take care of the way to handle list efficiently. As a matter of fact, we probably dont want to send all the list contents to update them. We should create a resource with a set of methods to manage the list itself:

  • An HTTP method POST to add an element to the list
  • An HTTP method DELETE to remove an element from the list

The following figure summarizes the aspect:

Resources for entity sublist management

Another aspect of the way to handle sublists is: should we must embed the complete element contents in the sublist or only the references to them? In fact, it depends on what we want to do but its clear that only the identifiers / references of sublist elements are required when managing them using the approach described above.

This resource has a path under the element resource. For example: /products/{productId}/categories or /products/{productId}/categories/.

Handling errors

The status code is the way to specific to the end user if processing succeeds or not. Always returning a status code 200 whatever happens and describing the error in the response content corresponds to an anti-pattern.

We must leverage HTTP status codes to notify end users of errors when calling a resource. The response content can provide in addition more details of the error(s). This content can be text or structured (lime JSON).

Never forget to add this since this is a key part that will make your APIs more robust. Feeling free to leverage supports provided by the technologies you use to implement server-side processing.

You can find out below the main status codes that are commonly used to notify errors.

The first family (status code 4xx) corresponds to errors that can occur if the end user provides wrong data or executes methods in a wrong context:

  • 400: the most generic one telling that the request sent isnt correct. An error message should neccessary be provided to give more details.
  • 401: this occurs when we arent authenticated unauthorized
  • 403: this occurs when we havent enough to access a resource
  • 404: probably one of the most well-known error. It occurs when trying to use à data that doesnt exist. It can be used for different HTTP méthode not only the GET one. For example when trying to update or delete data that dont exist.
  • 405: this occurs when trying to use an HTTP method that isnt supported for a resource.
  • 406: the server resource cant return the expect content specified in the header Accept
  • 409: conflict for optimistic locking
  • 412: this occurs when we arent in the right context to execute processing.
  • 415: this occurs when the structure of the sent values isnt correct (for example a JSON content that isnt well-formed).
  • 422: some REST dont like to use this code since it doesnt comme from the HTTP specification itself but from the WebDAV one. It tells that the validation of the data provided by the end user areat correct and cant be processing. It doesnt correspond to format errors but to value ones. However its widely used in Web APIs and describes well the problem.

The second familly (status code 5xx) corresponds to errors that can occur on the server side. We expect not to occur but we need to take into account within the clients that use the Web API. Here is a list of the common ones:

  • 500: something wrong within the server side processing when trying to execute the request.
  • 501: the corresponding server side processing isnt implemented for the sent request.
  • 503: the Web API isnt current available at the moment.

We describe here the foundations of designing a Web API. We will see in a next post some advanced aspects such as security and content negociation. Stay tuned!

Advertisements
This entry was posted in REST, Web API and tagged , , , , . Bookmark the permalink.

2 Responses to Designing a Web API

  1. Pingback: Can you give me some clarification about the concept of representation in REST service? | 我爱源码网

  2. Pingback: Can you give me some clarification about the concept of “representation” in a REST service? - Technology

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s