Implementing bulk updates within RESTful services

In a first post, we describe how to design a Web API (i.e. RESTful service). It deals with foundations on the way to implement REST principles. You can have a look at this post at the URL https://templth.wordpress.com/2014/12/15/designing-a-web-api/.

The latter mainly focuses on the way to implement CRUD operations but doesnt tackle bulk updates. The aspect is really important since it allows to minize the interactions with the server since they correspond to network round trips. In the current post, we will tackle with this aspect and the way to implement it independently from the server side implementations.

We use the contact sample all along the post that we will extend to support bulk operations.

Implementing bulk adds

In our previous post, we saw how to add a single contact within a Web API. We will describe here how to extend this feature to support the addition of a set of contacts.

Request

Commonly collection resource already uses the method POST to add a single element to the collection. Thats why we need to implement a mechanism to support several on a same method POST. As a matter of fact having another resource that uses an action name in resource path like /contacts/bulk isnt RESTful, so not the right approach.

Two approaches can be considered to support several actions and contents on a same method POST:

  • Content based. The collection resource accepts both single element and collection of elements for its method. According to the input payload, the processing detect if a single or a bulk adding must be done.
  • Action identifier based. An identifier of the action to handle is provided within the request using for example a custom header. multiple element input

For the first approach, we can have the following request in the case of a single element:

POST /contacts
Content-Type: application/json
{
    "firstName": "my first name",
    "lastName": "my last name",
    (...)
}

And the following in the case of a bulk update:

POST /contacts
Content-Type: application/json
[
    {
        "firstName": "my first name (1)",
        "lastName": "my last name (1)"
        (...)
    },
    {
        "firstName": "my first name (2)",
        "lastName": "my last name (2)"
        (...)
    },
    (...)
]

For the second approach, we can have the following request in the case of a single element:

POST /contacts
x-action: single
Content-Type: application/json
{
    "firstName": "my first name",
    "lastName": "my last name",
    (...)
}

And the following in the case of a bulk update:

POST /contacts
x-action: bulk
[
    {
        "firstName": "my first name (1)",
        "lastName": "my last name (1)"
        (...)
    },
    {
        "firstName": "my first name (2)",
        "lastName": "my last name (2)"
        (...)
    },
    (...)
]

Response

With a single adding, the response is quite straightforward and commonly contains the two things:

  • A status code 201 (Created)
  • An header Location contained the URL of the newly created element

The following snippet describes the content of such response:

201 Created
Location: http://(...)/elements/generated-id

In the context of bulk adding, things need to be a bit adapted. As a matter of the header accepts one value and can be defined once within a response.

That said, since the semantics of a Post method is up to the RESTful service designer, we can leverage the header Link to provide this hint, as described below:

201 Created
Link: <http://(...)/elements/generated-id1>, <http://(...)/elements/generated-id2>

Note about status code 202 that is particularly applicable in such case since bulk adding can be handled asynchronously. In such case, we need do pull à dedicated resource to know the status of this processing. Tell something about the response in such case.

Such approach can only work if we consider that the processing is transactional:

  • Everything works fine and all data are inserted
  • At least one element has validation errors and nothing is added
  • One or more inserts fail and everything is rollbacked.

In this case, if there are some validation errors, the response could be as described below:

422 Unprocessable Entity
Content-type: application/json
[
    {
        "index": 1,
        "messages": [
            {
                "firstName": "The fist name should at least have three characters."
             }
        ]
    },
    {
        "index": 1,
        "messages": [
            {
                "id": "The value of the field it isn't unique."
             }
        ]
    },
]

In the case of insertion errors:

500 Internal Server Error
Content-type: application/json
[
    {
        "index": 1,
        "messages": [
            "The contact can't be added because of the error #22 (description)"
        ]
    },
    (...)
]

In the case of non-transactional processing, we need to have to return the result of bulk adding by element contained in the request payload. The status code of the response will be always 200 and errors if any described in the response paoad, as described below:

200 OK
Content-type: application/json
[
    {
        "index": 1,
        "status": "error",
        "messages": [
            "The contact can't be added because of the error #22 (description)"
        ]
    },
    {
        "index": 2,
        "status": "success",
        "auto-generated-id": "43"
    },
    (...)
]

Another approach can be to replace the whole collection representation of a list resource.

Implementing bulk replaces

The method PUT can be also used on a collection resource. In this case, this means that we want to completely replace the content of the collection associated with the resource with a new one.

Request

In this case, we can simply send the whole collection content, as described below:

PUT /contacts
Content-Type: application/json
[
    {
        "firstName": "my first name (1)",
        "lastName": "my last name (1)"
        (...)
    },
    {
        "firstName": "my first name (2)",
        "lastName": "my last name (2)"
        (...)
    },
    (...)
]

Response

This approach requires to be transactional: either the representation is replaced, either it isnt.

If the request is successful, we can simply have the following response:

204 No Content

In the case of errors, we can have similar contents than with the bulk additions described in a previous section. For example:

422 Unprocessable Entity
Content-type: application/json
[
    {
        "index": 1,
        "messages": [
            {
                "firstName": "The fist name should at least have three characters."
             }
        ]
    },
    {
        "index": 1,
        "messages": [
            {
                "id": "The value of the field it isn't unique."
             }
        ]
    },
]

Before ending this section, we need to deal with the subject of auto-generated identifiers of elements. As a matter of fact, when providing a list content, the Web API might need to do some inserts into the store and have the strategy to auto-generate identifiers. We reach here the limit of the feature since a method PUT needs to be idempotent and with auto-generated identifiers, we wont have the exactly list representation content if we send again the same request. For that reason, we should use another approach for such use case.

HTTP also provides the method PATCH that allows to implement partial updates of the state of a resource.

Implementing bulk updates

In this section, we will tackle the way to implement bulk updates based on the HTTP method PATCH. The latter targets partial updates of resource states and is particularly suitable for bulk updates. We will describe in this section of to use it for such use case?

Request

Using the method PATCH is from far the most convenient way to partially update the collection associated with a resource. We dont have to send the whole collection but only the elements we want to update. In fact, such approach allows to control which updates we want to do (add, update or delete). Whereas we are free to use the format we want to describe the operations to execute on data, some standard formats are however available.

JSON Patch (https://tools.ietf.org/html/rfc6902) can be used in this context. It corresponds to a JSON document structure for expressing a sequence of operations to apply to a JSON document. A similar XML format named XML patch framework: http://tools.ietf.org/html/rfc5261 also exists.

We use below the JSON Patch format in JSON.

The content provided corresponds to an array of JSON structures that can have the attributes:

  • Attribute op that describes the operation on the described element. In our context, values add, remove and update are relevant.
  • Attribute path that allows to identify the element involved within the JSON document. This attribute can be omitted in the case of an add operation.
  • Attribute value that contains the content of the element to use for the operation

The following code describes how to update the list of contacts by adding a new one and removing the one with identifier 1:

PATCH /contacts
[
    {
        "op": "add", "value": {
            "firstName": "my first name",
            "lastName": "my last name"
        }
    },
    {
        "op": "remove", "path": "/contacts/1"
    }
]

Response

Regarding the response, we are exactly in the same scenario than for the element additions. We can consider that the bulk updates are transactional or not. We can notice that the specification JSON Patch describes anything regarding the response content. Its up to you to use the most appropriated format.

In the case of a transactional approach, we have the following scenario:

  • Everything works fine and all data are inserted
  • At least one element has validation errors and nothing is added
  • One or more inserts fail and everything is rollbacked.

In such case, we can have a response content as described below:

200 OK
Content-type: application/json
[
    {
        "index": 1,
        "status": "error",
        "messages": [
            "The contact can't be added because of the error #22 (description)"
        ]
    },
    {
        "index": 2,
        "status": "skipped"
    },
    (...)
]

In the case of a non transactional approach, elements can be added unitarily even if some of them cant be added because of errors. The response content would be

200 OK
Content-type: application/json
[
    {
        "index": 1,
        "status": "error",
        "messages": [
            "The contact can't be added because of the error #22 (description)"
        ]
    },
    {
        "index": 2,
        "status": "success",
        "auto-generated-id": "43"
    },
    (...)
]

Advertisements
This entry was posted in REST, Web API and tagged , , , , , . Bookmark the permalink.

One Response to Implementing bulk updates within RESTful services

  1. Pingback: Implementing bulk updates within RESTful services | Restlet - We Know About APIs

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s