Versatility of pagination strategies

[gcotelli]

Currently, we support 2 strategies for pagination of collections:

• Do not paginate at all
• Page-based pagination

In order to provide more versatility in the pagination strategies a user can use, we can also support:

• KeySet-based pagination
• Cursor-based pagination

A pagination policy now defines:

• Request parameters controlling the pagination
• Hypermedia affordances to provide, both as links or as headers in the response
• Support for using pagination data in query evaluation

Page-based pagination

This is the simplest and most common form of paging. The set is divided into pages.

• Use start and limit query parameters to control pagination
• Provide first, last, next, and previous hypermedia affordances. If needed can provide affordances for jumping to specific pages.

When using SQL in the persistence layer, It can do a query using LIMIT and OFFSET.

Pros:

• You can jump to any particular page
• It allows sending parallel requests with different pages.
• Stateless on the server-side

Cons:

• Bad performance for large OFFSET in SQL.
• It can return repeated or missing data if anything is added/deleted while paginating.

KeySet-based pagination

The API provides a key param that acts as a delimiter of the page. This key param should be the same key of the set sort order. For example, it can be an index (since_id), or timestamp (since_created_at , since_updated_at), etc.

The first request doesn’t contain the delimiter param. The response to this request will contain the value of the key for the last element of the set, so it can be used for subsequent queries.

• Use limit and an API-defined query parameter to control pagination.
• Provide next hypermedia affordance.

When using SQL in the persistence layer, it will end up doing a query including a WHERE and ORDER BY using the key param, and LIMIT for the limit.

Pros:

• The SQL query can be more efficient than using OFFSET
• New records inserted on previous pages won't cause duplicated elements.

Cons:

• It's tied to the sort order.
• There is no way to jump to a specific page. We need to iterate over all the previous pages.
• Missing items if they are added to the previous pages

Cursor-based pagination

Given a set, a cursor will be a piece of data that contains a pointer to an element and the info to get the next/previous elements. The server should return the cursor pointing to the next page in each request. In most cases, the cursor is opaque, so users cannot manipulate it.

The SQL query will depend on the implementation, but it will be similar to the query generated by the KeySet-based Pagination method using a WHERE condition.

Clients should not store the cursor on their side. Google API Documentation suggests adding an expiration date to the token and expiring cursors sent in requests.

• Use limit and after/before to control pagination.
• Provide next and/or previous hypermedia affordances.

Pros:

• If the cursor is opaque, the implementation underneath can change without having to introduce an API change.
• In SQL, for most of the cases, it is much faster than using page since it won’t use OFFSET in the database.
• There is no issue when a record is deleted as opposed to Page-based Pagination

Cons:

• There is no way to skip pages.
• It doesn’t allow sending parallel requests for different batches.
• The implementation is more complex than LIMIT/OFFSET.
• Hard to debug. Given a request, you have to unencode it to see what’s doing.
• Missing items if they are added to the previous pages

Reference

See here some examples of real usage