Normalizing Cursor, Offset, and Link Header Pagination

When a service combines data from several APIs, each API may use a different method to return the next batch of records. Pagination splits a large collection into pages, so clients can retrieve records a page at a time instead of all at once. One provider might use a cursor, another might use an offset, and a third might return a continuation URL in an HTTP Link header.

An integration layer can give clients one consistent contract: Clients request a page size and receive an array of items. They send an opaque continuation token with the next request. The adapter translates that token into the mechanism the upstream expects. This simplifies client code while preserving an important limit: a shared contract does not make every provider offer the same navigation, count, or consistency guarantees.

Give clients one way to continue

A practical contract uses a requested page size and a token that the client sends back to continue:

GET /items?page_size={requested-size}&page_token={opaque-token}

{
  "items": ["..."],
  "next_page_token": "..."
}

The first request omits page_token. The adapter returns the current page in items; when another page is available, it returns a next_page_token. The client uses the token as supplied and does not read or change its contents. When no next page remains, the adapter omits the token or returns null, according to the contract it documents.

This design follows the shape of Google’s AIP-158 pagination guidance, but it is an integration-layer recommendation, not a universal pagination standard. The adapter should document the default and maximum page sizes, and whether it reduces requests that exceed the maximum or rejects them. An upstream may also return fewer items than requested, including an empty page, while still providing a continuation token. Clients should use the token, not the length of items, to determine whether to continue.

Translate each upstream’s continuation mechanism

The adapter should keep its public contract stable and store or encode the upstream state it needs for the next request. The state might be a cursor, an offset, or a provider-supplied URL.

Cursor: continue from an ordered position

A cursor marks a position in an ordered list of results. The upstream returns a cursor or a record that the adapter uses to request the records after that position. For example, Stripe’s documented v1 list methods use object IDs with starting_after or ending_before, and return has_more to indicate whether another page exists. Stripe documents a separate pagination interface for v2, so the v1 behavior should not be assumed for every Stripe API.

A cursor depends on ordering. If the upstream sort does not uniquely order records, records with the same sort value can make page boundaries ambiguous. The JSON:API cursor profile calls for additional constraints when a partial sort does not establish a unique order.

Cursors also do not automatically freeze a collection in time. If records change between requests, later pages may reflect those changes unless the provider supports snapshot-style consistency. The adapter should describe the guarantees its upstream actually provides rather than promising that a cursor produces a point-in-time export.

Offset: skip records, then return a limit

With offset pagination, the adapter counts how many records to skip to reach a position. In OData, $skip sets that count and $top limits how many records the server returns. The adapter can advance the offset after each page, then wrap the next offset in its client-facing token.

The adapter needs a reliable upstream rule for detecting the end. A short page is not a portable signal: some APIs can return fewer items than requested while still indicating that more records remain. Use the provider’s documented end-of-collection signal where one exists; otherwise, document how the adapter determines that no next page remains.

Changes to earlier records can shift offset positions during traversal. An insertion or deletion before the current offset can cause a client to see a record twice or miss one. OData requires stable ordering for requests using $top or $skip when no unique order is specified, but stable ordering alone does not guarantee that the collection stays unchanged across requests.

Link headers: follow the server’s next URL

Some APIs place navigation links in an HTTP Link header. The Web Linking standard, RFC 8288, defines how servers format these links and how clients parse them. It does not define a universal pagination contract. Providers choose the link labels, called relation names, and the pagination behavior.

GitHub, for example, uses relation names such as next, prev, first, and last. The adapter should parse the header and choose the link labeled next. It should not assume a fixed query parameter or split the header on commas. The links can vary by response and endpoint.

Microsoft Graph provides a related pattern in the response body: it returns an @odata.nextLink URL for the client to request. Graph directs clients to follow that URL until it is no longer present. Whether the continuation appears in a header or a response body, the adapter should treat the URL as provider-supplied state instead of rebuilding it from guessed parameters.

Keep the token opaque and bound to the request

A client-facing token should represent enough state for the adapter to continue the same query. That includes the provider cursor, offset, or next URL, along with context such as the resource, filters, sort order, and account or tenant scope. The adapter should reject a token when the client changes parameters that the contract requires to remain the same, or when the token belongs to a different authorization context.

A token must not grant access by itself. The adapter should authorize every request using the caller’s current credentials and permissions. If a token contains provider state, the adapter should protect that state from tampering and avoid exposing sensitive upstream URLs or query details. For link-based pagination, the adapter should also validate that a supplied or stored target belongs to an allowed upstream before fetching it.

Provider continuation state may expire. OData describes advertised URLs as links that can expire, and providers set their own behavior. When a continuation reference is no longer valid, the adapter should return a clear error and explain that the client must begin a new traversal, rather than silently restarting and risking duplicate or missing records.

Document what normalization does not provide

A token-based contract lets clients request pages in order, but it does not automatically let them jump to any page number. Cursor and link-based providers generally direct clients to the next position; they do not necessarily let a client jump to page 37. Offset providers can support direct positioning by skip count, but their limits and performance depend on the provider.

The contract should promise exact totals or a snapshot of the collection only if the adapter can provide them. A total count can be optional, estimated, or unavailable; AIP-158 allows an estimated total when it is identified as an estimate. Likewise, a cursor does not guarantee that all pages reflect one fixed collection state. Expose provider-specific capabilities separately when clients need backward navigation, random access, exact counts, or snapshot exports.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com