Designing Dynamic Data Retrieval in Client-Server Architectures

A client in a client-server architecture requests data, and the server retrieves and returns it. Dynamic data retrieval lets the client ask for a useful subset, such as records matching a search, sorted by date, or limited to selected fields. The server must answer those requests without allowing response sizes or query costs to grow without bounds.

For endpoints that return collections of records, start with a clear request contract. Define which filters and sort orders clients can use, how many records each response can contain, and how clients request the next portion of results. Then decide how the server handles page boundaries, data changes, caching, and resource limits. These choices make results predictable for clients while keeping backend work manageable.

Define what clients can request

Specify the endpoint’s controls and guarantees before implementing its query logic. A client needs to know which filters and sort orders the server accepts, whether it can request particular fields, and what happens when it omits or exceeds the page-size limit. Field selection, also called projection, means returning only the fields the client asks for. Validate requested fields against an allowed set so clients cannot retrieve fields the API does not expose. Microsoft’s API design guidance describes filtering, sorting, and field selection as ways to reduce response size and tailor results.

Document whether the server rejects an oversized page request or caps it at a maximum. Also explain how a client recognizes the end of the collection. Google’s AIP-158 pagination pattern uses a page-size parameter and a continuation token, a value the client sends to request the next portion of the results. In that pattern, a service can return fewer records than requested, and an empty next-page token signals the end. Other APIs can choose different rules, but the client and server need to follow the same documented contract.

Choose pagination for how clients browse

Pagination divides a collection into smaller responses. Offset pagination identifies a page by how many records to skip and how many to return. It fits interfaces that show page numbers or let users jump to a known page. However, deep offsets can make the database compute rows it will discard, and inserts or deletes between requests can shift which records appear on later pages. PostgreSQL’s documentation notes that skipped rows still need to be computed.

Cursor pagination continues from a position in the ordered results rather than from a row count. In a relational database, keyset pagination is a common cursor approach: the server uses the last record’s sort values to find the next records. An index, a lookup structure that helps the database find records, can speed up the search when it supports the filter and sort. Ed-Fi’s paging guidance describes this approach for large API resources. Cursor pagination suits sequential browsing through large collections, but arbitrary jumps to a page number are less natural. The GraphQL pagination guidance discusses both offset and cursor approaches; the suitable approach depends on the endpoint.

Choose a method based on client navigation needs and measured backend behavior. Offset pagination may suit users who need numbered pages despite deep-page costs; test cursor or keyset pagination when users move forward through a large result set. Data size alone provides no universal threshold for choosing between them.

Make page boundaries predictable

Sort results by a stable, unique order before dividing them into pages. A sort on a field such as creation time can contain ties. Add a unique identifier as a secondary sort key so the server can distinguish records with the same timestamp. PostgreSQL warns that LIMIT and OFFSET without an order that uniquely determines row positions can return inconsistent subsets.

A continuation token should carry or refer to the position in that order. Keep the token opaque, meaning clients treat it as an uninterpreted value and do not build or edit it. Google’s AIP-158 guidance recommends opaque tokens and says the server must authorize every request, including requests that carry a token. A token describes where to continue; it does not prove that its holder can access the records.

Tie the continuation state to the query details, such as its filter and sort order. This prevents a client from using one query’s state to continue a different query. The token format depends on the service: the server might store the state or encode it in a protected token. Microsoft Graph provides a product-specific example in which the client follows the full @odata.nextLink URL rather than reconstructing its paging parameters, as described in its Graph best practices.

Decide how pages handle data changes

A cursor defines where to continue; it does not automatically freeze the collection. In a live traversal, each page reflects data as it exists when the server handles that request. Records added or changed between requests can therefore affect later pages. In a snapshot traversal, all pages refer to a captured state of the data, which requires the server to preserve or identify that state.

Choose and document the behavior that fits the use case. A changing activity feed may be useful as a live view. An export that must represent one consistent result set needs a snapshot mechanism if the backend supports one. Elasticsearch documents this distinction for its search API: an index refresh can change results between search_after requests, while a point-in-time context preserves index state for the traversal. That Elasticsearch mechanism illustrates the issue; other databases and APIs use their own consistency controls.

Also define the end-of-results rule precisely. In DynamoDB, a query response can be limited to 1 MB or less, and the client continues using LastEvaluatedKey. A nonempty key means the request stopped at a page boundary, but does not guarantee that another matching record exists. DynamoDB clients continue until the key is absent or empty, as its pagination documentation specifies. Other APIs should document their own completion signal, especially when filtering can leave a page empty while more results remain.

Set cache and resource limits

Caching affects both response speed and correctness. An HTTP cache stores responses so it can reuse them when the request and freshness rules allow. RFC 9111 describes how caches decide whether to reuse a response. They assess whether it is fresh and whether the request details match; validation also informs the decision. Set freshness rules, which determine how long a cached response counts as current, based on how quickly the underlying data changes. Configure the cache to distinguish requests by any headers that change the response.

Treat personalized responses and sensitive search criteria carefully. Query parameters contribute to a request URI, and URIs can be exposed through logs or other handling. RFC 9110 discusses the risks of placing sensitive information in URIs. For private or authenticated responses, verify the cache directives and intermediary behavior rather than assuming a shared cache will protect user-specific data.

Limit more than the number of records per response. A client can still create expensive work through a complex filter, a costly sort, or repeated requests. Set a maximum page size. Bound query complexity and execution time. Apply rate or cost controls where appropriate. OWASP identifies missing limits on page size, execution time, memory, and operations per request as resource-consumption risks in its API Security Top 10. Choose limits by measuring service load and defining acceptable response sizes and latency; example page-size limits from individual API guidelines are not universal defaults.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com