12 System Design Concepts Every Developer Should Know

When you build software that serves real users, system design means deciding how its parts store data, handle requests, and respond when traffic or failures increase. Your choices determine how quickly the software responds, how much traffic it handles, and what users experience when a service it depends on fails.

The 12 concepts below help you make those decisions based on user needs and measured behavior. They cover requirements, capacity, data, failure handling, recovery, and change. Each involves trade-offs: for example, a cache stores reusable data so the service can reduce database work, but it can return stale data. Repeating a request can recover from a brief fault, but it adds load to a struggling service.

Start with demand and request paths

1. Define requirements with SLIs and SLOs

Start by deciding what users need from the service. A service-level indicator (SLI) measures a user-visible behavior, such as the percentage of successful requests or how long requests take. A service-level objective (SLO) sets a target for that measurement. A service-level agreement (SLA) is a broader commitment that may carry consequences if the service misses its target. Google’s SRE guidance explains how teams use SLIs and SLOs to define service behavior.

Choose measurements that reflect actual user experience. If a service responds quickly to most requests but slowly to a small share, its average latency can hide the problem. A latency percentile shows the response time below which a given share of requests fall, making that slow tail visible. Set targets for the measures users care about before choosing infrastructure.

2. Find capacity limits before adding resources

A service’s capacity depends on its bottlenecks. A database that is saturated, a lock that makes requests wait, or a slow external dependency can limit the whole request path even when application servers have spare capacity.

Test with traffic that resembles real usage, then measure throughput, latency, and resource saturation, or how close resources are to their limits. Microsoft describes scalability as the relationship between added resources and throughput gained; added machines do not guarantee a proportional increase when a bottleneck remains. Google’s four monitoring signals - latency, traffic, errors, and saturation - offer a practical starting point for seeing how a service behaves under load (Google SRE monitoring guidance).

3. Scale out with replaceable instances

Vertical scaling gives one machine more resources. Horizontal scaling adds machines or service instances. Horizontal scaling works most easily when each service instance is stateless, meaning it does not rely on local memory or disk to hold data that must persist between requests. Store that state in a suitable shared data store so another instance can handle the next request.

This approach makes instances easier to add, remove, or replace, but the shared database, network, or another dependency can still limit capacity. Microsoft’s scale-out guidance describes why bottlenecks and synchronization points can prevent added resources from producing proportional gains. For a small service, adding resources to one machine may be simpler; measure the actual limit before changing its architecture.

4. Use load balancing to distribute traffic

A load balancer directs requests across multiple servers or service instances. Health checks let it stop sending traffic to an instance that fails a specified test. This improves how the service uses its available capacity and can keep requests flowing when an instance fails.

Choose health checks that reflect whether an instance can handle work, and consider what happens when a whole zone or region becomes unavailable. A load balancer cannot create capacity: if every instance depends on the same overloaded database, spreading requests across more instances will not remove that bottleneck. Google Cloud’s guidance on horizontal scalability also describes how stateless services make instances easier to add and remove.

Control data access and background work

5. Cache data that users request repeatedly

A cache stores reusable data near the service handling a request, so the service can return it without querying a slower database or calling another service again. A common approach, called cache-aside, checks the cache first; on a miss, the service reads from the primary store and fills the cache. When data changes, the service must refresh or invalidate the relevant cached value.

Caching reduces repeated work, but it introduces freshness decisions. A cached value can be out of date, and many simultaneous requests for an expired popular value can overload the data store while they refill the cache. Decide how long each value can be stale, how updates affect cached values, and what the service does if the cache is unavailable. Redis’s cache-aside documentation describes these implementation concerns.

6. Partition data around how you query it

Partitioning divides a large dataset into smaller pieces. Sharding generally means distributing those pieces across different machines or nodes. Partitioning helps when a query can target only the relevant piece. For example, a query for recent records can benefit from time-based partitions if it filters by date.

The partition key, the value the database uses to assign each row to a partition, matters. If most requests target one partition, that partition can become a hotspot while others remain underused. PostgreSQL documents both partitioning’s potential benefits and the need for queries to support partition pruning, which lets the database skip irrelevant partitions (PostgreSQL table partitioning). Judge whether partitioning or sharding will help by checking actual query patterns and growth constraints; table size alone is insufficient.

7. Choose replication and consistency guarantees deliberately

Replication keeps copies of data on multiple servers. The primary server accepts writes, and replicas receive copies. Replication can support failover or distribute reads, but replicas may lag behind the primary. With synchronous replication, a write waits for confirmation from the required replica or replicas. With asynchronous replication, the primary can acknowledge a write before every replica receives it, so a failover can lose recent writes.

The CAP theorem describes a trade-off during a network partition, when parts of a distributed service cannot communicate. A service that needs a consistent view of completed operations must delay or reject some requests during the partition; a service that responds to every request can return data that does not reflect the latest completed write. The theorem has specific assumptions and does not mean every database simply “picks two.” The original CAP paper sets out its scope.

Decide what stale reads or unavailable writes mean for the product. A profile page may tolerate a brief delay before showing an update. An operation that changes a balance or reserves inventory may require stronger coordination. PostgreSQL’s high-availability documentation describes the latency and data-loss trade-offs associated with its replication options.

8. Put non-urgent work on a queue

A queue holds work for a worker to process later. A service can acknowledge a request after accepting a task, then let a worker handle tasks such as sending email or generating a report. Queues absorb short-lived bursts and let the services that add tasks and the workers that process them run at different rates.

A queued task has not necessarily finished when the service accepts it. The product needs a way to show whether processing is pending, complete, or failed. Queue depth also needs attention: if tasks arrive faster than workers finish them, the backlog grows. When several workers process tasks at once, tasks can finish in a different order from the order they arrived. Some queue systems also redeliver messages, so workers should handle duplicates safely. Microsoft’s queue-based load-leveling pattern covers backlog monitoring, consumer rates, duplicate handling, and persistent failures.

Design for failure and change

9. Bound retries and make repeated requests safe

A timeout limits how long a caller waits for another service. A retry repeats a request after a failure that may be temporary. An idempotent operation has the same intended effect when repeated as when performed once. Idempotency matters when a caller times out without knowing whether the first request completed; a repeated payment or order request should not accidentally create a second charge or order.

Retries also create more traffic, which can worsen an overloaded dependency’s condition. Retry only failures that are appropriate to repeat, limit the number of attempts, and use backoff, which increases the delay between attempts, with jitter, a random delay that helps prevent many clients from retrying at once. A circuit breaker stops calls to a failing dependency for a period so the caller can fail quickly or use a fallback. Google’s production service guidance discusses how retries can amplify failures. AWS also explains how to design safe idempotent APIs.

10. Plan failover and recovery around business needs

Redundancy provides an alternative resource or path when a component fails. Failover moves work to that alternative. Two recovery measures make the business impact explicit: recovery time objective (RTO) is the acceptable time to restore service, and recovery point objective (RPO) is the acceptable amount of recent data loss.

A second server or region does not make recovery automatic. Decide how failover starts and how clients reconnect. Also define how operators restore data from backups. Then test those steps. Synchronous replication can reduce the risk of losing acknowledged writes, but it adds coordination and can increase latency. Asynchronous replication avoids waiting for replicas on each write, but it can leave recent data behind during failover. Microsoft’s design principles call for aligning recovery objectives with business needs.

11. Use observability to find user-facing failures

Observability means understanding a service’s behavior from the information it produces. Metrics reveal quantities and trends; logs capture events. Traces follow a request across services. Together, they help an operator see where a request failed or slowed down.

Track latency and traffic alongside errors and saturation, then connect alerts to user impact or an SLO. Include request or trace identifiers in relevant records so an operator can follow a slow request across service boundaries. An alert should point to an action, such as checking a failing dependency or reducing load, rather than paging on every unusual metric. Google’s monitoring guidance describes these signals and emphasizes actionable alerts.

12. Keep components loosely coupled so they can evolve

Loose coupling means components rely on clear interfaces rather than each other’s internal details. A versioned API gives callers a stable contract while the service changes how it works internally. Asynchronous events can also let one component publish work without requiring another component to finish it during the same request.

Decoupling adds responsibilities: callers and services need compatible interfaces, clear failure behavior, and ownership for changes. Separate deployments do not guarantee loose coupling. If two services must coordinate on every request or share tightly synchronized data, a change in one can still disrupt the other. Microsoft recommends well-defined, versioned APIs to support independent evolution.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com