SD notes

Scalability

Beginner · 7 min read · updated 29 October 2026

Quick answer

What is scalability in system design?

Scalability is a system's ability to handle more load — users, requests or data — by adding resources, without a redesign and without performance falling apart. You scale vertically by buying a bigger machine or horizontally by adding more machines, which works best when servers are stateless and the data tier is planned for growth.

On this page

Scale up or scale out
Stateless servers
Latency vs throughput
Bottlenecks move
Comparison
Interview questions
Common questions

Scalability is handling more load by adding resources, without rewriting the system.

Scale up or scale out

Vertical (scale up): a bigger machine — more CPU, RAM, faster disk.
Simple: no code changes, no distributed problems.
A hard ceiling, cost rises steeply near the top, still one point of failure.
Horizontal (scale out): more machines behind a load balancer.
No fixed ceiling; one machine failing loses a fraction of capacity.
Needs stateless servers and a data tier that can be split.
Real systems do both: scale up until it gets expensive, then scale out.

Vertical vs horizontal scaling as traffic grows

Traffic is 300 req/s: the left runs one 500/s machine; the right runs 1 server at 30%.

0:01 / 0:13Step 1 of 5

Transcript · 5 steps

Stateless servers

Stateful: the session lives in one server's memory — clients must stick to it, and a crash logs them out.
Stateless: every request carries what is needed (a token) or the server reads shared state (Redis, a database).
Stateless servers make scale-out trivial: add, remove or replace any server at any time.
The state still exists; it moved to a store built to be shared and replicated.

Stateful vs stateless servers: what a crash costs

Two clusters of two servers. On the left a session lives in the memory of the server that created it; on the right every server reads it from a shared Redis.

0:01 / 0:16Step 1 of 5

Transcript · 5 steps

Latency vs throughput

Latency: time for one request (ms). Watch p95/p99, not just the average.
Throughput: requests completed per second.
Near capacity, requests wait in queues: throughput flattens while latency shoots up.
Rule of thumb: plan for peak load with headroom (many teams target well under full utilisation).

Latency vs throughput: one server's queue as load nears capacity

One server takes 10 ms per request, so it can finish 100 a second. At 20 req/s (20% busy) a request takes 12.5 ms — barely more than the work itself.

0:01 / 0:15Step 1 of 5

Transcript · 5 steps

Bottlenecks move

The bottleneck is whichever component saturates first; nothing else matters until it is fixed.
Stateless app servers scale out easily, so the database is usually next.
Ways to relieve the data tier:
Caching for repeated reads.
Read replicas for read-heavy load.
Sharding when writes or data outgrow one machine.
Queues to absorb bursts and move slow work off the request path.

A bottleneck that moves: scale the app tier and the database is next

1,500 req/s across 2 app servers is 75% of the tier; every request makes one query, so the database is at 50%. No bottleneck yet.

0:01 / 0:15Step 1 of 5

Transcript · 5 steps

Which fix relieves this bottleneck? — Each round names what saturates first. Pick the change that helps most.

Product pages are read thousands of times between edits, and the database is busy. Add a cache for the pages. Repeated reads are served from memory and never reach the database.
Reads are varied and rarely repeat, writes are light, and the database CPU is at its limit. Add read replicas. Replicas each take a share of the reads; a cache helps little when reads rarely repeat.
Writes alone exceed what the largest database machine can handle. Shard the data. Every replica applies every write; only splitting the data spreads them.
Sign-ups are slow because each one sends a welcome email before answering. Put the email on a queue for a worker. The email leaves the request path, so the user gets an answer at once.
App servers are at full CPU while the database is mostly idle. Add stateless app servers behind the load balancer. The app tier is the bottleneck, and stateless servers scale out freely.

Comparison

	Vertical	Horizontal
How	Bigger machine	More machines
Ceiling	The largest machine	Practically none
Failure	One box down = outage	One box down = less capacity
Complexity	Low	Load balancing, shared state, distributed data
Typical fit	Early stage, databases	Stateless web/app tiers

Interview questions

How would you scale a web app from one server? Separate the database, make app servers stateless, add a load balancer and more servers, then cache, add read replicas and shard.
Why not always scale vertically? There is a largest machine, the price climbs steeply, and one machine is a single point of failure.
Where do sessions go in a scaled-out system? In a shared store such as Redis, or in a signed token the client sends with each request.
Your p99 latency spiked but average CPU is low — what do you check? A saturated dependency (database, lock, connection pool), queueing, garbage collection pauses or one slow instance.

Next: Load Balancing · Before this: Introduction to System Design

Common questions

What is the difference between vertical and horizontal scaling?

Vertical scaling (scaling up) moves the work to a bigger machine with more CPU, memory or disk; it is simple but has a hard ceiling and leaves one point of failure. Horizontal scaling (scaling out) adds more machines behind a load balancer; it has no fixed ceiling and survives a machine dying, but needs stateless servers and distributed data.

What is a stateless server?

A stateless server keeps no client data between requests in its own memory. Sessions, carts and uploads live in a shared store such as Redis or a database, so any server can answer any request, servers can be added or removed freely, and a crash loses nothing.

What is the difference between latency and throughput?

Latency is how long one request takes, measured in milliseconds. Throughput is how many requests the system completes per second. They are linked: as throughput approaches capacity, requests queue and latency climbs sharply, so systems run with headroom.

Why do we look at p99 latency instead of the average?

The average hides the slow tail. The 99th percentile is the time within which 99% of requests finish, so it shows what the slowest one in a hundred users experiences — and a page that makes many calls hits some tail request on most loads.

What is a bottleneck in system design?

A bottleneck is the component that reaches its limit first and so caps the whole system's throughput. Scaling anything else does not help. Fixing one usually moves the bottleneck elsewhere — often from the app tier to the database.

Scalability in System Design: Vertical vs Horizontal

Scalability for system design interviews: vertical vs horizontal scaling, stateless servers, latency vs throughput and moving bottlenecks, each animated.

Subject: System Design
Level: Beginner
Reading time: 7 min
Updated: 2026-10-29

What is scalability in system design?

Scalability is handling more load by adding resources, without rewriting the system.

Scale up or scale out

Vertical (scale up): a bigger machine — more CPU, RAM, faster disk.
Simple: no code changes, no distributed problems.
A hard ceiling, cost rises steeply near the top, still one point of failure.
Horizontal (scale out): more machines behind a load balancer.
No fixed ceiling; one machine failing loses a fraction of capacity.
Needs stateless servers and a data tier that can be split.
Real systems do both: scale up until it gets expensive, then scale out.

Vertical vs horizontal scaling as traffic grows.

Traffic is 300 req/s: the left runs one 500/s machine; the right runs 1 server at 30%.
Traffic is 700 req/s: the left upgrades to a 1,000/s machine; the right keeps 1 server at 70%.
Traffic is 1,500 req/s: the left upgrades to a 2,000/s machine; the right adds a server (2 now, 75% each).
Traffic is 2,600 req/s: the left has no bigger machine to buy and runs at 130% — overloaded; the right adds a server (3 now, 87% each).
At 3,600 req/s scaling up has hit its ceiling (180% of the biggest machine), while 4 small servers share the load at 90% each.

Stateless servers

Stateful: the session lives in one server's memory — clients must stick to it, and a crash logs them out.
Stateless: every request carries what is needed (a token) or the server reads shared state (Redis, a database).
Stateless servers make scale-out trivial: add, remove or replace any server at any time.
The state still exists; it moved to a store built to be shared and replicated.

Stateful vs stateless servers: what a crash costs.

Two clusters of two servers. On the left a session lives in the memory of the server that created it; on the right every server reads it from a shared Redis.
Alice logs in (L). The left stores her session in S1's memory and pins her there; the right writes it to Redis.
Alice's next request (R): the left sends her back to S1 where her session is; the right's round robin picks S2, which reads Redis. Both recognise her.
S1 crashes. On the left Alice's session dies with S1's memory; on the right nothing was lost — the session is in Redis.
Her next request (R) goes to S2 on the left, which has never seen her: logged out. The right sends it to S2, which reads Redis: recognised.

Latency vs throughput

Latency: time for one request (ms). Watch p95/p99, not just the average.
Throughput: requests completed per second.
Near capacity, requests wait in queues: throughput flattens while latency shoots up.
Rule of thumb: plan for peak load with headroom (many teams target well under full utilisation).

Latency vs throughput: one server's queue as load nears capacity.

One server takes 10 ms per request, so it can finish 100 a second. At 20 req/s (20% busy) a request takes 12.5 ms — barely more than the work itself.
At 50 req/s (50% busy) throughput still rises with load, but requests now queue: each takes 20 ms, up from 12.5.
At 80 req/s (80% busy) throughput still rises with load, but requests now queue: each takes 50 ms, up from 20.
At 90 req/s (90% busy) throughput still rises with load, but requests now queue: each takes 100 ms, up from 50.
At 95% busy, latency is 200 ms — 20× the work — for only 5 more req/s. Past 100 req/s the queue grows without limit. Run with headroom.

Bottlenecks move

The bottleneck is whichever component saturates first; nothing else matters until it is fixed.
Stateless app servers scale out easily, so the database is usually next.
Ways to relieve the data tier:
Caching for repeated reads.
Read replicas for read-heavy load.
Sharding when writes or data outgrow one machine.
Queues to absorb bursts and move slow work off the request path.

A bottleneck that moves: scale the app tier and the database is next.

1,500 req/s across 2 app servers is 75% of the tier; every request makes one query, so the database is at 50%. No bottleneck yet.
Traffic grows to 2,500 req/s. 2 app servers would need to run at 125%: the app tier is the bottleneck.
Stateless app servers scale out easily: a third brings the tier to 83%. The database, still one machine, is now at 83%.
Traffic reaches 4,000 req/s and the tier scales to 4 servers (100%), but the single database would need 133%. The bottleneck has moved to the data tier.
A cache answers 60% of reads, so the database sees 1,600 queries/s (53%). Replicas and shards are the other ways to relieve it.

Which fix relieves this bottleneck? — Each round names what saturates first. Pick the change that helps most.

Product pages are read thousands of times between edits, and the database is busy. Add a cache for the pages. Repeated reads are served from memory and never reach the database.
Reads are varied and rarely repeat, writes are light, and the database CPU is at its limit. Add read replicas. Replicas each take a share of the reads; a cache helps little when reads rarely repeat.
Writes alone exceed what the largest database machine can handle. Shard the data. Every replica applies every write; only splitting the data spreads them.
Sign-ups are slow because each one sends a welcome email before answering. Put the email on a queue for a worker. The email leaves the request path, so the user gets an answer at once.
App servers are at full CPU while the database is mostly idle. Add stateless app servers behind the load balancer. The app tier is the bottleneck, and stateless servers scale out freely.

Comparison

	Vertical	Horizontal
How	Bigger machine	More machines
Ceiling	The largest machine	Practically none
Failure	One box down = outage	One box down = less capacity
Complexity	Low	Load balancing, shared state, distributed data
Typical fit	Early stage, databases	Stateless web/app tiers

Interview questions

How would you scale a web app from one server? Separate the database, make app servers stateless, add a load balancer and more servers, then cache, add read replicas and shard.
Why not always scale vertically? There is a largest machine, the price climbs steeply, and one machine is a single point of failure.
Where do sessions go in a scaled-out system? In a shared store such as Redis, or in a signed token the client sends with each request.
Your p99 latency spiked but average CPU is low — what do you check? A saturated dependency (database, lock, connection pool), queueing, garbage collection pauses or one slow instance.