AI InterviewSearch…Search problems, topics, or pages…Ctrl KGet started
Skip to content
SD notes

System Design · Note 3 of 22

Load Balancing

Beginner · 6 min read · updated 29 October 2026

Quick answer

What is load balancing in system design?

Load balancing spreads incoming requests across several servers so no single server is overloaded and a failed server stops receiving traffic. A load balancer sits between clients and servers, picks a server for each request with an algorithm such as round robin, least connections or IP hash, and runs health checks to route around failures.

On this page
  1. What a load balancer does
  2. Algorithms
  3. Health checks
  4. L4 vs L7
  5. Interview questions
  6. Common questions

A load balancer is the front door of a scaled system: clients talk to one address, and it decides which server answers each request.

What a load balancer does

  • Spreads load across a pool of identical servers.
  • Hides failures: health checks take a broken server out of rotation.
  • Enables horizontal scaling: add a server to the pool, no client changes.
  • Optional extras (L7): TLS termination, path-based routing, compression.

Algorithms

Round robin

  • Servers take requests in a fixed order: S1, S2, S3, S1 …
  • Zero state, zero cost — but blind to how busy a server is.
  • Weighted round robin: a bigger server gets more turns.
Round robin: each server takes the next request in turn

Requests arrive at the load balancer, which keeps a pointer to the next server in a fixed order: S1, S2, S3, then S1 again.

0:01 / 0:23Step 1 of 8
Transcript · 8 steps

Least connections

  • Counts open connections per server; the next request goes to the fewest.
  • Best when request durations vary (uploads, websockets, slow queries).
Least connections: the next request goes to the server with the fewest open

Each request holds a connection for a different time (the number on it, in seconds). The balancer counts every server's open connections.

0:01 / 0:22Step 1 of 8
Transcript · 8 steps

IP hash (sticky routing)

  • server = hash(client IP) mod N — the same client lands on the same server.
  • Useful when session state lives in server memory.
  • Changing N remaps most clients — see consistent hashing.
IP hash: the same client always reaches the same server

Some apps keep a user's session in one server's memory. IP hash pins each client to a server by hashing its address, so no table is needed.

0:01 / 0:19Step 1 of 7
Transcript · 7 steps

Health checks

  • The balancer polls an endpoint such as /health every few seconds.
  • Fall threshold: out of rotation after N misses in a row (not one blip).
  • Rise threshold: back in after N passes in a row.
  • Users never see the dead server's errors.
Health checks: a failing server is taken out, then put back

Every few seconds the balancer calls each server's health endpoint. All three answer 200, so all three stay in rotation.

0:01 / 0:15Step 1 of 6
Transcript · 6 steps

L4 vs L7

L4 (transport)L7 (application)
ReadsIP, portURL, headers, cookies
SpeedFasterSlower per request
RoutingPer connectionPer request, by path or header
TLSPasses throughCan terminate
ExamplesAWS NLB, LVSNginx, HAProxy, AWS ALB

Interview questions

  • Where does a load balancer sit? Between clients and app servers, and often again between app servers and internal services.
  • Which algorithm would you pick? Round robin for short, uniform requests; least connections when durations vary; hashing when a client must stick to a server.
  • How do you avoid the balancer being a single point of failure? Run two or more (active-passive with a floating IP, or several behind DNS).
  • Why keep servers stateless? Any server can take any request, so the balancer is free to choose and a lost server loses no sessions.

Next: Caching · Before this: Scalability

Common questions

Why do we need a load balancer?

One server has a ceiling on requests per second and is a single point of failure. A load balancer lets you add servers behind one address, spreads the work across them and stops sending traffic to a server that fails its health checks.

What is the difference between round robin and least connections?

Round robin hands requests out in a fixed order and ignores how busy each server is. Least connections counts each server's open connections and picks the fewest, so it copes better when some requests take much longer than others.

What is the difference between an L4 and an L7 load balancer?

An L4 balancer routes on IP addresses and ports without reading the request, so it is fast and protocol-agnostic. An L7 balancer reads HTTP — paths, headers, cookies — so it can route /api and /images to different pools, terminate TLS and rewrite requests, at more cost per request.

Is the load balancer itself a single point of failure?

It would be if there were only one. In practice balancers run as a pair or a fleet — active-passive with a floating IP, or several behind DNS — and managed balancers such as AWS ELB are redundant by design.

What are sticky sessions?

Sticky sessions send every request from one user to the same server, by a cookie or by hashing the client's IP. They help when sessions live in a server's memory, but they unbalance load and lose sessions when that server goes away; storing sessions in Redis avoids both.

AI InterviewGet started

Load Balancing in System Design, Animated

Load balancing for system design interviews: round robin, least connections, IP hash and health checks, each shown as a looping animation with key points.

  • Subject: System Design
  • Level: Beginner
  • Reading time: 6 min
  • Updated: 2026-10-29

What is load balancing in system design?

Load balancing spreads incoming requests across several servers so no single server is overloaded and a failed server stops receiving traffic. A load balancer sits between clients and servers, picks a server for each request with an algorithm such as round robin, least connections or IP hash, and runs health checks to route around failures.

A load balancer is the front door of a scaled system: clients talk to one address, and it decides which server answers each request.

What a load balancer does

  • Spreads load across a pool of identical servers.
  • Hides failures: health checks take a broken server out of rotation.
  • Enables horizontal scaling: add a server to the pool, no client changes.
  • Optional extras (L7): TLS termination, path-based routing, compression.

Algorithms

Round robin

  • Servers take requests in a fixed order: S1, S2, S3, S1 …
  • Zero state, zero cost — but blind to how busy a server is.
  • Weighted round robin: a bigger server gets more turns.
ClientsLoad balancerS12 servedS22 servedS32 served6
Round robin: each server takes the next request in turn.
  1. Requests arrive at the load balancer, which keeps a pointer to the next server in a fixed order: S1, S2, S3, then S1 again.
  2. Request 1 reaches the load balancer. Round robin asks nothing about the servers — it only reads whose turn it is.
  3. Request 1 goes to S1, the server whose turn it was; the pointer moves on to S2, while request 2 reaches the balancer.
  4. Request 2 goes to S2, the server whose turn it was; the pointer moves on to S3, while request 3 reaches the balancer.
  5. Request 3 goes to S3, the server whose turn it was; the pointer moves on to S1, while request 4 reaches the balancer.
  6. Request 4 goes to S1, the server whose turn it was; the pointer moves on to S2, while request 5 reaches the balancer.
  7. Request 5 goes to S2, the server whose turn it was; the pointer moves on to S3, while request 6 reaches the balancer.
  8. Request 6 lands on S3. Six requests, 2/2/2 each: an even spread, as long as every request costs the same.

Least connections

  • Counts open connections per server; the next request goes to the fewest.
  • Best when request durations vary (uploads, websockets, slow queries).
Load balancert = 7sS11 open1S21 open2S31 open1
Least connections: the next request goes to the server with the fewest open.
  1. Each request holds a connection for a different time (the number on it, in seconds). The balancer counts every server's open connections.
  2. At 1s a 6-second request arrives. Open connections are 0, 0, 0, so it goes to S1, the first of the servers tied for fewest.
  3. At 2s a 1-second request arrives. Open connections are 1, 0, 0, so it goes to S2, the first of the servers tied for fewest.
  4. At 3s a 1-second request arrives. Open connections are 1, 0, 0, so it goes to S2, the first of the servers tied for fewest.
  5. At 4s a 5-second request arrives. Open connections are 1, 0, 0, so it goes to S2, the first of the servers tied for fewest.
  6. At 5s a 1-second request arrives. Open connections are 1, 1, 0, so it goes to S3, the least busy.
  7. At 6s a 2-second request arrives. Open connections are 1, 1, 0, so it goes to S3, the least busy.
  8. At 7s a 1-second request arrives. Open connections are 0, 1, 1, so it goes to S1, the least busy.

IP hash (sticky routing)

  • server = hash(client IP) mod N — the same client lands on the same server.
  • Useful when session state lives in server memory.
  • Changing N remaps most clients — see consistent hashing.
S1S2S3removed10.0.0.410.0.0.910.0.1.710.0.2.310.0.3.8
IP hash: the same client always reaches the same server.
  1. Some apps keep a user's session in one server's memory. IP hash pins each client to a server by hashing its address, so no table is needed.
  2. hash(10.0.0.4) mod 3 = 1, so 10.0.0.4 goes to S2 — and every later request from it does too.
  3. hash(10.0.0.9) mod 3 = 1, so 10.0.0.9 goes to S2 — and every later request from it does too.
  4. hash(10.0.1.7) mod 3 = 0, so 10.0.1.7 goes to S1 — and every later request from it does too.
  5. hash(10.0.2.3) mod 3 = 1, so 10.0.2.3 goes to S2 — and every later request from it does too.
  6. hash(10.0.3.8) mod 3 = 2, so 10.0.3.8 goes to S3 — and every later request from it does too.
  7. S3 is removed and the hash is now taken mod 2: 2 of 5 clients land on a different server and lose their session. Consistent hashing moves far fewer.

Health checks

  • The balancer polls an endpoint such as /health every few seconds.
  • Fall threshold: out of rotation after N misses in a row (not one blip).
  • Rise threshold: back in after N passes in a row.
  • Users never see the dead server's errors.
Load balancerrotation: S1, S2, S3200S1healthy200S2healthy200S3healthycheck round 6
Health checks: a failing server is taken out, then put back.
  1. Every few seconds the balancer calls each server's health endpoint. All three answer 200, so all three stay in rotation.
  2. S2 times out once. One miss could be a blip, so the balancer keeps it in rotation until 2 misses in a row.
  3. A second miss in a row: S2 is taken out of rotation. Requests now go only to S1 and S3 — users never see S2's errors.
  4. S2 is still failing its checks; it stays out, and S1 and S3 carry all the traffic.
  5. S2 answers again (1 pass in a row); it needs 2 before it is trusted with traffic again.
  6. A second pass in a row: S2 is put back into rotation and traffic is spread over all three servers again.

L4 vs L7

L4 (transport)L7 (application)
ReadsIP, portURL, headers, cookies
SpeedFasterSlower per request
RoutingPer connectionPer request, by path or header
TLSPasses throughCan terminate
ExamplesAWS NLB, LVSNginx, HAProxy, AWS ALB

Interview questions

  • Where does a load balancer sit? Between clients and app servers, and often again between app servers and internal services.
  • Which algorithm would you pick? Round robin for short, uniform requests; least connections when durations vary; hashing when a client must stick to a server.
  • How do you avoid the balancer being a single point of failure? Run two or more (active-passive with a floating IP, or several behind DNS).
  • Why keep servers stateless? Any server can take any request, so the balancer is free to choose and a lost server loses no sessions.

Next: Caching · Before this: Scalability

Common questions

Why do we need a load balancer?

One server has a ceiling on requests per second and is a single point of failure. A load balancer lets you add servers behind one address, spreads the work across them and stops sending traffic to a server that fails its health checks.

What is the difference between round robin and least connections?

Round robin hands requests out in a fixed order and ignores how busy each server is. Least connections counts each server's open connections and picks the fewest, so it copes better when some requests take much longer than others.

What is the difference between an L4 and an L7 load balancer?

An L4 balancer routes on IP addresses and ports without reading the request, so it is fast and protocol-agnostic. An L7 balancer reads HTTP — paths, headers, cookies — so it can route /api and /images to different pools, terminate TLS and rewrite requests, at more cost per request.

Is the load balancer itself a single point of failure?

It would be if there were only one. In practice balancers run as a pair or a fleet — active-passive with a floating IP, or several behind DNS — and managed balancers such as AWS ELB are redundant by design.

What are sticky sessions?

Sticky sessions send every request from one user to the same server, by a cookie or by hashing the client's IP. They help when sessions live in a server's memory, but they unbalance load and lose sessions when that server goes away; storing sessions in Redis avoids both.

← Scalability · Caching →