AI System Design
← Learn
Level 2beginner

Load Balancing

Spread the work so no single server drowns.

Depth:
1

Mission

Traffic is uneven and one server keeps melting while others idle. Distribute the load.
2

Interactive Simulation

Before we explain anything — play. Push it until it breaks, then fix it.

Requests
4.0K/s
Latency
140ms
p95 419ms
Error rate
0.0%
CPU
91%
Est. cost
$638/mo
illustrative
Accepted
4.0K/s
Latency (ms)
Error rate (%)

Load

High variance = a mix of quick lookups and slow reports. Equal request counts ≠ equal work.

Balancer

Algorithm

Break it

System Score90
3

What just happened?

Round-robin gave every server the same number of requests, but some requests take far longer — so one server melted while others idled. Least-connections routed by actual busyness and evened out the work. When a server died, only health checks stopped the balancer from sending it traffic into a black hole.

4

The concept

A load balancer distributes incoming requests across a pool of servers using strategies like round-robin, least-connections, or hashing. It also does health checks, removing dead servers from rotation. It's the enabler that makes horizontal scaling actually work.

5

Trade-offs

Nothing is free. Here's what this solution costs you.

Single point of failure
The LB itself must be redundant.
Stickiness vs balance
Session affinity can create uneven load.
Health-check tuning
Too slow and you route into dead servers; too twitchy and you eject healthy ones.
6

In the real world

Conceptually similar to a cloud L7 load balancer with target health checks, or NGINX/Envoy upstream pools.

7

Mini quiz

Question 1 of 30 correct

Least-connections routing is better than round-robin when…

8

Interview me

The app becomes your interviewer. One question, in your own words.

9

Boss challenge

Lose a server under load

6K req/s of mixed fast and slow requests. Then API 1 crashes.

Goal: Keep the error rate under 1% with request variance at 50%+ and a server down.

Use the simulator above with no hints. These checks update live as you play.

10

Interview question

“Explain L4 vs L7, round-robin vs least-connections, and how health checks work.”

Next: Message Queues