Design a Rate Limiter

Rate limiting exists for three reasons. It blunts denial-of-service attempts and abusive scraping, it controls cost when each request calls a paid third-party API, and it prevents one noisy tenant from starving everyone else. The limiter must enforce rules like 10 posts per second per user or 1,000 calls per day per API key, and it must do so with almost no added latency, because it sits on the path of every request.

Clarifying questions set the shape: server-side or client-side; throttle by user ID, IP or API key; how many requests at what scale; distributed or single-process; separate service or part of application code; should throttled users be told. A reasonable target is a server-side, distributed limiter with flexible rules, low latency, low memory, clear feedback to clients, and fault tolerance: if the limiter's store fails, the system should keep serving rather than go dark.

Assume 10 million daily users averaging 100 API calls each: 1 billion requests per day, or about 11,600 QPS on average and perhaps 35,000 at a 3x peak. Every request performs at least one counter operation, so the counter store must sustain tens of thousands of operations per second with sub-millisecond latency. A single Redis node handles on the order of 100,000 simple operations per second, so a small cluster gives ample headroom.

Memory depends heavily on the algorithm. A fixed-window counter needs one key per user per window, roughly 100 bytes with overhead, so 10 million users need about 1 GB. A sliding-window log storing every timestamp for a 100-requests-per-minute limit could hold 100 entries per user at about 64 bytes each in a sorted set: 6.4 KB x 10 million ≈ 64 GB. That 60x difference is a concrete reason to prefer counters over logs at scale.

Client-side limiting is unreliable because clients can be modified or forged; it is a courtesy, never a guarantee. Server-side limiting inside each API server works but duplicates logic and makes global limits hard. The most common choice is middleware in front of the APIs, often an API gateway that already handles authentication, TLS termination and IP allowlisting. A managed gateway is attractive when the stack already has one.

The decision depends on your context: the language and stack (can you implement it efficiently server-side?), whether you need full control of the algorithm (a third-party gateway may limit choices), and engineering capacity. Building your own takes time; buying a gateway trades flexibility for speed.

In a token bucket, each client has a bucket of capacity B refilled at rate r tokens per second. Each request consumes a token; if none remain, it is rejected. It is simple, memory-cheap and deliberately allows bursts up to B, which matches real traffic well; it is used widely by cloud providers. The challenge is tuning two parameters per rule, and you may need separate buckets per endpoint, per IP and globally.

A leaking bucket puts requests in a FIFO queue that is drained at a fixed rate; when the queue is full, new requests are dropped. Its strength is a perfectly smooth outflow, ideal when downstream capacity is fixed. Its weakness is that a burst fills the queue with old requests, so newer ones are dropped even if the old ones are no longer useful, and again two parameters need tuning.

A fixed window counter divides time into windows and counts requests per window, rejecting once the count exceeds the limit. It is cheap and easy to reason about, but it permits up to twice the limit around a window boundary: a client can send the full quota at the end of one minute and again at the start of the next. A sliding window log fixes that by storing each request timestamp, evicting those older than the window and counting the rest. It is exact but memory-heavy, and even rejected requests may be logged.

The sliding window counter blends the two. It estimates the rolling count as current window count plus previous window count weighted by how much of the previous window still overlaps. With a limit of 10 per minute, 5 requests last minute, 3 so far this minute, and 30% into the current minute, the estimate is 3 + 5 x 0.7 = 6.5, so the request is allowed. It smooths spikes with fixed memory, assuming requests in the prior window were evenly spread.

Counters belong in an in-memory store rather than a database, because every request touches them. Redis fits well: INCR atomically increments a key and EXPIRE sets a time-to-live so window keys clean themselves up. The middleware builds a key such as rl:user42:2026-10-02T10:15, increments it, sets an expiry on first creation, and compares the result to the rule's limit.

Rules live in configuration files on disk or in a config service, and workers load them into a local cache so the hot path never waits on a remote lookup. When a request is throttled, the API returns HTTP 429 Too Many Requests. Well-behaved clients rely on headers: X-Ratelimit-Limit (quota for the window), X-Ratelimit-Remaining, and X-Ratelimit-Retry-After (seconds to wait). Throttled requests can be dropped or, for things like orders, enqueued to process later.

A naive read-check-write sequence breaks under concurrency: two requests both read a counter of 3, both decide it is under the limit of 4, and both write 4, letting one extra request through. Locks would fix it but add latency. The idiomatic fix is to make the whole check atomic in Redis, either with a Lua script that reads, compares and increments in one step, or by using sorted sets for a sliding log where removal of stale entries and insertion happen in a single transaction.

With several limiter instances, each must see the same counters. Sticky routing of a client to one limiter does not scale and breaks on failover, so the better approach is a centralized store like a Redis cluster. For multi-region deployments, keep counters local to each edge region and synchronize with eventual consistency, accepting slight over-admission in exchange for latency.

A hard limit never lets the count exceed the threshold; a soft limit tolerates brief overshoot, which is cheaper to enforce across distributed instances. Limiting can also happen at other layers: application-level (HTTP, layer 7) rules are the focus here, but IP-level limits with tools like iptables operate at layer 3 and drop abusive traffic earlier and cheaper.

Decide failure behavior explicitly: if Redis is unreachable, most consumer APIs fail open (allow traffic) to avoid self-inflicted outages, while security-sensitive endpoints such as login may fail closed. Monitor how often rules trigger; too many 429s during a flash sale may mean the rule is too strict or a burst-friendly algorithm is needed. Clients should cache results, respect retry headers, and use exponential backoff with jitter.