Stock Exchange

An exchange matches buyers and sellers efficiently. Retail investors reach it through brokers, while institutional clients such as funds trade in large volumes and often split orders to avoid moving the price. A limit order buys or sells at a fixed price or better and may wait in the book; a market order executes immediately at the best available price and gives up price control for speed.

Market data comes in levels. L1 shows the best bid and best ask with quantities. L2 adds more price levels of depth. L3 shows every individual order at each level. Candlestick charts summarise prices over an interval with open, close, high and low. Many brokers and exchanges speak FIX, a long-established, vendor-neutral protocol for exchanging securities transaction messages.

Our exchange trades 100 symbols and handles billions of orders per day. Clients can place and cancel limit orders, receive real-time executions and market data, and must pass risk checks, for example a cap on how many shares of one symbol a user can trade per day. Each order must have funds locked in a wallet before submission, so buyers cannot spend the same money twice. Requirements emphasise availability, fault tolerance, and above all low and predictable latency, measured at the 99th percentile, because tail latency is what traders actually feel.

Assume 1 billion orders per day and trading hours of 6.5 hours, which is 6.5 x 3,600 = 23,400 seconds. Average QPS is 1e9 / 23,400 ≈ 42,700, about 43,000 QPS. Peaks at the open and close can be 5x, giving about 215,000 QPS. Double-check: 43,000 x 23,400 ≈ 1.006e9, matching the daily total. These rates are high but not extreme; the hard part is doing them with microsecond-level latency and strict ordering.

The trading flow is the critical path. A broker sends an order to the client gateway, which validates, authenticates and rate-limits it. The order manager runs risk checks and verifies funds in the wallet, then sends the order to the sequencer, which stamps it with a sequence ID. The matching engine matches it against the order book and emits executions, which pass back through the sequencer for outbound sequence IDs, then to the order manager and back to the client. Everything on this path must be as fast as possible.

The market data flow takes executions from the matching engine to the market data publisher, which builds order book snapshots and candlesticks and sends them to a data service that clients subscribe to. The reporting flow gathers orders and executions for reporting, settlement and compliance; it is not latency sensitive, so it can be off the critical path and favour accuracy and completeness.

The sequencer is what makes the exchange deterministic. It stamps every inbound order with a monotonically increasing sequence ID and every outbound execution likewise. Ordered inputs plus a deterministic matching engine mean that replaying the same sequence always produces the same executions: functional determinism. The sequenced stream also acts as an event store, enabling replay, audit, and detection of missing messages via gaps in IDs.

The matching engine maintains one order book per symbol: a list of buy orders and a list of sell orders grouped by price level. Each price level holds orders in a doubly linked list in arrival order, and a hash map from order ID to node allows cancel in O(1). Adding an order appends to the tail of its price level; matching consumes from the head of the best opposite level; cancelling unlinks a node. With price levels themselves indexed, the core operations stay O(1) or close to it, which is essential for predictable latency. The default algorithm is FIFO (price-time priority): best price first, then earliest arrival. Some markets use pro-rata, allocating fills at a price level in proportion to order size.

Latency equals the sum of time spent on each hop. Network round trips inside a data center cost tens to hundreds of microseconds and disk writes can cost milliseconds, so the most effective optimisation is to remove hops from the critical path. Modern exchanges put the gateway, order manager, sequencer and matching engine on one server, communicating via mmap-backed shared memory as a message bus rather than over the network. No disk sync or network call sits between receiving an order and matching it.

Each component runs an application loop that spins on a dedicated CPU core, with the thread pinned to that core so there are no context switches, no lock contention and warm caches. The loop polls for messages and processes them single-threaded, eliminating locks. State is managed with event sourcing: the sequenced input log on mmap is the truth, and in-memory state can be rebuilt by replaying it. The market data publisher uses ring buffers, pre-allocated circular arrays that avoid allocation and locking, the same idea behind the LMAX Disruptor.

A single server is a single point of failure, so we run a hot-warm pair. The hot matching engine processes orders and emits executions; the warm instance receives the same sequenced events and processes them too, keeping identical state, but its output is discarded. If the hot instance dies, the warm one takes over almost instantly because determinism guarantees identical state. Across machines, the event log is replicated using Raft, which tolerates failures of a minority of nodes and elects a new leader automatically.

There is a trade-off. Consensus requires a majority acknowledgment over the network before an event is committed, which adds latency to the critical path. Exchanges must decide what failure they protect against and accept a small latency cost, or batch and pipeline replication to amortise it. Recovery time objectives in finance are strict, and a mistaken failover is costly, so failure detection must also be conservative.

Determinism has two meanings. Functional determinism ensures identical outputs from identical inputs. Latency determinism means nearly every order takes about the same time; we track it with p99 and p99.9 latencies, because spikes from garbage collection, page faults or noisy neighbours are what hurt traders. Techniques include pre-allocating memory, avoiding GC languages on the hot path or tuning them carefully, and disabling CPU frequency scaling.

For fairness, market data should reach all subscribers at the same time. TCP unicast sends to each subscriber in turn, so whoever is first in the list gets an edge. UDP multicast delivers one packet to many subscribers simultaneously, making distribution fair; reliability is added with sequence numbers and retransmission. Colocation lets firms rent servers inside the exchange's data center, buying lower network latency; this is a paid service and does not change the ordering rules once orders arrive.

An exchange is a high-value DDoS target. Isolate public services like web and market data dashboards from the private trading network so an attack on one cannot touch the other. Cache read-heavy public data, harden URLs against cache-busting query strings, and apply rate limiting and allowlists at the gateway. For failures, the sequenced event log is the backbone: any component can be rebuilt by replay, warm instances take over immediately, and Raft protects the log itself. The overarching lesson is that predictability, not raw average speed, is the defining requirement, and almost every design choice serves determinism.