Design a Notification System

Many parts of a product need to tell users something: a payment cleared, a friend replied, a flight changed. Rather than each team integrating with Apple, Google, and SMS vendors separately, a shared notification platform accepts requests and handles delivery. Clarify which channels are needed, whether delivery must be real time or can tolerate small delays, which devices are supported, and whether users can opt out.

A reasonable scope is iOS push, Android push, SMS, and email; soft real time, meaning seconds of delay under load is acceptable; and support for user opt-out. Notifications can be triggered by client apps or scheduled on the server.

A typical volume is 10 million mobile pushes, 1 million SMS, and 5 million emails per day, about 16 million in total. Averaged over 86,400 seconds that is about 185 notifications per second; pushes alone are 10M / 86,400 ≈ 116 per second. Traffic is very bursty, though: a marketing campaign or breaking event can send millions within minutes, so design for peaks one or two orders of magnitude above the average.

Payloads are small, perhaps 1–5 KB including template data, so 16M × 5 KB = 80 GB per day of notification log at most. The bottleneck is not storage but third-party throughput limits and the cost of SMS, which is why rate limiting and channel choice matter.

Each channel has its own path. iOS push needs a provider (your server) to build a request with the device token and a JSON payload and send it to Apple Push Notification service (APNs), which delivers to the device. Android uses Firebase Cloud Messaging (FCM) in the same role. SMS goes through vendors such as Twilio or Nexmo, and email through providers such as SendGrid or Mailchimp, because running your own mail servers with good deliverability is hard.

To send anything you need contact details. When a user installs the app or signs up, the API servers record their email and phone in a user table and their device tokens in a device table. One user can have many devices, so the device table is one-to-many, and stale tokens must be removed when providers report them invalid.

A first draft puts one notification server between callers and providers. It works, but it has three problems: it is a single point of failure; it is hard to scale because the database, cache, and processing all live together; and it is a performance bottleneck because building HTML emails and waiting on slow third-party responses block everything else.

The improved design separates concerns. Notification servers become a stateless, horizontally scaled tier that authenticates callers, validates requests, fetches user and template data from the cache or database, and drops a message onto a queue. There is one message queue per channel, so an outage at the SMS vendor fills only the SMS queue while pushes and emails keep flowing. Workers pull from each queue and call the matching provider. Each tier can now scale on its own.

Notifications may be delayed or reordered, but they should not vanish. The system persists each notification in a notification log database before or as it enqueues it, and workers update its status. If a worker crashes mid-send, the record shows the notification as unsent, and a retry job can pick it up.

Exactly-once delivery is impossible across a network you do not control, because a provider may accept a message and then the acknowledgement may be lost. So the system aims for at-least-once delivery plus deduplication. Each event carries an ID; before sending, a worker checks whether that ID has already been processed and skips it if so. This gives behaviour that is close to exactly once in practice, while accepting rare duplicates in edge cases.

Templates keep formatting consistent and save callers from building full messages: a template has placeholders for name, item, or date, and the server fills them in. Notification settings store, per user and channel, whether they opted in; workers check these before sending, both for user experience and for legal compliance on marketing messages.

Rate limiting caps how many notifications a user receives in a period, because users who feel spammed turn notifications off entirely, which is far worse than one missed message. Retries with backoff handle transient provider failures; if a provider keeps failing, alert operators and consider failing over to a secondary vendor.

Only authorised internal clients should be able to send notifications, otherwise the platform becomes a spam cannon. Callers authenticate with an appKey and appSecret (or an equivalent token scheme), and the notification servers reject anything unsigned.

The most important operational metric is queue depth: a growing backlog means workers cannot keep up and more should be added. Beyond that, event tracking records open rates, click rates, and engagement by integrating with the analytics service, which lets product teams judge whether a notification was worth sending. A notification moves through states such as start, pending, sent, delivered, clicked, or error, and tracking those transitions makes delivery problems visible.

The final design trades simplicity for isolation and durability. Queues add latency and operational overhead, but without them one slow provider would stall all traffic. Persisting a log before sending costs a write per message but buys recoverability. Deduplication costs a lookup per message but keeps users from seeing the same alert twice.

In an interview, walk through the evolution explicitly: name each weakness of the single-server design and the component that removes it. Then cover reliability, settings, rate limits, retries, security, and monitoring as the features that turn a working prototype into a platform other teams can depend on.