Payment System

We are designing the payments backend for a marketplace in which buyers pay and sellers get paid. There are two flows: pay-in, where the buyer's money moves from their card to the marketplace's bank account, and pay-out, where the marketplace later sends sellers their share. The company does not process cards itself; it integrates with a payment service provider (PSP) such as Stripe or Adyen, which talks to the card schemes.

Scale is modest, about 1 million transactions per day, but the non-functional requirements are strict. The system must be reliable and fault tolerant, because failed payments must be handled carefully rather than dropped. It needs a reconciliation process between internal services and external providers to verify that payment information is consistent. The guiding principle is that correctness beats raw performance at every turn.

One million transactions per day divided by 1e5 seconds gives 10 transactions per second. Even a 10x holiday peak at 100 TPS is easy for one relational database. That tells us the design should not be driven by throughput at all. Instead it should be driven by correctness: idempotency, transactional state changes, audit trails and recovery from partial failures. Storage is similarly small, roughly 1 million rows per day for orders plus several ledger entries each, or a few hundred million rows a year.

The important consequence is that we can pick a mature ACID relational database with a long track record, strong tooling and good backups, rather than a NoSQL store chosen for scale we do not need.

When a buyer checks out, a payment event is created; one event can contain several payment orders, one per seller. The payment service receives the event, stores it, runs risk checks such as anti-money-laundering screening, and then hands each order to the payment executor, which calls the PSP. The PSP moves money through the card schemes (Visa, Mastercard) and banks. On success, the payment service updates the wallet, which tracks each seller's balance, and the ledger, which records every financial movement.

The ledger uses double-entry accounting: every transaction is recorded as a debit to one account and an equal credit to another, so the sum of all entries is always zero. If a buyer pays 1 dollar, the buyer account is debited 1 dollar and the seller account credited 1 dollar. Any imbalance points directly at a bug, which makes the ledger the primary tool for auditing and reconciliation.

Storing raw card numbers brings the full weight of PCI DSS compliance. Most companies avoid it with a hosted payment page. The payment service first registers the payment with the PSP, passing a UUID nonce for idempotency; the PSP returns a token. The client then renders the PSP's hosted page, via an SDK or iframe, which collects card details directly. Card data never touches our servers. After the PSP processes the payment, it redirects the browser to our success URL and also sends a webhook with the final status, which updates payment_order.status.

The pay-out flow mirrors pay-in, but instead of a PSP for card acceptance we use a pay-out provider such as Tipalti, which moves funds from the marketplace account to sellers' bank accounts, usually in scheduled batches with its own reporting.

Messages between services and providers can be lost or delayed, so the last line of defence is reconciliation. Every night the PSP or bank sends a settlement file listing balances and transactions. A reconciliation job compares it against our ledger and payment records. Mismatches fall into three classes: those an automated program can fix because the cause and fix are known; those whose cause is known but too costly to automate, which go to a queue for the finance team; and those with unknown cause, which are escalated for investigation.

Some payments take hours or days, for example when the PSP flags a payment as high risk or 3-D Secure requires extra authentication. The PSP returns a pending status, our service records it and the client shows it, and the final outcome arrives via a webhook or by our service polling the PSP. Designs must assume asynchronous completion as normal, not exceptional.

Synchronous HTTP between services is simple but couples availability and latency: if the ledger is slow, checkout is slow. Asynchronous messaging through a queue like Kafka decouples them. A single-receiver queue suits work items processed once; a multi-receiver log lets the same payment event drive the ledger, wallet, analytics and notifications independently. For durability, every state transition of a payment is recorded in an append-only table, so a crashed service knows exactly where to resume.

Failures are routed deliberately. Retryable errors go to a retry queue and are retried with exponential backoff; non-retryable errors, such as invalid input, are stored for analysis. Messages that keep failing after the retry budget move to a dead letter queue, where they can be debugged and replayed without blocking healthy traffic.

No network can guarantee exactly-once delivery, but we can build exactly-once processing from two halves. At-least-once comes from retries: if a call times out, try again, with backoff and jitter so retries do not overwhelm a recovering provider. At-most-once comes from idempotency: the client sends an Idempotency-Key header, and the server inserts it under a unique constraint before doing work. A duplicate either sees the stored result or collides on the constraint and returns the earlier outcome.

Two double-charge scenarios matter. A user clicking Pay twice is caught by the idempotency key on our API. A PSP that processed the payment but whose response was lost is caught by reusing the same nonce with the PSP, so the PSP recognises the retry and returns the original result instead of charging again. Together, retries plus idempotency give the effect of exactly once.

Consistency has several layers. Between internal services, idempotent processing plus a durable state log keeps state aligned. Between us and the PSP, idempotency plus reconciliation catches divergence. For database replicas, reading from a lagging replica could show stale payment status, so either serve all reads and writes from the primary or use a consensus-based database such as one built on Raft that guarantees linearisable reads.

Security spans the whole path: HTTPS against eavesdropping, encryption and integrity checks against tampering, nonces and idempotency against replay attacks, rate limiting against DDoS, tokenization so real card numbers are never stored, PCI DSS compliance for whatever card data remains, and fraud checks such as address verification, CVV validation and behavioural analysis. In short: low TPS, high stakes; be idempotent everywhere, record every state change, reconcile every day.