Pattern: Dealing with Contention

Contention happens when many requests try to change the same thing at the same time: the last ticket, the last unit of stock, a bank balance, a counter, a document, a seat. Without care, they overwrite each other and the system loses updates, oversells, or double-spends. With the wrong care, they queue behind locks and throughput collapses. This pattern appears in ticketing, inventory, booking, banking, auctions and gaming, and a prepared candidate can name the options and when each wins.

1. The problem in one example

Two customers buy the last unit of an item at the same instant.

Request A: read stock -> 1        Request B: read stock -> 1
Request A: write stock = 0        Request B: write stock = 0

Both read 1 before either wrote, both believe they got the item, and two units were sold. This is a lost update, and it is a race condition: the outcome depends on the timing of operations that must not interleave.

The core question for every technique below is: how do we make the check and the update happen as one indivisible step, or otherwise guarantee that only one request wins?

2. First: avoid the contention

The best solution to contention is often to remove it.

• Partition so that writers rarely meet. If each user updates only their own record, there is no contention between users. Choose the data model and shard key to give each hot item its own owner.
• Make operations commutative. Incrementing a counter is commutative: the order of two increments does not change the sum. Adding to a set is commutative. When updates commute, they can be applied in any order without coordination.
• Append instead of update. Record each purchase as a new row, and compute the stock from the rows. Appends do not conflict. (You must still enforce the limit somewhere.)
• Reduce the window. Keep the critical section as small as possible: do slow work such as calling a payment provider outside the lock.

When contention is unavoidable, such as selling one specific seat, use one of the following.

3. Let the database arbitrate: atomic operations

Push the check and the update into one database statement that the database executes atomically.

UPDATE inventory
SET stock = stock - 1
WHERE sku = 'A1' AND stock > 0;
-- rows affected = 1: you got it.  rows affected = 0: sold out.

The database guarantees that concurrent executions are serialised on that row, so exactly one request sees the last unit. No separate read, no race. This is the simplest correct solution and should be your default for single-row conditions.

Related tools:

• Unique constraints. A unique index on (event, seat) makes a second booking of the same seat fail with an error, no matter what the application does. Use constraints as the safety net for your invariants.
• Conditional writes. Many key-value stores support "write only if the current value or version equals X", the same idea.
• Atomic increments. Counters in databases and caches have atomic increment operations.

4. Pessimistic locking: lock, then work

Take a lock on the row before reading it, hold it while you decide and update, then release.

BEGIN;
SELECT * FROM seats WHERE id = 42 FOR UPDATE;   -- others wait here
-- check, decide, update
UPDATE seats SET status = 'held' WHERE id = 42;
COMMIT;

• Pros: simple to reason about, correct under high contention, no wasted work from retries.
• Cons: other requests wait, so throughput on a hot row is limited to one at a time. Long transactions hold locks and block others. Locks taken in different orders cause deadlocks, so access rows in a consistent order and keep transactions short. A lock holder that stalls blocks everyone behind it.
• Use when: conflicts are frequent, the critical section is short and the work inside is cheap.

For job queues, a variant locks rows while skipping rows that are already locked, so many workers can each claim different rows without waiting on each other.

5. Optimistic concurrency: check at the end, retry on conflict

Do not lock. Read the record with its version, do the work, and write back only if the version is unchanged. If another writer got there first, the write affects zero rows, and you retry from the read.

UPDATE accounts
SET balance = :new_balance, version = version + 1
WHERE id = :id AND version = :version_you_read;
-- 0 rows updated: someone else changed it, retry

• Pros: no locks held, so no blocking, and it scales when conflicts are rare.
• Cons: under heavy contention most attempts fail and retry, wasting work and possibly never succeeding (livelock). Add a retry limit, backoff and jitter.
• Use when: conflicts are rare and the work between read and write is substantial, such as editing a form.

Choosing between pessimistic and optimistic is a standard question. A one-line answer: "optimistic when contention is low, pessimistic or atomic updates when it is high".

6. Serialise through a single writer

If many requests fight over one key, make them take turns by routing all operations for that key to a single worker, through a partitioned queue keyed by the item.

• Pros: no locks, no conflicts, no retries. Each key's updates are processed in order, and the single writer can batch them. Throughput scales with the number of keys, since different keys go to different workers.
• Cons: each key's throughput is limited to one worker, the response is usually asynchronous (the client learns the outcome later), and you need a queue and care for failure and ordering.
• Use when: one item is extremely hot (the last tickets in a flash sale), or a strict sequence per key is needed.

This is the idea behind actor-style systems and single-threaded-per-partition designs.

7. Reservations and holds

When an action spans time, such as choosing a seat and then paying, you cannot hold a database lock for minutes. Instead, make the claim a recorded state with an expiry.

1. Atomically move the item from available to held, recording who holds it and when the hold expires.
2. The user completes the multi-step process.
3. On success, move it to booked. On timeout or cancel, the hold expires and the item returns to available.

A fast store with expiring keys makes the hold cheap, and the transactional database remains the final authority with a uniqueness constraint. Handle the expiry race: a payment that completes just after the hold expires must check that the hold is still valid, or re-acquire it, and otherwise refund. This is the core of the ticket booking design.

8. Distributed locks and their limits

When the contended resource is not in a single database, such as a file, an external API quota or a job that must run on only one node, you may use a distributed lock from a coordination service or a key-value store.

Important cautions, which interviewers like to hear:

• Always use an expiry (lease), or a crashed holder blocks everyone forever.
• A lease can expire while the holder is still working, for example during a long pause. Then two processes believe they hold the lock. Use a fencing token: an increasing number issued with each lock grant, which the protected resource checks and uses to reject operations from an older holder.
• A lock on a single node can be lost in a failover before it is replicated. For strong guarantees, use a consensus-based service.
• Prefer to avoid locks. An idempotent operation with a unique key, or a conditional write, is safer than a lock around a read-modify-write.

9. Contention on counters and aggregates

A single counter updated by every request is a hot key: even with atomic increments, throughput is limited to what one node does to one key.

• Salt the counter across several sub-counters and sum on read.
• Aggregate in memory per server and flush periodically.
• Approximate where exactness is unnecessary.
• Derive from events rather than maintaining a shared total.

10. Choosing among the options

11. A worked example

Problem. A flash sale of 500 units of one product; 100,000 people click "buy" within a minute.

Reasoning.

1. This is extreme contention on one inventory row. A pessimistic lock would serialise 100,000 requests behind one row and time most of them out.
2. Put the buyers through a waiting room that admits requests at a rate the system can handle, as in the ticketing design.
3. Reserve stock with an atomic conditional update, stock = stock - 1 WHERE stock > 0. Only 500 updates succeed, and the rest see zero rows and are told "sold out" immediately, without a lock wait.
4. Create the order in a held state with an expiry, then take payment. If the payment fails or times out, release the unit back with an atomic increment.
5. A unique constraint on (order id) and an idempotency key on checkout prevent double submission.
6. To lift throughput above one row's limit, split the stock into several buckets of 50 units each, and route each buyer to a bucket, falling back to others when it empties. The cost is some complexity near the last few units.

What I would say about the trade-off. "I accept that some buyers are rejected quickly rather than waiting, I never oversell, because the database arbitrates, and I choose the simplest mechanism, an atomic update, before adding queues or locks."

12. Interview questions and model answers

Q: How do you prevent two users buying the last item? Make the check and the update atomic: a conditional update that decrements stock only where it is above zero, so the database serialises concurrent requests and exactly one succeeds. A uniqueness constraint backs it up where an invariant exists.

Q: Optimistic or pessimistic locking? Optimistic when conflicts are rare, because it holds no locks. Under heavy contention, optimistic retries waste work, so I use an atomic update, a short row lock, or a single writer per key.

Q: A hot row cannot keep up. What next? Reduce its contention: split it into several sub-rows or buckets, serialise updates through a queue with a single writer per key, or batch updates. If exactness is not required, use an approximate or aggregated counter.

Q: How do you handle a multi-step purchase where the item must stay reserved? A hold with an expiry. The item moves atomically to held, the user pays, and on success it becomes booked. If the hold expires, it is released, and the final step re-checks the hold.

Q: Is a distributed lock safe? Only with caveats. It needs a lease, a lease can expire during a pause, so I use a fencing token checked by the resource. For strong guarantees I use a consensus-based coordination service, and I prefer conditional writes to locks where I can.

Q: What is a deadlock and how do you avoid it? Two transactions each hold a lock the other needs. Access resources in a consistent order, keep transactions short, set lock timeouts and retry on deadlock errors.

13. Common mistakes

• Reading, checking and writing in separate steps.
• Holding a database lock while calling a slow external service.
• Optimistic retries on a hot row with no limit or backoff.
• A distributed lock with no expiry, or one trusted without a fencing token.
• A single global counter updated by every request.
• Relying on application checks instead of a database constraint.
• Making users wait on a lock queue instead of rejecting quickly.