Backpressure, timeouts, and retries during a reservation surge

Begin with a failure that crosses a boundary

At Monday's release, the email provider begins taking five seconds per request. A naive API sends confirmation email before replying to a booking. Requests hold application workers, database connections remain occupied if the integration sits inside the transaction, and clients retry after their own deadlines. The provider's slowness has now damaged reservation availability.

Our design commits a reservation and its outbox work, then responds without waiting for email. The delivery worker has its own connection and concurrency limits. Members see the committed reservation in their account even when email is delayed. This is a concrete form of failure isolation: the dependency that sends a reminder cannot occupy all the capacity needed to decide ownership.

The compromise is delayed or duplicate notification. The system must expose delivery lag to operators and avoid telling members that email delivery is guaranteed within the booking response time. The Amazon Builders' Library discussion of timeouts and retries explains how waiting calls retain scarce resources and how retries can amplify overload.

Limit work before it occupies the scarce resource

For a teaching configuration, cap concurrent booking transactions at twenty per API replica, allow at most twenty additional booking requests to wait, and limit admission wait to forty milliseconds. These numbers illustrate independent bounds, not recommended defaults. Choose real values from the capacity envelope and verify them with the actual number of replicas.

A per-replica limit is not a global database limit. Three replicas with twenty concurrent transactions can offer sixty transactions, plus background and administrative work. Reserve a total database connection budget and partition it among booking, public reads, and delivery. A rolling deployment that temporarily doubles replicas must not silently double accepted database work.

Reject a request before starting the booking transaction if the queue is full or its admission deadline expires. Return a machine-readable temporary-overload result and a human-readable explanation. For an HTTP API, RFC 9110 defines 503 Service Unavailable for temporary overload or maintenance and allows a Retry-After suggestion.

Count that rejection as unsuccessful in the member-facing booking objective. Backpressure protects recovery and gives the client a usable result; it does not turn rejected bookings into successful service. Use per-member rate limits as a separate fairness control. A concurrency cap protects resources, while a rate limit controls how much traffic one participant may offer.

Propagate a remaining budget, not a fresh timeout at every hop

The illustrative booking deadline is 800 ms at the API boundary. If authentication and queueing consume 100 ms, downstream work has less than 700 ms left. Do not give every database call another full 800 ms. Budget connection acquisition, lock wait, statements, commit, and response serialization inside the original deadline.

Configure both connection and operation timeouts, and check what the client library actually includes. Establishing a new connection may involve DNS and TLS outside a narrowly configured read timeout. On cancellation, ask the database driver to cancel work where supported, release resources safely, and bound cleanup. Cancellation propagation is an implementation to verify, not a promise that an already committed transaction can be undone.

Before starting work, compare the remaining time with a conservative minimum useful budget. If almost no time remains, fail before acquiring a contested slot lock. After a commit has begun, a timeout can leave the outcome uncertain. The client must not display “nothing was booked” merely because the response did not arrive.

Use the same request key to recover the operation result, as described in Storage and consistency: make one reservation authoritative. Treat unknown, confirmed, conflict, and rejected before execution as distinct states. That vocabulary prevents the interface from converting a transport failure into a false business assertion.

Choose one retry owner for each operation

Suppose the browser, gateway, and API each allow three total attempts. A single user action can produce 3 × 3 × 3 = 27 downstream attempts. That multiplication is arithmetic, not a workload measurement. Inspect existing client and SDK behavior before adding another retry loop.

For this release, the browser owns recovery of a booking operation with its stable request key; the gateway does not retry booking writes. The API may retry a database transaction only for an explicitly retryable database failure, within the same logical operation and remaining deadline. It does not repeat an external side effect inside that retry.

A slot conflict is a completed business result, not a reason to replay the same request until another member cancels. Validation and permission failures require correction. For temporary overload, honor the server's retry guidance, add bounded jitter, and stop when the user action's recovery window expires. If there is insufficient time for another attempt, show the recoverable state instead of launching work that cannot finish usefully.

AWS guidance on limiting retries recommends bounded backoff and jitter, checking error semantics, and avoiding compounded retries across layers. Idempotency preserves one business intent across retries; it does not make the extra request free.

Give asynchronous work a recovery policy

Assume a provider outage lasts ten minutes while five notification events per second arrive. With no successful deliveries, the backlog becomes 10 × 60 × 5 = 3,000 events. If the recovered worker can safely deliver twenty per second while new events continue at five per second, net drain is fifteen per second and the simplified drain time is 200 seconds.

Do not immediately release every delayed event with unlimited concurrency. Respect provider limits, use a bounded retry budget, and spread attempts with jitter. Monitor oldest pending event age as well as queue depth. A small queue can still contain an old reservation confirmation that needs attention.

Classify events by usefulness. A confirmation remains meaningful after a short delay; a reminder for a session that has ended should expire. Recheck the booking's current state before sending a reminder, so a cancelled reservation does not receive an invitation to attend. Preserve failed event information for operator review instead of silently discarding it.

A durable outbox prevents a transient worker failure from losing delivery intent, but durable storage is finite. Set retention, disk alerts, and an explicit policy for sustained growth. If delivery storage threatens the authoritative database, pause nonessential reminders first. If the service cannot durably record required booking work, stop accepting new bookings before a disk failure makes outcomes harder to recover.

Specify degraded modes per operation

• Email unavailable: keep booking while durable notification work fits within its operating limits; show account confirmation and monitor delivery lag.
• Cache unavailable: bound database fallback reads; do not flood the database. Keep static instrument descriptions available where feasible.
• Database unavailable: reject new reservations; public availability is usable only within its established age policy, then becomes unavailable or explicitly unverified.
• One hot slot: keep transaction work short and return conflicts promptly; increasing all queues will not create another telescope.
• Delivery backlog near storage limit: pause optional work and apply the documented admission stop before required durable writes become unsafe.

A circuit breaker may help isolate repeated calls to an unhealthy provider, but it introduces open, closed, and recovery transitions that require tests. Use a small bounded probe allowance during recovery. A circuit breaker is not a replacement for timeouts, queue bounds, or idempotency.

These modes preserve the original product priority: ownership stays honest even when discovery or notification degrades. The requirements and constraints lesson makes that preference explicit, so an incident response does not have to invent it under pressure.

Practice: run one surge with three injected failures

Use a controlled local or staging test with synthetic members and no real email recipients. Offer the assumed release traffic, then separately delay the provider, disconnect the cache, and drop the booking response after a successful commit. Record offered load, admissions, queue age, active transactions, operation-result recovery, database waits, and oldest outbox age.

1. Under provider delay, booking latency should remain independent until a documented storage or resource limit is reached.
2. Under cache failure, fallback reads should stop at their cap and booking capacity should remain protected.
3. After a lost response, a same-key recovery should return the committed booking without another reservation or new notification intent.

Then remove the failures and measure recovery. A test passes only if queues drain within the agreed window, limits remain bounded, and no stale “unknown” state persists without a recovery path. Compare those observations with the proposed objectives; do not relabel the objectives as achieved until you have evidence.

Your final artifact is an operation policy sheet: admission bound, deadline, retry owner, idempotency scope, degradation behavior, and recovery signal. Review it alongside the architecture and failure sequences. A design is ready for implementation when the team can explain both how work succeeds and how the system stops accepting work it can no longer finish honestly.