Latency, throughput, and capacity: size the Monday release

Distinguish latency, throughput, and capacity

Latency is elapsed time for an individual operation. Throughput is completed operations per unit time. Capacity is the workload the system can sustain while satisfying its chosen latency, error, and resource limits. A server can increase completed requests per second while its queues become unacceptable to users.

The unit matters. A schedule read returns public availability for one observatory and day. A booking attempt authenticates a member, checks a contested slot, and commits several database changes. They are not equal-cost “requests.” Google's Handling Overload discusses why queries per second alone can conceal differences in resource cost. Our capacity experiment must preserve the mix and contention of the real task.

For NightLens, discovery targets 99% successful reads within 300 ms at the API boundary. Booking targets 99% eligible attempts with a committed outcome or an ordinary conflict within 800 ms. Overload rejections and internal errors count against that booking objective. We still measure ordinary conflict rates separately so competition for a popular telescope does not masquerade as capacity loss.

Build an arrival profile before a server count

Assume six thousand visitors arrive during a ninety-second release window. Each makes three schedule requests. The mean is 6,000 × 3 / 90 = 200 schedule requests/second. This is a planning average across that window, not the highest instant.

Add a separate burst scenario: 900 schedule reads per second for twenty seconds, plus 40 booking attempts per second. Include a case where half the booking attempts target the same ten popular slots. Distinct-slot requests measure broad throughput; same-slot requests reveal lock contention and conflict handling. Do not multiply visitors by an arbitrary “safety factor” and treat it as observed demand.

Also define ordinary traffic: assume 25 reads per second and two booking attempts per second. A load test should move between those states and back again. A system that survives the peak but takes fifteen minutes to drain its backlog has a recovery problem even if the peak chart looks impressive.

The test generator must send arrivals at the intended rate and record missed schedules. If it waits for each response before sending more work, a slowing service can reduce offered load and hide overload. Report offered requests, admitted requests, completed requests, rejections, and timeouts together.

Use a latency budget as a diagnostic tool

Allocate an illustrative 800 ms booking deadline: 60 ms for gateway and authentication, 40 ms for admission and scheduling, 300 ms for the database transaction including lock waits, 100 ms for application work and serialization, and 300 ms of contingency for variation and transport. These are design allocations to test, not measured component percentiles.

Do not add each component's p99 and call the sum the end-to-end p99. Different requests may encounter different slow components, and their timings may be correlated. Measure whole-request latency and use traces to investigate where a slow request spent its time. The budget tells you where to ask questions; it does not prove the objective.

For example, reducing JSON serialization from 8 ms to 4 ms will not rescue a booking that waits 600 ms for a slot lock. Splitting the API into more replicas may increase that lock pressure. First separate queue delay, connection-pool wait, transaction execution, and commit time in the trace. Fieldnote's checkout sequence releases database locks before a remote payment call, an example of reducing lock occupancy without making the dependency itself faster.

Estimate in-flight work with consistent units

Under a stable workload and a consistently defined system boundary, Little's law relates mean in-flight work to effective throughput and mean time: L = λ × W. The research paper Statistical Analysis with Little's Law discusses the relationship and the care needed when estimating it from observations. It is a statement about averages, not a percentile sizing formula.

If admitted schedule reads complete at 200 per second with mean residence time 0.12 seconds, mean in-flight work is 200 × 0.12 = 24 requests. At a stable 900 per second with mean residence time 0.30 seconds, it is 270. That second calculation is a hypothetical stable operating point; it cannot be assumed during a continuously growing queue.

Keep boundaries aligned. If the time includes a gateway queue, count work in that queue. If the throughput is admitted traffic after rejection, do not substitute all attempted traffic. Replacing the mean time with p99 gives no corresponding p99 concurrency guarantee.

Now isolate the database connection pool. Suppose booking transactions hold a connection for a mean of 0.05 seconds at 40 attempts per second. That implies two connections occupied on average for that workload. It does not prove that a two-connection pool is sufficient: bursts, variance, discovery reads, background work, and long lock waits still need headroom and admission limits.

Find the first scarce resource

Imagine a local experiment reports the following illustrative result: one API replica sustains 350 schedule reads per second within the objective, but a shared database becomes saturated at 600 uncached reads per second. Three replicas do not establish 1,050 requests per second of system capacity. They share a 600-read bottleneck before accounting for bookings.

Use the result to choose an experiment. Check query plans and returned columns, cap the day range, or add a cache for public advisory results. Remeasure with booking traffic present. If a cache serves 80% of a 900-read workload, the simple expected miss rate is 180 per second. That arithmetic assumes the miss distribution is representative; a cold cache or synchronized expiration can send far more work to the database.

Set an operating envelope with a recovery requirement: maximum admitted reads, maximum concurrent bookings, bounded queues, and a rule for rejecting excess work. The later backpressure and timeout lesson turns those bounds into failure behavior. A target without a limit tells operators when things are bad, but not how to stop the damage.

Estimate storage separately from request rate

Assume the audit log receives 40,000 events per day at an average encoded payload of 700 bytes. Raw payload growth is 28,000,000 bytes/day, or about 28 decimal MB. Over 365 days, that is 10.22 decimal GB before indexes, row overhead, free space, write-ahead logs, replicas, or backups.

If three database copies retain that raw data, the payload alone becomes 30.66 GB across those copies. Backup retention is a separate calculation because full and incremental backups have their own schedules and compression. This estimate is useful for finding missing terms, not choosing a disk size to two decimal places.

Distinguish durable reservation history from disposable cache entries. A cache eviction is an expected miss. Losing the authoritative booking record is a different failure. Agree on retention and recovery objectives for each dataset before applying one blanket replication or backup policy.

Practice: diagnose a superficially healthy release

Consider a hypothetical test: 900 reads per second are offered for twenty seconds. The service completes 700 per second, rejects none, and holds the remainder. Mean completion latency during the first second is 120 ms, and the dashboard displays only completed requests.

1. Calculate the backlog at the end of the burst under the simplified constant-rate assumption.
2. Estimate drain time if incoming load drops to 200 per second and service remains 700 per second.
3. Name two observations missing from the dashboard.

The backlog is (900 − 700) × 20 = 4,000 requests. Net drain capacity is 700 − 200 = 500/second, giving eight seconds to drain. These are fluid approximations; timeouts, cancellations, and changing service rates can alter the result. Missing observations include queue age, outstanding requests, offered load, and late timeouts. Early completed-request latency says little about the members still waiting.

For your own next step, write three load profiles, record the assumptions behind each, and list the resource that would stop scaling first. Then define what counts as passing: latency, unsuccessful outcomes, maximum queue age, and recovery time. Measure the envelope before buying capacity or dividing the application into services.