§11.5Practical Example: Creating a Performance Regression Checklist

Learning objective. Assemble the chapter into a runnable performance regression checklist: hold the network and workload constant with emulation, run a transfer, extract the golden signals from the qlog, and gate each against a threshold, the shape of a CI test that catches a performance regression before it ships.

The idea: a gate per golden signal #

A functional test asks "is it correct?"; a regression test asks "is it as fast as before?" The trap is that performance is noisy, so a naive "measure the time" test flaps. The fix combines two tools from this chapter: network emulation (§11.4) to hold the path constant, and the golden signals (§11.2) as the things measured. Each signal becomes a gate with a threshold; the release passes only if every gate passes.

Running it #

The lab pins a 50 ms emulated RTT (the delay proxy from §8.5), transfers 512 KB over HTTP/3, extracts the golden signals from the client qlog, and gates each:

scenario: emulated RTT 50 ms, clean path, 512 KB transfer

golden signal                                 measured   gate
------------------------------------------------------------------
min_rtt tracks the path                          51 ms   PASS
queueing delay (srtt - min_rtt) low               3 ms   PASS
no packet loss on a clean path                0 events   PASS
goodput above floor                           1.9 MB/s   PASS
TTFB within budget                               53 ms   PASS

check: all regression gates PASS — release is within budget

Reading the checklist #

Each gate maps a golden signal (§11.2) to a threshold, and the reasoning behind the threshold is what makes it a good gate rather than an arbitrary number:

• min_rtt tracks the path (51 ms ≈ the emulated 50 ms). This is a sanity gate: if min_rtt drifts far from the emulated RTT, the scenario itself is broken (the emulator misconfigured, the wrong path) and the other numbers are meaningless. Validate the harness before trusting it.
• Queueing delay low (smoothed_rtt − min_rtt = 3 ms). The single most useful derived signal (§11.2): near-zero on a clean path means no bufferbloat. A regression that over-buffers (§9.4) would inflate this while min_rtt stayed put: the gate catches added latency the average RTT would hide.
• No loss on a clean path (0 events). The emulated path drops nothing, so any loss is a bug in the sender (over-driving the path, a pacing regression). On a lossy scenario this gate would instead assert loss stays below an expected rate.
• Goodput above a floor (1.9 MB/s). Application throughput excluding overhead (§11.2); the floor is set from the scenario's BDP and window sizing (§8.4). A drop here with the path held constant means a throughput regression: the headline signal.
• TTFB within budget (53 ms ≈ 1 RTT). End-to-end first-byte latency (§8.5); on a warm connection it should be about one RTT. A jump signals a handshake or first-byte regression.

Every gate passes, so the checklist reports the release is within budget. The power is in the negative case: because the network and workload are held constant, a failing gate points at the code, and at which signal regressed, which localizes the fix (queueing → buffering, goodput → congestion/flow control, TTFB → handshake path).

Turning it into a real gate #

The lab is the skeleton; a production regression gate adds a few things:

• Multiple scenarios. Run the checklist across the named scenarios of §11.4 (lan, mobile, long-fat-network, lossy-satellite) because a change can regress one path and not another (a congestion tweak that helps a LAN can hurt satellite).
• Percentiles, not single runs. Run each scenario N times and gate on p50 and p99 (§11.2); timing noise means a single run flaps, but a tail percentile over many runs is stable and catches the regressions that hurt the worst-off users.
• Baselines, not absolutes. Gate against the previous release's numbers with a tolerance ("goodput ≥ 95% of baseline") rather than fixed constants, so the thresholds track the code instead of going stale.
• Artifacts on failure. Emit the qlog (and optionally a keylog + pcap, §11.3) for every run, so a failed gate hands the engineer the trace to debug rather than just a red build.

Worked example: catching a congestion regression #

Suppose a release changes the congestion controller to ramp more aggressively. Functional tests pass: data still transfers correctly. But the checklist, run on the lossy-satellite scenario, fails two gates: goodput drops below its baseline floor, and loss rises above the scenario's expected rate, while min_rtt still matches (so the harness is sound). Two signals moving together (lower goodput, higher loss, same path) localize it immediately to the recovery/congestion path, exactly the diff that changed. The regression is caught in CI, on an emulated satellite, before a single real satellite user sees it, which is the entire point of the checklist.

Takeaways #

A performance regression checklist holds the network (emulation, §11.4) and workload constant, runs a transfer, extracts the golden signals (§11.2) from the qlog, and gates each against a threshold, passing only if all pass. Real gates use multiple scenarios, percentiles over many runs, baseline-relative thresholds, and per-run trace artifacts, with queueing delay the highest-value signal and a harness-sanity check to keep the measurement honest. That completes the observability chapter and the performance story; the chapter quiz checks the model, and Chapter 12 turns to MASQUE — proxying UDP and IP inside HTTP/3 — before Chapter 13 closes the book with implementation and interoperability guidance.