Why open models and coordinated omission matter

The problem

A classic load generator runs N virtual users in a loop: send a request, wait for the response, send the next one. When the server stalls, every virtual user is stuck waiting. The tool silently stops sending requests, so the stall is represented by only N slow samples, while the thousands of requests that should have been sent during the stall are never issued and never measured. Percentiles computed from that data look healthy. Gil Tene called this coordinated omission: the measured system and the measuring tool coordinate to omit the bad data.

Real users do not wait politely. If your API freezes for two seconds, new visitors keep arriving at the rate they always did, and each of them experiences the freeze.

What loadsmith does about it

1. Open model (the default). Journeys are started on a schedule of intended start times drawn from a Poisson process (or evenly spaced with arrivals: constant). The schedule does not care whether earlier journeys have finished. Ramp stages are handled as a non-homogeneous Poisson process (see design.md).
2. Two latencies per request. - service latency: from the moment the request was actually sent to the end of the response. This is what a naive tool reports. - corrected latency: from the request's intended start to the end of the response. If the tool itself could not send on time (concurrency cap reached, event loop saturated), the waiting time is charged to the request.
3. Both are stored in separate histograms for every step and every second, and both appear in the console, JSON and HTML outputs. SLOs use corrected latency unless you ask for p99:service.

Where intended start applies

• Open model: the first request of each journey has intended = scheduled arrival time. Later requests in the journey start when the previous response (plus think time) is done, which is what a real user does.
• Closed model: without pace, there is no schedule, so corrected equals service (honest: a closed loop genuinely cannot observe omission). With load.pace: 500ms each virtual user gets a fixed iteration schedule (iteration n is intended at start + n * pace), and late iterations are charged for lateness.

max_concurrency and unbounded open models

The cap bounds the number of simultaneously active journeys per run (split across workers). Arrivals above the cap wait, and that wait shows up in corrected latency. Without a cap an open model launches every arrival immediately, so a stall is visible in both views (all those requests really were sent and really were slow) at the price of an unbounded number of connections. The cap models a realistic server-side or client-side limit; pick one close to what your production tier allows.

Worked example

examples/target_server.py freezes every request for 2 seconds out of each 15. examples/shop.yaml offers 40 journeys/s for 15 seconds with at most 5 active journeys, so one stall window falls inside the run.

What happens at the stall:

• 5 journeys are in flight and blocked: 5 requests with ~2s service latency. That is 5 of ~1300 requests, 0.4%, below the 1% needed to move p99.
• Meanwhile ~80 journeys (2s x 40/s) arrive on schedule but cannot start because the cap is full. When the stall ends they start immediately, with ~10 ms service latency each, but their intended start was up to 2 seconds ago.

Result of a real run (results/shop.json, also in examples/sample-results.json):

The naive p99 says "162 ms, fine"; the SLO p99 < 500ms on corrected latency fails with 1.89 s, which is what the people who arrived during those two seconds actually saw. Per step, login (the first request of every journey) carries the damage: p95 1.85 s corrected versus a service p99 of 215 ms.

The deterministic version of this experiment is tests/test_coordinated_omission.py: a fake clock and a fake transport with a stall window, run through the real engine, asserting service p99 < 50 ms and corrected p99 > 1.5 s.

Practical advice

• Prefer model: open with a rate close to production traffic; use closed only to model a fixed pool of clients (workers, bots) or add pace.
• Always look at both rows. Corrected much larger than service means the system, or the load generator, could not keep up with the schedule. Check the generator is not the bottleneck (use --workers) before blaming the server.
• Keep dropped at 0: journeys still unfinished after load.grace seconds (default 10 s) at the end are cancelled and counted.