# loadsmith

Scenario-based HTTP load testing that tells the truth about latency. You describe realistic multi-step user journeys in
YAML, drive them with an open (arrival-rate) or closed (virtual users) model, and get percentiles that are corrected for
**coordinated omission**: every request is timed both from when it was actually sent and from when it *should* have been
sent. SLOs turn the run into a pass/fail exit code for CI.

## Features

- YAML scenarios: variables, CSV feeders (`circular`/`random`/`unique`), templated URLs, headers and JSON bodies
  (`{{ user.email }}`, `{{ random.uuid }}`, `{{ prev.json.id }}`), extraction via JSONPath subset / regex / header,
  checks (status, JSON path, body contains, latency), think time (constant/uniform/exponential), weighted scenario
  mix, `before`/`after` hooks (log in once per virtual user).
- Load models: open (Poisson or constant arrivals; recommended) and closed, ramp stages, concurrency cap.
- Engine: asyncio + httpx with keep-alive pools (optional HTTP/2), several worker processes with merged results,
  intended-start scheduling.
- Statistics: own HDR-style histogram (configurable significant digits, mergeable), p50/p90/p95/p99/p99.9/max per step,
  per scenario and per second; throughput; errors by type.
- SLOs (`p95 < 300ms`, `error_rate < 1%`, `rps > 200`) with exit code.
- Live terminal dashboard, JSON results, self-contained HTML report (inline SVG), `compare`, `record --har`.

## Install

```bash
uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'     # add ',http2' for HTTP/2
source .venv/bin/activate
```

## Quickstart

```bash
python examples/target_server.py --port 8146 &        # demo API that stalls for 2s out of every 15s
loadsmith run examples/shop.yaml --html report.html -o results.json
```

Exit code: `0` all SLOs pass, `1` an SLO failed, `2` invalid scenario/arguments, `3` runtime failure.

## Usage

### Run

```
loadsmith run SCENARIO [-w WORKERS] [--rate N | --users N] [--duration 15s] [--max-concurrency N]
                       [--base-url URL] [--slo 'p99 < 500ms']... [--seed N] [-o results.json] [--html report.html]
                       [--no-live]
```

Real output of `loadsmith run examples/shop.yaml --no-live --seed 1` against the example server (the stall falls inside
the run):

```
Running shop: open model, 15s, 1 worker(s), target http://127.0.0.1:8146

shop  1294 requests in 15s (86.3 req/s), 1 errors (0.08%), 495 journeys completed, 0 dropped
                                  Latency (ms)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃                           ┃    p50 ┃    p90 ┃    p95 ┃   p99 ┃ p99.9 ┃   max ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━┩
│ corrected (from intended  │ 32.6ms │  143ms │  1.48s │ 1.89s │ 2.03s │ 2.05s │
│ start)                    │        │        │        │       │       │       │
│ service (from actual      │ 29.9ms │ 65.0ms │ 79.1ms │  162ms │ 2.03s │ 2.05s │
│ send)                     │        │        │        │       │       │       │
└───────────────────────────┴────────┴────────┴────────┴───────┴───────┴───────┘
                                     Steps
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃ scenario / step          ┃ reqs ┃ errs ┃    p50 ┃    p95 ┃    p99 ┃ p99 svc ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ browse / list products   │  343 │    0 │ 17.7ms │ 31.2ms │  114ms │   114ms │
│ browse / login           │  343 │    0 │ 66.9ms │  1.85s │  1.90s │   215ms │
│ purchase / add to cart   │  152 │    0 │ 22.7ms │ 40.5ms │ 58.5ms │  58.5ms │
│ purchase / checkout      │  152 │    1 │ 62.9ms │  112ms │  280ms │   280ms │
│ purchase / list products │  152 │    0 │ 16.9ms │ 28.7ms │ 42.2ms │  42.2ms │
│ purchase / login         │  152 │    0 │ 60.8ms │  1.83s │  1.95s │   1.95s │
└──────────────────────────┴──────┴──────┴────────┴────────┴────────┴─────────┘
Errors by type: check=1
  latency >= 400ms: 1
                  SLOs
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ objective           ┃ actual ┃ result ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ p99 < 500ms         │  1.89s │ FAIL   │
│ p99:service < 500ms │  162ms │ PASS   │
│ error_rate < 2%     │  0.08% │ PASS   │
│ rps > 80            │   86.3 │ PASS   │
└─────────────────────┴────────┴────────┘
SLO check failed
```

The naive p99 is 162 ms; the corrected p99 is 1.89 s. That gap is the stall that a closed-loop tool would never show you.
See [docs/coordinated-omission.md](docs/coordinated-omission.md). The matching reports are in
[`examples/sample-report.html`](examples/sample-report.html) and `examples/sample-results.json`.

When stdout is a terminal, a live dashboard shows elapsed time, active VUs, RPS, total requests/errors, corrected and
service percentiles and sparklines of RPS and errors. `--no-live` (or a non-TTY) prints a progress line every 5 s instead.

### Compare two runs

```
loadsmith compare a.json b.json [--max-regression 20]
```

Prints each metric for A and B with absolute and percentage change (red = worse, green = better) and a per-step table
of corrected p95/p99. `--max-regression PCT` exits `1` if overall corrected p95 or p99 grew by more than PCT percent.

### Record a scenario from a browser HAR

```
loadsmith record --har examples/sample.har [-o scenario.yaml] [--base-url URL] [--include-static]
```

Static assets and other hosts are dropped; cookies, user-agent and similar headers are removed; JSON bodies become `json:`;
gaps between requests become `think:`; recorded 2xx/3xx statuses become `status` checks. Output (abridged):

```yaml
name: recorded-journey
base_url: http://127.0.0.1:8146
load: {model: open, rate: 2, duration: 30s, max_concurrency: 50}
steps:
- name: POST /login
  method: POST
  url: /login
  json: {email: alice@example.com, password: secret}
  checks: [{status: 200}]
  think: 1.8
- name: GET /products
  ...
slos: [p95 < 500ms, error_rate < 1%]
```

Review the result: replace hard-coded values with templates and add `extract:`.

## Scenario file reference

```yaml
name: shop
base_url: http://127.0.0.1:8146
timeout: 10s                 # per request
http2: false
headers: {X-Env: load}       # default headers
variables: {qty: 2}          # available as {{ qty }}
precision: 3                 # histogram significant digits (1-5)

feeders:                     # CSV with header row; row bound to each virtual user
  user: {file: users.csv, mode: circular}   # circular | random | unique

load:
  model: open                # open (default) | closed
  stages:                    # linear ramp from the previous target; first stage starts at `start` (default 0)
    - {duration: 2s, target: 40}     # open: arrivals/s, closed: virtual users
    - {duration: 12s, target: 40}
    - {duration: 1s, target: 0}
  # shorthand instead of stages:  rate: 40 / users: 10  +  duration: 15s
  max_concurrency: 5         # cap on active journeys (open model)
  arrivals: poisson          # poisson | constant
  pace: 500ms                # closed model only: fixed iteration schedule (enables correction)
  grace: 10s                 # time allowed to finish after the schedule ends

slos: [p99 < 500ms, error_rate < 2%, rps > 80]

scenarios:                   # or put before/steps/after at the top level for a single scenario
  - name: purchase
    weight: 3
    before: [...]            # once per virtual user
    steps:
      - name: add to cart
        method: POST
        url: /cart
        headers: {Authorization: "Bearer {{ token }}"}
        json: {product_id: "{{ product_id }}", qty: "{{ qty }}"}   # or body: "raw text"
        extract:
          product_id: "$.items[3].id"                  # JSONPath shorthand
          sess: {header: X-Session}
          oid: {regex: 'order=(\w+)', from: body}      # from: body | a header name
        checks:
          - status: 200                                # or a list: [200, 201]
          - {json: $.items, equals: 1}                 # also: exists: true|false, contains: x
          - {body_contains: order_id}
          - {latency_lt: 400ms}                        # service latency; bare numbers are ms
        think: {uniform: [100ms, 400ms]}               # 0.5 | constant | uniform | exponential (mean)
    after: [...]
```

**Templates.** `{{ var }}`, `{{ user.email }}` (feeder row), `{{ prev.status }}`, `{{ prev.json.items[0].id }}`,
`{{ prev.headers.location }}`, `{{ prev.body }}`, `{{ vu.id }}`, `{{ iteration }}`, `{{ env.NAME }}`, `{{ timestamp }}`,
`{{ random.uuid }}`, `{{ random.int(1, 9) }}`, `{{ random.float(0, 1) }}`, `{{ random.choice('a', 'b') }}`,
`{{ random.string(8) }}`, `{{ random.email }}`. Filters: `| default('x')`, `| urlencode`, `| lower`, `| upper`.
In `json:` bodies a value that is a single expression keeps its type (`"{{ qty }}"` becomes a number).

**JSONPath subset.** `$`, `.key`, `['key']`, `[n]` (negative allowed), `[*]`, `.*`, `..key`.

**SLO syntax.** `metric[:corrected|:service][[step]] op value`; metrics `p50`..`p99.9` (any `pNN`), `min`, `mean`, `max`,
`error_rate`, `rps`, `requests`, `errors`; operators `< <= > >= ==`. Examples: `p95 < 300ms`, `p99:service < 1s`,
`p95[checkout] < 800ms`, `error_rate < 1%`. Latency SLOs use corrected latency by default.

**Semantics worth knowing.** In the open model each arrival is one virtual user running `before`, the steps once, then
`after`. In the closed model each virtual user runs `before` once, loops the steps until the end, then `after`. HTTP
status >= 400 is an error unless a `status` check lists it. Transport errors, status errors and failed extractions end
the current journey; failed checks do not.

## Configuration

Everything is in the scenario file; `run` options override `base_url`, rate/users, duration, concurrency, and add SLOs.
`--workers N` spawns N processes, each taking `1/N` of the arrival rate and concurrency cap.
`--seed` makes arrival times, mix choices and `random.*` reproducible per worker.

## Architecture

```
src/loadsmith/
  scenario.py   YAML -> dataclasses, feeders, overrides      engine.py     virtual users, open/closed loops, checks
  template.py   {{ }} rendering                             schedule.py   Poisson / ramp arrival times
  jsonpath.py   JSONPath subset                             runner.py     thread or process workers, aggregation
  hist.py       HDR-style histogram                         slo.py        SLO parse/evaluate
  stats.py      per-step + per-second recorder              results.py    JSON results
  dashboard.py  rich live view                              report_html.py  HTML + SVG
  compare.py    result diffing                              har.py        HAR -> scenario
  cli.py        click commands
examples/       target_server.py (port 8146), shop.yaml, users.csv, sample.har, sample report/results
docs/           coordinated-omission.md, design.md
```

See [docs/design.md](docs/design.md) for the algorithms (histogram, arrival inversion, error semantics).

## Tests

```bash
.venv/bin/pytest -q          # 93 tests, ~25 s (includes short end-to-end runs and a 2-process run)
```

Covers templating and extraction, histogram accuracy against numpy and merging, Poisson arrival statistics, coordinated
omission on a simulated stall with a deterministic fake clock, SLOs, HAR conversion, scenario validation, engine
semantics, end-to-end runs against the example server, multi-process merge, CLI exit codes and `compare`.

## Limitations

- JSONPath has no filters or slices; no WebSocket, gRPC or HTTP/3.
- Corrected latency starts the clock at a journey's first request; later steps in a journey start when the user is ready.
- `latency_lt` checks use service latency.
- The load generator's own overhead is included in latency; use `--workers` beyond a few thousand requests per second
  per core, and keep an eye on the gap between corrected and service latency.
- Intended for local or staging targets you own. Do not aim it at systems you are not permitted to load test.