loadsmith

Scenario-based HTTP load testing that tells the truth about latency. You describe realistic multi-step user journeys in YAML, drive them with an open (arrival-rate) or closed (virtual users) model, and get percentiles that are corrected for coordinated omission: every request is timed both from when it was actually sent and from when it should have been sent. SLOs turn the run into a pass/fail exit code for CI.

Features

• YAML scenarios: variables, CSV feeders (circular/random/unique), templated URLs, headers and JSON bodies ({{ user.email }}, {{ random.uuid }}, {{ prev.json.id }}), extraction via JSONPath subset / regex / header, checks (status, JSON path, body contains, latency), think time (constant/uniform/exponential), weighted scenario mix, before/after hooks (log in once per virtual user).
• Load models: open (Poisson or constant arrivals; recommended) and closed, ramp stages, concurrency cap.
• Engine: asyncio + httpx with keep-alive pools (optional HTTP/2), several worker processes with merged results, intended-start scheduling.
• Statistics: own HDR-style histogram (configurable significant digits, mergeable), p50/p90/p95/p99/p99.9/max per step, per scenario and per second; throughput; errors by type.
• SLOs (p95 < 300ms, error_rate < 1%, rps > 200) with exit code.
• Live terminal dashboard, JSON results, self-contained HTML report (inline SVG), compare, record --har.

Install

uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'     # add ',http2' for HTTP/2
source .venv/bin/activate

Quickstart

python examples/target_server.py --port 8146 &        # demo API that stalls for 2s out of every 15s
loadsmith run examples/shop.yaml --html report.html -o results.json

Exit code: 0 all SLOs pass, 1 an SLO failed, 2 invalid scenario/arguments, 3 runtime failure.

Usage

Run

loadsmith run SCENARIO [-w WORKERS] [--rate N | --users N] [--duration 15s] [--max-concurrency N]
                       [--base-url URL] [--slo 'p99 < 500ms']... [--seed N] [-o results.json] [--html report.html]
                       [--no-live]

Real output of loadsmith run examples/shop.yaml --no-live --seed 1 against the example server (the stall falls inside the run):

Running shop: open model, 15s, 1 worker(s), target http://127.0.0.1:8146

shop  1294 requests in 15s (86.3 req/s), 1 errors (0.08%), 495 journeys completed, 0 dropped
                                  Latency (ms)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃                           ┃    p50 ┃    p90 ┃    p95 ┃   p99 ┃ p99.9 ┃   max ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━┩
│ corrected (from intended  │ 32.6ms │  143ms │  1.48s │ 1.89s │ 2.03s │ 2.05s │
│ start)                    │        │        │        │       │       │       │
│ service (from actual      │ 29.9ms │ 65.0ms │ 79.1ms │  162ms │ 2.03s │ 2.05s │
│ send)                     │        │        │        │       │       │       │
└───────────────────────────┴────────┴────────┴────────┴───────┴───────┴───────┘
                                     Steps
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃ scenario / step          ┃ reqs ┃ errs ┃    p50 ┃    p95 ┃    p99 ┃ p99 svc ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ browse / list products   │  343 │    0 │ 17.7ms │ 31.2ms │  114ms │   114ms │
│ browse / login           │  343 │    0 │ 66.9ms │  1.85s │  1.90s │   215ms │
│ purchase / add to cart   │  152 │    0 │ 22.7ms │ 40.5ms │ 58.5ms │  58.5ms │
│ purchase / checkout      │  152 │    1 │ 62.9ms │  112ms │  280ms │   280ms │
│ purchase / list products │  152 │    0 │ 16.9ms │ 28.7ms │ 42.2ms │  42.2ms │
│ purchase / login         │  152 │    0 │ 60.8ms │  1.83s │  1.95s │   1.95s │
└──────────────────────────┴──────┴──────┴────────┴────────┴────────┴─────────┘
Errors by type: check=1
  latency >= 400ms: 1
                  SLOs
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ objective           ┃ actual ┃ result ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ p99 < 500ms         │  1.89s │ FAIL   │
│ p99:service < 500ms │  162ms │ PASS   │
│ error_rate < 2%     │  0.08% │ PASS   │
│ rps > 80            │   86.3 │ PASS   │
└─────────────────────┴────────┴────────┘
SLO check failed

The naive p99 is 162 ms; the corrected p99 is 1.89 s. That gap is the stall that a closed-loop tool would never show you. See docs/coordinated-omission.md. The matching reports are in examples/sample-report.html and examples/sample-results.json.

When stdout is a terminal, a live dashboard shows elapsed time, active VUs, RPS, total requests/errors, corrected and service percentiles and sparklines of RPS and errors. --no-live (or a non-TTY) prints a progress line every 5 s instead.

Compare two runs

loadsmith compare a.json b.json [--max-regression 20]

Prints each metric for A and B with absolute and percentage change (red = worse, green = better) and a per-step table of corrected p95/p99. --max-regression PCT exits 1 if overall corrected p95 or p99 grew by more than PCT percent.

Record a scenario from a browser HAR

loadsmith record --har examples/sample.har [-o scenario.yaml] [--base-url URL] [--include-static]

Static assets and other hosts are dropped; cookies, user-agent and similar headers are removed; JSON bodies become json:; gaps between requests become think:; recorded 2xx/3xx statuses become status checks. Output (abridged):

name: recorded-journey
base_url: http://127.0.0.1:8146
load: {model: open, rate: 2, duration: 30s, max_concurrency: 50}
steps:
- name: POST /login
  method: POST
  url: /login
  json: {email: alice@example.com, password: secret}
  checks: [{status: 200}]
  think: 1.8
- name: GET /products
  ...
slos: [p95 < 500ms, error_rate < 1%]

Review the result: replace hard-coded values with templates and add extract:.

Scenario file reference

name: shop
base_url: http://127.0.0.1:8146
timeout: 10s                 # per request
http2: false
headers: {X-Env: load}       # default headers
variables: {qty: 2}          # available as {{ qty }}
precision: 3                 # histogram significant digits (1-5)

feeders:                     # CSV with header row; row bound to each virtual user
  user: {file: users.csv, mode: circular}   # circular | random | unique

load:
  model: open                # open (default) | closed
  stages:                    # linear ramp from the previous target; first stage starts at `start` (default 0)
    - {duration: 2s, target: 40}     # open: arrivals/s, closed: virtual users
    - {duration: 12s, target: 40}
    - {duration: 1s, target: 0}
  # shorthand instead of stages:  rate: 40 / users: 10  +  duration: 15s
  max_concurrency: 5         # cap on active journeys (open model)
  arrivals: poisson          # poisson | constant
  pace: 500ms                # closed model only: fixed iteration schedule (enables correction)
  grace: 10s                 # time allowed to finish after the schedule ends

slos: [p99 < 500ms, error_rate < 2%, rps > 80]

scenarios:                   # or put before/steps/after at the top level for a single scenario
  - name: purchase
    weight: 3
    before: [...]            # once per virtual user
    steps:
      - name: add to cart
        method: POST
        url: /cart
        headers: {Authorization: "Bearer {{ token }}"}
        json: {product_id: "{{ product_id }}", qty: "{{ qty }}"}   # or body: "raw text"
        extract:
          product_id: "$.items[3].id"                  # JSONPath shorthand
          sess: {header: X-Session}
          oid: {regex: 'order=(\w+)', from: body}      # from: body | a header name
        checks:
          - status: 200                                # or a list: [200, 201]
          - {json: $.items, equals: 1}                 # also: exists: true|false, contains: x
          - {body_contains: order_id}
          - {latency_lt: 400ms}                        # service latency; bare numbers are ms
        think: {uniform: [100ms, 400ms]}               # 0.5 | constant | uniform | exponential (mean)
    after: [...]

Templates. {{ var }}, {{ user.email }} (feeder row), {{ prev.status }}, {{ prev.json.items[0].id }}, {{ prev.headers.location }}, {{ prev.body }}, {{ vu.id }}, {{ iteration }}, {{ env.NAME }}, {{ timestamp }}, {{ random.uuid }}, {{ random.int(1, 9) }}, {{ random.float(0, 1) }}, {{ random.choice('a', 'b') }}, {{ random.string(8) }}, {{ random.email }}. Filters: | default('x'), | urlencode, | lower, | upper. In json: bodies a value that is a single expression keeps its type ("{{ qty }}" becomes a number).

JSONPath subset. $, .key, ['key'], [n] (negative allowed), [*], .*, ..key.

SLO syntax. metric[:corrected|:service][[step]] op value; metrics p50..p99.9 (any pNN), min, mean, max, error_rate, rps, requests, errors; operators < <= > >= ==. Examples: p95 < 300ms, p99:service < 1s, p95[checkout] < 800ms, error_rate < 1%. Latency SLOs use corrected latency by default.

Semantics worth knowing. In the open model each arrival is one virtual user running before, the steps once, then after. In the closed model each virtual user runs before once, loops the steps until the end, then after. HTTP status >= 400 is an error unless a status check lists it. Transport errors, status errors and failed extractions end the current journey; failed checks do not.

Configuration

Everything is in the scenario file; run options override base_url, rate/users, duration, concurrency, and add SLOs. --workers N spawns N processes, each taking 1/N of the arrival rate and concurrency cap. --seed makes arrival times, mix choices and random.* reproducible per worker.

Architecture

src/loadsmith/
  scenario.py   YAML -> dataclasses, feeders, overrides      engine.py     virtual users, open/closed loops, checks
  template.py   {{ }} rendering                             schedule.py   Poisson / ramp arrival times
  jsonpath.py   JSONPath subset                             runner.py     thread or process workers, aggregation
  hist.py       HDR-style histogram                         slo.py        SLO parse/evaluate
  stats.py      per-step + per-second recorder              results.py    JSON results
  dashboard.py  rich live view                              report_html.py  HTML + SVG
  compare.py    result diffing                              har.py        HAR -> scenario
  cli.py        click commands
examples/       target_server.py (port 8146), shop.yaml, users.csv, sample.har, sample report/results
docs/           coordinated-omission.md, design.md

See docs/design.md for the algorithms (histogram, arrival inversion, error semantics).

Tests

.venv/bin/pytest -q          # 93 tests, ~25 s (includes short end-to-end runs and a 2-process run)

Covers templating and extraction, histogram accuracy against numpy and merging, Poisson arrival statistics, coordinated omission on a simulated stall with a deterministic fake clock, SLOs, HAR conversion, scenario validation, engine semantics, end-to-end runs against the example server, multi-process merge, CLI exit codes and compare.

Limitations

• JSONPath has no filters or slices; no WebSocket, gRPC or HTTP/3.
• Corrected latency starts the clock at a journey's first request; later steps in a journey start when the user is ready.
• latency_lt checks use service latency.
• The load generator's own overhead is included in latency; use --workers beyond a few thousand requests per second per core, and keep an eye on the gap between corrected and service latency.
• Intended for local or staging targets you own. Do not aim it at systems you are not permitted to load test.