# benchdiff

Is my branch actually faster, or is it noise? benchdiff runs your benchmarks for two or more git refs in isolated
worktrees, interleaves the runs to cancel machine drift, and reports each change with a bootstrap confidence interval,
a significance test and an effect size. Results are stored per commit so you can see trends, find the commit that
introduced a regression, and gate CI on it.

## Features

- Benchmarks as shell commands (`benchdiff.toml`) or as Python functions discovered in `benchmarks/*.py` with `@bench` (parameter grids, optional GC disabling).
- `benchdiff run REF [REF...]`: one `git worktree` per ref, optional per-ref venv and setup command, automatic cleanup.
- Interleaved ABBA scheduling, discarded warmups, adaptive run count until the CI is tight (or a time/run cap).
- Wall time, CPU time and peak RSS; environment metadata and noise warnings (load, CPU governor).
- Tukey outlier detection (reported, never silently dropped), median/mean, BCa bootstrap CI of the ratio of medians, Mann-Whitney U, Cliff's delta, Holm-Bonferroni across benchmarks.
- Terminal table with unicode box plots, Markdown (PR comments), JSON, and self-contained HTML with SVG violin/box plots.
- CI gate: `--fail-on-regression 5%` sets the exit status.
- History in `.benchdiff/results.sqlite`: `benchdiff history` with sparkline and change-point detection.
- `benchdiff bisect`: automated bisect driven by the statistical verdict.
- `benchdiff compare a.json b.json`: compare saved results without rerunning.

## Install

Python 3.12+.

```bash
cd benchdiff
uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'
export PATH="$PWD/.venv/bin:$PATH"
benchdiff --version
```

## Quickstart

Build the demo repository (6 commits; commit 4 makes `word_freq` about 30% slower, commit 5 makes `normalize` much faster):

```bash
examples/demo_repo_builder.sh /tmp/benchdiff-demo
cd /tmp/benchdiff-demo
cat benchdiff.toml

# 1. compare the first commit with HEAD
benchdiff run HEAD~5 HEAD

# 2. measure every commit in one interleaved run, then look at the trend
benchdiff run $(git rev-list --reverse main)
benchdiff history 'word_freq_bench[words=20000]'

# 3. find the culprit automatically
benchdiff bisect 'word_freq_bench[words=20000]' HEAD~5 HEAD --threshold 5%
```

## Usage

### Defining benchmarks

Shell commands, in `benchdiff.toml` (run with the worktree as working directory):

```toml
[settings]
threshold = "5%"
min_runs = 12
max_runs = 60

[setup]                       # optional, once per ref, inside its worktree
cmd = "pip install -e ."
venv = true                   # give every ref its own venv (default false)

[bench.cli_count]
cmd = "python -m textkit data/corpus.txt"
setup = "true"                # optional per-benchmark setup/teardown
teardown = "true"
warmup = 2                    # any [settings] key can be overridden per benchmark
```

Python functions, in `benchmarks/*.py` of each ref:

```python
from benchdiff import bench

@bench(params={"words": [5000, 20000]})      # -> word_freq_bench[words=5000], ...
def word_freq_bench(words):
    word_freq(TEXTS[words])

@bench(disable_gc=True, warmup=3)
def normalize_bench():
    normalize(TEXTS[20000])
```

Each Python sample runs in a fresh subprocess using `time.perf_counter_ns`. Benchmarks missing in some ref are skipped with a warning.
In every command `python` resolves to the interpreter running benchdiff (or the per-ref venv) and these variables are set:
`BENCHDIFF_REF`, `BENCHDIFF_SHA`, `BENCHDIFF_WORKTREE`, `BENCHDIFF_PYTHON`. `PYTHONPATH` gets the worktree
(configurable with `[python] pythonpath = ["src"]`).

### `benchdiff run`

The first ref is the baseline. Output from the demo:

```
benchdiff (wall time, base: HEAD~5, threshold 5.0%)                                                                                                                         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Benchmark                    ┃ Ref    ┃           Median ┃ Change ┃           95% CI ┃ p (Holm) ┃          Cliff d ┃      RSS ┃ Verdict   ┃ Distribution                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ cli_count                    │ HEAD~5 │ 18.5 ms +/- 0.9% │        │                  │          │                  │ 98.7 MiB │ baseline  │                 ├─▒┃▒─┤ •  • │
│                              │ HEAD   │ 12.3 ms +/- 1.4% │ -33.8% │ [-34.9%, -32.6%] │   <0.001 │      -1.00 large │ 98.7 MiB │ faster    │ ├─▒┃▒─┤••                    │
│ word_freq_bench[words=20000] │ HEAD~5 │ 1.15 ms +/- 0.6% │        │                  │          │                  │ 98.9 MiB │ baseline  │ ├┃▒┤         •               │
│                              │ HEAD   │ 1.55 ms +/- 1.5% │ +34.9% │ [+32.1%, +36.7%] │   <0.001 │      +1.00 large │ 98.9 MiB │ slower    │             ├─▒┃▒▒───┤ •   • │
│ normalize_bench              │ HEAD~5 │  7.4 ms +/- 1.6% │        │                  │          │                  │ 98.9 MiB │ baseline  │                        ├┃▒─┤ │
│                              │ HEAD   │  189 us +/- 7.5% │ -97.4% │ [-97.7%, -97.2%] │   <0.001 │      -1.00 large │ 98.9 MiB │ faster    │ ┃•                           │
│ checksum_bench               │ HEAD~5 │ 6.35 ms +/- 0.4% │        │                  │          │                  │ 99.0 MiB │ baseline  │ ├┃▒─┤ •••                  • │
│                              │ HEAD   │ 6.36 ms +/- 1.0% │  +0.1% │   [-0.9%, +1.5%] │    0.815 │ -0.03 negligible │ 99.0 MiB │ no change │ ├┃▒──•• •                    │
└──────────────────────────────┴────────┴──────────────────┴────────┴──────────────────┴──────────┴──────────────────┴──────────┴───────────┴──────────────────────────────┘
note: outliers in cli_count (kept in the statistics): HEAD~5: 3/60, HEAD: 2/60
note: cli_count: stopped at max_runs before reaching the CI target
note: outliers in word_freq_bench[words=20000] (kept in the statistics): HEAD~5: 1/60 (1 severe), HEAD: 3/60 (1 severe)
note: word_freq_bench[words=20000]: stopped at max_runs before reaching the CI target
note: outliers in normalize_bench (kept in the statistics): HEAD: 1/60
note: normalize_bench: stopped at max_runs before reaching the CI target
note: outliers in checksum_bench (kept in the statistics): HEAD~5: 5/60 (3 severe), HEAD: 3/60
note: checksum_bench: stopped at max_runs before reaching the CI target
warning: CPU governor is 'powersave': frequency scaling can add noise (use 'performance' for stable runs)
AMD Ryzen 5 7600X 6-Core Processor x12, python 3.12.12, load 1.62, governor powersave
```

Useful options: `-f table|markdown|json|html`, `-o FILE`, `-b NAME_OR_GLOB` (repeatable), `--threshold 2%`,
`--fail-on-regression 5%`, `--metric wall|cpu`, `--min-runs`, `--max-runs`, `--min-time`, `--max-time`,
`--target-ci 1%`, `--warmup`, `--seed`, `--keep-worktrees`, `--no-store`, `-C REPO`, `-c CONFIG`.

CI gate (exit status 1 when a benchmark is significantly slower by at least 5%, 0 otherwise, 2 on errors):

```
$ benchdiff run HEAD~5 HEAD -b 'word_freq_bench*' --fail-on-regression 5%; echo $?
regression: word_freq_bench[words=5000] HEAD +32.4% (limit 5.0%)
regression: word_freq_bench[words=20000] HEAD +33.1% (limit 5.0%)
1
```

An HTML report with SVG violin/box plots from the same demo is in [`examples/sample-report.html`](examples/sample-report.html).

### `benchdiff compare`

Save results with `-f json` and compare later (no rerun; each file contributes its last ref unless `--ref-a/--ref-b` is given):

```
$ benchdiff run HEAD~5 -b cli_count -f json -o a.json
$ benchdiff run HEAD -b cli_count -f json -o b.json
$ benchdiff compare a.json b.json -f markdown
### benchdiff: `HEAD~5`, `HEAD`

wall time, base `HEAD~5`, threshold 2.0%, Holm-adjusted p-values.

| Benchmark | Ref | Median | Change | 95% CI | p (Holm) | Cliff's d | Verdict |
|---|---|---:|---:|---:|---:|---:|---|
| cli_count | HEAD~5 | 18.9 ms | | | | | baseline |
| cli_count | HEAD | 11.8 ms | -37.6% | [-38.9%, -36.7%] | <0.001 | -1.00 (large) | **faster** |

- outliers in `cli_count` (kept): HEAD~5: 4/60, HEAD: 3/60
- warning: environments may differ between a.json and b.json; runs were not interleaved
```

Runs that were not interleaved carry a warning, because drift between them is not cancelled.

### `benchdiff history`

```
$ benchdiff history 'word_freq_bench[words=20000]'
word_freq_bench[words=20000] (wall time, 6 commits)  ▁▁▁█▇▇
commit    date            median   vs prev  subject
eb507ac3  2026-01-01     1.16 ms            feat: initial textkit with word_freq, normalize an
bdd886d9  2026-01-02     1.16 ms     +0.4%  feat: add top_words helper
cdf6998e  2026-01-03     1.15 ms     -1.0%  docs: add README
6a324704  2026-01-04     1.62 ms    +40.5%  fix: count words case-insensitively  <-- change point +37.4%
f9f42e2b  2026-01-05      1.6 ms     -1.1%  perf: normalize with str.translate
d8fa0174  2026-01-06     1.55 ms     -3.0%  docs: document the command line
```

Change points come from binary segmentation (see [docs/statistics.md](docs/statistics.md)); `--min-change` sets the smallest level shift that counts.

### `benchdiff bisect`

```
$ benchdiff bisect 'word_freq_bench[words=20000]' HEAD~5 HEAD --threshold 5%
verifying that HEAD is slower than HEAD~5
preparing HEAD~5 (eb507ac3)
preparing d8fa0174 (d8fa0174)
benchmarking word_freq_bench[words=20000]
  60 samples per ref (max_runs)
  d8fa0174 +34.1% -> slower
preparing HEAD~5 (eb507ac3)
preparing cdf6998e (cdf6998e)
benchmarking word_freq_bench[words=20000]
  60 samples per ref (max_runs)
  cdf6998e +0.2% -> no change
preparing HEAD~5 (eb507ac3)
preparing 6a324704 (6a324704)
benchmarking word_freq_bench[words=20000]
  60 samples per ref (max_runs)
  6a324704 +34.7% -> slower
first bad commit: 6a324704e75e  fix: count words case-insensitively
word_freq_bench is +34.1% slower at HEAD than at HEAD~5; probes:
  d8fa0174    +34.1%  slower
  cdf6998e     +0.2%  no change
  6a324704    +34.7%  slower
```

Every probe is measured interleaved with the good commit. Your checkout and `git bisect` state are never touched.

## Configuration

`benchdiff.toml` in the repo (or `-c FILE`). All percentages accept `"2%"`, `"2"` or `2`.

| Key | Default | Meaning |
|---|---|---|
| `[settings] warmup` | 1 | discarded runs per ref |
| `min_runs` / `max_runs` | 10 / 200 | samples per ref |
| `min_time` / `max_time` | 1 / 30 | seconds spent per benchmark |
| `target_ci` | 1% | stop when the median's CI half-width is below this |
| `threshold` | 2% | smallest change reported as faster/slower |
| `metric` | wall | `wall` or `cpu` |
| `repeat` | 3 | timed calls per subprocess for Python benchmarks |
| `[setup] cmd`, `venv` | none, false | per-ref setup command; per-ref venv |
| `[python] dir`, `pythonpath`, `disable_gc` | `benchmarks`, `["."]`, false | Python benchmark discovery |
| `[bench.NAME] cmd`, `setup`, `teardown` | | a shell benchmark (plus any `[settings]` key) |

Precedence: command line, then `[bench.NAME]`, then `[settings]`.

## Architecture

| Module | Role |
|---|---|
| `cli.py` | click commands: `run`, `compare`, `history`, `bisect`; exit codes |
| `config.py` | `benchdiff.toml` parsing, settings merge |
| `engine.py` | worktree preparation, ABBA scheduling, sampling loop, `run_refs` |
| `pyrunner.py`, `registry.py` | `@bench` decorator and the per-ref subprocess that discovers and times it |
| `gitutil.py` | ref resolution, `Workspace` (temporary worktrees), commit ranges |
| `stats.py` | outliers, BCa/percentile bootstrap, Mann-Whitney, Cliff's delta, Holm, verdict |
| `analysis.py` | raw samples -> summaries, comparisons, verdicts |
| `report.py`, `html_report.py`, `sketch.py` | terminal/Markdown/JSON, HTML+SVG, box plot and sparkline text |
| `store.py`, `history.py` | SQLite results, change-point detection |
| `bisect.py` | verdict-driven binary search over a commit range |
| `envinfo.py` | CPU/governor/load metadata and noise warnings |

## Tests

```bash
.venv/bin/pytest -q                    # everything (56 tests, ~20 s)
.venv/bin/pytest -q -m "not slow"      # skip the end-to-end tests that benchmark the demo repo
```

The slow tests build the demo repo, run real benchmarks, check the verdicts, the history change point, the CI exit code and that bisect finds commit 4.

## Limitations

- Timing noise cannot be removed, only measured. On a loaded laptop with the `powersave` governor, expect wide intervals and `max_runs` stops; the report says so.
- The config file is read from the repository you run in, not from each ref, so a ref that lacks a command fails with that command's stderr.
- History compares medians from different runs (different machine conditions); use `run`/`bisect` to confirm a suspected change point.
- Peak RSS for Python benchmarks is the whole subprocess, including interpreter and imports.
- Bisect assumes a regression that persists once introduced. Linux is the supported platform (`wait4` rusage, `/proc`, `/sys` metadata).