benchdiff

Is my branch actually faster, or is it noise? benchdiff runs your benchmarks for two or more git refs in isolated worktrees, interleaves the runs to cancel machine drift, and reports each change with a bootstrap confidence interval, a significance test and an effect size. Results are stored per commit so you can see trends, find the commit that introduced a regression, and gate CI on it.

Features

• Benchmarks as shell commands (benchdiff.toml) or as Python functions discovered in benchmarks/*.py with @bench (parameter grids, optional GC disabling).
• benchdiff run REF [REF...]: one git worktree per ref, optional per-ref venv and setup command, automatic cleanup.
• Interleaved ABBA scheduling, discarded warmups, adaptive run count until the CI is tight (or a time/run cap).
• Wall time, CPU time and peak RSS; environment metadata and noise warnings (load, CPU governor).
• Tukey outlier detection (reported, never silently dropped), median/mean, BCa bootstrap CI of the ratio of medians, Mann-Whitney U, Cliff's delta, Holm-Bonferroni across benchmarks.
• Terminal table with unicode box plots, Markdown (PR comments), JSON, and self-contained HTML with SVG violin/box plots.
• CI gate: --fail-on-regression 5% sets the exit status.
• History in .benchdiff/results.sqlite: benchdiff history with sparkline and change-point detection.
• benchdiff bisect: automated bisect driven by the statistical verdict.
• benchdiff compare a.json b.json: compare saved results without rerunning.

Install

Python 3.12+.

cd benchdiff
uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'
export PATH="$PWD/.venv/bin:$PATH"
benchdiff --version

Quickstart

Build the demo repository (6 commits; commit 4 makes word_freq about 30% slower, commit 5 makes normalize much faster):

examples/demo_repo_builder.sh /tmp/benchdiff-demo
cd /tmp/benchdiff-demo
cat benchdiff.toml

# 1. compare the first commit with HEAD
benchdiff run HEAD~5 HEAD

# 2. measure every commit in one interleaved run, then look at the trend
benchdiff run $(git rev-list --reverse main)
benchdiff history 'word_freq_bench[words=20000]'

# 3. find the culprit automatically
benchdiff bisect 'word_freq_bench[words=20000]' HEAD~5 HEAD --threshold 5%

Usage

Defining benchmarks

Shell commands, in benchdiff.toml (run with the worktree as working directory):

[settings]
threshold = "5%"
min_runs = 12
max_runs = 60

[setup]                       # optional, once per ref, inside its worktree
cmd = "pip install -e ."
venv = true                   # give every ref its own venv (default false)

[bench.cli_count]
cmd = "python -m textkit data/corpus.txt"
setup = "true"                # optional per-benchmark setup/teardown
teardown = "true"
warmup = 2                    # any [settings] key can be overridden per benchmark

Python functions, in benchmarks/*.py of each ref:

from benchdiff import bench

@bench(params={"words": [5000, 20000]})      # -> word_freq_bench[words=5000], ...
def word_freq_bench(words):
    word_freq(TEXTS[words])

@bench(disable_gc=True, warmup=3)
def normalize_bench():
    normalize(TEXTS[20000])

Each Python sample runs in a fresh subprocess using time.perf_counter_ns. Benchmarks missing in some ref are skipped with a warning. In every command python resolves to the interpreter running benchdiff (or the per-ref venv) and these variables are set: BENCHDIFF_REF, BENCHDIFF_SHA, BENCHDIFF_WORKTREE, BENCHDIFF_PYTHON. PYTHONPATH gets the worktree (configurable with [python] pythonpath = ["src"]).

benchdiff run

The first ref is the baseline. Output from the demo:

benchdiff (wall time, base: HEAD~5, threshold 5.0%)                                                                                                                         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Benchmark                    ┃ Ref    ┃           Median ┃ Change ┃           95% CI ┃ p (Holm) ┃          Cliff d ┃      RSS ┃ Verdict   ┃ Distribution                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ cli_count                    │ HEAD~5 │ 18.5 ms +/- 0.9% │        │                  │          │                  │ 98.7 MiB │ baseline  │                 ├─▒┃▒─┤ •  • │
│                              │ HEAD   │ 12.3 ms +/- 1.4% │ -33.8% │ [-34.9%, -32.6%] │   <0.001 │      -1.00 large │ 98.7 MiB │ faster    │ ├─▒┃▒─┤••                    │
│ word_freq_bench[words=20000] │ HEAD~5 │ 1.15 ms +/- 0.6% │        │                  │          │                  │ 98.9 MiB │ baseline  │ ├┃▒┤         •               │
│                              │ HEAD   │ 1.55 ms +/- 1.5% │ +34.9% │ [+32.1%, +36.7%] │   <0.001 │      +1.00 large │ 98.9 MiB │ slower    │             ├─▒┃▒▒───┤ •   • │
│ normalize_bench              │ HEAD~5 │  7.4 ms +/- 1.6% │        │                  │          │                  │ 98.9 MiB │ baseline  │                        ├┃▒─┤ │
│                              │ HEAD   │  189 us +/- 7.5% │ -97.4% │ [-97.7%, -97.2%] │   <0.001 │      -1.00 large │ 98.9 MiB │ faster    │ ┃•                           │
│ checksum_bench               │ HEAD~5 │ 6.35 ms +/- 0.4% │        │                  │          │                  │ 99.0 MiB │ baseline  │ ├┃▒─┤ •••                  • │
│                              │ HEAD   │ 6.36 ms +/- 1.0% │  +0.1% │   [-0.9%, +1.5%] │    0.815 │ -0.03 negligible │ 99.0 MiB │ no change │ ├┃▒──•• •                    │
└──────────────────────────────┴────────┴──────────────────┴────────┴──────────────────┴──────────┴──────────────────┴──────────┴───────────┴──────────────────────────────┘
note: outliers in cli_count (kept in the statistics): HEAD~5: 3/60, HEAD: 2/60
note: cli_count: stopped at max_runs before reaching the CI target
note: outliers in word_freq_bench[words=20000] (kept in the statistics): HEAD~5: 1/60 (1 severe), HEAD: 3/60 (1 severe)
note: word_freq_bench[words=20000]: stopped at max_runs before reaching the CI target
note: outliers in normalize_bench (kept in the statistics): HEAD: 1/60
note: normalize_bench: stopped at max_runs before reaching the CI target
note: outliers in checksum_bench (kept in the statistics): HEAD~5: 5/60 (3 severe), HEAD: 3/60
note: checksum_bench: stopped at max_runs before reaching the CI target
warning: CPU governor is 'powersave': frequency scaling can add noise (use 'performance' for stable runs)
AMD Ryzen 5 7600X 6-Core Processor x12, python 3.12.12, load 1.62, governor powersave

Useful options: -f table|markdown|json|html, -o FILE, -b NAME_OR_GLOB (repeatable), --threshold 2%, --fail-on-regression 5%, --metric wall|cpu, --min-runs, --max-runs, --min-time, --max-time, --target-ci 1%, --warmup, --seed, --keep-worktrees, --no-store, -C REPO, -c CONFIG.

CI gate (exit status 1 when a benchmark is significantly slower by at least 5%, 0 otherwise, 2 on errors):

$ benchdiff run HEAD~5 HEAD -b 'word_freq_bench*' --fail-on-regression 5%; echo $?
regression: word_freq_bench[words=5000] HEAD +32.4% (limit 5.0%)
regression: word_freq_bench[words=20000] HEAD +33.1% (limit 5.0%)
1

An HTML report with SVG violin/box plots from the same demo is in examples/sample-report.html.

benchdiff compare

Save results with -f json and compare later (no rerun; each file contributes its last ref unless --ref-a/--ref-b is given):

$ benchdiff run HEAD~5 -b cli_count -f json -o a.json
$ benchdiff run HEAD -b cli_count -f json -o b.json
$ benchdiff compare a.json b.json -f markdown
### benchdiff: `HEAD~5`, `HEAD`

wall time, base `HEAD~5`, threshold 2.0%, Holm-adjusted p-values.

| Benchmark | Ref | Median | Change | 95% CI | p (Holm) | Cliff's d | Verdict |
|---|---|---:|---:|---:|---:|---:|---|
| cli_count | HEAD~5 | 18.9 ms | | | | | baseline |
| cli_count | HEAD | 11.8 ms | -37.6% | [-38.9%, -36.7%] | <0.001 | -1.00 (large) | **faster** |

- outliers in `cli_count` (kept): HEAD~5: 4/60, HEAD: 3/60
- warning: environments may differ between a.json and b.json; runs were not interleaved

Runs that were not interleaved carry a warning, because drift between them is not cancelled.

benchdiff history

$ benchdiff history 'word_freq_bench[words=20000]'
word_freq_bench[words=20000] (wall time, 6 commits)  ▁▁▁█▇▇
commit    date            median   vs prev  subject
eb507ac3  2026-01-01     1.16 ms            feat: initial textkit with word_freq, normalize an
bdd886d9  2026-01-02     1.16 ms     +0.4%  feat: add top_words helper
cdf6998e  2026-01-03     1.15 ms     -1.0%  docs: add README
6a324704  2026-01-04     1.62 ms    +40.5%  fix: count words case-insensitively  <-- change point +37.4%
f9f42e2b  2026-01-05      1.6 ms     -1.1%  perf: normalize with str.translate
d8fa0174  2026-01-06     1.55 ms     -3.0%  docs: document the command line

Change points come from binary segmentation (see docs/statistics.md); --min-change sets the smallest level shift that counts.

benchdiff bisect

$ benchdiff bisect 'word_freq_bench[words=20000]' HEAD~5 HEAD --threshold 5%
verifying that HEAD is slower than HEAD~5
preparing HEAD~5 (eb507ac3)
preparing d8fa0174 (d8fa0174)
benchmarking word_freq_bench[words=20000]
  60 samples per ref (max_runs)
  d8fa0174 +34.1% -> slower
preparing HEAD~5 (eb507ac3)
preparing cdf6998e (cdf6998e)
benchmarking word_freq_bench[words=20000]
  60 samples per ref (max_runs)
  cdf6998e +0.2% -> no change
preparing HEAD~5 (eb507ac3)
preparing 6a324704 (6a324704)
benchmarking word_freq_bench[words=20000]
  60 samples per ref (max_runs)
  6a324704 +34.7% -> slower
first bad commit: 6a324704e75e  fix: count words case-insensitively
word_freq_bench is +34.1% slower at HEAD than at HEAD~5; probes:
  d8fa0174    +34.1%  slower
  cdf6998e     +0.2%  no change
  6a324704    +34.7%  slower

Every probe is measured interleaved with the good commit. Your checkout and git bisect state are never touched.

Configuration

benchdiff.toml in the repo (or -c FILE). All percentages accept "2%", "2" or 2.

Precedence: command line, then [bench.NAME], then [settings].

Architecture

Tests

.venv/bin/pytest -q                    # everything (56 tests, ~20 s)
.venv/bin/pytest -q -m "not slow"      # skip the end-to-end tests that benchmark the demo repo

The slow tests build the demo repo, run real benchmarks, check the verdicts, the history change point, the CI exit code and that bisect finds commit 4.

Limitations

• Timing noise cannot be removed, only measured. On a loaded laptop with the powersave governor, expect wide intervals and max_runs stops; the report says so.
• The config file is read from the repository you run in, not from each ref, so a ref that lacks a command fails with that command's stderr.
• History compares medians from different runs (different machine conditions); use run/bisect to confirm a suspected change point.
• Peak RSS for Python benchmarks is the whole subprocess, including interpreter and imports.
• Bisect assumes a regression that persists once introduced. Linux is the supported platform (wait4 rusage, /proc, /sys metadata).