Methodology

Window arithmetic

A series is a list of samples; sample i holds the good and total event counts for the interval [ts_i, ts_i + step). Prefix sums of both columns are built once. The counts for a window ending at t with length W are the samples whose start lies in [t - W, t), found with two binary searches, so every window query is O(log n) and exact at the boundaries (a sample belongs to exactly one window position). Gaps and irregular timestamps are fine because lookup is by timestamp, not index. A window shorter than the step is widened to one step.

Error budget

With objective O percent the budget ratio is b = 1 - O/100. For a 30 day window and 99.9% that is 0.001, i.e. 0.001 x 43,200 = 43.2 minutes of full outage.

Consumed share = bad / (b x expected events in the full window). Expected events are the observed total scaled by full_window / elapsed, so a window that is only partly covered by data (new service, calendar month in progress) is judged against the traffic it will likely have:

• 30 days at exactly 0.1% errors consumes 100% of the budget.
• 10 days at 0.1% errors consumes 33.3% (the other 20 days are still to come).

Consumed minutes = share x budget minutes. For rolling windows errors drop out as they age; for calendar windows the window starts at the 1st of the month (UTC) or Monday.

Burn rate

burn = (bad / total over window) / b. A burn rate of 1 uses exactly the whole budget over the SLO window; 14.4 uses it up in 30d / 14.4 = 2d 2h.

Budget consumed when a rule just trips = burn x long_window / SLO_window, which gives the workbook figures:

Exhaustion projection

If the remaining budget share is r and the last 6 hours burned at rate x, the budget lasts r x W / x. Example: 66.7% left, 10x burn, W = 30d gives 0.667 x 30d / 10 = 2d. Zero burn means never; a negative remainder means already exhausted.

Multi-window multi-burn-rate alerts

A policy fires when burn(long) >= threshold AND burn(short) >= threshold. The long window gives precision (the problem consumed meaningful budget), the short window (1/12 of the long one) makes the alert reset quickly once the problem stops, instead of staying red for the whole long window. Fast burns page, slow burns open tickets.

Generated PromQL records `slo:sli_error:ratio_rate` and alerts on

``` slo:sli_error:ratio_rate1h{service="x",slo="y"} > 0.0144 and slo:sli_error:ratio_rate5m{service="x",slo="y"} > 0.0144 ```

where 0.0144 = 14.4 x 0.001.

Alert replay

For every policy and every tick (default: each sample) from start + long window onward the engine computes both burns and tracks open/closed alert intervals. Ticks before a full long window of history exists are skipped, otherwise a partial window would trip spuriously.

Ground truth

Incidents are found independently of the alert rules: hourly buckets whose burn is >= 1x are merged (gaps of one bucket allowed), trimmed to the first and last sample that individually burns >= 1x, and kept only if they consume at least 1% of the budget. This ignores noise-level errors and tiny blips.

Scoring

• An alert is a true positive if it overlaps an incident (extended by 30 minutes after its end).
• Precision = true-positive alerts / alerts. Recall = incidents overlapped by at least one alert / incidents.
• Detection time = first overlapping alert fire - incident start. Reset time = last overlapping alert resolve - incident end (both floored at 0).

Worked example (bundled synthetic data)

The 45 minute outage at 30% errors burns 229x. The 14.4x/1h rule needs 1.44% of an hour in errors, reached five minutes into the incident, so it fires at 03:20 (detection 5m) and resolves at 04:05 when the 5m window clears (reset 5m). The slow leak burns only ~1.4x, so only the 1x/3d/6h ticket rule fires, 47 hours after it started: slow leaks are found late by design, but before they exhaust the budget.