# logsieve Point logsieve at gigabytes of messy logs and it learns the message templates (using a from-scratch implementation of the Drain algorithm), then tells you what is common, what is new, and what spiked. It answers "what changed in the logs between yesterday and today?" in seconds, from the command line, in pure Python. ## Features - Streaming input: files, globs, gzip (`.gz`) and stdin; nothing is loaded whole into memory. - Per-file format auto-detection by sampling: JSON lines, syslog (BSD and ISO), nginx/apache combined access logs, Python `logging` default, and generic `timestamp level message`. Timestamps: ISO 8601, syslog (no year), epoch (s/ms/us), nginx `[10/Oct/2026:13:55:36 +0000]`. - Masking pre-processor with typed placeholders (``, ``, ``, ``, ``, ``, ``) plus your own regex masks from a TOML file. - Drain parse tree with configurable depth and similarity threshold; template ids are a stable hash of the template text. - Persistent SQLite model (`learn`) holding templates, counts, level mix and per-minute histograms; reusable by `match`, `diff`, `grep`, `tail` and `summarize`. - `summarize`, `diff` (two-proportion z-test), `spikes` (median + k*MAD or EWMA, with sparklines), `grep` (raw lines plus parameter values and per-slot value frequencies), `learn`, `match`, and a live `tail`. - Output as rich tables, JSON or Markdown. - Measured throughput of 160k+ lines/sec end to end on simple lines (see [Performance](#performance)). ## Install Requires Python 3.12+. ```sh cd logsieve uv venv .venv uv pip install -p .venv/bin/python -e '.[dev]' .venv/bin/logsieve --help ``` ## Quickstart The `examples/` directory has a generated web-app incident: at 14:00 on day 2 database timeouts spike, and a never-before-seen "circuit breaker" error appears. ```sh logsieve summarize examples/app_today.log --top 5 logsieve diff --baseline examples/app_yesterday.log --target examples/app_today.log logsieve spikes examples/app_today.log --bucket 5m logsieve grep da87be7b examples/app_today.log --params ``` Regenerate the sample data with `python examples/generate.py` (deterministic; every file is under 5 MB). It writes `app_yesterday.log` / `app_today.log` (generic format), `access.log` (nginx combined), `worker.jsonl.gz` (JSON lines), `syslog.log` and `python.log`. ## Usage All commands accept `--config PATH` (default `./logsieve.toml`), `--depth`, `--sim` and `--input-format [auto|json|nginx|syslog|python|generic|raw]`. Informational lines (totals) go to stderr so stdout stays clean for `--format json` pipelines. ### summarize `logsieve summarize FILES... [--top N] [--level LEVEL] [--format table|json|markdown] [--db MODEL]` Top templates by count with share, level mix, first/last seen and an example line. `--level ERROR` counts only lines at that severity or higher. With `--db` and no files it summarises a saved model. ``` Id Count % Levels First seen Last seen Template Example ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── bd89b417 9,731 44.8% INFO 100% 2026-10-10 00:00:32 2026-10-10 23:59:45 [web] Request completed <*> 2026-10-10T00:00:32.163Z INFO <*> status= [web] Request completed duration= method=GET user_id= path=/api/orders/96627 … 8d656a78 2,003 9.23% INFO 100% 2026-10-10 00:00:31 2026-10-10 23:59:01 [auth] User <*> logged in from 2026-10-10T00:00:31.368Z INFO [auth] User carol logged in from 10.0.27.177 4993b746 1,582 7.29% INFO 100% 2026-10-10 00:02:06 2026-10-10 23:59:48 [worker] Job <*> finished in 2026-10-10T00:02:06.086Z INFO processed= [worker] Job invoice-sync items finished in 0.3s processed=248 i… 7ee47531 1,525 7.03% INFO 100% 2026-10-10 00:11:40 2026-10-10 23:59:25 [cache] Cache hit ratio % 2026-10-10T00:11:40.180Z INFO for <*> [cache] Cache hit ratio 76% for namespace=pricing 2bf32e13 1,351 6.22% INFO 100% 2026-10-10 00:00:27 2026-10-10 23:59:44 [auth] Session expired 2026-10-10T00:00:27.002Z INFO [auth] Session e28e434c-8505-1890-bf11-0e279e a1eb4b expired 21,705 lines, 13 templates (showing 5) ``` ### diff `logsieve diff --baseline A --target B [--top N] [--min-count 5] [--min-z 3.0] [--format ...]` `A` and `B` are each one or more log files, globs, or saved models (`.db`), and can be repeated. Templates are mined once over both sides, so ids are comparable. Output has three sections: new templates, vanished templates, and frequency changes ranked by |z| on normalised rates (details in [docs/algorithms.md](docs/algorithms.md)). ``` New templates (absent from baseline) Id Base Base % Target Target % log2 ratio z Template ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── b7099add 0 0.00% 670 3.09% +10.23 +24.7 [db] Retrying database query attempt for order 6cf4bd19 0 0.00% 54 0.25% +6.61 +7.0 [payments] Circuit breaker OPEN for service payments-gateway after consecutive failures Vanished templates (absent from target) Id Base Base % Target Target % log2 ratio z Template ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 83d8eaac 382 1.97% 0 0.00% -9.74 -20.8 [cache] Using legacy cache client, migrate to v2 Frequency changes (ranked by |z|) Id Base Base % Target Target % log2 ratio z Template ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── da87be7b 9 0.05% 1,207 5.56% +6.83 +33.0 [db] Database connection timeout after (pool=primary host=db-) bd89b417 9,895 51.0% 9,731 44.8% -0.19 -12.4 [web] Request completed <*> <*> status= duration= user_id= 41314549 393 2.02% 771 3.55% +0.81 +9.3 [billing] Payment declined for order : card_declined d02206f2 763 3.93% 1,151 5.30% +0.43 +6.6 [db] Slow query detected: on table <*> baseline 19,414 lines, target 21,705 lines ``` ### spikes `logsieve spikes FILES... [--bucket 5m] [--method mad|ewma] [-k 5] [--min-count 3] [--top N] [--width 60]` Builds a per-template time series, flags buckets above the robust baseline, and prints an ASCII sparkline per template plus the anomalous buckets. ``` Templates with spikes (5m buckets, mad, k=5.0) Id Total Spikes Peak at Peak Expected Series Template ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── da87be7b 1,207 6 2026-10-10 14:05:00 224 0.0 ▁ ▁ ▁ ▁ ▁ ▁██ [db] Database connection timeout ▁ ▁ after (pool=primary host=db-) b7099add 670 6 2026-10-10 14:15:00 122 0.0 ██ [db] Retrying database query attempt for order d02206f2 1,151 7 2026-10-10 14:20:00 69 2.0 ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▂▁▁▁██▁ [db] Slow query detected: ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ on table <*> Anomalous buckets Bucket start Id Count Expected Threshold Score ─────────────────────────────────────────────────────────────────────── 2026-10-10 08:15:00 d02206f2 10 2.0 9.41 5.4 2026-10-10 14:00:00 b7099add 117 0.0 5.0 117.0 ``` ### grep `logsieve grep TEMPLATE_ID FILES... [--db MODEL] [--params] [--limit N] [--top-values N]` The id may be a prefix. Lines are printed with their parameter values; `--params` instead shows the value frequency of each variable slot, answering "which host/user produced this error?". ``` template: [db] Database connection timeout after (pool=primary host=db-) Slot Token Value Count % ────────────────────────────────────────────────── $1 30000ms 1,207 100.0% $2 host=db-) 1 413 34.2% $2 host=db-) 2 408 33.8% $2 host=db-) 3 386 32.0% 1,207 matching lines ``` ``` template: [payments] Circuit breaker OPEN for service payments-gateway after consecutive failures 2026-10-10T14:03:07.823Z ERROR [payments] Circuit breaker OPEN for service payments-gateway after 5 consecutive failures params: $1=5 2026-10-10T14:03:09.562Z ERROR [payments] Circuit breaker OPEN for service payments-gateway after 5 consecutive failures params: $1=5 2 matching lines ``` ### learn and match `logsieve learn FILES... [--db .logsieve.db] [--bucket 1m] [--reset]` saves (or extends) a model. `logsieve match FILES... --db MODEL [--unknown-only] [--summary]` classifies lines against it without changing it; lines matching no template are `UNKNOWN`. ``` learned 19,414 lines -> 12 templates (12 new), saved to /tmp/ls.db UNKNOWN 2026-10-10T14:00:04.132Z WARN [db] Retrying database query attempt 1 for order 76942 UNKNOWN 2026-10-10T14:00:05.519Z WARN [db] Retrying database query attempt 3 for order 66602 UNKNOWN 2026-10-10T14:00:06.084Z WARN [db] Retrying database query attempt 3 for order 32772 attempt 3 for order 32772; 2026-10-10T14:00:07.022Z WARN [db] Retrying database query atte 2026-10-10T14:00:07.395Z WARN [db] Retrying database query attempt 2 for order 63347 724 unknown of 21,705 lines ``` ### tail `logsieve tail FILE [--db MODEL] [--lines 1000] [--from-start] [--refresh 0.5] [--duration SECS]` A live-updating table of templates, with templates never seen before highlighted red and marked `NEW`. With `--db`, "seen" means present in the saved model; otherwise the last `--lines` lines of the file form the baseline. Truncation and rotation are handled. Stop with Ctrl-C (or `--duration`); the new templates are listed on exit. ## Configuration `logsieve.toml` (see `examples/logsieve.toml`): ```toml [drain] depth = 4 # tree depth; (depth - 2) leading tokens route a line sim_threshold = 0.4 # minimum share of equal tokens to join a cluster max_children = 100 # per-node fan-out before overflow goes to <*> [json] # keys tried in order for JSON-lines input message_keys = ["message", "msg"] level_keys = ["level"] timestamp_keys = ["timestamp", "ts"] [[mask]] # extra masks run before the built-in ones name = "ORDER" # becomes pattern = 'ORD-\d+' ``` A saved model remembers its Drain parameters, bucket size and extra masks, and those win over the command line when the model is loaded. ## Architecture | Module | Role | | --- | --- | | `src/logsieve/cli.py` | click commands; thin glue | | `src/logsieve/inputs.py` | file/glob/gzip/stdin streaming, per-file format sampling | | `src/logsieve/formats.py` | format parsers, detection, level normalisation | | `src/logsieve/timeparse.py` | timestamp parsers (ISO, syslog, nginx, epoch) | | `src/logsieve/masking.py` | typed placeholder masking | | `src/logsieve/drain.py` | Drain tree, clusters, parameter extraction | | `src/logsieve/miner.py` | masking + Drain + statistics | | `src/logsieve/store.py` | SQLite model persistence | | `src/logsieve/stats.py` | diff statistics, spike detection, sparklines | | `src/logsieve/render.py` | table / JSON / Markdown output | | `src/logsieve/tail.py` | follow mode | | `scripts/bench.py` | throughput benchmark | | `examples/generate.py` | sample log generator | ## Performance `python scripts/bench.py` writes 500,000 simple log lines (45 MB, eight message shapes with random parameters) and times each stage. Measured on this machine (Python 3.12.12): ``` python 3.12.12, 500,000 lines, 44.9 MB read + parse 299,997 lines/sec (1.67s) masking only 376,163 lines/sec (1.33s) full pipeline (mine) 316,157 lines/sec (1.58s) end-to-end from file 169,109 lines/sec (2.96s) templates found: 8 ``` Throughput depends on how many parameters each line carries and on how often timestamps repeat; real logs with many lines per second parse a little faster than this synthetic file, which has a distinct timestamp on every line. Masking is gated by cheap substring checks so lines without, say, an IP never run the IP regex. ## Tests ```sh .venv/bin/python -m pytest ``` 80 tests cover masking, format detection, timestamp parsing, Drain correctness, diff statistics, spike detection, the SQLite model, tail, and every CLI command end to end (including the bundled examples). ## Limitations - Drain branches on the first `depth - 2` tokens. A variable value in those positions (for example `User alice ...`) creates one cluster per value until `max_children` is reached; mask such values via config or raise `--sim`/lower `--depth`. - Lines are templated one at a time, in input order. Like all online Drain implementations the result is slightly order dependent, and a template's id changes if later lines generalise it. Saved models store the final ids. - Timestamps without a time zone are read as UTC; syslog's missing year is taken from the file's modification time. - Lines indented with whitespace are treated as continuation lines (stack traces) and dropped. Other unparseable lines in an otherwise detected format are kept as raw records. - `spikes` and `learn` keep one histogram bucket per template per time bucket in memory; a huge time range with a tiny `--bucket` is refused. - Multi-line log events are not joined. - `grep` without `--db` reads the input twice, so it cannot take stdin.