logsieve
Point logsieve at gigabytes of messy logs and it learns the message templates (using a from-scratch implementation of the Drain algorithm), then tells you what is common, what is new, and what spiked. It answers "what changed in the logs between yesterday and today?" in seconds, from the command line, in pure Python.
Features
- Streaming input: files, globs, gzip (
.gz) and stdin; nothing is loaded whole into memory. - Per-file format auto-detection by sampling: JSON lines, syslog (BSD and ISO), nginx/apache combined access logs, Python
loggingdefault, and generictimestamp level message. Timestamps: ISO 8601, syslog (no year), epoch (s/ms/us), nginx[10/Oct/2026:13:55:36 +0000]. - Masking pre-processor with typed placeholders (
<IP>,<UUID>,<HEX>,<NUM>,<EMAIL>,<DURATION>,<ID>) plus your own regex masks from a TOML file. - Drain parse tree with configurable depth and similarity threshold; template ids are a stable hash of the template text.
- Persistent SQLite model (
learn) holding templates, counts, level mix and per-minute histograms; reusable bymatch,diff,grep,tailandsummarize. summarize,diff(two-proportion z-test),spikes(median + k*MAD or EWMA, with sparklines),grep(raw lines plus parameter values and per-slot value frequencies),learn,match, and a livetail.- Output as rich tables, JSON or Markdown.
- Measured throughput of 160k+ lines/sec end to end on simple lines (see Performance).
Install
Requires Python 3.12+.
cd logsieve
uv venv .venv
uv pip install -p .venv/bin/python -e '.[dev]'
.venv/bin/logsieve --help
Quickstart
The examples/ directory has a generated web-app incident: at 14:00 on day 2 database timeouts spike, and a never-before-seen "circuit breaker" error appears.
logsieve summarize examples/app_today.log --top 5
logsieve diff --baseline examples/app_yesterday.log --target examples/app_today.log
logsieve spikes examples/app_today.log --bucket 5m
logsieve grep da87be7b examples/app_today.log --params
Regenerate the sample data with python examples/generate.py (deterministic; every file is under 5 MB). It writes app_yesterday.log / app_today.log (generic format), access.log (nginx combined), worker.jsonl.gz (JSON lines), syslog.log and python.log.
Usage
All commands accept --config PATH (default ./logsieve.toml), --depth, --sim and --input-format [auto|json|nginx|syslog|python|generic|raw]. Informational lines (totals) go to stderr so stdout stays clean for --format json pipelines.
summarize
logsieve summarize FILES... [--top N] [--level LEVEL] [--format table|json|markdown] [--db MODEL]
Top templates by count with share, level mix, first/last seen and an example line. --level ERROR counts only lines at that severity or higher. With --db and no files it summarises a saved model.
Id Count % Levels First seen Last seen Template Example
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
bd89b417 9,731 44.8% INFO 100% 2026-10-10 00:00:32 2026-10-10 23:59:45 [web] Request completed <*> 2026-10-10T00:00:32.163Z INFO
<*> status=<NUM> [web] Request completed
duration=<DURATION> method=GET
user_id=<NUM> path=/api/orders/96627 …
8d656a78 2,003 9.23% INFO 100% 2026-10-10 00:00:31 2026-10-10 23:59:01 [auth] User <*> logged in from 2026-10-10T00:00:31.368Z INFO
<IP> [auth] User carol logged in
from 10.0.27.177
4993b746 1,582 7.29% INFO 100% 2026-10-10 00:02:06 2026-10-10 23:59:48 [worker] Job <*> finished in 2026-10-10T00:02:06.086Z INFO
<DURATION> processed=<NUM> [worker] Job invoice-sync
items finished in 0.3s processed=248
i…
7ee47531 1,525 7.03% INFO 100% 2026-10-10 00:11:40 2026-10-10 23:59:25 [cache] Cache hit ratio <NUM>% 2026-10-10T00:11:40.180Z INFO
for <*> [cache] Cache hit ratio 76%
for namespace=pricing
2bf32e13 1,351 6.22% INFO 100% 2026-10-10 00:00:27 2026-10-10 23:59:44 [auth] Session <UUID> expired 2026-10-10T00:00:27.002Z INFO
[auth] Session
e28e434c-8505-1890-bf11-0e279e
a1eb4b expired
21,705 lines, 13 templates (showing 5)
diff
logsieve diff --baseline A --target B [--top N] [--min-count 5] [--min-z 3.0] [--format ...]
A and B are each one or more log files, globs, or saved models (.db), and can be repeated. Templates are mined once over both sides, so ids are comparable. Output has three sections: new templates, vanished templates, and frequency changes ranked by |z| on normalised rates (details in docs/algorithms.md).
New templates (absent from baseline)
Id Base Base % Target Target % log2 ratio z Template
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
b7099add 0 0.00% 670 3.09% +10.23 +24.7 [db] Retrying database query attempt <NUM> for order <NUM>
6cf4bd19 0 0.00% 54 0.25% +6.61 +7.0 [payments] Circuit breaker OPEN for service payments-gateway after <NUM>
consecutive failures
Vanished templates (absent from target)
Id Base Base % Target Target % log2 ratio z Template
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
83d8eaac 382 1.97% 0 0.00% -9.74 -20.8 [cache] Using legacy cache client, migrate to v2
Frequency changes (ranked by |z|)
Id Base Base % Target Target % log2 ratio z Template
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
da87be7b 9 0.05% 1,207 5.56% +6.83 +33.0 [db] Database connection timeout after <DURATION> (pool=primary
host=db-<NUM>)
bd89b417 9,895 51.0% 9,731 44.8% -0.19 -12.4 [web] Request completed <*> <*> status=<NUM> duration=<DURATION>
user_id=<NUM>
41314549 393 2.02% 771 3.55% +0.81 +9.3 [billing] Payment declined for order <NUM>: card_declined
d02206f2 763 3.93% 1,151 5.30% +0.43 +6.6 [db] Slow query detected: <DURATION> on table <*>
baseline 19,414 lines, target 21,705 lines
spikes
logsieve spikes FILES... [--bucket 5m] [--method mad|ewma] [-k 5] [--min-count 3] [--top N] [--width 60]
Builds a per-template time series, flags buckets above the robust baseline, and prints an ASCII sparkline per template plus the anomalous buckets.
Templates with spikes (5m buckets, mad, k=5.0)
Id Total Spikes Peak at Peak Expected Series Template
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
da87be7b 1,207 6 2026-10-10 14:05:00 224 0.0 ▁ ▁ ▁ ▁ ▁ ▁██ [db] Database connection timeout
▁ ▁ after <DURATION> (pool=primary
host=db-<NUM>)
b7099add 670 6 2026-10-10 14:15:00 122 0.0 ██ [db] Retrying database query attempt
<NUM> for order <NUM>
d02206f2 1,151 7 2026-10-10 14:20:00 69 2.0 ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▂▁▁▁██▁ [db] Slow query detected: <DURATION>
▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ on table <*>
Anomalous buckets
Bucket start Id Count Expected Threshold Score
───────────────────────────────────────────────────────────────────────
2026-10-10 08:15:00 d02206f2 10 2.0 9.41 5.4
2026-10-10 14:00:00 b7099add 117 0.0 5.0 117.0
grep
logsieve grep TEMPLATE_ID FILES... [--db MODEL] [--params] [--limit N] [--top-values N]
The id may be a prefix. Lines are printed with their parameter values; --params instead shows the value frequency of each variable slot, answering "which host/user produced this error?".
template: [db] Database connection timeout after <DURATION> (pool=primary host=db-<NUM>)
Slot Token Value Count %
──────────────────────────────────────────────────
$1 <DURATION> 30000ms 1,207 100.0%
$2 host=db-<NUM>) 1 413 34.2%
$2 host=db-<NUM>) 2 408 33.8%
$2 host=db-<NUM>) 3 386 32.0%
1,207 matching lines
template: [payments] Circuit breaker OPEN for service payments-gateway after <NUM> consecutive failures
2026-10-10T14:03:07.823Z ERROR [payments] Circuit breaker OPEN for service payments-gateway after 5 consecutive failures
params: $1=5
2026-10-10T14:03:09.562Z ERROR [payments] Circuit breaker OPEN for service payments-gateway after 5 consecutive failures
params: $1=5
2 matching lines
learn and match
logsieve learn FILES... [--db .logsieve.db] [--bucket 1m] [--reset] saves (or extends) a model. logsieve match FILES... --db MODEL [--unknown-only] [--summary] classifies lines against it without changing it; lines matching no template are UNKNOWN.
learned 19,414 lines -> 12 templates (12 new), saved to /tmp/ls.db
UNKNOWN 2026-10-10T14:00:04.132Z WARN [db] Retrying database query attempt 1 for order 76942
UNKNOWN 2026-10-10T14:00:05.519Z WARN [db] Retrying database query attempt 3 for order 66602
UNKNOWN 2026-10-10T14:00:06.084Z WARN [db] Retrying database query attempt 3 for order 32772
attempt 3 for order 32772; 2026-10-10T14:00:07.022Z WARN [db] Retrying database query atte
2026-10-10T14:00:07.395Z WARN [db] Retrying database query attempt 2 for order 63347
724 unknown of 21,705 lines
tail
logsieve tail FILE [--db MODEL] [--lines 1000] [--from-start] [--refresh 0.5] [--duration SECS]
A live-updating table of templates, with templates never seen before highlighted red and marked NEW. With --db, "seen" means present in the saved model; otherwise the last --lines lines of the file form the baseline. Truncation and rotation are handled. Stop with Ctrl-C (or --duration); the new templates are listed on exit.
Configuration
logsieve.toml (see examples/logsieve.toml):
[drain]
depth = 4 # tree depth; (depth - 2) leading tokens route a line
sim_threshold = 0.4 # minimum share of equal tokens to join a cluster
max_children = 100 # per-node fan-out before overflow goes to <*>
[json] # keys tried in order for JSON-lines input
message_keys = ["message", "msg"]
level_keys = ["level"]
timestamp_keys = ["timestamp", "ts"]
[[mask]] # extra masks run before the built-in ones
name = "ORDER" # becomes <ORDER>
pattern = 'ORD-\d+'
A saved model remembers its Drain parameters, bucket size and extra masks, and those win over the command line when the model is loaded.
Architecture
| Module | Role |
|---|---|
src/logsieve/cli.py |
click commands; thin glue |
src/logsieve/inputs.py |
file/glob/gzip/stdin streaming, per-file format sampling |
src/logsieve/formats.py |
format parsers, detection, level normalisation |
src/logsieve/timeparse.py |
timestamp parsers (ISO, syslog, nginx, epoch) |
src/logsieve/masking.py |
typed placeholder masking |
src/logsieve/drain.py |
Drain tree, clusters, parameter extraction |
src/logsieve/miner.py |
masking + Drain + statistics |
src/logsieve/store.py |
SQLite model persistence |
src/logsieve/stats.py |
diff statistics, spike detection, sparklines |
src/logsieve/render.py |
table / JSON / Markdown output |
src/logsieve/tail.py |
follow mode |
scripts/bench.py |
throughput benchmark |
examples/generate.py |
sample log generator |
Performance
python scripts/bench.py writes 500,000 simple log lines (45 MB, eight message shapes with random parameters) and times each stage. Measured on this machine (Python 3.12.12):
python 3.12.12, 500,000 lines, 44.9 MB
read + parse 299,997 lines/sec (1.67s)
masking only 376,163 lines/sec (1.33s)
full pipeline (mine) 316,157 lines/sec (1.58s)
end-to-end from file 169,109 lines/sec (2.96s)
templates found: 8
Throughput depends on how many parameters each line carries and on how often timestamps repeat; real logs with many lines per second parse a little faster than this synthetic file, which has a distinct timestamp on every line. Masking is gated by cheap substring checks so lines without, say, an IP never run the IP regex.
Tests
.venv/bin/python -m pytest
80 tests cover masking, format detection, timestamp parsing, Drain correctness, diff statistics, spike detection, the SQLite model, tail, and every CLI command end to end (including the bundled examples).
Limitations
- Drain branches on the first
depth - 2tokens. A variable value in those positions (for exampleUser alice ...) creates one cluster per value untilmax_childrenis reached; mask such values via config or raise--sim/lower--depth. - Lines are templated one at a time, in input order. Like all online Drain implementations the result is slightly order dependent, and a template's id changes if later lines generalise it. Saved models store the final ids.
- Timestamps without a time zone are read as UTC; syslog's missing year is taken from the file's modification time.
- Lines indented with whitespace are treated as continuation lines (stack traces) and dropped. Other unparseable lines in an otherwise detected format are kept as raw records.
spikesandlearnkeep one histogram bucket per template per time bucket in memory; a huge time range with a tiny--bucketis refused.- Multi-line log events are not joined.
grepwithout--dbreads the input twice, so it cannot take stdin.