Design notes

Scanning pipeline

Every mode funnels text into one function, Scanner.scan_text(text, path):

1. Lowercase the text once; a rule runs only if one of its keywords is a substring and its path_regex (if any) matches the path.
2. finditer the rule regex; take the secret group.
3. Drop it if an allowlist regex hits or its Shannon entropy is below the rule's floor.
4. Drop it if its span overlaps an already-accepted finding (the first rule in rule order wins, so one secret yields one finding even if two rules describe it, such as an npm token in .npmrc) or the line carries secretsweep:allow.
5. Run the validator to set confidence: valid becomes high, invalid low, otherwise the rule's base confidence.
6. Apply pairing (both AWS rules present means both high).
7. Run the entropy detector over what the rules did not claim.

Because tree, staged and history scans share this function, behaviour is identical everywhere.

Entropy

Shannon entropy is H = -sum(p_i * log2 p_i) over character frequencies. Examples measured with the implementation:

Entropy alone is a poor secret test (a git SHA scores like a key), so the detector also requires context: api_token = "<value>" is reported, file_hash = "<value>" and build_id = "<value>" are not.

Checksum validation: GitHub tokens

Since 2021 a ghp_/gho_/ghu_/ghs_/ghr_ token is prefix_ + 30 random chars + a 6-char checksum: the CRC32 of the 30 chars, base62 encoded (0-9A-Za-z) and left-padded with 0. Worked example:

payload   abcdefghijklmnopqrstuvwxyz0123
crc32     2154666580
base62    2LolCm
token     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

A scanned token whose last 6 characters equal that value is valid (high confidence). A random 36-char string after ghp_ is almost certainly a dummy: confidence low, hidden by --min-confidence medium. The example repository's fixture tokens are generated with correct checksums, so they scan as high.

Structure checks for other formats: a JWT must have header and payload that base64url-decode to JSON objects (header with alg); a PEM body must be strict base64 whose first byte is the ASN.1 SEQUENCE tag 0x30 (OpenSSH keys must start with openssh-key-v1\0); this also works for PEM text embedded in JSON/notebooks with literal \n escapes.

History scanning

git log --all --reverse --topo-order -p -U0 --format=@@SSCOMMIT@@... is streamed line by line, oldest commit first:

@@SSCOMMIT@@50b48dc...<TAB>Bob<TAB>bob@example.com<TAB>2024-02-02T14:30:00+00:00<TAB>Add local environment file
diff --git a/.env b/.env
+++ b/.env
@@ -0,0 +1,4 @@
+GITHUB_TOKEN=ghp_...

A small state machine tracks the current commit, the +++ path and the new-side line number from each @@ -a,b +c,d @@ header. Only added lines are kept (a deleted secret was already reported when it was added), joined per file per commit with their real line numbers, and passed to scan_text, so multi-line PEM blocks survive and reported lines match the file at that commit. Using -U0 keeps the stream small.

Dedupe: findings are keyed by fingerprint sha256(rule \0 secret \0 path)[:32]. Because commits arrive oldest first, the first sighting is the introducing commit; later sightings only bump occurrences. After the stream ends, each distinct finding is checked against git cat-file -p HEAD:<path> to set in_head.

Fingerprints deliberately exclude line numbers and commits so they survive edits above the secret, rebases and filter-repo rewrites.

Baselines

A baseline is a list of fingerprints (plus rule id and path for human review). Scans drop findings whose fingerprint is present and report how many were baselined. A baseline made with --history also covers the working tree and staged scans, since the fingerprint is mode independent.

Parallelism

Tree scans of 300+ files use a ProcessPoolExecutor (regex matching is CPU bound and holds the GIL, so threads would not help). Each worker compiles its own rules from a picklable ScannerConfig once in the initializer; chunks of 32 files amortize IPC. Smaller scans run in-process to avoid pool start-up cost. History scanning is a single sequential stream because it is bounded by git log throughput.

Redaction

Finding.secret holds the raw value only in memory. Every serializer goes through display_secret(show); repr() hides it; baselines contain fingerprints only. The --verify path is the only code that transmits a secret, and only for rules declaring a verifier.