# secretsweep A secret leak scanner for working trees, staged changes and the **full git history**, built for CI. It ships 54 provider rules defined in TOML, a context-aware entropy detector, checksum and structure validation that turns matches into confidence levels, baselines so legacy repositories can adopt it without a flag day, and SARIF output for GitHub code scanning. Secrets are redacted in every output unless you ask otherwise, and nothing leaves your machine unless you explicitly pass `--verify`. ## Features - 54 built-in rules (AWS key id + secret pairing, GitHub `ghp_`/`gho_`/`github_pat_`, GitLab, Slack, Stripe, Google, Twilio, SendGrid, OpenAI/Anthropic, JWT, PEM private keys, Azure, DB URLs, `.npmrc`, Docker config, and more). Add your own in TOML. - Entropy detector (Shannon, hex and base64 thresholds) that only fires on assignments to credential-like names, so UUIDs, lockfile hashes and base64 images stay quiet. - Validation: GitHub token CRC32 checksum, JWT header/payload must decode to JSON, PEM bodies must be valid base64 with the right ASN.1/OpenSSH framing, Docker `auth` must decode to `user:pass`. Failed validation downgrades confidence; passed validation upgrades it. - Scan modes: working tree (respects `.gitignore`, skips binaries and big files, parallel), `--staged` (reads the index), `--history` (every commit on every ref, reports the introducing commit/author/date and whether the secret is still at HEAD), `--archives` (zip/tar members). - Baselines (`baseline create`), inline `secretsweep:allow reason` comments, `.secretsweepignore` globs. - Opt-in `--verify` for GitHub and Slack tokens only. - Output: rich table, JSON, SARIF 2.1.0, JUnit. Exit codes for CI. - `install-hook` writes a pre-commit hook; `rewrite-guide` prints rotate-then-purge remediation. ## Install ```bash uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]' source .venv/bin/activate ``` Requires Python 3.11+ and `git` on the PATH. ## Quickstart ```bash examples/make_leaky_repo.sh /tmp/leaky # build a repo with planted FAKE secrets cd /tmp/leaky secretsweep scan . # working tree secretsweep scan --history # every commit on every ref secretsweep baseline create --history # accept today's findings secretsweep scan --history --format sarif -o secretsweep.sarif secretsweep install-hook # block leaks at commit time ``` ## Usage ``` secretsweep scan [PATH] [--staged | --history] [--format table|json|sarif|junit] [-o FILE] [--baseline FILE | --no-baseline] [--rules FILE]... [--disable RULE]... [--no-entropy] [--max-size BYTES] [--jobs N] [--archives] [--min-confidence low|medium|high] [--fail-on low|medium|high|critical] [--show-secrets] [--verify] [--quiet] secretsweep baseline create [PATH] [--history | --staged] [-o FILE] secretsweep install-hook [--repo DIR] [--force] secretsweep rewrite-guide FINGERPRINT [--repo DIR | --report report.json] [--show-secrets] secretsweep rules list [--rules FILE]... ``` Exit codes: `0` clean (or everything baselined / below `--fail-on`), `1` findings, `2` usage or runtime error. ### Working tree ``` $ secretsweep scan . secretsweep: 11 finding(s) ┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Severity ┃ Conf. ┃ Rule ┃ Location ┃ Secret ┃ ... 13 files scanned, 1 skipped in 0.01s ``` Files in `.gitignore` (such as `.env.local`), `vendor/**` from `.secretsweepignore`, the line marked `secretsweep:allow`, the lockfile hashes, UUIDs and the base64 PNG produce nothing. ### Git history Real output against the example repository: ``` $ secretsweep scan --history secretsweep: 17 finding(s) ┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┓ ┃ Severity ┃ Conf. ┃ Rule ┃ Location ┃ Secret ┃ Introduced ┃ HEAD ┃ ┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━┩ │ critical │ high │ stripe-secret-key │ .env:2 │ sk_l**** (40 chars) │ 50b48dcf Bob 2024-02-02 │ no │ │ critical │ high │ aws-access-key-id │ app/config.py:1 │ AKIA**** (20 chars) │ 83d68bc1 Alice 2024-01-10 │ no │ │ critical │ high │ aws-secret-access-key │ app/config.py:2 │ wJal**** (40 chars) │ 83d68bc1 Alice 2024-01-10 │ no │ │ critical │ high │ azure-storage-connection-stri │ config/settings.json:5 │ nmr6**** (88 chars) │ 00acf7c7 Dave 2024-04-20 │ yes │ │ critical │ high │ private-key-pem │ deploy/id_rsa:1 │ ----**** (476 chars) │ afafd502 Erin 2024-05-05 │ no │ │ critical │ high │ openai-api-key │ notebooks/analysis.ipynb:3 │ sk-p**** (88 chars) │ f1a549f4 Carol 2024-03-15 │ yes │ │ high │ high │ github-token │ .env:1 │ ghp_**** (40 chars) │ 50b48dcf Bob 2024-02-02 │ no │ │ high │ high │ database-url-credentials │ .env:3 │ Sup3**** (13 chars) │ 50b48dcf Bob 2024-02-02 │ no │ ... 8 commits in 0.04s ``` Secrets that were added and later deleted (`.env`, `deploy/id_rsa`, the first AWS keys) are found, the commit that introduced each one is named, and `HEAD = yes` marks the ones still live in the current tree. Identical findings across commits are collapsed to the first sighting (JSON includes `occurrences`). ### Baseline workflow ``` $ secretsweep baseline create --history Baseline with 17 finding(s) written to .secretsweep-baseline.json $ secretsweep scan --history No secrets found (8 commits in 0.03s, 17 baselined) $ echo "K=sk_live_..." > new_leak.env && secretsweep scan; echo $? ... new_leak.env:1 stripe-secret-key ... 1 ``` The baseline stores only fingerprints (`sha256(rule + secret + path)`, truncated), never secrets, and is auto-discovered at `/.secretsweep-baseline.json`. Fingerprints ignore line numbers, so moving a baselined secret within its file does not resurrect it. Commit the baseline file; when you rotate a secret, its fingerprint changes and the old entry is simply unused. ### SARIF, JSON, JUnit ```bash secretsweep scan --history --format sarif -o secretsweep.sarif # upload with github/codeql-action/upload-sarif secretsweep scan --format json | jq '.findings[] | {rule_id, path, line}' secretsweep scan --format junit -o secretsweep-junit.xml ``` The SARIF run on the example repository validates against the bundled official SARIF 2.1.0 schema (`tests/data/sarif-schema-2.1.0.json`) with 17 results. Each result carries `partialFingerprints` (for stable alert tracking), a `security-severity` rating on the rule, and for history scans the introducing commit in `properties`. ### Redaction Output shows the first 4 characters plus the length (`AKIA**** (20 chars)`; secrets shorter than 12 characters show 2 characters). This applies to the table, JSON, SARIF and JUnit alike. `--show-secrets` prints full values. ### Allowing findings ```python EXAMPLE_KEY = "AKIAIOSFODNN7EXAMPLE" # secretsweep:allow documented AWS example key ``` `.secretsweepignore` uses gitignore syntax, applies to tree, staged and history scans, and lives in the scan root: ``` vendor/** docs/examples/*.md ``` ### Pre-commit hook ``` $ secretsweep install-hook Installed pre-commit hook at /path/to/repo/.git/hooks/pre-commit ``` The hook runs `secretsweep scan --staged --quiet`, which scans the staged blob contents and exits 1 to abort the commit. It will not overwrite a hook it did not write unless you pass `--force`, and honours `core.hooksPath`. ### Live verification (`--verify`) Off by default and never implied. With `--verify`, GitHub tokens (`ghp_`, `github_pat_` ...) are sent to `https://api.github.com/user` and Slack tokens to `https://slack.com/api/auth.test`, which are read-only "who am I" calls, and the result (`valid`, `invalid`, `unknown`) is attached to the finding. No other secret type is ever transmitted, and each token goes only to its own provider. A notice is printed on stderr whenever the flag is used. The test suite stubs the HTTP layer and never touches the network. ### Remediation ``` $ secretsweep rewrite-guide 238d26722d60 Remediation for stripe-secret-key (critical) in .env:2 Introduced by 50b48dcfc4 (Bob, 2024-02-02); no longer at HEAD, but still in history. 1. ROTATE FIRST. ... 4. Purge it from history (requires https://github.com/newren/git-filter-repo). ... git filter-repo --path '.env' --invert-paths ``` The fingerprint comes from any report. See `docs/remediation.md`. ## Configuration - `/.secretsweep.toml` is loaded automatically; `--rules FILE` adds more. Both use the rule schema in `docs/rules-authoring.md`, plus `disabled_rules = ["rule-id"]`. - `--disable RULE` turns a rule off for one run; `--no-entropy` disables the entropy detector. - `--max-size` (default 1,000,000 bytes) skips large files; files with a NUL byte in the first 8 KiB are treated as binary. - `--min-confidence` and `--fail-on` tune what is reported and what fails CI (see `docs/precision-tuning.md`). ## Architecture | Module | Role | | --- | --- | | `rules.py`, `builtin_rules.toml` | Rule schema, loading, user overrides | | `detect.py` | `Scanner.scan_text`: keyword prefilter, regex, allowlist, entropy floor, validators, pairing, inline allow | | `entropy.py` | Shannon entropy and the context-aware detector | | `validators.py` | GitHub CRC32, JWT, PEM, Docker auth | | `walk.py` | File listing (via `git ls-files`, falling back to `.gitignore` walking), archives, process-pool scanning | | `gitscan.py` | Streaming `git log -p` parser for history, staged scanning, HEAD-presence check | | `baseline.py`, `ignore.py` | Baseline files and `.secretsweepignore` | | `engine.py` | Orchestrates a scan: rules, mode, baseline, verification | | `report.py` | Table, JSON, SARIF, JUnit | | `verify.py`, `hook.py`, `guide.py`, `cli.py` | Verification, pre-commit hook, remediation text, CLI | See `docs/design.md` for how the history scan and validation work. ## Tests ```bash .venv/bin/pytest -q ``` 175 tests: every rule has a true-positive and a false-positive fixture (enforced by a test that fails if a rule lacks one), entropy and checksum maths, history scans that locate deleted secrets and their introducing commit on real temporary git repositories, baselines, inline allow, staged mode, archives, parallel/serial equivalence, SARIF validated against the official schema with `jsonschema`, JUnit, redaction, the pre-commit hook end to end, and verification against a stubbed HTTP layer. All fixture secrets are generated fake values. ## Performance Measured on a 12-core Linux machine with 5,000 synthetic Python files (about 5 KB each, 120 lines): about 1,900 files/s with the process pool and about 350 files/s single-process (`--jobs 1`). The pool is only used from 300 files up. Real repositories are faster because most files contain none of the rule keywords. The example repository's 8-commit history scans in about 40 ms. ## Limitations - History scanning inspects added lines of non-merge diffs (`git log -p` default), so a secret that only appears via a conflict resolution in a merge commit is seen only if it is also added elsewhere. - Multi-line PEM detection works across consecutive added lines in one file of one commit. - Entropy detection requires a credential-like variable name; a bare random string in prose is not reported. - The HEAD-presence check is by path: a secret in a file that was renamed after the leak shows `HEAD = no`. - Archives are scanned one level deep (no archives inside archives). - Rules are regex based; novel token formats need a custom rule. Revoking a leaked credential is always the real fix.