secretsweep

A secret leak scanner for working trees, staged changes and the full git history, built for CI. It ships 54 provider rules defined in TOML, a context-aware entropy detector, checksum and structure validation that turns matches into confidence levels, baselines so legacy repositories can adopt it without a flag day, and SARIF output for GitHub code scanning. Secrets are redacted in every output unless you ask otherwise, and nothing leaves your machine unless you explicitly pass --verify.

Features

• 54 built-in rules (AWS key id + secret pairing, GitHub ghp_/gho_/github_pat_, GitLab, Slack, Stripe, Google, Twilio, SendGrid, OpenAI/Anthropic, JWT, PEM private keys, Azure, DB URLs, .npmrc, Docker config, and more). Add your own in TOML.
• Entropy detector (Shannon, hex and base64 thresholds) that only fires on assignments to credential-like names, so UUIDs, lockfile hashes and base64 images stay quiet.
• Validation: GitHub token CRC32 checksum, JWT header/payload must decode to JSON, PEM bodies must be valid base64 with the right ASN.1/OpenSSH framing, Docker auth must decode to user:pass. Failed validation downgrades confidence; passed validation upgrades it.
• Scan modes: working tree (respects .gitignore, skips binaries and big files, parallel), --staged (reads the index), --history (every commit on every ref, reports the introducing commit/author/date and whether the secret is still at HEAD), --archives (zip/tar members).
• Baselines (baseline create), inline secretsweep:allow reason comments, .secretsweepignore globs.
• Opt-in --verify for GitHub and Slack tokens only.
• Output: rich table, JSON, SARIF 2.1.0, JUnit. Exit codes for CI.
• install-hook writes a pre-commit hook; rewrite-guide prints rotate-then-purge remediation.

Install

uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'
source .venv/bin/activate

Requires Python 3.11+ and git on the PATH.

Quickstart

examples/make_leaky_repo.sh /tmp/leaky          # build a repo with planted FAKE secrets
cd /tmp/leaky
secretsweep scan .                              # working tree
secretsweep scan --history                      # every commit on every ref
secretsweep baseline create --history           # accept today's findings
secretsweep scan --history --format sarif -o secretsweep.sarif
secretsweep install-hook                        # block leaks at commit time

Usage

secretsweep scan [PATH] [--staged | --history] [--format table|json|sarif|junit] [-o FILE]
                 [--baseline FILE | --no-baseline] [--rules FILE]... [--disable RULE]...
                 [--no-entropy] [--max-size BYTES] [--jobs N] [--archives]
                 [--min-confidence low|medium|high] [--fail-on low|medium|high|critical]
                 [--show-secrets] [--verify] [--quiet]
secretsweep baseline create [PATH] [--history | --staged] [-o FILE]
secretsweep install-hook [--repo DIR] [--force]
secretsweep rewrite-guide FINGERPRINT [--repo DIR | --report report.json] [--show-secrets]
secretsweep rules list [--rules FILE]...

Exit codes: 0 clean (or everything baselined / below --fail-on), 1 findings, 2 usage or runtime error.

Working tree

$ secretsweep scan .
secretsweep: 11 finding(s)
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity ┃ Conf.  ┃ Rule                             ┃ Location                  ┃ Secret               ┃
...
13 files scanned, 1 skipped in 0.01s

Files in .gitignore (such as .env.local), vendor/** from .secretsweepignore, the line marked secretsweep:allow, the lockfile hashes, UUIDs and the base64 PNG produce nothing.

Git history

Real output against the example repository:

$ secretsweep scan --history
secretsweep: 17 finding(s)
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ Severity ┃ Conf.  ┃ Rule                          ┃ Location                   ┃ Secret               ┃ Introduced                ┃ HEAD ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━┩
│ critical │ high   │ stripe-secret-key             │ .env:2                     │ sk_l**** (40 chars)  │ 50b48dcf Bob 2024-02-02   │ no   │
│ critical │ high   │ aws-access-key-id             │ app/config.py:1            │ AKIA**** (20 chars)  │ 83d68bc1 Alice 2024-01-10 │ no   │
│ critical │ high   │ aws-secret-access-key         │ app/config.py:2            │ wJal**** (40 chars)  │ 83d68bc1 Alice 2024-01-10 │ no   │
│ critical │ high   │ azure-storage-connection-stri │ config/settings.json:5     │ nmr6**** (88 chars)  │ 00acf7c7 Dave 2024-04-20  │ yes  │
│ critical │ high   │ private-key-pem               │ deploy/id_rsa:1            │ ----**** (476 chars) │ afafd502 Erin 2024-05-05  │ no   │
│ critical │ high   │ openai-api-key                │ notebooks/analysis.ipynb:3 │ sk-p**** (88 chars)  │ f1a549f4 Carol 2024-03-15 │ yes  │
│ high     │ high   │ github-token                  │ .env:1                     │ ghp_**** (40 chars)  │ 50b48dcf Bob 2024-02-02   │ no   │
│ high     │ high   │ database-url-credentials      │ .env:3                     │ Sup3**** (13 chars)  │ 50b48dcf Bob 2024-02-02   │ no   │
...
8 commits in 0.04s

Secrets that were added and later deleted (.env, deploy/id_rsa, the first AWS keys) are found, the commit that introduced each one is named, and HEAD = yes marks the ones still live in the current tree. Identical findings across commits are collapsed to the first sighting (JSON includes occurrences).

Baseline workflow

$ secretsweep baseline create --history
Baseline with 17 finding(s) written to .secretsweep-baseline.json
$ secretsweep scan --history
No secrets found (8 commits in 0.03s, 17 baselined)
$ echo "K=sk_live_..." > new_leak.env && secretsweep scan; echo $?
...  new_leak.env:1  stripe-secret-key ...
1

The baseline stores only fingerprints (sha256(rule + secret + path), truncated), never secrets, and is auto-discovered at <root>/.secretsweep-baseline.json. Fingerprints ignore line numbers, so moving a baselined secret within its file does not resurrect it. Commit the baseline file; when you rotate a secret, its fingerprint changes and the old entry is simply unused.

SARIF, JSON, JUnit

secretsweep scan --history --format sarif -o secretsweep.sarif   # upload with github/codeql-action/upload-sarif
secretsweep scan --format json | jq '.findings[] | {rule_id, path, line}'
secretsweep scan --format junit -o secretsweep-junit.xml

The SARIF run on the example repository validates against the bundled official SARIF 2.1.0 schema (tests/data/sarif-schema-2.1.0.json) with 17 results. Each result carries partialFingerprints (for stable alert tracking), a security-severity rating on the rule, and for history scans the introducing commit in properties.

Redaction

Output shows the first 4 characters plus the length (AKIA**** (20 chars); secrets shorter than 12 characters show 2 characters). This applies to the table, JSON, SARIF and JUnit alike. --show-secrets prints full values.

Allowing findings

EXAMPLE_KEY = "AKIAIOSFODNN7EXAMPLE"  # secretsweep:allow documented AWS example key

.secretsweepignore uses gitignore syntax, applies to tree, staged and history scans, and lives in the scan root:

vendor/**
docs/examples/*.md

Pre-commit hook

$ secretsweep install-hook
Installed pre-commit hook at /path/to/repo/.git/hooks/pre-commit

The hook runs secretsweep scan --staged --quiet, which scans the staged blob contents and exits 1 to abort the commit. It will not overwrite a hook it did not write unless you pass --force, and honours core.hooksPath.

Live verification (--verify)

Off by default and never implied. With --verify, GitHub tokens (ghp_, github_pat_ ...) are sent to https://api.github.com/user and Slack tokens to https://slack.com/api/auth.test, which are read-only "who am I" calls, and the result (valid, invalid, unknown) is attached to the finding. No other secret type is ever transmitted, and each token goes only to its own provider. A notice is printed on stderr whenever the flag is used. The test suite stubs the HTTP layer and never touches the network.

Remediation

$ secretsweep rewrite-guide 238d26722d60
Remediation for stripe-secret-key (critical) in .env:2
Introduced by 50b48dcfc4 (Bob, 2024-02-02); no longer at HEAD, but still in history.

1. ROTATE FIRST. ...
4. Purge it from history (requires https://github.com/newren/git-filter-repo). ...
     git filter-repo --path '.env' --invert-paths

The fingerprint comes from any report. See docs/remediation.md.

Configuration

• <root>/.secretsweep.toml is loaded automatically; --rules FILE adds more. Both use the rule schema in docs/rules-authoring.md, plus disabled_rules = ["rule-id"].
• --disable RULE turns a rule off for one run; --no-entropy disables the entropy detector.
• --max-size (default 1,000,000 bytes) skips large files; files with a NUL byte in the first 8 KiB are treated as binary.
• --min-confidence and --fail-on tune what is reported and what fails CI (see docs/precision-tuning.md).

Architecture

See docs/design.md for how the history scan and validation work.

Tests

.venv/bin/pytest -q

175 tests: every rule has a true-positive and a false-positive fixture (enforced by a test that fails if a rule lacks one), entropy and checksum maths, history scans that locate deleted secrets and their introducing commit on real temporary git repositories, baselines, inline allow, staged mode, archives, parallel/serial equivalence, SARIF validated against the official schema with jsonschema, JUnit, redaction, the pre-commit hook end to end, and verification against a stubbed HTTP layer. All fixture secrets are generated fake values.

Performance

Measured on a 12-core Linux machine with 5,000 synthetic Python files (about 5 KB each, 120 lines): about 1,900 files/s with the process pool and about 350 files/s single-process (--jobs 1). The pool is only used from 300 files up. Real repositories are faster because most files contain none of the rule keywords. The example repository's 8-commit history scans in about 40 ms.

Limitations

• History scanning inspects added lines of non-merge diffs (git log -p default), so a secret that only appears via a conflict resolution in a merge commit is seen only if it is also added elsewhere.
• Multi-line PEM detection works across consecutive added lines in one file of one commit.
• Entropy detection requires a credential-like variable name; a bare random string in prose is not reported.
• The HEAD-presence check is by path: a secret in a file that was renamed after the leak shows HEAD = no.
• Archives are scanned one level deep (no archives inside archives).
• Rules are regex based; novel token formats need a custom rule. Revoking a leaked credential is always the real fix.