# sitecheck

Static website auditor. Point it at a build folder (`dist/`, `public/`, `_site/`, `site/`) or a local/remote URL and it reports broken links and anchors, missing or duplicated SEO metadata, accessibility basics, and performance-budget violations, with exit codes, a baseline file and JSON, JUnit, SARIF and HTML reports that drop straight into CI.

## Features

- **Two crawlers.** Filesystem mode serves your output folder virtually (root-relative and relative URLs, `index.html` directory indexes, `<base href>`, case-sensitivity checks). HTTP mode is asyncio + httpx: same-origin crawl with a concurrency limit, robots.txt, sitemap seeding, depth/page limits, retries with exponential backoff.
- **Links.** Broken links and resources (including `srcset` and CSS `url()`), missing `#fragment` anchors, redirects and redirect chains/loops, mixed content, case mismatches (works on macOS, 404s on Linux hosting), orphan pages, and optional external link checking (`--external`, HEAD then GET fallback, cache, per-host rate limit).
- **SEO.** Title and description presence/length, duplicates across pages, canonical validity, Open Graph/Twitter completeness, h1 count, sitemap consistency in both directions, noindex warnings, hreflang validity and reciprocity.
- **Accessibility.** Images without alt, unnamed links and buttons, unlabeled form controls, heading skips, missing `lang`, duplicate ids, positive tabindex, missing `<main>`, and WCAG contrast for inline-styled text when both colors are statically known.
- **Performance budgets.** Page weight, request count, per-asset size, oversized images vs their declared width, missing lazy-loading, gzip savings estimate, render-blocking scripts. Budgets live in `sitecheck.toml`.
- **CI friendly.** `--fail-on` severity threshold, baseline file to ignore known issues, per-rule severity overrides, JSON/JUnit/SARIF/HTML output.

## Install

Requires Python 3.11+.

```bash
uv venv .venv
uv pip install -p .venv/bin/python -e '.[dev]'
.venv/bin/sitecheck --help
```

## Quickstart

```bash
# audit the bundled example site (a deliberately broken ~20-page site)
.venv/bin/sitecheck check examples/site --config examples/sitecheck.toml

# same thing over HTTP
(cd examples/site && python -m http.server 8159) &
.venv/bin/sitecheck check http://127.0.0.1:8159/ --config examples/sitecheck.toml --quiet
kill %1

# all report formats at once, fail the build on warnings
.venv/bin/sitecheck check examples/site --config examples/sitecheck.toml \
  --json reports/r.json --junit reports/r.xml --sarif reports/r.sarif --html reports/r.html --fail-on warning
```

`examples/site` is regenerable: `.venv/bin/python examples/make_assets.py` rebuilds the images (Pillow), fonts and bulk CSS/JS.

## Usage

```
sitecheck check TARGET [options]     TARGET is a directory or an http(s):// URL
sitecheck rules [--category C]       list every rule with default severity
```

| Option | Meaning |
|---|---|
| `--config FILE` | `sitecheck.toml`; default is `./sitecheck.toml`, then `TARGET/sitecheck.toml` |
| `--external / --no-external` | check external links over the network (off by default) |
| `--site-url URL` | public origin; absolute URLs on it are verified locally |
| `--max-pages N`, `--max-depth N`, `--concurrency N`, `--no-robots` | crawl limits |
| `--only links,seo,a11y,perf` | report only these categories |
| `--fail-on error\|warning\|info\|none` | exit 1 if an issue at/above this level remains (default `error`) |
| `--baseline FILE` / `--write-baseline FILE` | hide known issues / record current issues |
| `--json/--junit/--sarif/--html FILE` | write a report |
| `--quiet` | summary only |

Exit codes: `0` clean at the threshold, `1` issues at or above `--fail-on`, `2` usage/config/crawl-start error.

### Example: filesystem mode

`sitecheck check examples/site --config examples/sitecheck.toml` (output trimmed; the real run lists all 21 pages, exit code 1):

```
sitecheck examples/site  (fs mode, 21 pages, 52 URLs checked)

/
info  text-uncompressed  /css/site.css is 119 KB raw, about 14 KB gzipped

/about.html
error    case-mismatch          /Team/ exists as /team/index.html; this 404s on case-sensitive hosts  (line   
                                22)                                                                           
warning  contrast               #999999 on #ffffff is 2.85:1 (needs 4.5:1)  (line 23)                         
warning  duplicate-description  same description "Acme Widgets builds reliable, well documented widg" as      
                                /team/                                                                        
warning  duplicate-title        same title "About Acme Widgets - our story" as /team/                         
warning  heading-skip           h2 followed by h4 ("Awards")  (line 21)                                       
info     redirect               /team redirects (301): /team -> /team/  (line 22)                             

/blog/
error  broken-link  /blog/post-3.html returns 404  (line 20)

/blog/post-1.html
warning  description-missing  no meta description                                                             
warning  duplicate-title      same title "Announcing the Widget 2.0 release with a very long" as              
                              /blog/post-2.html                                                               
warning  heading-skip         h1 followed by h3 ("Details")  (line 16)                                        
warning  title-length         title is 76 characters (allowed 10-60)                                          

/blog/post-2.html
warning  canonical-invalid  canonical "/blog/post-2.html" is not an absolute URL                              
warning  duplicate-title    same title "Announcing the Widget 2.0 release with a very long" as                
                            /blog/post-1.html                                                                 
warning  title-length       title is 76 characters (allowed 10-60)                                            

/contact.html
error    button-name        <button> has no accessible name  (line 24)            
error    duplicate-id       id "name" used 2 times  (line 20)
...

Summary                              
┏━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ category ┃ error ┃ warning ┃ info ┃
┡━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ links    │    11 │       3 │    2 │
│ seo      │     2 │      19 │    2 │
│ a11y     │    10 │       7 │    0 │
│ perf     │     0 │       5 │    7 │
│ total    │    23 │      34 │   11 │
└──────────┴───────┴─────────┴──────┘
```

Notable findings in the full output, each planted in `examples/site`:

```
/about.html        error    case-mismatch      /Team/ exists as /team/index.html; this 404s on case-sensitive hosts
/products/         error    broken-anchor      /products/widget.html#reviews: no element with that id or name
/products/         warning  image-oversized    /img/hero-huge.jpg is 2400px wide but displayed at 600px
/products/         warning  page-weight        2433 KB total exceeds budget of 1024 KB
/contact.html      warning  contrast           #444444 on #336699 is 1.62:1 (needs 4.5:1)
/mixed.html        error    mixed-content      insecure script http://cdn.example.com/widget.js loaded from an https page
/en/               warning  hreflang-no-return /fr/ (fr) has no hreflang link back
/sitemap.xml       error    sitemap-url-broken sitemap lists /gone.html which returns 404
/hidden.html       warning  orphan-page        no page links here (listed in the sitemap)
```

### Example: HTTP mode

Against `python -m http.server 8159` serving the same folder:

```
sitecheck http://127.0.0.1:8159/  (http mode, 18 pages, 44 URLs checked)

Summary                              
┏━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ category ┃ error ┃ warning ┃ info ┃
┡━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ links    │    10 │       1 │    3 │
│ seo      │     2 │      17 │    2 │
│ a11y     │    10 │       7 │    0 │
│ perf     │     0 │       5 │    7 │
│ total    │    22 │      30 │   12 │
└──────────┴───────┴─────────┴──────┘
```

The numbers differ slightly from filesystem mode: a real Linux server cannot serve `/Team/`, so the case mismatches show up as plain 404s (`broken-link`); `robots.txt` is honoured so `/private/` is skipped (`robots-blocked`); and pages that nothing links to and the sitemap does not list (`/orphan.html`) cannot be discovered over HTTP at all.

### Baselines

```
$ sitecheck check examples/site --config examples/sitecheck.toml --write-baseline reports/baseline.json
wrote baseline with 68 issue(s) to reports/baseline.json
$ sitecheck check examples/site --config examples/sitecheck.toml --baseline reports/baseline.json --fail-on warning --quiet
...
68 issue(s) hidden by baseline
$ echo $?
0
```

### Report formats

- **JSON**: `{tool, target, mode, summary, pages[], issues[]}`; each issue has `code, severity, category, page, target, line, message`.
- **JUnit**: one suite per page, one testcase per issue; errors and warnings are `<failure>`.
- **SARIF 2.1.0**: upload to GitHub code scanning.
- **HTML**: a single self-contained file with a sortable, filterable issue table and a page list.

## Configuration

`sitecheck.toml` controls crawl limits, external checking, budgets, SEO thresholds, per-rule severities (`error|warning|info|off`) and ignores. The commented example is `examples/sitecheck.toml`; every key is documented in [docs/configuration.md](docs/configuration.md). Every rule is described in [docs/checks.md](docs/checks.md).

## Architecture

```
sitecheck/
  cli.py            click commands, option merging, exit codes
  config.py         sitecheck.toml dataclasses and validation
  rules.py          rule catalog (code, category, default severity)
  urls.py           resolve/normalize URLs
  parser.py         html.parser state machine -> ParsedPage
  contrast.py       WCAG color parsing and ratio math
  images.py         PNG/GIF/JPEG pixel size from headers
  fetch.py          Fetcher base (redirects), FsFetcher, HttpFetcher (retries)
  crawler.py        async crawler, seeding (sitemap, disk, robots)
  sitemap.py        sitemap/sitemapindex parsing
  external.py       external link probing with host rate limits
  model.py          Resource, Page, Ref, Site, Issue
  checks_links.py   checks_seo.py   checks_a11y.py   checks_perf.py
  audit.py          orchestration, severity/ignore handling, baselines
  report/           console, json, junit, sarif, html
tests/              92 tests (parser, each check, HTTP crawler against a local server, reports, CLI)
docs/               checks.md, configuration.md, design.md
examples/           site/ (planted issues), sitecheck.toml, make_assets.py
```

Design notes and worked examples (URL resolution, contrast math, page weight, redirects, identity of `/` vs `/index.html`) are in [docs/design.md](docs/design.md).

## Tests

```bash
.venv/bin/python -m pytest -q
```

HTTP-mode tests run against a throwaway `http.server` on an ephemeral port; no external network is used.

## Limitations

- Static analysis only: no JavaScript execution, so links and images inserted client-side are invisible.
- Contrast is limited to inline styles; stylesheet rules are not evaluated.
- Page weight ignores external resources and does not follow `@import` chains or `srcset` selection.
- Filesystem mode does not apply `robots.txt` and does not emulate clean URLs, `_redirects` or server rewrite rules.
- Fragment checks only work for pages that were crawled (inside `max_depth`/`max_pages`).
- External checks cannot distinguish bot-blocking (403/429) from dead links, so those are warnings.
- Sitemaps are read as plain XML (gzip-compressed sitemaps are not supported).