Design notes

Pipeline

target --> crawler --> Site (resources, pages, refs) --> checks --> issues --> finalize --> reports
             |                                              ^
         Fetcher (fs | http)                       external.py (optional)
  1. Crawl produces a Site: every fetched Resource (status, size, redirect chain, headers, body for text, pixel size for images), every parsed Page, and a flat list of Refs (who referenced which URL, with what kind and line).
  2. Checks are pure functions of the Site. They never touch the network or disk, which is why nearly all of them are tested with fixture files.
  3. finalize applies severity overrides, ignores and de-duplication, then sorts.

One crawler, two fetchers

Both modes use the same async crawler. A Fetcher only has to implement fetch_one(url); redirect following (up to 10 hops, loop detection, recording each hop) lives in the base class.

FsFetcher behaves like a conservative static host:

HttpFetcher uses one httpx.AsyncClient, never follows redirects itself, and retries connection errors, 429 and 502-504 with exponential backoff. Size is Content-Length (the transfer size) when present.

Visiting rules

A URL is parsed only if it is an HTML page, was reached by a link at depth <= max_depth, and the max_pages budget has room. Everything else a page references (assets, links beyond the depth limit, canonical and hreflang targets) is fetched but not parsed, so its existence and size are known. That is why a link at the depth limit is still verified.

In filesystem mode every .html file on disk and every sitemap entry is also a depth-0 seed. That is what makes orphan detection possible: a page found on disk with no inbound link is an orphan.

Same page, several URLs

/ and /index.html are the same file. Parsing it twice would produce duplicate-title noise, so resources carry an identity (relative file path in filesystem mode; the URL with a trailing index.html removed in HTTP mode). The first URL that reaches an identity becomes canonical and later URLs resolve to it through final_url.

URL resolution

urls.resolve(base, href) wraps urljoin and then normalises: scheme and host lower-cased, default ports dropped, empty path becomes /, fragment split off and percent-decoded. Trailing slashes are significant (/docs and /docs/ are different URLs, and relative links resolve differently from each).

Worked example, page http://h/a/b/page.html:

href result
c.html http://h/a/b/c.html
../c.html?x=1#s http://h/a/c.html?x=1, fragment s
/c.html http://h/c.html
//cdn.test/x.js http://cdn.test/x.js
mailto:x@y.z, tel:, javascript:, data: skipped

When the page has <base href="/docs/">, relative hrefs resolve against http://h/docs/ instead. Absolute URLs on a known alias origin (site_url or the sitemap origin) are rewritten to the audited origin before fetching.

Accessible names and labels

The parser is a stack machine on top of html.parser. Each open element is a frame that accumulates text, inherits inline colors, and remembers whether a descendant <img alt="..."> contributes a name. Implied end tags (<li>, <p>, <td>, ...) are handled so unclosed list items do not swallow each other's text.

Contrast

Only statically determinable colors are checked, otherwise the check would guess. For each element carrying color and/or background in its inline style, the effective foreground and background are the element's own values, falling back to the nearest ancestor with an inline value. If either is unknown (or uses alpha, var(), gradients) the element is skipped.

Relative luminance per WCAG 2.x: L = 0.2126 R + 0.7152 G + 0.0722 B with each channel linearised (c/12.92 or ((c+0.055)/1.055)^2.4). Ratio (L1 + 0.05) / (L2 + 0.05) with L1 the lighter.

Worked example: #777777 on #ffffff: channel 0x77/255 = 0.4667, linearised 0.1845, so L = 0.1845. Ratio 1.05 / 0.2345 = 4.48, which fails the 4.5:1 AA threshold by a hair. #767676 gives 4.54 and passes. The same pair at 30px passes against the 3:1 large-text threshold.

Page weight

For a page: its own HTML size, plus every distinct same-origin asset it references with a 200 response, plus the url() references found inside its stylesheets (one level, which covers fonts and CSS images). srcset candidates other than src are verified for existence but not counted, since a browser downloads only one. External assets count toward the request budget but not toward weight because their size is unknown.

Sitemap consistency

Sitemap URLs are localized, fetched as seeds, and compared by final_url, so https://example.com/ in the sitemap matches the page served at /index.html. A listed URL that does not return 200 is sitemap-url-broken; an indexable page absent from the list is sitemap-missing-page. Pages with noindex, and pages whose canonical points elsewhere, are not expected in the sitemap.

All external URLs are collected, de-duplicated, then probed with HEAD; a GET follows when HEAD fails with 400/403/404/405/406/501 or errors. A per-host lock enforces rate_per_host, and a semaphore bounds total concurrency. Results are cached for the run in Site.external.

Reports