Design notes

Pipeline

DDL / SQLite / Django --> Schema --> draft plan (inference) + user plan --> Engine --> Sink
                                                                              |
                                                       validate <-- database -+--> mask

Schema is a small dataclass model (tables, columns with a coarse kind, PK, uniques, FKs, CHECK text). Everything downstream only sees that model, which is why DDL, live SQLite and Django models are interchangeable. SQLite introspection reads sqlite_master and feeds the stored DDL back through the DDL parser, so there is one parser to trust.

Deterministic seeding

Each (table, column) gets its own random.Random, seeded from sha256(seed, table, column, purpose). Row counts use the stream (table, "__rows__"), deferred FK fills (table, column, "deferred").

make_rng(42, "customers", "email").random()  -> 0.468...
make_rng(42, "customers", "phone").random()  -> 0.938...

Consequences: the same seed gives byte-identical files (test_same_seed_gives_identical_files); adding a table or column leaves every other stream untouched (test_adding_a_table_does_not_change_existing_data); changing --scale changes counts but not how a given column behaves. Retries consume the column's own stream, so determinism survives rejection sampling.

Ordering and cycles

Tables are emitted parents-first (Kahn's algorithm, ties broken alphabetically). Self references are not edges: a self-referencing FK picks among the rows already generated (the first row points at itself, or is NULL when nullable), so manager/employee trees are acyclic by construction.

For a real cycle, a.b_id -> b and b.a_id -> a:

a(id, b_id NULL)      b(id, a_id NOT NULL)      order = [a, b], deferred = {a: [b_id]}

a.b_id is generated as NULL, b is generated against the complete a, and then one UPDATE a SET b_id = ? WHERE id = ? pass fills it with random b keys. If every FK in the cycle is NOT NULL no valid insertion order exists and generation stops with foreign key cycle with no nullable column to defer: a, b.

Keys and pools

A column that is referenced by an FK gets a Pool. For a unit-step integer sequence the pool is just (start, n), so referencing 10 million parents costs nothing and a zipf or uniform pick is arithmetic. Other referenced columns (uuid or text keys) keep a list of values. A via lookup (a payment dated after its order) additionally keeps the parent's column in a list indexed by the parent's position.

Uniqueness and CHECKs in one loop

After a row's columns are generated in dependency order, the engine looks for violations:

  1. any CHECK that evaluates to false (SQL three-valued logic: NULL/unknown passes, like a database would),
  2. a single-column UNIQUE/PK value already in that column's seen set,
  3. a composite key tuple already in the table's key set.

The violated constraint names the columns to blame. Those columns, plus every column that depends on them (so line_total = quantity * unit_price is recomputed when quantity is redrawn), are regenerated, never columns fixed by the per-parent loop or sequences. This repeats up to 200 times. Worked example, order_items UNIQUE (order_id, product_id) with order_id fixed by the parent loop: a duplicate pair redraws only product_id, using the same zipf stream, until a free pair appears. For text columns the fourth failed attempt switches to a numeric suffix (smith -> smith4021, a@x.org -> a4021@x.org) because natural-language domains such as 100 surnames can never reach 5000 unique values.

Failure is explicit:

t: could not satisfy CHECK (n > 100) after 200 attempts (last row: {'id': 1, 'n': 7}). Adjust the generators or ranges in the plan.

To avoid paying rejection costs in the common cases, init reads simple CHECKs first: status IN (...) becomes a weighted choice, price > 0 / qty BETWEEN 1 AND 10 become ranges, and end >= start between date columns becomes an after generator.

Computed columns

orders.total = sum(order_items.line_total) cannot be known while orders is being streamed. The parent is written with a placeholder and each child row adds to a dict[parent_key] -> running total as it is accepted. When all tables are finished the engine hands the sink one mapping per computed column; sinks apply it natively (UPDATE batches, trailing UPDATE statements in a SQL dump, in-place rewrite of CSV/JSONL). Memory is O(parents) per aggregate, which is the one exception to constant memory besides unique-value sets.

Distributions

Correlated values

Generators receive the partially built row, so they can read earlier columns: city reads state/country, postal_code reads the state's leading digit, phone reads the country's dial code, email is built from the first/last name actually generated. Dependencies are declared by each generator and drive the column order.

Masking

Masker.fake(kind, value) seeds a PRNG from HMAC-SHA256(key, kind || salt || value). The same value always maps to the same fake regardless of table, order or run; a different key gives unrelated fakes. Kinds with a uniqueness expectation (email, phone, id, ip) keep an owner map and re-derive with salt + 1 on a collision, so the mapping stays injective for the run.

Format preservation: digits stay digits and letters keep their case (AB-12-cd becomes another XX-99-xx shaped value); phone numbers keep +91 / +49 / +1 and Indian numbers keep a leading 6-9; dates shift by up to 180 days in the same format; emails stay valid addresses on example.* domains.

FK joins: after detection, a fixpoint pass copies each masked column's kind to every column it references or is referenced by (logins.account -> users.email). Both sides are therefore transformed by the same pure function and still join. Surrogate integer keys are never masked, so those joins are untouched.

Detection order: explicit config, column-name rules, then value sniffing on up to 200 sample values (80% must match email / IPv4 / SSN-like / phone shapes).