# apiprobe

apiprobe is an OpenAPI contract tester and property-based API fuzzer. Point it at an OpenAPI 3.0/3.1
spec and a running API: it generates valid, boundary and deliberately invalid requests for every
operation, sends them, and checks every response against the spec. It finds 500s, schema violations,
undocumented status codes, inputs that should have been rejected, and stateful bugs such as deleted
resources that can still be fetched. Each failure is shrunk to a minimal request and printed as a
`curl` command you can paste into a bug report. Schemathesis-style testing, written from scratch with
three small dependencies (click, rich, PyYAML).

## Features

- **Spec loader**: YAML or JSON, local file or URL; internal, relative-file and recursive `$ref`s
  (cycle-safe); `allOf`/`oneOf`/`anyOf`; path/query/header/cookie parameters; JSON, form-urlencoded and
  multipart bodies; apiKey, bearer and basic security with credentials from flags or environment.
- **Own data generator** (seeded): types, formats (date, date-time, email, uuid, uri, ipv4), bounds incl.
  exclusive bounds in both 3.0 and 3.1 style, `multipleOf`, regex `pattern`s, enum/const, arrays with
  `uniqueItems`, objects with `required`/`additionalProperties`, nullable, `oneOf`; spec examples first,
  then boundary values, then random values (including non-ASCII strings).
- **Negative testing**: requests that violate exactly one constraint and must be answered with 4xx.
- **Shrinking**: failing requests are minimised (optional fields removed, strings/arrays shortened,
  numbers reduced) into a minimal `curl` reproducer.
- **Checks**: no 5xx, documented status codes, response schema (with JSON pointer), content type,
  documented response headers, negative-data rejection, response-time budget.
- **Stateful testing**: OpenAPI `links` plus heuristics (POST `/pets` returning `id` links to
  `/pets/{petId}`) drive create -> read -> update -> read -> delete -> read(404) sequences.
- **CLI**: `run`, `lint` (spec smells), `replay` (re-run a saved failure), `coverage`.
- **Reports**: live progress, summary table, JUnit XML for CI, self-contained HTML report.

## Install

```bash
cd apiprobe
uv venv .venv
uv pip install -p .venv/bin/python -e '.[dev]'
source .venv/bin/activate
```

## Quickstart

The repository ships a deliberately buggy Pet Store (stdlib `http.server` only) with five planted bugs.

```bash
python examples/buggy_api/server.py 8141 &          # start the target API
apiprobe run examples/buggy_api/openapi.yaml \
    --base-url http://localhost:8141 --seed 1 --workers 4 \
    --auth apikey:letmein --report report.html --junit junit.xml
```

## Finding the five planted bugs

Real output of the command above against a freshly started server:

```
apiprobe Buggy Pet Store -> http://localhost:8141  (seed 1, 5 operations)

Summary                                                         
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Operation            ┃ Requests ┃ Statuses        ┃ Failures ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ GET /pets            │       75 │ 200 400         │        0 │
│ POST /pets           │      167 │ 201 400 409 500 │        4 │
│ GET /pets/{petId}    │       87 │ 200 400 404     │        2 │
│ PUT /pets/{petId}    │       81 │ 200 400 404     │        0 │
│ DELETE /pets/{petId} │       81 │ 204 400 404     │        0 │
└──────────────────────┴──────────┴─────────────────┴──────────┘

1. response_schema_conformance GET /pets/{petId}
   response body does not match schema: 'created_at' is a required property  (at /created_at)
   Reproducer:
     curl -X GET http://localhost:8141/pets/1

2. stateful:deleted_resource_still_available GET /pets/{petId}
   resource is still retrievable via GET /pets/{petId} after DELETE (status 200, expected 404/410)
   Reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "K.B", "age": 16}'
     curl -X GET http://localhost:8141/pets/51
     curl -X PUT http://localhost:8141/pets/51 -H 'Content-Type: application/json' --data-binary '{"name": "xnY9NsVPu1sFDYZ", "age": 22, "tag": "bird"}'
     curl -X DELETE http://localhost:8141/pets/51 -H 'X-API-Key: letmein'
     curl -X GET http://localhost:8141/pets/51

3. negative_data_rejection POST /pets
   invalid request was accepted with status 201 (body /age: violates minimum (0))
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "Z", "age": -1, "tag": "dog"}'

4. not_a_server_error POST /pets
   server error 500
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "ø", "age": 0}'

5. response_schema_conformance POST /pets
   response body does not match schema: -1 is less than 0  (at /age)
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "W", "age": -1, "tag": "dog"}'

6. status_code_conformance POST /pets
   status 409 is not documented (documented: 201, 400)
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "a", "age": 0}'

6 failures in 491 requests (0.1s)
HTML report: report.html
JUnit XML:   junit.xml
Failures saved to .apiprobe/failures (replay with: apiprobe replay 
.apiprobe/failures/001-response-schema-conformance-get-pets-petid.json)
```

| # | Planted bug (see `examples/buggy_api/server.py`) | How apiprobe reports it |
|---|------------------------------------------------|--------------------------|
| 1 | 500 on a non-ASCII pet name | failure 4, `not_a_server_error`, shrunk to `{"name": "ø", "age": 0}` |
| 2 | `GET /pets/{petId}` omits the required `created_at` | failure 1, `response_schema_conformance` at `/created_at` |
| 3 | duplicate names answered with an undocumented 409 | failure 6, `status_code_conformance` |
| 4 | DELETE only soft-deletes; GET still returns the pet | failure 2, `stateful:deleted_resource_still_available`, full 5-step sequence |
| 5 | negative `age` accepted | failure 3, `negative_data_rejection` (`/age: violates minimum (0)`); failure 5 is the knock-on effect: the 201 response echoes `age: -1`, which violates the response schema |

The end-to-end test `tests/test_e2e.py::test_all_five_planted_bugs_are_found` asserts all five.

## Usage

### `apiprobe run SPEC`

```
apiprobe run spec.yaml --base-url http://localhost:8000 --max-examples 50 --seed 1 --workers 4 \
    --include 'tag:pets' --exclude 'DELETE *' --auth bearer:$TOKEN --report report.html --junit junit.xml
```

| Option | Default | Meaning |
|--------|---------|---------|
| `--base-url` | first `servers` entry | API root |
| `--max-examples` | 50 | positive cases per operation (example + min + max + random); negative cases = max(#violations, N/2) |
| `--seed` | 0 | RNG seed; identical seeds give identical requests per operation |
| `--workers` | 1 | operations tested in parallel |
| `--include` / `--exclude` | | repeatable: `tag:pets`, `id:getPet` (glob), `'GET /pets/*'` (method glob + path glob), `/pets*` (path glob) |
| `--auth` | | repeatable: `bearer:TOKEN`, `basic:user:pass`, `apikey:VALUE`, `apikey:X-API-Key=VALUE` |
| `--header` | | extra `'Name: value'` header on every request |
| `--time-budget` | 2000 | response-time budget in ms |
| `--timeout` | 10 | per-request timeout in seconds |
| `--no-negative` / `--no-stateful` / `--no-shrink` | | turn phases off |
| `--report` / `--junit` | | HTML / JUnit XML output paths |
| `--state-dir` | `.apiprobe` | where `coverage.json` and `failures/*.json` are written |

Exit status: 0 when no check failed, 1 otherwise.

### `apiprobe replay FAILURE.json`

Every failure is saved to `.apiprobe/failures/NNN-<check>-<operation>.json` with the spec path, the
request(s) and the base URL. Replay re-sends it and re-runs the checks (exit 1 when it still reproduces):

```
$ apiprobe replay .apiprobe/failures/004-not-a-server-error-post-pets.json
step 1 -> 500
  curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "ø", "age": 0}'
  ! not_a_server_error: server error 500
reproduced not_a_server_error
```

Stateful failures are replayed as a sequence; ids from the new create response are substituted into
later steps via the recorded link bindings. `--base-url` and `--spec` override the stored values.

### `apiprobe coverage`

Reads `.apiprobe/coverage.json` from the last run (documented codes seen are green, unseen are red):

```
$ apiprobe coverage
Coverage
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Operation            ┃ Requests ┃ Status codes (documented: seen) ┃ Parameters ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ GET /pets            │ 75       │ 200 400                         │ 2/2        │
│ POST /pets           │ 167      │ 201 400  undocumented: 409 500  │ 0/0        │
│ GET /pets/{petId}    │ 87       │ 200 400 404                     │ 1/1        │
│ PUT /pets/{petId}    │ 81       │ 200 400 404                     │ 1/1        │
│ DELETE /pets/{petId} │ 81       │ 204 400 401 404                 │ 1/1        │
└──────────────────────┴──────────┴─────────────────────────────────┴────────────┘
operations 5/5 | documented status codes 13/14 | parameters 5/5
```

(401 on DELETE is never seen because every request carried a valid API key.)

### `apiprobe lint SPEC`

```
$ apiprobe lint examples/smelly_api.yaml
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity ┃ Rule                            ┃ Location                     ┃ Message                                  ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ error    │ duplicate-operation-id          │ DELETE /stock-items/{itemId} │ operationId 'getStockItem' also used by  │
│          │                                 │                              │ GET /stock-items/{itemId}                │
│ error    │ path-parameter-mismatch         │ DELETE /stock-items/{itemId} │ path variable {itemId} has no parameter  │
│          │                                 │                              │ definition                               │
│ error    │ path-parameter-mismatch         │ GET /stock-items/{itemId}    │ path parameter 'id' does not appear in   │
│          │                                 │                              │ the path                                 │
│ ...      │                                 │                              │                                          │
│ warning  │ inconsistent-property-naming    │ components.schemas.StockItem │ property 'unitPrice' is camelCase but    │
│          │                                 │                              │ most are snake_case                      │
│ warning  │ missing-operation-id            │ GET /warehouse_locations     │ operation has no operationId             │
│ warning  │ missing-response-schema         │ POST /stock-items            │ 201 response has no schema               │
│ info     │ no-error-response               │ GET /stock-items             │ no 4xx response documented               │
└──────────┴─────────────────────────────────┴──────────────────────────────┴──────────────────────────────────────────┘
```

`--json` prints machine-readable issues. Exit status is 1 when any error-level issue exists.

## Configuration

| Environment variable | Equivalent flag |
|----------------------|-----------------|
| `APIPROBE_BEARER` | `--auth bearer:...` |
| `APIPROBE_BASIC` | `--auth basic:user:pass` |
| `APIPROBE_APIKEY` | `--auth apikey:...` (applies to every apiKey scheme) |

Credentials are only attached to operations whose `security` requirement uses a matching scheme
(header, query or cookie for apiKey).

## Architecture

```
src/apiprobe/
  spec.py       load YAML/JSON (file or URL), resolve $refs into a (possibly cyclic) graph, extract Operations
  schema.py     dialect helpers: types/nullable, 3.0 vs 3.1 exclusive bounds, allOf flattening
  validator.py  JSON Schema validator returning every error with a JSON pointer
  regexgen.py   strings matching / not matching a regex, via Python's regex parse tree
  generator.py  seeded value generation: random, example, min/max boundary modes
  negative.py   enumerate constraint sites and build values violating exactly one of them
  cases.py      Case model, request building (json/form/multipart), curl rendering, auth, HTTP client
  checks.py     response checks and the Failure model
  shrink.py     greedy minimisation that keeps positive cases valid and negative cases invalid
  stateful.py   OpenAPI links + heuristic link inference, runtime expressions
  runner.py     per-operation plans, threading, dedupe, shrinking, stateful sequences, coverage
  report.py     JUnit XML, HTML report, coverage and failure files
  lint.py       spec smell rules
  replay.py     re-run saved failures
  cli.py        click commands with rich output
examples/
  buggy_api/server.py, openapi.yaml   Pet Store with five planted bugs
  smelly_api.yaml                     spec for the lint demo
docs/
  generation-and-shrinking.md   design of generation, negative cases, shrinking, stateful runs
  check-catalog.md              every check and lint rule
```

## Running the tests

```bash
.venv/bin/pytest -q
```

The suite (86 tests, about 3 seconds) covers `$ref` resolution including cycles and URL loading, the
validator against `jsonschema` as an oracle, generator validity across hundreds of random schemas (own
validator and `jsonschema`), single-violation negative generation, shrinking, every check, link inference,
JUnit XML structure, and end-to-end runs (library and CLI) against the buggy server on an ephemeral port.

## Limitations

- Targets OpenAPI 3.0/3.1 only (no Swagger 2.0). XML bodies are not generated or validated.
- `patternProperties`, `prefixItems`, `dependentRequired`, `if/then/else`, `contains` and
  `unevaluatedProperties` are ignored by generator and validator.
- Regex support covers the common subset; lookarounds are reported as generation notes and skipped.
- Parameter serialisation uses the default styles (`form` + explode for query, `simple` for headers);
  `deepObject` and matrix/label path styles are not implemented.
- Shrinking and replay talk to the live API, so side effects apply: a replay of a create can fail
  differently (for example with 409) once the resource already exists.
- Stateful testing follows a single create -> read -> update -> delete chain per collection; it does
  not explore arbitrary operation orderings.