apiprobe

apiprobe is an OpenAPI contract tester and property-based API fuzzer. Point it at an OpenAPI 3.0/3.1 spec and a running API: it generates valid, boundary and deliberately invalid requests for every operation, sends them, and checks every response against the spec. It finds 500s, schema violations, undocumented status codes, inputs that should have been rejected, and stateful bugs such as deleted resources that can still be fetched. Each failure is shrunk to a minimal request and printed as a curl command you can paste into a bug report. Schemathesis-style testing, written from scratch with three small dependencies (click, rich, PyYAML).

Features

• Spec loader: YAML or JSON, local file or URL; internal, relative-file and recursive $refs (cycle-safe); allOf/oneOf/anyOf; path/query/header/cookie parameters; JSON, form-urlencoded and multipart bodies; apiKey, bearer and basic security with credentials from flags or environment.
• Own data generator (seeded): types, formats (date, date-time, email, uuid, uri, ipv4), bounds incl. exclusive bounds in both 3.0 and 3.1 style, multipleOf, regex patterns, enum/const, arrays with uniqueItems, objects with required/additionalProperties, nullable, oneOf; spec examples first, then boundary values, then random values (including non-ASCII strings).
• Negative testing: requests that violate exactly one constraint and must be answered with 4xx.
• Shrinking: failing requests are minimised (optional fields removed, strings/arrays shortened, numbers reduced) into a minimal curl reproducer.
• Checks: no 5xx, documented status codes, response schema (with JSON pointer), content type, documented response headers, negative-data rejection, response-time budget.
• Stateful testing: OpenAPI links plus heuristics (POST /pets returning id links to /pets/{petId}) drive create -> read -> update -> read -> delete -> read(404) sequences.
• CLI: run, lint (spec smells), replay (re-run a saved failure), coverage.
• Reports: live progress, summary table, JUnit XML for CI, self-contained HTML report.

Install

cd apiprobe
uv venv .venv
uv pip install -p .venv/bin/python -e '.[dev]'
source .venv/bin/activate

Quickstart

The repository ships a deliberately buggy Pet Store (stdlib http.server only) with five planted bugs.

python examples/buggy_api/server.py 8141 &          # start the target API
apiprobe run examples/buggy_api/openapi.yaml \
    --base-url http://localhost:8141 --seed 1 --workers 4 \
    --auth apikey:letmein --report report.html --junit junit.xml

Finding the five planted bugs

Real output of the command above against a freshly started server:

apiprobe Buggy Pet Store -> http://localhost:8141  (seed 1, 5 operations)

Summary                                                         
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Operation            ┃ Requests ┃ Statuses        ┃ Failures ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ GET /pets            │       75 │ 200 400         │        0 │
│ POST /pets           │      167 │ 201 400 409 500 │        4 │
│ GET /pets/{petId}    │       87 │ 200 400 404     │        2 │
│ PUT /pets/{petId}    │       81 │ 200 400 404     │        0 │
│ DELETE /pets/{petId} │       81 │ 204 400 404     │        0 │
└──────────────────────┴──────────┴─────────────────┴──────────┘

1. response_schema_conformance GET /pets/{petId}
   response body does not match schema: 'created_at' is a required property  (at /created_at)
   Reproducer:
     curl -X GET http://localhost:8141/pets/1

2. stateful:deleted_resource_still_available GET /pets/{petId}
   resource is still retrievable via GET /pets/{petId} after DELETE (status 200, expected 404/410)
   Reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "K.B", "age": 16}'
     curl -X GET http://localhost:8141/pets/51
     curl -X PUT http://localhost:8141/pets/51 -H 'Content-Type: application/json' --data-binary '{"name": "xnY9NsVPu1sFDYZ", "age": 22, "tag": "bird"}'
     curl -X DELETE http://localhost:8141/pets/51 -H 'X-API-Key: letmein'
     curl -X GET http://localhost:8141/pets/51

3. negative_data_rejection POST /pets
   invalid request was accepted with status 201 (body /age: violates minimum (0))
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "Z", "age": -1, "tag": "dog"}'

4. not_a_server_error POST /pets
   server error 500
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "ø", "age": 0}'

5. response_schema_conformance POST /pets
   response body does not match schema: -1 is less than 0  (at /age)
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "W", "age": -1, "tag": "dog"}'

6. status_code_conformance POST /pets
   status 409 is not documented (documented: 201, 400)
   Minimal reproducer:
     curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "a", "age": 0}'

6 failures in 491 requests (0.1s)
HTML report: report.html
JUnit XML:   junit.xml
Failures saved to .apiprobe/failures (replay with: apiprobe replay 
.apiprobe/failures/001-response-schema-conformance-get-pets-petid.json)

The end-to-end test tests/test_e2e.py::test_all_five_planted_bugs_are_found asserts all five.

Usage

apiprobe run SPEC

apiprobe run spec.yaml --base-url http://localhost:8000 --max-examples 50 --seed 1 --workers 4 \
    --include 'tag:pets' --exclude 'DELETE *' --auth bearer:$TOKEN --report report.html --junit junit.xml

Exit status: 0 when no check failed, 1 otherwise.

apiprobe replay FAILURE.json

Every failure is saved to .apiprobe/failures/NNN-<check>-<operation>.json with the spec path, the request(s) and the base URL. Replay re-sends it and re-runs the checks (exit 1 when it still reproduces):

$ apiprobe replay .apiprobe/failures/004-not-a-server-error-post-pets.json
step 1 -> 500
  curl -X POST http://localhost:8141/pets -H 'Content-Type: application/json' --data-binary '{"name": "ø", "age": 0}'
  ! not_a_server_error: server error 500
reproduced not_a_server_error

Stateful failures are replayed as a sequence; ids from the new create response are substituted into later steps via the recorded link bindings. --base-url and --spec override the stored values.

apiprobe coverage

Reads .apiprobe/coverage.json from the last run (documented codes seen are green, unseen are red):

$ apiprobe coverage
Coverage
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Operation            ┃ Requests ┃ Status codes (documented: seen) ┃ Parameters ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ GET /pets            │ 75       │ 200 400                         │ 2/2        │
│ POST /pets           │ 167      │ 201 400  undocumented: 409 500  │ 0/0        │
│ GET /pets/{petId}    │ 87       │ 200 400 404                     │ 1/1        │
│ PUT /pets/{petId}    │ 81       │ 200 400 404                     │ 1/1        │
│ DELETE /pets/{petId} │ 81       │ 204 400 401 404                 │ 1/1        │
└──────────────────────┴──────────┴─────────────────────────────────┴────────────┘
operations 5/5 | documented status codes 13/14 | parameters 5/5

(401 on DELETE is never seen because every request carried a valid API key.)

apiprobe lint SPEC

$ apiprobe lint examples/smelly_api.yaml
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity ┃ Rule                            ┃ Location                     ┃ Message                                  ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ error    │ duplicate-operation-id          │ DELETE /stock-items/{itemId} │ operationId 'getStockItem' also used by  │
│          │                                 │                              │ GET /stock-items/{itemId}                │
│ error    │ path-parameter-mismatch         │ DELETE /stock-items/{itemId} │ path variable {itemId} has no parameter  │
│          │                                 │                              │ definition                               │
│ error    │ path-parameter-mismatch         │ GET /stock-items/{itemId}    │ path parameter 'id' does not appear in   │
│          │                                 │                              │ the path                                 │
│ ...      │                                 │                              │                                          │
│ warning  │ inconsistent-property-naming    │ components.schemas.StockItem │ property 'unitPrice' is camelCase but    │
│          │                                 │                              │ most are snake_case                      │
│ warning  │ missing-operation-id            │ GET /warehouse_locations     │ operation has no operationId             │
│ warning  │ missing-response-schema         │ POST /stock-items            │ 201 response has no schema               │
│ info     │ no-error-response               │ GET /stock-items             │ no 4xx response documented               │
└──────────┴─────────────────────────────────┴──────────────────────────────┴──────────────────────────────────────────┘

--json prints machine-readable issues. Exit status is 1 when any error-level issue exists.

Configuration

Credentials are only attached to operations whose security requirement uses a matching scheme (header, query or cookie for apiKey).

Architecture

src/apiprobe/
  spec.py       load YAML/JSON (file or URL), resolve $refs into a (possibly cyclic) graph, extract Operations
  schema.py     dialect helpers: types/nullable, 3.0 vs 3.1 exclusive bounds, allOf flattening
  validator.py  JSON Schema validator returning every error with a JSON pointer
  regexgen.py   strings matching / not matching a regex, via Python's regex parse tree
  generator.py  seeded value generation: random, example, min/max boundary modes
  negative.py   enumerate constraint sites and build values violating exactly one of them
  cases.py      Case model, request building (json/form/multipart), curl rendering, auth, HTTP client
  checks.py     response checks and the Failure model
  shrink.py     greedy minimisation that keeps positive cases valid and negative cases invalid
  stateful.py   OpenAPI links + heuristic link inference, runtime expressions
  runner.py     per-operation plans, threading, dedupe, shrinking, stateful sequences, coverage
  report.py     JUnit XML, HTML report, coverage and failure files
  lint.py       spec smell rules
  replay.py     re-run saved failures
  cli.py        click commands with rich output
examples/
  buggy_api/server.py, openapi.yaml   Pet Store with five planted bugs
  smelly_api.yaml                     spec for the lint demo
docs/
  generation-and-shrinking.md   design of generation, negative cases, shrinking, stateful runs
  check-catalog.md              every check and lint rule

Running the tests

.venv/bin/pytest -q

The suite (86 tests, about 3 seconds) covers $ref resolution including cycles and URL loading, the validator against jsonschema as an oracle, generator validity across hundreds of random schemas (own validator and jsonschema), single-violation negative generation, shrinking, every check, link inference, JUnit XML structure, and end-to-end runs (library and CLI) against the buggy server on an ephemeral port.

Limitations

• Targets OpenAPI 3.0/3.1 only (no Swagger 2.0). XML bodies are not generated or validated.
• patternProperties, prefixItems, dependentRequired, if/then/else, contains and unevaluatedProperties are ignored by generator and validator.
• Regex support covers the common subset; lookarounds are reported as generation notes and skipped.
• Parameter serialisation uses the default styles (form + explode for query, simple for headers); deepObject and matrix/label path styles are not implemented.
• Shrinking and replay talk to the live API, so side effects apply: a replay of a create can fail differently (for example with 409) once the resource already exists.
• Stateful testing follows a single create -> read -> update -> delete chain per collection; it does not explore arbitrary operation orderings.