Specify & generate
API operations and behavioral stories turn into executable pytest cases.
Agent QA Harness
DEVELOPER TOOLS / EVALUATIONCLI · reproducible local demo
A test is only useful if you can trust the run.
THE CONTEXT
Generated API tests need both useful coverage and execution checks.
THE KEY DECISION
Validate generated Python before running it; retain evidence from every run.
THE RESULT
Reproducible local runs, regression reports and a configurable CI quality gate.
Agent QACONTRACT → EVIDENCE
MECHANISM STUDY / 03SCHEMATIC
TAKE A CLOSER LOOKSELECT A STAGE ↓
SCROLL TO TRACE. SELECT TO EXPLORE.
Follow the connections as you scroll, or select a stage to see what happens there.
Inspect before running
An AST policy rejects unsafe behavior before bounded child-process execution.
Compare & gate
Persist observations, compare runs and surface regressions in a quality gate.
An API QA harness that connects specifications, guarded test generation, bounded execution and regression evidence in a single command-line workflow.
PROJECT TYPE
Developer tooling & QA infrastructure
FOCUS
Developer tools + Backend + AI / ML
STATUS
CLI · reproducible local demo
/01
The question.
Generated tests introduce two questions: do they cover the contract, and what will their code do when executed? A passing run also needs context from previous runs.
/02
The approach.
OpenAPI 3.x or behavioral stories feed deterministic or provider-assisted generation. AST validation rejects prohibited Python behavior before a child-process pytest runner executes accepted tests. Telemetry, schema checks and evaluator signals persist with each run.
/03
What came out of it.
Markdown and JSON reports expose results, latency percentiles and regression deltas. The CI workflow can block a pull request on the quality gate. A credential-free FastAPI demo makes the local path reproducible.
01
Two generation paths
Deterministic generation supports the local demo; a configured provider can handle broader prose and repair attempts.
02
A risk-reducing boundary
AST inspection, time limits and a child process are useful controls. They do not constitute a container or operating-system sandbox.
03
Regression as a first-class output
Run persistence and explicit delta calculations make pass/fail and percentile changes inspectable, with both machine-readable and reviewer-facing reports.
Under the surface.
- Inputs & generation
- OpenAPI 3.x · behavioral stories · Python AST · optional LiteLLM
- Execution & evaluation
- pytest subprocess · HTTP telemetry · deterministic metrics
- Persistence & delivery
- SQLAlchemy · SQLite · Markdown/JSON · GitHub Actions
SCOPE & LIMITS
- AST checks and a child process do not replace an OS sandbox.
- Provider-free factuality evaluation is heuristic.
- The diagram is a schematic run, not a new measured test report.
NEXT PROJECT / 04
Have an idea of your own? Let’s talk