Center
The training center and the check center — API and web. Design, not yet built.
Planned
The center ships after the library and the integrations. This page is the design. You will be able to self-host it (a docker-compose for PostgreSQL, Redis, the API and the web app) or use the hosted instance.
The center is where evals become a practice shared by a team. It is two halves on one API and one PostgreSQL database.
Training center
- Datasets with immutable versions. Content-hashed snapshots; a diff between any two versions; runs pin a version.
- Harvest from traffic. Pull traces a monitor scored low, user-flagged ones, or a random sample into a dataset; near-duplicates are dropped by embedding similarity (pgvector).
- Labeling queue. Tasks to grade, compare A/B, or verify an expected value; assignees; keyboard-driven UI; labels feed the golden set.
- Judge calibration. Run a judge version over a labeled set; get agreement, Cohen's kappa and a confusion matrix; promote the judge when it clears your threshold. Re-run when the judge model changes.
- Iteration log. Prompt and model versions and the runs they produced, so "did v7 beat v6" is a click.
Check center
- Run history. Every run from CLI, CI, MCP or a monitor; baselines; a sample-by-sample diff with the judge's reasons.
- Catalog. The built-in checks and your organization's, versioned, so a check means the same thing in every project.
- Monitors and alerts. Standing checks over sampled production traces with a windowed alert policy; email and webhook notifications.
- Live updates. Run progress and alerts over a WebSocket.
API
Everything above is a REST endpoint with OpenAPI docs: organizations, projects, ingest keys, datasets and versions, checks, suites, runs and results, label queues, judges and calibrations, traces, monitors, alerts. Users authenticate with JWT; the library, CI and proxies use a per-project ingest key.
Stack
FastAPI · SQLAlchemy 2 + Alembic · PostgreSQL 16 with pgvector · Redis · Celery for calibration jobs, harvest embedding and monitor scoring. Next.js for the web app.