Open source · MIT · built in the open

Know your LLM feature works before your users do.

Evalstone is the plug-and-play evals layer for any LLM-based project: a Python library, a learning center, a check center, an API and an MCP server. First it teaches you what evals are. Then it runs them for you, in CI and in production.

Start with Evals 101How Evalstone helpsGitHub
evals/support_bot.py
import evalstone as es

suite = es.Suite(
    "support-bot",
    dataset=es.Dataset.from_jsonl("evals/golden.jsonl"),
    checks=[
        es.checks.json_schema(schema=Answer),
        es.checks.judge("helpfulness", rubric=RUBRIC),
        es.checks.rag.faithfulness(),
    ],
    gate=es.Gate(min_pass_rate=0.95),
)

run = suite.run(target=answer)   # your function
run.gate()                       # non-zero exit in CI

The problem

Shipping LLM features by vibes works until it doesn't

Every team that puts a model in front of users hits the same four walls.

"It looks fine" is not a test

You tweak a prompt, try three questions, and ship. The fourth question, the one a customer asks, is the one that broke. Non-deterministic systems need a repeatable answer to "is it better?".

Model updates move the floor

Providers retire and replace models. The same prompt gives different answers next month. Without a golden set and a baseline run you only find out from support tickets.

A judge you never measured is a guess

LLM-as-judge is powerful and cheap, and it is also biased, position-sensitive and happy to agree with itself. A judge earns trust by being calibrated against humans.

Production is the eval nobody runs

Offline suites catch regressions you anticipated. Real traffic finds the rest. Sampling and scoring live exchanges turns "we think it is fine" into a number with a trend.

Evals, in one minute

An eval is a test for a system that does not give the same answer twice

Unit tests assert equality. Evals measure quality over a dataset and decide whether the number is good enough. The loop is small, and it is the same loop whether you are grading a chatbot, a RAG pipeline, an extraction prompt or an agent.
  1. 1

    Dataset

    A set of inputs with, where possible, what a good answer looks like. Real traffic, hand-written, or synthetic.

  2. 2

    Run

    Send every input through your feature and keep the outputs. Pin the dataset version and the git commit.

  3. 3

    Check

    Score each output: deterministic rules, an LLM judge with a rubric, a human, or a comparison against another run.

  4. 4

    Gate

    Turn scores into a decision: pass rate above a threshold, no regression versus the baseline, cost under budget.

Read the full chapter

How Evalstone helps

One layer, from the first golden set to production monitoring

Everything is designed to be plugged into an existing project with no rewrite: a pip install, a YAML file, and a base of checks you extend in plain Python.

Learning center

live
A short course on evals: what they are, why they matter, the kinds of checks, judges, gates, production, and the pitfalls. Read it before you write a single test.

Library

building
Datasets, checks, suites, runs and gates as plain Python. Deterministic checks, LLM judges, RAG, safety, pairwise and agent-trajectory checks built in.

Training center

planned
Versioned golden sets harvested from real traffic, a human labeling queue, and judge calibration so your LLM judge agrees with your people.

Check center

planned
Run history with baselines and diffs, a catalog of reusable checks, and alerts when a monitor sees quality drift.

CI gate

building
A GitHub Action and a pytest plugin. A pull request that lowers the pass rate or regresses the baseline goes red.

Production monitoring

planned
Sample live exchanges, score them continuously, and see the trend. Works with any proxy or SDK that can post a trace.

API

planned
Everything the center does is a REST endpoint with OpenAPI docs, so other languages and other tools can push runs and read results.

MCP server

building
Let Claude Code, Cursor or any MCP client run a suite, judge an output or record a sample while you work.

How it works

Three steps, no rewrite

  1. 1

    Describe the suite

    One YAML file: where the dataset is, what produces outputs (a function, an HTTP endpoint, a prompt), which checks to run, what the gate is.

  2. 2

    Run it anywhere

    evalstone run on your laptop, in pytest, in a GitHub Action, or from an MCP client. Cheap checks run first; judge calls are cached and priced.

  3. 3

    Keep the history

    Point it at a center, self-hosted or ours, and every run lands in PostgreSQL: baselines, diffs, labels, calibrations, monitors.

evalstone.yaml
# evalstone.yaml
project: support-bot
datasets:
  golden: evals/golden.jsonl
suites:
  nightly:
    dataset: golden
    target: { kind: http, url: http://localhost:8000/answer }
    checks: [json_schema, helpfulness, faithfulness]
    gate: { min_pass_rate: 0.95, max_regression: 0.02 }
judges:
  default_model: anthropic/claude-haiku-5-5
center:                      # optional — persists runs in PostgreSQL
  url: https://api.evalstone.com

Learning center

Evals 101, nine short chapters

Written for people who ship LLM features and have never run an eval. Each chapter explains why, shows the pitfall, then the practice. Read in order or jump in.

1 · What are evals?

Tests for non-deterministic systems, and why "it looks fine" is not one.

2 · Why you need them

The regressions you did not see: model updates, prompt drift, and silent cost.

3 · The eval loop

Dataset → run → inspect → fix → re-run → gate. Small loop, run often.

4 · Kinds of checks

Deterministic, LLM-as-judge, human, pairwise, RAG, agent trajectory, safety.

5 · Building a golden set

Harvest, label, split, version. How big is big enough.

6 · LLM-as-judge done right

Rubrics, scales, position bias, calibration against humans, cost control.

7 · Gates and CI

Thresholds, baselines, flakiness, and what to actually block on.

8 · Evals in production

Sampling, monitors, drift, and avoiding alert fatigue.

9 · Pitfalls

Overfitting the golden set, leaking answers, judge collusion, green-but-broken.

Status

Built in the open, in this order

  1. live

    Site + learning center

  2. next

    Library + CLI on PyPI

  3. then

    MCP server · pytest · GitHub Action

  4. then

    Center: API + web, self-hosted or hosted

The roadmap, the design decisions and every commit are public. Star the repository to follow along, or open an issue with the eval problem you are stuck on; that is how the chapters get better.

Follow on GitHubRead the design