Evals 101
A short course on evaluating LLM features, for people who ship them.
Nine chapters, each a few minutes. Read them in order the first time; each one builds on the previous. Nothing here requires Evalstone; the concepts apply to any tool, including a spreadsheet.
1 · What are evals?
Tests for a system that does not answer the same way twice.
2 · Why you need them
The regressions you did not see.
3 · The eval loop
Dataset → run → inspect → fix → re-run → gate.
4 · Kinds of checks
When to use rules, judges, humans, comparisons.
5 · Building a golden set
Harvest, label, split, version.
6 · LLM-as-judge done right
Rubrics, bias, calibration, cost.
7 · Gates and CI
What to block a pull request on.
8 · Evals in production
Sampling, monitors, drift.
9 · Pitfalls
The ways evals go green while the product is broken.
The one-paragraph version
An LLM feature is a function whose output you cannot predict exactly, so you cannot assert equality like a unit test. Instead you keep a dataset of representative inputs, run your feature over all of them, check each output with a rule, a model, or a person, and turn the scores into a gate: a yes or no that a change is good enough to ship. Then you do it again every time something changes. That loop is "evals".