An evaluation harness for LLM features: golden datasets, LLM-as-judge scoring, and regression gates in CI — plus cost, latency, and quality tracked on every run.
Verdict is the tool I built after shipping enough AI features to be scared of shipping them blind.
It runs prompts against a golden dataset, scores the outputs with a mix of deterministic checks and an LLM-as-judge, and fails the build when quality regresses — the same way a test suite fails on a broken function. Alongside correctness it tracks cost and latency per run, so a change that quietly doubles the token bill or the p95 shows up as a number, not a surprise on the invoice.
The challenge
Traditional tests assume a deterministic answer.
LLM outputs are fuzzy, non-deterministic, and judged on qualities like 'does it cite a source' or 'is the tone right' — none of which a strict equality check captures. Without a real eval loop, every prompt tweak or model swap is a leap of faith: you fix one case by hand, ship it, and discover a week later that you broke three others you never looked at.
Fuzzy outputsno strict equality check
Silent regressionsfix one, break three
Hidden spendtokens and p95 drift
The solution
Treat evaluation as CI.
A golden dataset pins the inputs and expected properties. Each case is scored by the right tool for the job — exact match and schema validation where the answer is structured, an LLM-as-judge with a rubric where it's qualitative. Results roll up to a pass rate with a threshold; drop below it and the gate blocks the merge. Every run records tokens, cost, and latency so regressions in spend and speed are caught with the same rigor as regressions in quality. Traces make it trivial to open a failing case and see exactly what the model saw.
Golden setScoreGate
Architecture
A test runner for prompts.
An eval pipeline modeled on a test runner: datasets in, scored results out, a gate in the middle, and full observability on every dimension that matters.
Datasets
Golden input sets
Expected properties
Versioned fixtures
Scoring
Deterministic checks
Schema validation
LLM-as-judge with rubric
Gates
Pass-rate thresholds
CI merge blocking
Regression diffs
Observability
Per-run cost & tokens
p95 latency
Full request traces
One request, start to finish
A change touches a prompt or model
The suite runs against the golden set
Each case is scored by check or judge
The gate passes or blocks on the threshold
Cost, latency, and traces are recorded
Decisions
Four calls and why I made them.
Call 01
Evals run in CI, not by hand
An eval you run manually is an eval you skip under deadline. Wiring the suite into the pipeline so it blocks a bad merge is what makes it actually protect quality instead of being a nice idea in a notebook.
Call 02
Right scorer for each case
LLM-as-judge is powerful and also slower, costlier, and noisier. Where an output is structured, a schema check or exact match is faster and more reliable; the judge is reserved for the genuinely qualitative cases where nothing else works.
Call 03
Track cost and latency as first-class results
Quality isn't the only thing that regresses. A prompt that adds 40% more tokens or a model swap that doubles p95 is a real regression too — and invisible unless you measure it every run, right next to correctness.
Call 04
A judge with a rubric, not a vibe
Asking a model 'is this good?' gives you noise. Giving the judge an explicit rubric and specific criteria per case makes its scores stable enough to gate on, and makes a failure explainable rather than mysterious.
Reflections
What I took away.
You can't ship AI features you can't measure
Every AI project I've built eventually hit the same wall: I'd improve one case and silently regress others, with no way to know until a user found it. Verdict came out of the realization that the eval harness isn't overhead on top of the AI work — it's the thing that lets you change the AI work at all without holding your breath.
LLM-as-judge is a tool, not a default
It's tempting to grade everything with a model because it's easy to wire up. But a judge is another non-deterministic call with its own cost and failure modes. The evals I trust most are the boring deterministic ones; the judge earns its place only where the property really is subjective, and even then only with a tight rubric.
Outcomes
What shipped.
GateIn CIBlocks on regression
ScoringHybridChecks + LLM judge
TrackedCost·p95Every run
FailuresTracedSee what the model saw
Stack
Technology
AI
Claude API
LLM-as-judge
Structured output
Eval
Golden datasets
Deterministic checks
Schema validation
Rubric scoring
CI
Merge gates
Regression diffs
Threshold policies
Observability
Cost & token tracking
p95 latency
Request traces
Features
What it does
Golden datasets that pin inputs and expected properties as versioned fixtures
Hybrid scoring: deterministic checks and schema validation where possible, LLM-as-judge with a rubric where needed
Regression gates wired into CI that block a merge when the pass rate drops below threshold
Per-run cost, token, and p95 latency tracking so spend and speed regressions surface like quality ones
Full request traces on every case to open a failure and see exactly what the model received
Model- and prompt-swap safety: change the AI layer and know within one run whether anything broke