Verdict's owl in a judge's collar with a gavel and golden scales

AI productAI LabCase study

Verdict

LLM eval & release gate.

An evaluation harness for LLM features: golden datasets, LLM-as-judge scoring, and regression gates in CI — plus cost, latency, and quality tracked on every run.

Role
Design and engineering
When
2025 to now
Team
Solo build
Read
About 4 min
Verdict's owl in a judge's collar with a gavel and golden scales
Verdict console in pen and ink: the golden set as a court docket with PASS and FAIL stamps, the judge owl and a blocked release-gate stamp

Overview

Overview

Verdict is the tool I built after shipping enough AI features to be scared of shipping them blind.

It runs prompts against a golden dataset, scores the outputs with a mix of deterministic checks and an LLM-as-judge, and fails the build when quality regresses — the same way a test suite fails on a broken function. Alongside correctness it tracks cost and latency per run, so a change that quietly doubles the token bill or the p95 shows up as a number, not a surprise on the invoice.

The challenge

Traditional tests assume a deterministic answer.

LLM outputs are fuzzy, non-deterministic, and judged on qualities like 'does it cite a source' or 'is the tone right' — none of which a strict equality check captures. Without a real eval loop, every prompt tweak or model swap is a leap of faith: you fix one case by hand, ship it, and discover a week later that you broke three others you never looked at.

  • Fuzzy outputsno strict equality check
  • Silent regressionsfix one, break three
  • Hidden spendtokens and p95 drift

The solution

Treat evaluation as CI.

A golden dataset pins the inputs and expected properties. Each case is scored by the right tool for the job — exact match and schema validation where the answer is structured, an LLM-as-judge with a rubric where it's qualitative. Results roll up to a pass rate with a threshold; drop below it and the gate blocks the merge. Every run records tokens, cost, and latency so regressions in spend and speed are caught with the same rigor as regressions in quality. Traces make it trivial to open a failing case and see exactly what the model saw.

Architecture

A test runner for prompts.

An eval pipeline modeled on a test runner: datasets in, scored results out, a gate in the middle, and full observability on every dimension that matters.

Datasets
  • Golden input sets
  • Expected properties
  • Versioned fixtures
Scoring
  • Deterministic checks
  • Schema validation
  • LLM-as-judge with rubric
Gates
  • Pass-rate thresholds
  • CI merge blocking
  • Regression diffs
Observability
  • Per-run cost & tokens
  • p95 latency
  • Full request traces

One request, start to finish

  1. A change touches a prompt or model
  2. The suite runs against the golden set
  3. Each case is scored by check or judge
  4. The gate passes or blocks on the threshold
  5. Cost, latency, and traces are recorded

Decisions

Four calls and why I made them.

Call 01

Evals run in CI, not by hand

An eval you run manually is an eval you skip under deadline. Wiring the suite into the pipeline so it blocks a bad merge is what makes it actually protect quality instead of being a nice idea in a notebook.

Call 02

Right scorer for each case

LLM-as-judge is powerful and also slower, costlier, and noisier. Where an output is structured, a schema check or exact match is faster and more reliable; the judge is reserved for the genuinely qualitative cases where nothing else works.

Call 03

Track cost and latency as first-class results

Quality isn't the only thing that regresses. A prompt that adds 40% more tokens or a model swap that doubles p95 is a real regression too — and invisible unless you measure it every run, right next to correctness.

Call 04

A judge with a rubric, not a vibe

Asking a model 'is this good?' gives you noise. Giving the judge an explicit rubric and specific criteria per case makes its scores stable enough to gate on, and makes a failure explainable rather than mysterious.

Reflections

What I took away.

You can't ship AI features you can't measure

Every AI project I've built eventually hit the same wall: I'd improve one case and silently regress others, with no way to know until a user found it. Verdict came out of the realization that the eval harness isn't overhead on top of the AI work — it's the thing that lets you change the AI work at all without holding your breath.

LLM-as-judge is a tool, not a default

It's tempting to grade everything with a model because it's easy to wire up. But a judge is another non-deterministic call with its own cost and failure modes. The evals I trust most are the boring deterministic ones; the judge earns its place only where the property really is subjective, and even then only with a tight rubric.

Outcomes

What shipped.

  • GateIn CIBlocks on regression
  • ScoringHybridChecks + LLM judge
  • TrackedCost·p95Every run
  • FailuresTracedSee what the model saw

Stack

Technology

AI

  • Claude API
  • LLM-as-judge
  • Structured output

Eval

  • Golden datasets
  • Deterministic checks
  • Schema validation
  • Rubric scoring

CI

  • Merge gates
  • Regression diffs
  • Threshold policies

Observability

  • Cost & token tracking
  • p95 latency
  • Request traces

Features

What it does

  • Golden datasets that pin inputs and expected properties as versioned fixtures
  • Hybrid scoring: deterministic checks and schema validation where possible, LLM-as-judge with a rubric where needed
  • Regression gates wired into CI that block a merge when the pass rate drops below threshold
  • Per-run cost, token, and p95 latency tracking so spend and speed regressions surface like quality ones
  • Full request traces on every case to open a failure and see exactly what the model received
  • Model- and prompt-swap safety: change the AI layer and know within one run whether anything broke

Keep reading

More case studies

Your project

Building something like this?

AI, voice, privacy constraints or a mobile app that can’t afford to break. Free 30-minute call, no pitch, clear next steps either way.

Email
Back to
All work