Verdict
LLM eval & observability
Live Demo
Overview
Verdict is the tool I built after shipping enough AI features to be scared of shipping them blind. It runs prompts against a golden dataset, scores the outputs with a mix of deterministic checks and an LLM-as-judge, and fails the build when quality regresses — the same way a test suite fails on a broken function. Alongside correctness it tracks cost and latency per run, so a change that quietly doubles the token bill or the p95 shows up as a number, not a surprise on the invoice.
The Challenge
Traditional tests assume a deterministic answer. LLM outputs are fuzzy, non-deterministic, and judged on qualities like 'does it cite a source' or 'is the tone right' — none of which a strict equality check captures. Without a real eval loop, every prompt tweak or model swap is a leap of faith: you fix one case by hand, ship it, and discover a week later that you broke three others you never looked at.
The Solution
Treat evaluation as CI. A golden dataset pins the inputs and expected properties. Each case is scored by the right tool for the job — exact match and schema validation where the answer is structured, an LLM-as-judge with a rubric where it's qualitative. Results roll up to a pass rate with a threshold; drop below it and the gate blocks the merge. Every run records tokens, cost, and latency so regressions in spend and speed are caught with the same rigor as regressions in quality. Traces make it trivial to open a failing case and see exactly what the model saw.
Architecture
An eval pipeline modeled on a test runner: datasets in, scored results out, a gate in the middle, and full observability on every dimension that matters.
Key Decisions & Tradeoffs
Evals run in CI, not by hand
Why: An eval you run manually is an eval you skip under deadline. Wiring the suite into the pipeline so it blocks a bad merge is what makes it actually protect quality instead of being a nice idea in a notebook.
Right scorer for each case
Why: LLM-as-judge is powerful and also slower, costlier, and noisier. Where an output is structured, a schema check or exact match is faster and more reliable; the judge is reserved for the genuinely qualitative cases where nothing else works.
Track cost and latency as first-class results
Why: Quality isn't the only thing that regresses. A prompt that adds 40% more tokens or a model swap that doubles p95 is a real regression too — and invisible unless you measure it every run, right next to correctness.
A judge with a rubric, not a vibe
Why: Asking a model 'is this good?' gives you noise. Giving the judge an explicit rubric and specific criteria per case makes its scores stable enough to gate on, and makes a failure explainable rather than mysterious.
What I took away from this project
Lessons that still shape how I build — in my own words.
You can't ship AI features you can't measure
Every AI project I've built eventually hit the same wall: I'd improve one case and silently regress others, with no way to know until a user found it. Verdict came out of the realization that the eval harness isn't overhead on top of the AI work — it's the thing that lets you change the AI work at all without holding your breath.
LLM-as-judge is a tool, not a default
It's tempting to grade everything with a model because it's easy to wire up. But a judge is another non-deterministic call with its own cost and failure modes. The evals I trust most are the boring deterministic ones; the judge earns its place only where the property really is subjective, and even then only with a tight rubric.
Impact
Technology Stack
AI
Eval
CI
Observability
Key Features
Want something like this, built for you?
If the Verdict story sounds close to what you're trying to ship, a 30-min call is the fastest way to find out if we're a fit.