Conductor's owl, waving a baton at three small robot agents

AI productAI LabCase study

Conductor

Multi-agent orchestration engine.

A from-scratch agent framework: a planner decomposes a goal, tool-using worker agents execute in parallel, and a critic verifies before anything ships — with the whole run streamed as a trace.

Role
Design and engineering
When
2025 to now
Team
Solo build
Read
About 4 min
Conductor's owl, waving a baton at three small robot agents
Conductor console in pen and ink: a finished run engraved as an orchestral score, with the planner, three workers and the critic as staves and a verified stamp

Overview

Overview

Conductor is a multi-agent orchestration engine I built to understand what agent frameworks actually buy you, by building one myself.

A planner agent breaks a goal into subtasks, worker agents execute them with real tools — web search, code execution, structured extraction — and a critic agent verifies the result against the sources before it's returned. Every step is streamed to a live run trace so you can watch the system think. It's a lab project, but the primitives are the ones I reach for in production.

The challenge

Agents are easy to demo and hard to trust.

A single model call with tools looks magical until it silently invents a citation, loops forever, or blows the budget on a task it should have refused. The interesting engineering isn't getting an agent to run — it's getting a group of them to run reliably: bounded steps, verifiable outputs, parallel work without race conditions, and a trace you can actually debug when something goes sideways.

  • Bounded stepsno runaway loops
  • Verifiable outputchecked against sources
  • Debuggable runsa trace for every step

The solution

Three roles with strict contracts.

The planner emits a typed task list via structured output, so its plan is data the runtime can schedule, not prose to re-parse. Workers run concurrently, each with a scoped tool set and a step budget. A critic re-reads the final answer against the retrieved sources and can send the plan back for one revision. The orchestration layer is hand-written — a small state machine over the Claude API's tool-use loop — because the moment I tried to hide it behind a framework, debugging a bad run became guesswork.

Architecture

Planner, workers, critic.

A planner-worker-critic loop over a hand-written state machine. Tool calls, budgets, and verification are enforced by the runtime, not left to the model's good intentions.

Planner
  • Goal decomposition
  • Typed task list (structured output)
  • Dependency ordering
Workers
  • Parallel tool-using agents
  • Scoped tool sets
  • Per-agent step budgets
Critic
  • Answer-vs-source check
  • One bounded revision
  • Refusal on low confidence
Runtime
  • State machine over tool-use loop
  • Streaming run trace
  • Token & cost accounting

One request, start to finish

  1. Goal arrives
  2. Planner emits a typed task graph
  3. Workers execute subtasks in parallel with tools
  4. Critic verifies output against sources
  5. Verified result streams back with a full trace

Decisions

Four calls and why I made them.

Call 01

Structured output for the plan, not free text

If the planner returns prose, the runtime has to guess what it meant. Forcing the plan through a tool schema makes it a typed object the scheduler can execute directly — no fragile parsing, no ambiguity about what runs next.

Call 02

Hand-written orchestration over a framework

I started with an off-the-shelf agent framework. The first bad run — a worker looping on a tool — took an hour to trace through abstractions I didn't write. A ~300-line state machine over the raw tool-use loop gave me the same capabilities and a stack I could actually reason about.

Call 03

A critic that can say no

The biggest reliability win wasn't a smarter planner — it was a separate agent whose only job is to check the answer against its sources and refuse when confidence is low. Verification as its own role catches the failures a single confident model hides.

Call 04

Hard step and token budgets per agent

Autonomy without a ceiling is how a $0.05 task becomes a $5 one. Every agent runs under an explicit budget; the runtime halts and reports instead of letting a loop run the bill up.

Reflections

What I took away.

The framework was the abstraction I wanted to own

I set out to learn agent frameworks and came away convinced that for anything I need to debug in production, the orchestration is the part I want to write. Frameworks are a great way to learn the shape of the problem — and the exact thing I'll rip out once the shape is clear. Conductor is small on purpose: every decision it makes is one I can point to in code.

Verification is a role, not a prompt

For a while I tried to make one agent both answer and self-check in the same call. It reliably graded its own homework an A. Splitting the critic into its own agent, with only the answer and the sources in context, is what turned a demo into something I'd actually trust to return a result unattended.

Outcomes

What shipped.

  • Roles3Planner · workers · critic
  • Runtime~300 LOCHand-written state machine
  • WorkersParallelConcurrent tool use
  • Every runTracedStep-by-step observability

Stack

Technology

AI

  • Claude API
  • Tool use
  • Structured output
  • Streaming

Runtime

  • TypeScript
  • State machine
  • Concurrency control

Tools

  • Web search
  • Code execution
  • Structured extraction

Observability

  • Run traces
  • Token accounting
  • Cost budgets

Features

What it does

  • Planner agent that decomposes a goal into a typed, dependency-ordered task graph via structured output
  • Worker agents that run concurrently, each with a scoped tool set and an enforced step budget
  • Critic agent that verifies the final answer against retrieved sources and can trigger one bounded revision
  • Hand-written orchestration: a compact state machine over the tool-use loop, built to be debugged
  • Live run trace streaming every agent, tool call, and decision as the system works
  • Per-agent token and cost budgets so an autonomous run can never quietly overspend

Keep reading

More case studies

Your project

Building something like this?

AI, voice, privacy constraints or a mobile app that can’t afford to break. Free 30-minute call, no pitch, clear next steps either way.

Email
Back to
All work