Converse's owl wearing a headset next to a sound-wave speech bubble

AI productAI LabCase study

Converse

Realtime streaming voice agent.

A low-latency voice agent: streaming speech-to-text into a tool-using model into streaming speech-out, with barge-in — built to hold a natural sub-second conversation.

Role
Design and engineering
When
2025 to now
Team
Solo build
Read
About 4 min
Converse's owl wearing a headset next to a sound-wave speech bubble
Converse console in pen and ink: a live call drawn as a comic strip, with speech balloons, a tool call inset and the inked call waveform

Overview

Overview

Converse is a realtime voice agent that turns the request/response chatbot loop into an actual conversation.

Audio streams in and is transcribed as the user speaks; a tool-using model reasons and starts responding before the sentence is even finished; speech streams back out; and if the user cuts in, the agent stops and listens. The whole thing is engineered around one number — perceived turn latency — because below a certain threshold a voice agent feels like a person, and above it feels like a phone tree.

The challenge

Voice is unforgiving about latency in a way text never is.

A 300ms delay in a chat window is invisible; the same delay in speech is an awkward pause that makes the whole thing feel broken. Getting there means every stage has to overlap — you can't wait for full transcription before thinking, or a full answer before speaking — and you have to handle the messiness of real speech: people interrupt, pause mid-thought, and talk over each other.

  • Sub-second turnsor it feels like a phone tree
  • Overlapping stagesnever wait for a full answer
  • Real interruptionspeople talk over each other

The solution

A fully streaming pipeline with no blocking stages.

Partial transcripts flow to the model as they arrive; the model streams tokens the moment it has them; those tokens are chunked into speech on the fly. Barge-in is first-class — incoming user audio interrupts playback and cancels the in-flight response so the agent yields instantly instead of talking over you. Endpointing decides when a turn is actually done, and tool calls happen mid-conversation so the agent can look something up without dropping the thread.

Architecture

Three overlapping streams.

Three streams overlapped end to end — speech in, reasoning, speech out — with interruption handling as a first-class path, not an afterthought.

Listen
  • Streaming speech-to-text
  • Endpoint detection
  • Barge-in capture
Think
  • Streaming LLM responses
  • Mid-turn tool calls
  • Interruptible generation
Speak
  • On-the-fly text-to-speech
  • Chunked playback
  • Instant cancellation
Control
  • Turn-taking state machine
  • Latency budget tracking
  • Session, zero audio kept

One request, start to finish

  1. User speaks; partial transcripts stream in
  2. The model starts reasoning before the turn ends
  3. Response tokens stream straight into speech
  4. User can barge in — playback stops instantly
  5. Turn-taking resets and the loop continues

Decisions

Four calls and why I made them.

Call 01

Overlap every stage, block on none

The only way to hit sub-second turns is to stop treating STT, reasoning, and TTS as a sequence. Feeding partial transcripts into the model and streaming its tokens into speech as they arrive is what collapses the dead air between someone finishing and the agent starting.

Call 02

Barge-in as a first-class path

Nothing makes an agent feel more robotic than talking over the user. Treating incoming audio as an interrupt that cancels the in-flight response — instead of a queued next turn — is the single change that made conversations feel natural.

Call 03

Endpointing over fixed timeouts

A fixed 'wait N seconds of silence' either cuts people off mid-thought or leaves them hanging. Detecting the actual end of a turn from the speech signal is messier to build but is the difference between a conversation and an interrogation.

Call 04

Ephemeral sessions, no stored audio

Carried over from Intervoo: a voice tool that banks your audio is a trust problem. Processing is in-session and audio is discarded, which also keeps the data-retention surface close to zero.

Reflections

What I took away.

Latency is a product feature, not a metric

I kept thinking of latency as something to optimize after the features worked. With voice it is the feature — the experience is either fast enough to feel human or it isn't, and no amount of clever behavior rescues it once it's slow. Building the whole pipeline around a latency budget from the start changed every architectural call downstream.

Real conversations are full of interruptions

My first version assumed clean turn-taking: you talk, then I talk. Real people don't work that way — they interrupt, back-channel, and change their mind mid-sentence. Handling the messy overlap instead of pretending it doesn't exist is most of what separates a voice demo from something you'd actually want to talk to.

Outcomes

What shipped.

  • Turn latencySub-secOverlapped streaming
  • Barge-inInstantInterrupt & yield
  • PipelineStreamingNo blocking stages
  • Audio0 storedEphemeral sessions

Stack

Technology

AI

  • Claude API
  • Streaming
  • Tool use

Voice

  • Streaming STT
  • Streaming TTS
  • Endpointing
  • Barge-in

Realtime

  • WebRTC
  • WebSockets
  • Turn-taking state machine

Privacy

  • In-session processing
  • Zero audio retention

Features

What it does

  • Fully streaming pipeline — partial transcripts feed the model, whose tokens stream straight into speech
  • First-class barge-in: user audio interrupts playback and cancels the in-flight response instantly
  • Turn endpointing from the speech signal instead of fixed silence timeouts
  • Mid-conversation tool calls so the agent can look things up without dropping the thread
  • A turn-taking state machine that tracks who holds the floor and enforces a latency budget
  • Ephemeral, in-session audio processing with zero retention — carried over from Intervoo

Keep reading

More case studies

Your project

Building something like this?

AI, voice, privacy constraints or a mobile app that can’t afford to break. Free 30-minute call, no pitch, clear next steps either way.

Email
Back to
All work