Converse
Realtime streaming voice agent
Live Demo
Overview
Converse is a realtime voice agent that turns the request/response chatbot loop into an actual conversation. Audio streams in and is transcribed as the user speaks; a tool-using model reasons and starts responding before the sentence is even finished; speech streams back out; and if the user cuts in, the agent stops and listens. The whole thing is engineered around one number — perceived turn latency — because below a certain threshold a voice agent feels like a person, and above it feels like a phone tree.
The Challenge
Voice is unforgiving about latency in a way text never is. A 300ms delay in a chat window is invisible; the same delay in speech is an awkward pause that makes the whole thing feel broken. Getting there means every stage has to overlap — you can't wait for full transcription before thinking, or a full answer before speaking — and you have to handle the messiness of real speech: people interrupt, pause mid-thought, and talk over each other.
The Solution
A fully streaming pipeline with no blocking stages. Partial transcripts flow to the model as they arrive; the model streams tokens the moment it has them; those tokens are chunked into speech on the fly. Barge-in is first-class — incoming user audio interrupts playback and cancels the in-flight response so the agent yields instantly instead of talking over you. Endpointing decides when a turn is actually done, and tool calls happen mid-conversation so the agent can look something up without dropping the thread.
Architecture
Three streams overlapped end to end — speech in, reasoning, speech out — with interruption handling as a first-class path, not an afterthought.
Key Decisions & Tradeoffs
Overlap every stage, block on none
Why: The only way to hit sub-second turns is to stop treating STT, reasoning, and TTS as a sequence. Feeding partial transcripts into the model and streaming its tokens into speech as they arrive is what collapses the dead air between someone finishing and the agent starting.
Barge-in as a first-class path
Why: Nothing makes an agent feel more robotic than talking over the user. Treating incoming audio as an interrupt that cancels the in-flight response — instead of a queued next turn — is the single change that made conversations feel natural.
Endpointing over fixed timeouts
Why: A fixed 'wait N seconds of silence' either cuts people off mid-thought or leaves them hanging. Detecting the actual end of a turn from the speech signal is messier to build but is the difference between a conversation and an interrogation.
Ephemeral sessions, no stored audio
Why: Carried over from Intervoo: a voice tool that banks your audio is a trust problem. Processing is in-session and audio is discarded, which also keeps the data-retention surface close to zero.
What I took away from this project
Lessons that still shape how I build — in my own words.
Latency is a product feature, not a metric
I kept thinking of latency as something to optimize after the features worked. With voice it is the feature — the experience is either fast enough to feel human or it isn't, and no amount of clever behavior rescues it once it's slow. Building the whole pipeline around a latency budget from the start changed every architectural call downstream.
Real conversations are full of interruptions
My first version assumed clean turn-taking: you talk, then I talk. Real people don't work that way — they interrupt, back-channel, and change their mind mid-sentence. Handling the messy overlap instead of pretending it doesn't exist is most of what separates a voice demo from something you'd actually want to talk to.
Impact
Technology Stack
AI
Voice
Realtime
Privacy
Key Features
Related case studies
Want something like this, built for you?
If the Converse story sounds close to what you're trying to ship, a 30-min call is the fastest way to find out if we're a fit.