A low-latency voice agent: streaming speech-to-text into a tool-using model into streaming speech-out, with barge-in — built to hold a natural sub-second conversation.
Converse is a realtime voice agent that turns the request/response chatbot loop into an actual conversation.
Audio streams in and is transcribed as the user speaks; a tool-using model reasons and starts responding before the sentence is even finished; speech streams back out; and if the user cuts in, the agent stops and listens. The whole thing is engineered around one number — perceived turn latency — because below a certain threshold a voice agent feels like a person, and above it feels like a phone tree.
The challenge
Voice is unforgiving about latency in a way text never is.
A 300ms delay in a chat window is invisible; the same delay in speech is an awkward pause that makes the whole thing feel broken. Getting there means every stage has to overlap — you can't wait for full transcription before thinking, or a full answer before speaking — and you have to handle the messiness of real speech: people interrupt, pause mid-thought, and talk over each other.
Sub-second turnsor it feels like a phone tree
Overlapping stagesnever wait for a full answer
Real interruptionspeople talk over each other
The solution
A fully streaming pipeline with no blocking stages.
Partial transcripts flow to the model as they arrive; the model streams tokens the moment it has them; those tokens are chunked into speech on the fly. Barge-in is first-class — incoming user audio interrupts playback and cancels the in-flight response so the agent yields instantly instead of talking over you. Endpointing decides when a turn is actually done, and tool calls happen mid-conversation so the agent can look something up without dropping the thread.
ListenThinkSpeak
Architecture
Three overlapping streams.
Three streams overlapped end to end — speech in, reasoning, speech out — with interruption handling as a first-class path, not an afterthought.
Listen
Streaming speech-to-text
Endpoint detection
Barge-in capture
Think
Streaming LLM responses
Mid-turn tool calls
Interruptible generation
Speak
On-the-fly text-to-speech
Chunked playback
Instant cancellation
Control
Turn-taking state machine
Latency budget tracking
Session, zero audio kept
One request, start to finish
User speaks; partial transcripts stream in
The model starts reasoning before the turn ends
Response tokens stream straight into speech
User can barge in — playback stops instantly
Turn-taking resets and the loop continues
Decisions
Four calls and why I made them.
Call 01
Overlap every stage, block on none
The only way to hit sub-second turns is to stop treating STT, reasoning, and TTS as a sequence. Feeding partial transcripts into the model and streaming its tokens into speech as they arrive is what collapses the dead air between someone finishing and the agent starting.
Call 02
Barge-in as a first-class path
Nothing makes an agent feel more robotic than talking over the user. Treating incoming audio as an interrupt that cancels the in-flight response — instead of a queued next turn — is the single change that made conversations feel natural.
Call 03
Endpointing over fixed timeouts
A fixed 'wait N seconds of silence' either cuts people off mid-thought or leaves them hanging. Detecting the actual end of a turn from the speech signal is messier to build but is the difference between a conversation and an interrogation.
Call 04
Ephemeral sessions, no stored audio
Carried over from Intervoo: a voice tool that banks your audio is a trust problem. Processing is in-session and audio is discarded, which also keeps the data-retention surface close to zero.
Reflections
What I took away.
Latency is a product feature, not a metric
I kept thinking of latency as something to optimize after the features worked. With voice it is the feature — the experience is either fast enough to feel human or it isn't, and no amount of clever behavior rescues it once it's slow. Building the whole pipeline around a latency budget from the start changed every architectural call downstream.
Real conversations are full of interruptions
My first version assumed clean turn-taking: you talk, then I talk. Real people don't work that way — they interrupt, back-channel, and change their mind mid-sentence. Handling the messy overlap instead of pretending it doesn't exist is most of what separates a voice demo from something you'd actually want to talk to.
Outcomes
What shipped.
Turn latencySub-secOverlapped streaming
Barge-inInstantInterrupt & yield
PipelineStreamingNo blocking stages
Audio0 storedEphemeral sessions
Stack
Technology
AI
Claude API
Streaming
Tool use
Voice
Streaming STT
Streaming TTS
Endpointing
Barge-in
Realtime
WebRTC
WebSockets
Turn-taking state machine
Privacy
In-session processing
Zero audio retention
Features
What it does
Fully streaming pipeline — partial transcripts feed the model, whose tokens stream straight into speech
First-class barge-in: user audio interrupts playback and cancels the in-flight response instantly
Turn endpointing from the speech signal instead of fixed silence timeouts
Mid-conversation tool calls so the agent can look things up without dropping the thread
A turn-taking state machine that tracks who holds the floor and enforces a latency budget
Ephemeral, in-session audio processing with zero retention — carried over from Intervoo