3 min read
Anatomy of a real-time AI voice agent
What it takes for an AI to hold a phone call: voice activity detection, streaming speech-to-text, a grounded LLM, fast speech synthesis, and a clean hand-off to a human.
A chatbot can take a few seconds to answer and nobody minds. On a phone call, a two-second pause feels like the line dropped. That one constraint shapes every decision in Zap, the AI voice agent I built to make B2B discovery calls.
Zap listens, answers questions about the product from a knowledge base, scores the lead, and hands hot prospects to a human sales rep. Here's how the pieces fit.
The loop
Every turn of the conversation runs through the same pipeline, built on Pipecat:
caller audio → VAD → speech-to-text → LLM (+ RAG) → text-to-speech → caller
| Stage | Choice |
|---|---|
| Turn detection | Voice activity detection (VAD) |
| Speech-to-text | Whisper, served on Groq |
| Reasoning | Llama 3.1 |
| Grounding | RAG over the product knowledge base |
| Speech | Kokoro TTS |
| Hand-off | n8n workflow |
| Storage | MongoDB |
Rule one: everything streams
The naive version waits for each stage to finish before starting the next: record the whole sentence, transcribe it, generate the whole reply, synthesise the whole reply, then play it. Add those up and you get a long, awkward silence.
A voice agent feels responsive only when the stages overlap. Transcription runs while the caller is still talking. The model's reply is spoken sentence by sentence as it's generated, instead of after the last token. Pipecat's frame-based pipeline makes this the default rather than something you bolt on later.
Rule two: know when the caller has stopped
The hardest problem in voice isn't speech recognition, it's turn-taking. Respond too early and you talk over someone mid-thought. Respond too late and the call feels sluggish.
Voice activity detection decides when a turn has ended. It's also what makes barge-in work: if the caller starts speaking while Zap is talking, playback stops and Zap listens. People interrupt each other all the time on real calls, so an agent that can't be interrupted feels robotic within seconds.
Rule three: the model only says what the knowledge base supports
A sales agent that invents a feature or a price is worse than no agent. Zap answers from a RAG knowledge base: before the model replies, the relevant facts about the product are retrieved and put in front of it, and the prompt tells it to stay inside them.
This also keeps answers short. A focused context produces focused replies, and on a phone line short replies are what you want.
Rule four: end every call with a decision
The point of a discovery call isn't the conversation, it's knowing what to do next. When a call ends, Zap scores the lead. A hot lead triggers an n8n workflow that alerts a sales rep straight away, with the context of the call, so a person follows up while the prospect still remembers talking to us.
The AI handles the volume; humans handle the deals.
What I'd do again
- Pick the pipeline framework first. Getting streaming, interruptions and audio transport right from scratch is a project of its own. Pipecat let me spend my time on the agent instead of the plumbing.
- Measure the gap, not the parts. The number that matters is the silence between the caller finishing and the agent starting. Measure that, end to end, on a real phone line.
- Design the hand-off before the conversation. An agent that knows when to step aside earns more trust than one that tries to close every deal itself.
Zap is live at neuraconnects.xyz, and there's a narrated walkthrough in the Selected work section.