Bonus dive

#Realtime Voice: A Guided Deep Dive

A hands-on playground for the one category of AI product the rest of this series can't prepare you to build: realtime conversational voice. You'll build a from-scratch simulator of a realtime session and understand every moving part. Audio as a stream of frames, turn detection, the STT then LLM then TTS pipeline, the turn-taking state machine, barge-in (interrupting the agent mid-sentence), latency budgets, and the architectural fork between a pipeline and a single speech-to-speech model. No framework magic, just enough code to see how a voice agent actually keeps a conversation going.

One honest constraint shapes this dive. A genuine realtime voice session needs low-latency, full-duplex audio, meaning a microphone, a speaker, and a WebSocket or WebRTC transport, and none of that can be shown honestly in a small, from-scratch, offline example. So this repo is a deterministic simulator. Audio is modeled as timestamped frames and each stage carries a latency budget in milliseconds, so you can watch turn-taking, barge-in, and the latency math play out exactly, offline, with no key, for $0. The transport is the one thing the simulation stands in for. The state machine, the architectures, and the reasoning are the real ones, and the README maps each to production.

This is a bonus dive. It picks up exactly where Multimodal stops. That dive covers batch speech-to-text and text-to-speech and lists realtime as out of scope. This is that scope. Its code depends on none of the others.

Like its siblings, walk through it. Each section ends with something to run, and every section runs offline and free. EXERCISES.md has a predict-then-run prompt for each one, and TEXTBOOK.md is the lecture behind this lab: Chapter 12 of the AI Engineering Textbook, on why conversational time is a harder constraint than machine time.

#0. The one big idea

Realtime voice is a low-latency, full-duplex loop. Audio streams in and out at once, the agent can be interrupted mid-sentence, and every hundred milliseconds gets felt. The engineering is a turn-taking state machine over that stream, plus one architectural choice: an STT-LLM-TTS pipeline, or a single speech-to-speech model.

That's the whole repo. Batch audio, from the Multimodal dive, is upload a file, wait, get a result. Realtime is a conversation. You never have the whole recording, the user can cut in at any moment, and a one-second pause feels broken. Everything below, from frames and turn detection to barge-in, the latency budget, and the two architectures, is one face of that sentence. Hold onto it and none of this feels complicated.

#1. Setup (5 minutes)

# 1. Create an isolated Python environment
python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

# 2. Install dependencies (tiny: the repo is an offline simulator)
pip install -r requirements.txt

# 3. Copy the env file: this dive is a fully offline simulator (no key needed)
cp .env.example .env
#    (Wiring up a real realtime API? Its key goes in your OS keychain, not .env.
#     See ../docs/SECRETS.md in the series repo: https://github.com/alexvervloet/ai-engineering-deep-dive)

# 4. Confirm everything is wired up (makes no API call, costs nothing)
python check_setup.py

There's one provider here, mock, and it needs no key. Unlike the sibling repos, this dive doesn't switch between OpenAI and Claude. The box below says why. Everything runs offline and deterministically.

Why a simulator rather than a real provider. Realtime voice needs full-duplex audio I/O and a streaming transport. Wiring that to a real API means a mic, a speaker, WebRTC or WebSocket plumbing, and platform-specific audio libraries, none of which belongs in a small, readable, offline teaching repo. So we simulate the mechanics, meaning frames, latency, turn-taking, and barge-in, exactly and deterministically. The state machine and the architecture choices are real. Only the transport is mocked. Production uses a speech-to-speech API over WebSocket or WebRTC, with OpenAI's Realtime API and Google's Gemini Live API the two obvious ones, or a streaming STT, LLM, and TTS pipeline, both mapped in "From teaching code to production."

#2. Audio is a stream of frames

python examples/01_audio_is_frames.py        # offline

Here's the first mental shift. You never have the recording. Audio arrives as a continuous stream of tiny frames, about 10 to 20 ms each, and you work with the frames so far. The example builds one simulated utterance as frames, a speech frame per word followed by a run of silence, and a simple voice-activity rule finds the end of the turn. Turn detection is a judgment call over silence. Too eager and you cut the user off. Too patient and the agent feels slow. Everything else is built on this stream. (voice/audio.py)

#3. The pipeline, STT then LLM then TTS

python examples/02_pipeline.py

The first way to build a voice agent is three models in series: speech-to-text, then the LLM, then text-to-speech. Each hop adds delay, and the number the user feels is time-to-first-audio, meaning how long after they stop talking before they hear anything. That's the end-pointing wait first, because nothing downstream starts until the VAD decides the user is done, then STT plus LLM plus TTS stacked on top. 1500 ms in this repo's budget. The example prints it line by line so you see where the dead air comes from, and why streaming each stage so they overlap is the fix. What the pipeline buys you is control. There's a text transcript in the middle you can log, moderate, and edit. (voice/stages.py)

#4. The turn-taking state machine

python examples/03_turn_taking.py

A voice agent is a state machine over the stream. LISTENING, then the user stops, then THINKING, then first audio, then SPEAKING, then done, then LISTENING again. The example runs a clean, non-overlapping three-turn dialogue so you can watch the machine cycle once per turn. Real conversations aren't this tidy, which is the next section. (voice/session.py)

#5. Barge-in, where the human interrupts

python examples/04_barge_in.py

This is the feature that separates a voice agent from a walkie-talkie. People interrupt ("no wait, actually...") and a good agent stops talking on the spot, discards the rest of its planned audio, and listens. An agent that talks over you feels broken, and it's the most common thing that ruins a voice demo. The example sends a long agent response, has the user cut in partway through, and shows the session fire a barge-in and re-enter LISTENING mid-sentence. It works because of full-duplex audio, since you're still listening while speaking, and fast cancellation, killing the TTS stream and flushing the buffer the instant the user's voice is detected.

#6. The latency budget

python examples/05_latency_budget.py

Latency is voice's make-or-break metric. Humans notice a gap past about 300 to 500 ms, and past that the agent feels sluggish or gets talked over. The example splits time-to-first-audio into end-pointing, processing, and the felt total, both ways on the same turn. Speech-to-speech is 2× faster on the processing it controls and only 1.5× faster overall, 1000 ms against 1500 ms, because both designs wait out the same silence window before either of them starts. That third column is the honest one. Engineer against the number your users feel rather than the one on a spec sheet, and remember that the biggest single line in the budget is usually a threshold you chose, not a model you bought.

#7. Speech-to-speech, and when to choose it

python examples/06_speech_to_speech.py

Speech-to-speech uses a single multimodal model that hears audio and speaks audio directly, with no transcript in the middle. It wins on latency, since there's one hop, and on naturalness, since it hears tone and pacing and can speak with them. It gives up control and observability, because there's no text step to log, moderate, redact, or hand to a tool. The example lays out the decision. Speech-to-speech for consumer assistants and companions where latency and feel dominate. The pipeline when you need the transcript for guardrails, tools and RAG, auditing, or per-stage vendor choice. And often a hybrid, with a speech-to-speech turn plus a parallel transcript for safety. Same discipline as the whole series. Pick the simplest architecture that meets your real constraints.

#The capstone: voice_agent.py

Everything assembled into a simulated voice agent you can drive. Interactive typed turns, a latency readout per turn, a choice of architecture, and scripted demos of a clean dialogue and a barge-in.

# Interactive: each line you type is one user turn (type 'quit' to exit)
python hands_on/voice_agent.py

# Choose the architecture:
python hands_on/voice_agent.py --mode speech_to_speech

# Scripted demos:
python hands_on/voice_agent.py --demo dialogue
python hands_on/voice_agent.py --demo barge-in

Read hands_on/voice_agent.py. It's the library, RealtimeSession plus utterance plus merge, wired to a CLI. Suggested exercise: run --demo barge-in in both --mode pipeline and --mode speech_to_speech. The interruption arrives at the same moment in both and lands in a different state. Speech-to-speech is already telling the joke and gets cut off mid-sentence, while the pipeline is still thinking and never says a word. Latency changes how the agent feels and it also changes which code path runs.

#Where to go next

You've built the mechanics of a realtime voice agent. What comes next is wiring them to real audio and hardening the conversation.

• A real transport. A speech-to-speech API over WebSocket or WebRTC (OpenAI's Realtime API, Google's Gemini Live API). Send mic frames, receive audio frames, and handle the session events. This dive's state machine is what you drive with it.
• A real pipeline. A streaming STT model (batch Whisper is the wrong tool here, so reach for a realtime transcription endpoint or a vendor built for it, like Deepgram or AssemblyAI), a streaming LLM, and a streaming TTS vendor, with each stage overlapped so the latency stacks less.
• Not writing the plumbing yourself. Pipecat and LiveKit Agents are the open-source orchestrators most teams reach for. They own the transport, the VAD, and the interruption handling. Read one after this dive and you'll recognise every moving part, which is the point of building it from scratch first.
• Better turn detection. A trained VAD or end-pointing model instead of a silence threshold, plus handling backchannels ("mm-hm") that aren't interruptions.
• Tools and RAG in a voice loop. Let the agent call functions or retrieve (RAG dive) mid-conversation without killing latency, and speak a "let me check..." while it works.
• Telephony. SIP/PSTN integration, echo cancellation, and jitter buffers for real phone calls.
• Emotion and prosody. Using and producing tone as well as words, which is the edge speech-to-speech models have.
• Evaluating voice. Latency percentiles, interruption handling, and transcription accuracy as numbers you track (Evals dive).

#From teaching code to production

This repo simulates the transport so the mechanics stay visible. Here's what each piece becomes when it's real.

The general ops machinery (observability, cost, reliability, caching, guardrails, eval gates) gets built from scratch and wired into one running app in Production, #8, which runs offline on a mock provider.

#File map

check_setup.py              ← run first: verifies Python + packages (no key needed)
README.md                   ← this guide (the lab)
TEXTBOOK.md                 ← Chapter 12: the lecture behind the lab
EXERCISES.md                ← predict-then-run prompts, one per section
LESSONS.md                  ← what this repo got wrong, and how it was caught
voice/                      ← the from-scratch simulator (read it!)
  audio.py                  ← audio as a stream of timestamped frames (+ builders)
  stages.py                 ← the two architectures as latency-annotated stages
  session.py                ← the turn-taking state machine (VAD turn detection + barge-in)
  providers.py              ← the mock-only provider shim (parity with the series)
hands_on/
  voice_agent.py            ← capstone: a simulated voice agent (interactive + demos)
examples/
  01_audio_is_frames.py     ← audio is a stream of frames; VAD finds the turn end
  02_pipeline.py            ← STT→LLM→TTS and its latency budget
  03_turn_taking.py         ← the LISTENING→THINKING→SPEAKING state machine
  04_barge_in.py            ← the user interrupts; the agent yields instantly
  05_latency_budget.py      ← time-to-first-audio: pipeline vs speech-to-speech
  06_speech_to_speech.py    ← one model; when to pick it over the pipeline
tests/
  test_session.py           ← locks the timelines the docs quote

#Troubleshooting

Run python check_setup.py first. Then, by symptom:

Still stuck? Every file is small and self-contained. Open it, read the docstring at the top, and run it directly. voice/session.py is the whole story: the turn-taking machine, with barge-in.

#The series

This is one of the standalone, hands-on deep dives into building with LLM APIs: eight core, plus the bonus dives. Each stands on its own, with its own setup, examples, and capstone, and they share one house style: provider-agnostic where it makes sense, built from scratch (no frameworks), offline-first examples, and a real capstone. Do them in any order; this sequence builds naturally:

1. OpenAI API: the API from zero
2. Claude API: the same ideas, the Anthropic way
3. Prompt Engineering: shape model behavior with better prompts
4. RAG: answer questions over your own documents
5. Evals: measure whether a change actually helps
6. Agents: give a model tools and a loop so it can act
7. Prompt Injection & Guardrails: attack and defend all of the above
8. Production: operate one app end to end

Bonus dives, standalone and slotting in where they're most useful:

• Agent Harnesses: build on the loop, adding hooks, permissions, sandboxing, and subagents
• Context Engineering: manage what's in the window
• AI Data Engineering: the corpus behind the index, with versions, lineage, ACLs, and deletes
• Multimodal: images and audio as well as text
• Realtime Voice: low-latency speech-to-speech agents
• Fine-tuning: teach a model new behavior by example
• MCP: serve tools, data, and prompts over a standard protocol
• Local Models: run open-weight models on your own machine
• Observability: watch a running app over time, covering drift, quality, alerting, and the feedback loop
• Architecture: the seams between the components, each decision measured rather than asserted
• GenAI Security: treat the model as an untrusted principal, and put identity, supply chain, isolation, budgets, and release gates around it
• Inference Platform Engineering: turn finite GPU memory and a request queue into latency, throughput, and a fleet size you can defend
• Testing & Delivery: decide whether a build is fit to promote, using evidence, gates, staged rollout, and rollback
• Professional Tools: rebuild each hand-written piece with the tool professionals reach for, and measure both

And the whole series lands in one codebase in the capstone: a codebase Q&A tool built step by step, one tag per dive.

Realtime Voice is a bonus dive. It slots right after Multimodal, since that dive does batch speech-to-text and text-to-speech and marks realtime out of scope; this is that scope.