#Exercises: make the learning stick

Because there's no end to wait for. Audio streams continuously, in both directions at once, and you only ever have the frames so far. That forces turn detection: you have to decide, from silence, when the user is done talking. Too eager and you cut them off; too patient and the agent feels slow. Batch audio never has to make that call.

1500 ms. The stages run in series (you can't synthesize speech until the LLM has produced text until STT has produced a transcript), so their delays add rather than overlap, and all three sit behind the 500 ms end-pointing wait, because nothing can start until the agent concludes the turn is over. If you only counted the models you'd get 1000 ms and be wrong by a third. Streaming the stages so they overlap is the main way to shrink the processing half; tightening the VAD window shrinks the other half, at the cost of cutting people off mid-thought.

Control and observability. There's a text transcript in the middle you can log, moderate, edit, hand to a tool or RAG, and audit, and you can swap any stage's vendor independently. Speech-to-speech hides all of that.

LISTENING → (VAD sees the user stop) → THINKING → (first audio is ready) → SPEAKING → (response finishes) → LISTENING. And a fifth edge that section 5 adds: SPEAKING → (user starts talking = barge-in) → LISTENING, cancelling the output.

It fires a barge-in: the agent stops speaking, the rest of the planned joke is discarded (never "played"), and the new speech is treated as the next turn. A good voice agent yields the instant it hears you. An agent that finishes its sentence over you feels broken.

Full-duplex audio (you're still listening while you speak, so you can even detect the interruption) and fast cancellation (kill the TTS stream, flush the playback buffer, and cancel the in-flight model response quickly enough to feel instant). If you only listened between turns, you couldn't be interrupted.

Speech-to-speech, by 2× on processing (500 ms vs 1000 ms) but only 1.5× on the gap the user actually hears (1000 ms vs 1500 ms), because both architectures wait out the same 500 ms of end-pointing before either of them starts. The second number is the honest one for a product decision, and the habit worth taking: a shared fixed cost always compresses a ratio, so quote the improvement your user can perceive, not the one your component benchmark shows. Latency is voice's make-or-break metric, since humans notice a gap past ~300-500 ms, so that difference is felt directly.

It costs control and observability: no transcript to log, moderate, redact, or feed to a tool, and it's harder to steer. Choose the pipeline when you need that transcript: guardrails/moderation, tool or RAG calls on the text, an auditable log, or independent vendor choice per stage. Many production systems go hybrid: speech-to-speech for the turn, a parallel transcript for safety.

No. The interruption arrives at the same moment (1800 ms) in both runs, but it lands in a different state. Speech-to-speech began speaking at 1600 ms, so it's cut off mid-joke and you see a response_start before the barge-in. The pipeline wouldn't have spoken until 2100 ms, so it's still in THINKING; the turn is superseded and its planned reply is dropped without a sound ever coming out. Same interruption, different code path, because latency changed when the agent was speaking.