How speech-to-speech models become believable voice agents.
Speech agents feel magical only when several technical loops line up: audio tokens, conversational context, turn-taking, streaming generation, interruption handling, and latency budgets. This page lets you poke each layer instead of reading a static essay.
Speech-to-speech is not just ASR + LLM + TTS.
The easy mental model is a three-box cascade. Modern systems are more nuanced: they preserve timing, speaker cues, prosody, uncertainty, and interruption state while streaming partial output.
CSM generates speech as conversational audio tokens.
Sesame’s open CSM is a contextual speech generation model. It is not a general multimodal assistant by itself; it is the voice generator that turns text plus conversational audio context into natural speech.
Click a CSM component
CSM uses a Llama-style backbone and a smaller audio decoder. It predicts Mimi/RVQ audio codes, then the codec decodes those codes into waveform audio.
Voice UX is won or lost in hundreds of milliseconds.
The user does not care where delay came from. Capture, VAD, network, inference, codec decoding, and playback buffering all add up. Drag the sliders to feel why “fast model” is not enough.
End-to-end response budget
Approximate time from “user stops or yields” to first audible assistant audio. Lower is better, but too-aggressive turn detection causes interruptions.
The hard part is deciding when not to talk.
Humans use breaths, prosody, gaze, hesitation, and semantics to coordinate turns. Voice agents fake this with signal processing, prediction, and carefully tuned state machines.
Interactive turn-state machine
Step through the runtime loop. The best systems can barge in, back off, fill pauses, and repair misunderstandings without exposing machinery.
Three ways to build a spoken assistant.
Each design has a different failure mode. Most production systems are hybrids because reliability, latency, and expressivity pull in different directions.
| Pattern | How it works | Strength | Failure mode |
|---|---|---|---|
| Cascade ASR → LLM → TTS | Transcribe speech to text, generate text, synthesize voice. | Debuggable, controllable, easy to integrate tools and safety. | Often loses prosody, timing, emotion, and interruption nuance. |
| End-to-end S2S audio in → audio out | Model consumes audio features/tokens and emits audio tokens directly. | Can preserve paralinguistic information and reduce handoff friction. | Harder to inspect; may hallucinate what it heard or be weaker at tool use. |
| Contextual TTS LLM text → expressive voice | Separate reasoning model writes text; a speech model uses context audio/text to speak it naturally. | Great control plus strong voice presence when context is rich. | Requires orchestration: context windows, speaker state, and timing policy. |
“Insane UX” is a stack, not one model trick.
The demos that feel alive combine model capability with product discipline. The model supplies expressive speech; the runtime makes it conversational.
Voice presence
Voice carries confidence, uncertainty, warmth, and timing. Sesame explicitly frames the goal as “voice presence,” not just clean audio.
- consistent persona
- emotionally appropriate prosody
- style adapts to context
Contextual generation
CSM can condition on prior text and audio segments, so a line can be spoken as a continuation of the conversation instead of a standalone clip.
- speaker history
- previous rhythm
- content-aware emphasis
Low-latency streaming
Great agents start speaking before the whole answer is done. Audio tokens are decoded in chunks and scheduled into a smooth playback buffer.
- first-audio target
- jitter buffer
- incremental decoding
Turn prediction
VAD alone is not enough. A model needs to know if a silence means “done,” “thinking,” or “about to continue.”
- pause thresholds
- semantic endpointing
- barge-in policy
Repair behavior
Mishearing is inevitable. The UX win is a short, specific repair instead of a generic “I did not understand.”
- repeat only when needed
- quote what was heard
- ask narrow clarifiers
Agent speech
The assistant should not read every internal thought. It needs spoken phrasing: shorter clauses, fewer lists, and more timing-aware delivery.
- speech-first copy
- backchannels
- natural stop points
Quick comprehension test.
If these feel easy, you understand the real engineering split: model architecture, runtime orchestration, and product UX.