valenti/vocero-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consolidate English documentation under root docs

Browse Source
This commit is contained in:
alvaro
2026-10-05 10:11:35 -03:00
parent b91762a927
commit 6b8c27fd76
38 changed files with 2261 additions and 5594 deletions
Download Patch File Download Diff File Expand all files Collapse all files
speech_to_speech/LLM/README.md
+5 -138
View File
@@ -1,139 +1,6 @@
# LLM Summary
# Language model handlers
## Available LLM backends (`--llm_backend`)
Runtime-supported values in `s2s_pipeline.py`:
- `transformers` → `language_model.py` (Transformers backend)
- `mlx-lm` → `language_model.py` (MLX backend)
- `responses-api` → `responses_api_language_model.py`
## Usage
### 1) Transformers (`--llm_backend transformers`)
- Handler: `LanguageModelHandler`
- Typical use: local GPU/CPU inference using Hugging Face Transformers
- Backend-specific args prefix: `--llm_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
```bash
python s2s_pipeline.py \
--llm_backend transformers \
--model_name Qwen/Qwen3-4B-Instruct-2507 \
--llm_device cuda \
--llm_torch_dtype float16 \
--llm_gen_max_new_tokens 128
```
Common options:
- `--llm_gen_min_new_tokens`
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`
### 2) MLX-LM (`--llm_backend mlx-lm`)
- Handler: `LanguageModelHandler`
- Typical use: Apple Silicon local inference
- Backend-specific args prefix: same as Transformers (`--llm_*`)
```bash
python s2s_pipeline.py \
--llm_backend mlx-lm \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
--llm_device mps \
--llm_gen_max_new_tokens 128
```
Common options:
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`
### 3) OpenAI-compatible API (`--llm_backend responses-api`)
- Handler: `ResponsesApiModelHandler`
- Typical use: remote model serving via OpenAI-compatible endpoints
- Backend-specific args prefix: `--responses_api_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
```bash
python s2s_pipeline.py \
--llm_backend responses-api \
--model_name gpt-5.4-mini \
--responses_api_api_key YOUR_API_KEY \
--responses_api_base_url https://api.example.com/v1 \
--responses_api_stream true
```
Common options:
- `--chat_size`
- `--init_chat_prompt`
- `--user_role`
## LLM Behavior
When STT is set to language auto-detection (`--language auto`), LLM handlers can receive `(text, language_code)` and prepend a language control instruction like:
- `Please reply to my message in <language>.`
This helps the assistant respond in the detected language. The behavior is opt-in via `--enable_lang_prompt` (shared across all backends); it defaults to `False`.
## Setup
### CUDA setup
```bash
python s2s_pipeline.py \
--llm_backend transformers \
--model_name microsoft/Phi-3-mini-4k-instruct
```
### Local Mac setup
```bash
python s2s_pipeline.py \
--local_mac_optimal_settings \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16
```
`--local_mac_optimal_settings` already sets `--llm_backend mlx-lm` and will default the model to `mlx-community/Qwen3-4B-Instruct-2507-bf16` if not overridden.
### Realtime (OpenAI-compatible) setup
Run the server in realtime mode, then connect with the realtime client:
```bash
# 1. Start the pipeline in realtime mode
python s2s_pipeline.py \
--mode realtime \
--llm_backend mlx-lm \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
--ws_host 0.0.0.0 \
--ws_port 8765
# 2. Connect with the realtime client
python listen_and_play_realtime.py --host 127.0.0.1 --port 8765
```
Or with `--local_mac_optimal_settings` on Apple Silicon:
```bash
python s2s_pipeline.py \
--local_mac_optimal_settings \
--mode realtime \
--ws_host 0.0.0.0 \
--ws_port 8765
```
### Remote API setup
```bash
python s2s_pipeline.py \
--llm_backend responses-api \
--model_name gpt-5.4-mini \
--responses_api_api_key YOUR_API_KEY
```
This directory contains local Transformers/MLX and remote Responses/Chat
Completions handlers. See [language model backends](../../docs/speech/llm.md)
for configuration and [realtime API](../../docs/speech/realtime-api.md) for the
tool-call and conversation flow.
speech_to_speech/STT/README.md
+4 -162
View File
@@ -1,163 +1,5 @@
# STT Summary
# Speech-to-text handlers
This document summarizes the Speech-to-Text (STT) implementations in the `STT/` folder, including language support, language abbreviations, and usage in `s2s_pipeline.py`.
## Available STT Modes (`--stt`)
- `whisper` → `STT/whisper_stt_handler.py`
- `whisper-mlx` → `STT/lightning_whisper_mlx_handler.py`
- `mlx-audio-whisper` → `STT/mlx_audio_whisper_handler.py`
- `faster-whisper` → `STT/faster_whisper_handler.py`
- `parakeet-tdt` → `STT/parakeet_tdt_handler.py`
- `paraformer` → `STT/paraformer_handler.py`
## Language Support by Handler
### 1) Whisper (`--stt whisper`)
- Handler: `WhisperSTTHandler`
- Language input flag: `--language` (from shared Whisper args)
- Supports fixed language (e.g. `en`) or `auto`
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- Detects language from token output
- If detected language is outside the supported list, it falls back to the previous language
### 2) Lightning Whisper MLX (`--stt whisper-mlx`)
- Handler: `LightningWhisperSTTHandler`
- Uses same shared `--language` argument as Whisper
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- If `--language auto`, model auto-detects each utterance
- If detected language is unsupported, falls back to last supported language
### 3) MLX Audio Whisper (`--stt mlx-audio-whisper`)
- Handler: `MLXAudioWhisperSTTHandler`
- Model flag: `--mlx_audio_whisper_model_name`
- Language still comes from shared `--language` flag (wired in pipeline)
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- Uses fixed language unless `--language auto`
- Falls back to last known supported language when needed
### 4) Faster-Whisper (`--stt faster-whisper`)
- Handler: `FasterWhisperSTTHandler`
- Language flag: `--faster_whisper_stt_gen_language`
- Default language: `en`
- Note:
- This handler passes generation kwargs directly to `faster_whisper.WhisperModel.transcribe(...)`
- Effective language coverage depends on selected Faster-Whisper/OpenAI Whisper model
### 5) Parakeet TDT (`--stt parakeet-tdt`)
- Handler: `ParakeetTDTSTTHandler`
- Language flag: `--parakeet_tdt_language` (optional)
- Supports auto language detection when language not specified
- Declared supported language list (25 European languages):
- `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `pl`, `ru`, `uk`, `cs`, `sk`, `hu`, `ro`, `bg`, `hr`, `sl`, `sr`, `da`, `no`, `sv`, `fi`, `et`, `lv`, `lt`
- Backend behavior:
- On macOS/MPS: MLX (`mlx-community/parakeet-tdt-0.6b-v3`)
- On CUDA/CPU: nano-parakeet (`nvidia/parakeet-tdt-0.6b-v3`)
### 6) Paraformer (`--stt paraformer`)
- Handler: `ParaformerSTTHandler`
- Model flag: `--paraformer_stt_model_name`
- Default model: `paraformer-zh`
- No dedicated language flag in current args class
- Practical support:
- Depends on selected FunASR model checkpoint
- Default setup is Chinese-oriented (`zh`)
## Language Abbreviations (ISO-style codes seen in STT handlers)
| Code | Language |
|---|---|
| `en` | English |
| `fr` | French |
| `es` | Spanish |
| `zh` | Chinese |
| `ja` | Japanese |
| `ko` | Korean |
| `hi` | Hindi |
| `de` | German |
| `pt` | Portuguese |
| `pl` | Polish |
| `it` | Italian |
| `nl` | Dutch |
| `ru` | Russian |
| `uk` | Ukrainian |
| `cs` | Czech |
| `sk` | Slovak |
| `hu` | Hungarian |
| `ro` | Romanian |
| `bg` | Bulgarian |
| `hr` | Croatian |
| `sl` | Slovenian |
| `sr` | Serbian |
| `da` | Danish |
| `no` | Norwegian |
| `sv` | Swedish |
| `fi` | Finnish |
| `et` | Estonian |
| `lv` | Latvian |
| `lt` | Lithuanian |
| `auto` | Per-utterance automatic language detection |
## Usage Examples
### Whisper (Transformers)
```bash
python s2s_pipeline.py --stt whisper --language en
python s2s_pipeline.py --stt whisper --language auto
```
### Whisper MLX (LightningWhisperMLX)
```bash
python s2s_pipeline.py --stt whisper-mlx --language auto --device mps
```
### MLX Audio Whisper
```bash
python s2s_pipeline.py --stt mlx-audio-whisper \
--mlx_audio_whisper_model_name mlx-community/whisper-large-v3-turbo \
--language auto
```
### Faster-Whisper
```bash
python s2s_pipeline.py --stt faster-whisper \
--faster_whisper_stt_model_name large-v3 \
--faster_whisper_stt_gen_language en
```
### Parakeet TDT
```bash
python s2s_pipeline.py --stt parakeet-tdt --parakeet_tdt_device auto
python s2s_pipeline.py --stt parakeet-tdt --parakeet_tdt_language de
```
With live transcription (MLX or CUDA/nano-parakeet backend):
```bash
python s2s_pipeline.py --stt parakeet-tdt \
--enable_live_transcription \
--live_transcription_update_interval 0.25
```
### Paraformer
```bash
python s2s_pipeline.py --stt paraformer --paraformer_stt_model_name paraformer-zh
```
This directory contains Whisper, Parakeet, and Paraformer handlers and live
transcription helpers. See [speech-to-text backends](../../docs/speech/stt.md)
for supported modes, language codes, and configuration.
speech_to_speech/TTS/README.md
+4 -154
View File
@@ -1,155 +1,5 @@
# TTS Summary
# Text-to-speech handlers
## Available TTS modes (`--tts`)
Runtime-supported values in `s2s_pipeline.py`:
- `chatTTS` → `chatTTS_handler.py`
- `facebookMMS` → `facebookmms_handler.py`
- `pocket` → `pocket_tts_handler.py`
- `kokoro` → `kokoro_handler.py`
- `qwen3` → `qwen3_tts_handler.py`
Deprecated TTS implementations, including MeloTTS, live in [`../../../archive/TTS`](../../../archive/TTS) and are no longer wired into `s2s_pipeline.py`.
## Usage
### 1) ChatTTS (`--tts chatTTS`)
Primary args prefix: `--chat_tts_*`
```bash
python s2s_pipeline.py \
--tts chatTTS \
--chat_tts_device cuda \
--chat_tts_stream true \
--chat_tts_chunk_size 512
```
### 2) Facebook MMS (`--tts facebookMMS`)
Primary args prefix: `--facebook_mms_*` plus `--tts_language`
```bash
python s2s_pipeline.py \
--tts facebookMMS \
--facebook_mms_device cuda \
--tts_language en
```
This handler maps STT language codes (e.g. `en`, `fr`, `es`) to MMS model suffixes (e.g. `eng`, `fra`, `spa`) and reloads the model on language changes.
### 3) Pocket TTS (`--tts pocket`)
Primary args prefix: `--pocket_tts_*`
```bash
python s2s_pipeline.py \
--tts pocket \
--pocket_tts_voice jean \
--pocket_tts_device cpu \
--pocket_tts_sample_rate 16000
```
Available preset voices include:
`alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`, `azelma`.
### 4) Kokoro (`--tts kokoro`)
Primary args prefix: `--kokoro_*`
```bash
python s2s_pipeline.py \
--tts kokoro \
--kokoro_device auto \
--kokoro_voice bm_fable \
--kokoro_lang_code b
```
Behavior:
- Uses MLX backend on Apple Silicon (`mlx-community/Kokoro-82M-bf16`)
- Uses native kokoro pipeline otherwise (`hexgrad/Kokoro-82M`)
- Can auto-switch voice/language based on STT language code mapping
### 5) Qwen3-TTS (`--tts qwen3`)
Primary args prefix: `--qwen3_tts_*`
```bash
python s2s_pipeline.py \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_device cuda \
--qwen3_tts_backend ggml \
--qwen3_tts_speaker Aiden \
--qwen3_tts_language auto \
--qwen3_tts_non_streaming_mode True
```
Behavior:
- Uses `faster-qwen3-tts` on non-macOS platforms, defaulting to the GGML backend. Pass `--qwen3_tts_backend torch` to use the CUDA-graphs backend instead.
- Uses `mlx-audio` on Apple Silicon and auto-maps `Qwen/...` model IDs to `mlx-community/...`, defaulting to the `6bit` MLX variant unless the model name already pins a suffix.
- Supports MLX quantization overrides on Apple Silicon via `--qwen3_tts_mlx_quantization bf16|4bit|6bit|8bit`.
- Keeps the existing voice-clone/custom-voice/voice-design handler flow intact.
- Defaults to the CustomVoice model with speaker `Aiden`, so no reference audio is required. Voice-clone/base models can still use `--qwen3_tts_ref_audio`.
Install notes for Linux GGML:
- The default PyPI `qwentts-cpp-python` wheel targets CUDA 12.8.
- If that wheel does not match your CUDA runtime, install one of the Hugging Face wheelhouse builds before installing `speech-to-speech`.
```bash
pip install "qwentts-cpp-python==0.3.1+cu130" \
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130
pip install speech-to-speech
```
Available wheelhouse directories include `cu124`, `cu128`, `cu130`, and `cpu`.
Example for Apple Silicon using the default 6-bit MLX variant:
```bash
python s2s_pipeline.py \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_speaker Aiden
```
You can override the default and select `bf16`, `4bit`, or `8bit` explicitly:
```bash
python s2s_pipeline.py \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_mlx_quantization 4bit \
--qwen3_tts_speaker Aiden
```
To benchmark the Apple Silicon MLX variants side by side:
```bash
.venv/bin/python benchmark_tts.py \
--handlers qwen3 \
--iterations 3 \
--qwen3_mlx_quantizations bf16 4bit 6bit 8bit
```
This will run separate benchmark entries for `qwen3[bf16]`, `qwen3[4bit]`, `qwen3[6bit]`, and `qwen3[8bit]`.
## Setup
### Low-latency GPU setup
```bash
python s2s_pipeline.py \
--stt whisper \
--tts pocket
```
### Apple Silicon setup
```bash
python s2s_pipeline.py \
--local_mac_optimal_settings
```
`--tts pocket` and `--tts kokoro` are also valid options on macOS.
This directory contains ChatTTS, Facebook MMS, Pocket, Kokoro, and Qwen3-TTS
handlers. See [text-to-speech backends](../../docs/speech/tts.md) for configuration,
voice selection, and the optional reference-audio example.
speech_to_speech/TTS/ref_audio.wav
BIN
View File
Binary file not shown.
speech_to_speech/api/openai_realtime/README.md
+5 -215
View File
@@ -1,216 +1,6 @@
## Realtime Engine -- High-Level Architecture
# Realtime API implementation
The server runs as a handler thread inside the existing pipeline. A FastAPI/uvicorn instance serves the WebSocket endpoint at `/v1/realtime`, while the same queue-driven handler chain (VAD, STT, LLM, TTS) processes audio in parallel threads.
```mermaid
flowchart LR
subgraph client [Client]
WS["WebSocket /v1/realtime"]
end
subgraph server [Realtime Server]
Router["WebSocket Router"]
Service["RealtimeService"]
Config["RuntimeConfig"]
end
subgraph pipeline [Pipeline Threads]
VAD["VAD"]
STT["STT"]
TN["TranscriptionNotifier"]
LLM["LLM"]
Proc["LMOutputProcessor"]
TTS["TTS"]
end
WS -- "client events (JSON)" --> Router
Router -- "parse / dispatch" --> Service
Service -- "PCM chunks" --> VAD
Service -- "session config" --> Config
Config -. "read by" .-> VAD & LLM & TTS
VAD -- "speech segments" --> STT
STT -- "transcript" --> TN
TN -- "transcript + notify" --> LLM
LLM -- "text + tools" --> Proc
Proc -- "clean text" --> TTS
TTS -- "PCM audio" --> Router
Proc -- "assistant_text / tools" --> Router
VAD -- "speech_started/stopped" --> Router
TN -- "transcription events" --> Router
Router -- "server events (JSON)" --> WS
```
**Key flow:**
1. **Inbound audio**: Client sends `input_audio_buffer.append` with base64 PCM. `RealtimeService` decodes, resamples to 16 kHz, splits into 512-sample chunks, and puts them on the `recv_audio_chunks_queue` for VAD.
2. **Speech detection**: VAD detects speech boundaries and emits `speech_started` / `speech_stopped` events on the `text_output_queue`. Full utterance audio goes to STT.
3. **Transcription**: STT output passes through `TranscriptionNotifier`, which taps the transcript for `transcription.delta` / `transcription.completed` events before forwarding to the LLM.
4. **Generation**: The LLM generates text (and optional tool calls). `LMOutputProcessor` splits the output: clean text goes to TTS, and `assistant_text` + tool call dicts go to the `text_output_queue`.
5. **Outbound audio**: TTS writes PCM chunks to `send_audio_chunks_queue`. The router's async `_send_loop` drains both queues, encoding PCM as `response.output_audio.delta` events and translating internal messages into protocol events.
6. **Session config**: `session.update` events deep-merge into `RuntimeConfig`, which is a shared Pydantic model read by VAD (turn detection thresholds), LLM (instructions, tools), and TTS (voice) at processing time.
---
## Supported OpenAI Realtime Events
### Client -> Server
| Event | Description |
|---|---|
| `input_audio_buffer.append` | Stream base64 PCM audio. Decoded, resampled to 16 kHz, and chunked for the VAD. |
| `session.update` | Deep-merge session config (instructions, tools, voice, turn detection, audio format). |
| `conversation.item.create` | Inject `input_text` or `function_call_output` into the LLM context without triggering generation. |
| `response.create` | Trigger LLM generation. Supports per-response `instructions` and `tool_choice` overrides. |
| `response.cancel` | Cancel the in-progress response and re-enable listening. |
### Server -> Client
| Event | Description |
|---|---|
| `session.created` | Sent on connection with current session config. |
| `error` | Protocol errors (`session_limit_reached`, `unknown_or_invalid_event`, `invalid_session_type`, `conversation_already_has_active_response`, etc.) |
| `input_audio_buffer.speech_started` | VAD detected user speech. |
| `input_audio_buffer.speech_stopped` | End of user speech segment. |
| `conversation.item.created` | Acknowledges injected `input_text` from `conversation.item.create`. |
| `conversation.item.input_audio_transcription.delta` | Streaming partial transcript (when live transcription is enabled). |
| `conversation.item.input_audio_transcription.completed` | Final transcript for the user turn (with duration usage). |
| `response.created` | Emitted on the first outbound audio chunk (response is `in_progress`). |
| `response.output_audio.delta` | Base64 PCM audio chunk from TTS. |
| `response.output_audio.done` | Audio stream complete for the current output item. |
| `response.output_audio_transcript.done` | Full assistant text transcript for the turn. |
| `response.function_call_arguments.done` | Tool call with `call_id`, `name`, and JSON `arguments`. |
| `response.done` | Response finished (`completed`, `cancelled` with reason `turn_detected` or `client_cancelled`). |
---
## Tool Calling Design
Tool calling works through two distinct paths depending on the LLM backend, but both converge to the same wire protocol for the client.
### Local LLM path (`LanguageModelHandler` -- transformers / mlx-lm)
Tools defined in `session.update` are converted to `FunctionTool` objects. Each tool's JSON Schema `parameters` are turned into a Python `inspect.Signature` (via `signature_from_schema`), and `to_code_prompt()` renders a human-readable `def name(...): """docstring"""` block.
These tool signatures are injected into the system prompt using a Jinja2 template (`tool_prompt.py`) that instructs the model to wrap tool calls in `<code>...</code>` delimiters:
```
<code>
function_name(arg_name_1=value1, arg_name_2='string_value')
</code>
```
After generation, `_extract_tools` uses a regex to find `<code>` blocks, then `extract_function_calls_from_text` parses each `name(kwargs)` call and validates it against registered tools. Valid calls become `ResponseFunctionToolCall` dicts with generated `call_id`s.
### OpenAI API path (`ResponsesApiModelHandler`)
Tools are passed natively as the `tools=` parameter to `client.responses.create`. The API returns structured `function_call` items directly -- no prompt engineering or regex parsing needed. Per-response `tool_choice` overrides from `response.create` are supported.
### Common output path
Both handlers yield `(text, language_code, tools)` tuples. `LMOutputProcessor` forwards clean text to TTS and puts `{"type": "assistant_text", "text": ..., "tools": [...]}` on the `text_output_queue`. The router's `_send_loop` translates these into:
- `response.output_audio_transcript.done` for the text
- `response.function_call_arguments.done` for each tool call
### Tool result flow
1. Client executes the tool and sends `conversation.item.create` with `type: "function_call_output"` and `output: "<result>"`
2. `RealtimeService` appends the tool output to the chat context and emits `conversation.item.created`; this does not trigger generation.
3. If the tool result needs to be spoken to the user, such as camera/search/data results, the client sends `response.create` to trigger follow-up generation.
4. For fire-and-forget robot actions such as dance, emotion, head movement, stop, or idle tools, the client can stop after `conversation.item.created`; the assistant should already have spoken the natural lead-in before the tool call.
---
## Interruption Handling
Barge-in (user speaks while the assistant is playing audio) is handled cooperatively between the VAD, the `_send_loop`, and the LLM/TTS handlers via a shared `CancelScope` object (`cancel_scope.py`).
### CancelScope design
`CancelScope` replaces the old two-signal pattern (`cancel_response` Event + `discard_stale_output` boolean) with a single object that manages:
- **Generation counter** (`cancel_scope.generation`): pipeline threads (LLM, TTS) capture the current generation at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token. When `cancel()` is called, the generation increments and all prior generations become stale -- no timing games required.
- **Discard flag** (`cancel_scope.discarding`): set by `cancel()`, checked by the async `_send_loop` to drop output from superseded generations that arrives between `cancel()` and `response_done()`. Cleared by `response_done(generation)` (only when the sentinel's generation matches the discarded or current one -- sentinels from unrelated older generations are ignored), by `new_response()` on an explicit `response.create`, or by `reset()` on session claim/release.
Pipeline output is **generation-tagged**: `AudioOutput` chunks and `AssistantTextEvent`s carry a `cancel_generation` field stamped by the handler that produced them. The send loop's `_generation_is_discardable` drops an item if its generation is stale, or if `discarding` is set and the item is not from the current generation. Output from the *current* generation always passes through, so a fresh response is never swallowed by a lingering discard window (e.g. a superseded speculative turn whose TTS never emitted a `__RESPONSE_DONE__` sentinel).
```mermaid
sequenceDiagram
participant User
participant VAD
participant SendLoop as _send_loop
participant LLM
participant TTS
participant Client
Note over TTS,Client: Response active or pending (in_response / response_pending)
User->>VAD: speaks
VAD->>SendLoop: speech_started on text_output_queue
SendLoop->>Client: input_audio_buffer.speech_started
SendLoop->>Client: response.done (status=cancelled, reason=turn_detected)
SendLoop->>SendLoop: cancel_scope.cancel() (gen++ & discarding=True)
SendLoop->>SendLoop: flush output_queue + text_output_queue
SendLoop->>SendLoop: response_playing.clear()
LLM->>LLM: is_stale(gen) → True, aborts generation
TTS->>TTS: is_stale(gen) → True, aborts generation
TTS->>SendLoop: __RESPONSE_DONE__ (tagged with gen)
SendLoop->>SendLoop: cancel_scope.response_done(gen) (discarding=False)
Note over VAD,Client: Pipeline is now processing the new user utterance
```
**Step by step:**
1. **VAD detects speech**: puts a `SpeechStartedEvent` on `text_output_queue`.
2. **`_send_loop` processes text events first** (priority over audio): translates `speech_started` into protocol events. If an active response was in progress, `RealtimeService.dispatch_pipeline_event` emits `response.output_audio.done` + `response.done` with `status="cancelled"` and `reason="turn_detected"`.
3. **Cancel + queue flush**: if a response is active (`in_response`) *or* pending (`response_pending` -- accepted `response.create`, no audio yet), and interrupts are enabled (see step 4), the send loop calls `cancel_scope.cancel()` (increments generation, enables discard), clears `response_pending`, drains `output_queue` (preserving `__RESPONSE_DONE__` sentinels) and `text_output_queue` (preserving user-side events: `speech_stopped`, partial/completed transcriptions, token usage), then clears `response_playing`.
4. **Interrupt gating**: the cancel only fires if the `SpeechStartedEvent.interrupt_response` flag is set *and* the session config allows it (`turn_detection.interrupt_response`, read via `RuntimeConfig.interrupt_response_enabled`, default true). When disabled, user speech during a response is transcribed but the response keeps playing.
5. **LLM/TTS cancellation**: handlers capture `gen = cancel_scope.generation` at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token, aborting early when stale.
6. **Discard guard**: while `cancel_scope.discarding` is True, the send loop drops audio chunks and assistant text whose `cancel_generation` is not current (see `_generation_is_discardable` above). The guard clears when a `__RESPONSE_DONE__` with a matching generation arrives (via `cancel_scope.response_done(gen)`), or when an explicit `response.create` starts a new response (`cancel_scope.new_response()`).
7. **Client-initiated cancel**: `response.cancel` calls `cancel_scope.cancel()` (only if a response was active), flushes both queues with the same preservation rules, triggers `finish_response(status="cancelled", reason="client_cancelled")`, re-enables `should_listen`, and clears `response_playing`.
8. **Spurious cancel safety**: if no response is active, `cancel_scope.cancel()` is not called, preventing the discard guard from being set without a `__RESPONSE_DONE__` to clear it.
---
## Testing
### Local LLM with Transformers
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend transformers \
--tts kokoro \
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
### Local LLM with MLX-LM
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend mlx-lm \
--tts kokoro \
--model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
### Remote LLM with OpenAI-compatible API
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend responses-api \
--tts kokoro \
--model_name "openai/gpt-oss-20b:groq" \
--responses_api_base_url "https://router.huggingface.co/v1" \
--responses_api_api_key "$HF_TOKEN" \
--responses_api_stream \
--enable_live_transcription
```
This directory implements the OpenAI Realtime-compatible WebSocket service,
session configuration, event handlers, and pipeline routing. See the
[realtime API guide](../../../docs/speech/realtime-api.md) for architecture,
supported events, tool calling, cancellation, and startup examples.