first git

src/speech_to_speech/api/openai_realtime/README.md

@@ -0,0 +1,216 @@
+## Realtime Engine -- High-Level Architecture
+
+The server runs as a handler thread inside the existing pipeline. A FastAPI/uvicorn instance serves the WebSocket endpoint at `/v1/realtime`, while the same queue-driven handler chain (VAD, STT, LLM, TTS) processes audio in parallel threads.
+
+```mermaid
+flowchart LR
+    subgraph client [Client]
+        WS["WebSocket /v1/realtime"]
+    end
+
+    subgraph server [Realtime Server]
+        Router["WebSocket Router"]
+        Service["RealtimeService"]
+        Config["RuntimeConfig"]
+    end
+
+    subgraph pipeline [Pipeline Threads]
+        VAD["VAD"]
+        STT["STT"]
+        TN["TranscriptionNotifier"]
+        LLM["LLM"]
+        Proc["LMOutputProcessor"]
+        TTS["TTS"]
+    end
+
+    WS -- "client events (JSON)" --> Router
+    Router -- "parse / dispatch" --> Service
+    Service -- "PCM chunks" --> VAD
+    Service -- "session config" --> Config
+    Config -. "read by" .-> VAD & LLM & TTS
+    VAD -- "speech segments" --> STT
+    STT -- "transcript" --> TN
+    TN -- "transcript + notify" --> LLM
+    LLM -- "text + tools" --> Proc
+    Proc -- "clean text" --> TTS
+    TTS -- "PCM audio" --> Router
+    Proc -- "assistant_text / tools" --> Router
+    VAD -- "speech_started/stopped" --> Router
+    TN -- "transcription events" --> Router
+    Router -- "server events (JSON)" --> WS
+```
+
+**Key flow:**
+
+1. **Inbound audio**: Client sends `input_audio_buffer.append` with base64 PCM. `RealtimeService` decodes, resamples to 16 kHz, splits into 512-sample chunks, and puts them on the `recv_audio_chunks_queue` for VAD.
+2. **Speech detection**: VAD detects speech boundaries and emits `speech_started` / `speech_stopped` events on the `text_output_queue`. Full utterance audio goes to STT.
+3. **Transcription**: STT output passes through `TranscriptionNotifier`, which taps the transcript for `transcription.delta` / `transcription.completed` events before forwarding to the LLM.
+4. **Generation**: The LLM generates text (and optional tool calls). `LMOutputProcessor` splits the output: clean text goes to TTS, and `assistant_text` + tool call dicts go to the `text_output_queue`.
+5. **Outbound audio**: TTS writes PCM chunks to `send_audio_chunks_queue`. The router's async `_send_loop` drains both queues, encoding PCM as `response.output_audio.delta` events and translating internal messages into protocol events.
+6. **Session config**: `session.update` events deep-merge into `RuntimeConfig`, which is a shared Pydantic model read by VAD (turn detection thresholds), LLM (instructions, tools), and TTS (voice) at processing time.
+
+---
+
+## Supported OpenAI Realtime Events
+
+### Client -> Server
+
+| Event | Description |
+|---|---|
+| `input_audio_buffer.append` | Stream base64 PCM audio. Decoded, resampled to 16 kHz, and chunked for the VAD. |
+| `session.update` | Deep-merge session config (instructions, tools, voice, turn detection, audio format). |
+| `conversation.item.create` | Inject `input_text` or `function_call_output` into the LLM context without triggering generation. |
+| `response.create` | Trigger LLM generation. Supports per-response `instructions` and `tool_choice` overrides. |
+| `response.cancel` | Cancel the in-progress response and re-enable listening. |
+
+### Server -> Client
+
+| Event | Description |
+|---|---|
+| `session.created` | Sent on connection with current session config. |
+| `error` | Protocol errors (`session_limit_reached`, `unknown_or_invalid_event`, `invalid_session_type`, `conversation_already_has_active_response`, etc.) |
+| `input_audio_buffer.speech_started` | VAD detected user speech. |
+| `input_audio_buffer.speech_stopped` | End of user speech segment. |
+| `conversation.item.created` | Acknowledges injected `input_text` from `conversation.item.create`. |
+| `conversation.item.input_audio_transcription.delta` | Streaming partial transcript (when live transcription is enabled). |
+| `conversation.item.input_audio_transcription.completed` | Final transcript for the user turn (with duration usage). |
+| `response.created` | Emitted on the first outbound audio chunk (response is `in_progress`). |
+| `response.output_audio.delta` | Base64 PCM audio chunk from TTS. |
+| `response.output_audio.done` | Audio stream complete for the current output item. |
+| `response.output_audio_transcript.done` | Full assistant text transcript for the turn. |
+| `response.function_call_arguments.done` | Tool call with `call_id`, `name`, and JSON `arguments`. |
+| `response.done` | Response finished (`completed`, `cancelled` with reason `turn_detected` or `client_cancelled`). |
+
+---
+
+## Tool Calling Design
+
+Tool calling works through two distinct paths depending on the LLM backend, but both converge to the same wire protocol for the client.
+
+### Local LLM path (`LanguageModelHandler` -- transformers / mlx-lm)
+
+Tools defined in `session.update` are converted to `FunctionTool` objects. Each tool's JSON Schema `parameters` are turned into a Python `inspect.Signature` (via `signature_from_schema`), and `to_code_prompt()` renders a human-readable `def name(...): """docstring"""` block.
+
+These tool signatures are injected into the system prompt using a Jinja2 template (`tool_prompt.py`) that instructs the model to wrap tool calls in `<code>...</code>` delimiters:
+
+```
+<code>
+function_name(arg_name_1=value1, arg_name_2='string_value')
+</code>
+```
+
+After generation, `_extract_tools` uses a regex to find `<code>` blocks, then `extract_function_calls_from_text` parses each `name(kwargs)` call and validates it against registered tools. Valid calls become `ResponseFunctionToolCall` dicts with generated `call_id`s.
+
+### OpenAI API path (`ResponsesApiModelHandler`)
+
+Tools are passed natively as the `tools=` parameter to `client.responses.create`. The API returns structured `function_call` items directly -- no prompt engineering or regex parsing needed. Per-response `tool_choice` overrides from `response.create` are supported.
+
+### Common output path
+
+Both handlers yield `(text, language_code, tools)` tuples. `LMOutputProcessor` forwards clean text to TTS and puts `{"type": "assistant_text", "text": ..., "tools": [...]}` on the `text_output_queue`. The router's `_send_loop` translates these into:
+- `response.output_audio_transcript.done` for the text
+- `response.function_call_arguments.done` for each tool call
+
+### Tool result flow
+
+1. Client executes the tool and sends `conversation.item.create` with `type: "function_call_output"` and `output: "<result>"`
+2. `RealtimeService` appends the tool output to the chat context and emits `conversation.item.created`; this does not trigger generation.
+3. If the tool result needs to be spoken to the user, such as camera/search/data results, the client sends `response.create` to trigger follow-up generation.
+4. For fire-and-forget robot actions such as dance, emotion, head movement, stop, or idle tools, the client can stop after `conversation.item.created`; the assistant should already have spoken the natural lead-in before the tool call.
+
+---
+
+## Interruption Handling
+
+Barge-in (user speaks while the assistant is playing audio) is handled cooperatively between the VAD, the `_send_loop`, and the LLM/TTS handlers via a shared `CancelScope` object (`cancel_scope.py`).
+
+### CancelScope design
+
+`CancelScope` replaces the old two-signal pattern (`cancel_response` Event + `discard_stale_output` boolean) with a single object that manages:
+
+- **Generation counter** (`cancel_scope.generation`): pipeline threads (LLM, TTS) capture the current generation at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token. When `cancel()` is called, the generation increments and all prior generations become stale -- no timing games required.
+- **Discard flag** (`cancel_scope.discarding`): set by `cancel()`, checked by the async `_send_loop` to drop output from superseded generations that arrives between `cancel()` and `response_done()`. Cleared by `response_done(generation)` (only when the sentinel's generation matches the discarded or current one -- sentinels from unrelated older generations are ignored), by `new_response()` on an explicit `response.create`, or by `reset()` on session claim/release.
+
+Pipeline output is **generation-tagged**: `AudioOutput` chunks and `AssistantTextEvent`s carry a `cancel_generation` field stamped by the handler that produced them. The send loop's `_generation_is_discardable` drops an item if its generation is stale, or if `discarding` is set and the item is not from the current generation. Output from the *current* generation always passes through, so a fresh response is never swallowed by a lingering discard window (e.g. a superseded speculative turn whose TTS never emitted a `__RESPONSE_DONE__` sentinel).
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant VAD
+    participant SendLoop as _send_loop
+    participant LLM
+    participant TTS
+    participant Client
+
+    Note over TTS,Client: Response active or pending (in_response / response_pending)
+    User->>VAD: speaks
+    VAD->>SendLoop: speech_started on text_output_queue
+    SendLoop->>Client: input_audio_buffer.speech_started
+    SendLoop->>Client: response.done (status=cancelled, reason=turn_detected)
+    SendLoop->>SendLoop: cancel_scope.cancel() (gen++ & discarding=True)
+    SendLoop->>SendLoop: flush output_queue + text_output_queue
+    SendLoop->>SendLoop: response_playing.clear()
+    LLM->>LLM: is_stale(gen) → True, aborts generation
+    TTS->>TTS: is_stale(gen) → True, aborts generation
+    TTS->>SendLoop: __RESPONSE_DONE__ (tagged with gen)
+    SendLoop->>SendLoop: cancel_scope.response_done(gen) (discarding=False)
+    Note over VAD,Client: Pipeline is now processing the new user utterance
+```
+
+**Step by step:**
+
+1. **VAD detects speech**: puts a `SpeechStartedEvent` on `text_output_queue`.
+2. **`_send_loop` processes text events first** (priority over audio): translates `speech_started` into protocol events. If an active response was in progress, `RealtimeService.dispatch_pipeline_event` emits `response.output_audio.done` + `response.done` with `status="cancelled"` and `reason="turn_detected"`.
+3. **Cancel + queue flush**: if a response is active (`in_response`) *or* pending (`response_pending` -- accepted `response.create`, no audio yet), and interrupts are enabled (see step 4), the send loop calls `cancel_scope.cancel()` (increments generation, enables discard), clears `response_pending`, drains `output_queue` (preserving `__RESPONSE_DONE__` sentinels) and `text_output_queue` (preserving user-side events: `speech_stopped`, partial/completed transcriptions, token usage), then clears `response_playing`.
+4. **Interrupt gating**: the cancel only fires if the `SpeechStartedEvent.interrupt_response` flag is set *and* the session config allows it (`turn_detection.interrupt_response`, read via `RuntimeConfig.interrupt_response_enabled`, default true). When disabled, user speech during a response is transcribed but the response keeps playing.
+5. **LLM/TTS cancellation**: handlers capture `gen = cancel_scope.generation` at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token, aborting early when stale.
+6. **Discard guard**: while `cancel_scope.discarding` is True, the send loop drops audio chunks and assistant text whose `cancel_generation` is not current (see `_generation_is_discardable` above). The guard clears when a `__RESPONSE_DONE__` with a matching generation arrives (via `cancel_scope.response_done(gen)`), or when an explicit `response.create` starts a new response (`cancel_scope.new_response()`).
+7. **Client-initiated cancel**: `response.cancel` calls `cancel_scope.cancel()` (only if a response was active), flushes both queues with the same preservation rules, triggers `finish_response(status="cancelled", reason="client_cancelled")`, re-enables `should_listen`, and clears `response_playing`.
+8. **Spurious cancel safety**: if no response is active, `cancel_scope.cancel()` is not called, preventing the discard guard from being set without a `__RESPONSE_DONE__` to clear it.
+
+---
+
+## Testing
+
+### Local LLM with Transformers
+
+```bash
+uv run speech-to-speech \
+  --mode realtime \
+  --stt parakeet-tdt \
+  --llm_backend transformers \
+  --tts kokoro \
+  --model_name "Qwen/Qwen3-4B-Instruct-2507" \
+  --llm_device mps \
+  --llm_torch_dtype float16 \
+  --enable_live_transcription
+```
+
+### Local LLM with MLX-LM
+
+```bash
+uv run speech-to-speech \
+  --mode realtime \
+  --stt parakeet-tdt \
+  --llm_backend mlx-lm \
+  --tts kokoro \
+  --model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" \
+  --llm_device mps \
+  --llm_torch_dtype float16 \
+  --enable_live_transcription
+```
+
+### Remote LLM with OpenAI-compatible API
+
+```bash
+uv run speech-to-speech \
+  --mode realtime \
+  --stt parakeet-tdt \
+  --llm_backend responses-api \
+  --tts kokoro \
+  --model_name "openai/gpt-oss-20b:groq" \
+  --responses_api_base_url "https://router.huggingface.co/v1" \
+  --responses_api_api_key "$HF_TOKEN" \
+  --responses_api_stream \
+  --enable_live_transcription
+```

src/speech_to_speech/api/openai_realtime/delay_fillers.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import logging
+import os
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+
+from speech_to_speech.pipeline.messages import TTSInput
+
+logger = logging.getLogger(__name__)
+
+_TRUE_VALUES = {"1", "true", "yes", "on"}
+
+
+@dataclass(slots=True)
+class DelayFillerConfig:
+    enabled: bool
+    threshold_s: float
+    short_phrases: list[str]
+    full_phrases: list[str]
+
+
+def _split_env_list(name: str) -> list[str]:
+    raw = os.environ.get(name, "").strip()
+    return [part.strip() for part in raw.split("|") if part.strip()]
+
+
+def _env_flag(name: str, *, default: bool) -> bool:
+    raw = os.environ.get(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() in _TRUE_VALUES
+
+
+def load_delay_filler_config() -> DelayFillerConfig:
+    short_phrases = _split_env_list("TTS_DELAY_FILLERS_SHORT_ES")
+    full_phrases = _split_env_list("TTS_DELAY_FILLERS_FULL_ES")
+
+    # Backward compatibility with the old single-list variable.
+    if not short_phrases and not full_phrases:
+        full_phrases = _split_env_list("TTS_DELAY_FILLERS_ES")
+
+    enabled = _env_flag("TTS_DELAY_FILLERS_ENABLED", default=bool(short_phrases or full_phrases))
+    threshold_ms_raw = os.environ.get("TTS_DELAY_FILLERS_THRESHOLD_MS", "1200").strip() or "1200"
+    try:
+        threshold_ms = max(0, int(threshold_ms_raw))
+    except ValueError:
+        logger.warning("Invalid TTS_DELAY_FILLERS_THRESHOLD_MS=%r; using 1200", threshold_ms_raw)
+        threshold_ms = 1200
+    return DelayFillerConfig(
+        enabled=enabled and bool(short_phrases or full_phrases),
+        threshold_s=threshold_ms / 1000.0,
+        short_phrases=short_phrases,
+        full_phrases=full_phrases,
+    )
+
+
+def _chunk_to_bytes(chunk: Any) -> bytes:
+    audio = getattr(chunk, "audio", chunk)
+    if isinstance(audio, np.ndarray) or hasattr(audio, "tobytes"):
+        return audio.tobytes()
+    if isinstance(audio, bytes):
+        return audio
+    raise TypeError(f"Unsupported delay filler audio chunk type: {type(audio)!r}")
+
+
+def pregenerate_delay_filler_clips(tts_handler: Any, phrases: list[str]) -> list[bytes]:
+    clips: list[bytes] = []
+    for phrase in phrases:
+        try:
+            clip = b"".join(
+                _chunk_to_bytes(chunk)
+                for chunk in tts_handler.process(TTSInput(text=phrase, language_code="es"))
+            )
+        except Exception:
+            logger.exception("Failed to pregenerate TTS delay filler for phrase %r", phrase)
+            continue
+        if not clip:
+            logger.warning("Discarding empty TTS delay filler for phrase %r", phrase)
+            continue
+        clips.append(clip)
+    return clips

src/speech_to_speech/api/openai_realtime/handlers/__init__.py

@@ -0,0 +1,11 @@
+from speech_to_speech.api.openai_realtime.handlers.audio import AudioHandler
+from speech_to_speech.api.openai_realtime.handlers.conversation import ConversationHandler
+from speech_to_speech.api.openai_realtime.handlers.response import ResponseHandler
+from speech_to_speech.api.openai_realtime.handlers.session import SessionHandler
+
+__all__ = [
+    "AudioHandler",
+    "ConversationHandler",
+    "ResponseHandler",
+    "SessionHandler",
+]

src/speech_to_speech/api/openai_realtime/handlers/audio.py

@@ -0,0 +1,199 @@
+from __future__ import annotations
+
+import base64
+import logging
+from typing import TYPE_CHECKING
+
+from openai.types.realtime import (
+    InputAudioBufferAppendEvent,
+    InputAudioBufferSpeechStartedEvent,
+    InputAudioBufferSpeechStoppedEvent,
+    RealtimeErrorEvent,
+    ResponseAudioDeltaEvent,
+    ResponseCreatedEvent,
+)
+
+from speech_to_speech.api.openai_realtime.handlers.base import RealtimeBaseHandler
+from speech_to_speech.api.openai_realtime.utils import resample
+from speech_to_speech.pipeline.events import SpeechStartedEvent, SpeechStoppedEvent
+
+if TYPE_CHECKING:
+    from speech_to_speech.api.openai_realtime.service import ServerEvent
+
+logger = logging.getLogger(__name__)
+
+PIPELINE_SAMPLE_RATE = 16000
+CHUNK_SAMPLES = 512
+BYTES_PER_SAMPLE = 2
+CHUNK_SIZE_BYTES = CHUNK_SAMPLES * BYTES_PER_SAMPLE
+
+
+class AudioHandler(RealtimeBaseHandler):
+    """Owns inbound audio decoding/chunking and outbound audio encoding."""
+
+    def _start_input_item(self, conn_id: str, *, preserve_active_response: bool = False) -> str:
+        response = self._service.response
+        st = self._state(conn_id)
+        if not preserve_active_response:
+            item_id = response._start_item(conn_id)
+        else:
+            response_item_id = st.current_item_id
+            response_content_index = st.content_index
+            item_id = response._start_item(conn_id)
+            st.current_item_id = response_item_id
+            st.content_index = response_content_index
+        st.input_content_index = 0
+        return item_id
+
+    def handle_audio_append(self, conn_id: str, event: InputAudioBufferAppendEvent) -> list[bytes]:
+        """Decode base64 audio, resample to pipeline rate, and split into 512-sample PCM16 chunks for the VAD."""
+        try:
+            pcm_bytes = base64.b64decode(event.audio)
+        except Exception as e:
+            logger.error(f"Base64 decode error: {e}")
+            return []
+
+        st = self._state(conn_id)
+
+        audio_cfg = st.runtime_config.session.audio
+        if audio_cfg is not None and audio_cfg.input is not None:
+            client_in_rate = getattr(audio_cfg.input.format, "rate", None) or PIPELINE_SAMPLE_RATE
+        else:
+            client_in_rate = PIPELINE_SAMPLE_RATE
+        pcm_bytes = resample(pcm_bytes, client_in_rate, PIPELINE_SAMPLE_RATE)
+
+        pcm_bytes = st.audio_remainder + pcm_bytes
+
+        chunks = []
+        for i in range(0, len(pcm_bytes), CHUNK_SIZE_BYTES):
+            chunk = pcm_bytes[i : i + CHUNK_SIZE_BYTES]
+            if len(chunk) == CHUNK_SIZE_BYTES:
+                chunks.append(chunk)
+            else:
+                st.audio_remainder = chunk
+                break
+        else:
+            st.audio_remainder = b""
+
+        if chunks:
+            st.audio_buffer_has_data = True
+        return chunks
+
+    def handle_audio_commit(self, conn_id: str) -> RealtimeErrorEvent | None:
+        """Commit the audio buffer. Returns an error if no audio was appended."""
+        st = self._state(conn_id)
+        if not st.audio_buffer_has_data:
+            return self.make_error(
+                message="Input audio buffer is empty, nothing to commit.",
+                _type="input_audio_buffer_commit_empty",
+            )
+        st.audio_buffer_has_data = False
+        logger.debug("Audio buffer committed")
+        return None
+
+    # ── Pipeline event handlers ────────────────────
+
+    def on_speech_started(self, conn_id: str, event: SpeechStartedEvent) -> list[ServerEvent]:
+        """Handle VAD speech_started: cancel active response if interrupts enabled, start new input item."""
+        response = self._service.response
+        events: list[ServerEvent] = []
+        st = self._state(conn_id)
+        if st.in_response and event.interrupt_response and st.runtime_config.interrupt_response_enabled:
+            events.extend(response.finish_response(conn_id, status="cancelled", reason="turn_detected"))
+        is_reopen = bool(event.reopened and event.turn_id is not None and event.turn_id == st.speculative_turn_id)
+        preserve_active_response = st.in_response
+        if is_reopen:
+            input_item_id = st.speculative_input_item_id
+            if input_item_id is None:
+                input_item_id = self._start_input_item(
+                    conn_id,
+                    preserve_active_response=preserve_active_response,
+                )
+                st.speculative_input_item_id = input_item_id
+            elif not preserve_active_response:
+                st.current_item_id = input_item_id
+                st.content_index = 0
+            st.input_audio_duration_s = 0.0
+            st.input_content_index = 0
+        else:
+            input_item_id = self._start_input_item(
+                conn_id,
+                preserve_active_response=preserve_active_response,
+            )
+            st.speculative_input_item_id = input_item_id
+            st.response_usage.turns += 1
+        st.speculative_turn_id = event.turn_id
+        st.speculative_turn_revision = event.turn_revision
+        st.last_item_id = input_item_id
+        events.append(
+            InputAudioBufferSpeechStartedEvent(
+                type="input_audio_buffer.speech_started",
+                event_id=self._next_event_id(),
+                audio_start_ms=event.audio_start_ms,
+                item_id=input_item_id,
+            )
+        )
+        return events
+
+    def on_speech_stopped(self, conn_id: str, event: SpeechStoppedEvent) -> list[ServerEvent]:
+        """Handle VAD speech_stopped: record duration and emit stopped event."""
+        if event.duration_s:
+            self._state(conn_id).input_audio_duration_s = event.duration_s
+        return [
+            InputAudioBufferSpeechStoppedEvent(
+                type="input_audio_buffer.speech_stopped",
+                event_id=self._next_event_id(),
+                audio_end_ms=event.audio_end_ms,
+                item_id=self._input_item_id(conn_id),
+            )
+        ]
+
+    # ── Outbound audio encoding ──────────────────
+
+    def encode_audio_chunk(self, conn_id: str, audio: bytes) -> list[ServerEvent]:
+        """Encode a raw PCM audio chunk, emitting ResponseCreated on the first chunk.
+
+        When ``handle_response_create`` already allocated the response,
+        ``current_response_id`` is set and no duplicate event is emitted.
+        For the implicit-response path (VAD -> STT -> LLM -> TTS, no
+        ``response.create``), ``current_response_id`` is still ``None``
+        and the event is emitted here on the first audio chunk.
+        """
+        response = self._service.response
+        st = self._state(conn_id)
+
+        events: list[ServerEvent] = []
+        need_created = st.current_response_id is None
+        resp_id, item_id = response._ensure_response(conn_id)
+        if need_created:
+            events.append(
+                ResponseCreatedEvent(
+                    type="response.created",
+                    event_id=self._next_event_id(),
+                    response=response._build_response(conn_id, "in_progress"),
+                )
+            )
+        rp = st.current_response_params
+        client_out_rate = None
+        if rp and rp.audio and rp.audio.output and rp.audio.output.format:
+            client_out_rate = getattr(rp.audio.output.format, "rate", None)
+        if client_out_rate is None:
+            audio_cfg = st.runtime_config.session.audio
+            if audio_cfg is not None and audio_cfg.output is not None:
+                client_out_rate = getattr(audio_cfg.output.format, "rate", None) or PIPELINE_SAMPLE_RATE
+            else:
+                client_out_rate = PIPELINE_SAMPLE_RATE
+        audio = resample(audio, PIPELINE_SAMPLE_RATE, client_out_rate)
+        b64 = base64.b64encode(audio).decode("ascii")
+        events.append(
+            ResponseAudioDeltaEvent(
+                type="response.output_audio.delta",
+                event_id=self._next_event_id(),
+                content_index=response._next_content_index(conn_id),
+                delta=b64,
+                item_id=item_id,
+                output_index=0,
+                response_id=resp_id,
+            )
+        )
+        return events

src/speech_to_speech/api/openai_realtime/handlers/base.py

@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+import logging
+from queue import Queue
+from threading import Event as ThreadingEvent
+from typing import TYPE_CHECKING
+
+from openai.types.realtime import RealtimeErrorEvent
+
+from speech_to_speech.utils.utils import _generate_id
+
+if TYPE_CHECKING:
+    from speech_to_speech.api.openai_realtime.service import ConnState, RealtimeService
+
+logger = logging.getLogger(__name__)
+
+
+class RealtimeBaseHandler:
+    """Shared base for domain handlers.
+
+    Provides conn_id-keyed accessors for per-connection state, config,
+    and queues.  Each connection owns its own ``RuntimeConfig`` via
+    ``ConnState``.
+    """
+
+    def __init__(self, service: RealtimeService) -> None:
+        self._service = service
+
+    # ── conn_id-keyed accessors ──
+
+    def _queue(self, conn_id: str) -> Queue | None:
+        return self._service.text_prompt_queue
+
+    def _should_listen(self, conn_id: str) -> ThreadingEvent | None:
+        return self._service.should_listen
+
+    # ── shared helpers ────────────────────────────────
+
+    def _state(self, conn_id: str) -> ConnState:
+        return self._service._state(conn_id)
+
+    def _input_item_id(self, conn_id: str) -> str:
+        st = self._state(conn_id)
+        return st.speculative_input_item_id or self._service.response._current_item_id(conn_id)
+
+    def _next_input_content_index(self, conn_id: str) -> int:
+        st = self._state(conn_id)
+        idx = st.input_content_index
+        st.input_content_index += 1
+        return idx
+
+    @staticmethod
+    def _next_event_id() -> str:
+        return _generate_id("event")
+
+    def make_error(self, message: str, _type: str) -> RealtimeErrorEvent:
+        return self._service.make_error(message, _type)

src/speech_to_speech/api/openai_realtime/handlers/conversation.py

@@ -0,0 +1,128 @@
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+from openai.types.realtime import (
+    ConversationItem,
+    ConversationItemCreatedEvent,
+    ConversationItemCreateEvent,
+    ConversationItemInputAudioTranscriptionCompletedEvent,
+    ConversationItemInputAudioTranscriptionDeltaEvent,
+)
+from openai.types.realtime.conversation_item_input_audio_transcription_completed_event import (
+    UsageTranscriptTextUsageDuration,
+)
+
+from speech_to_speech.api.openai_realtime.handlers.base import RealtimeBaseHandler
+from speech_to_speech.LLM.chat import ChatItemError, add_supported_item
+from speech_to_speech.pipeline.events import PartialTranscriptionEvent, TranscriptionCompletedEvent
+
+if TYPE_CHECKING:
+    from speech_to_speech.api.openai_realtime.service import ServerEvent
+
+logger = logging.getLogger(__name__)
+
+
+class ConversationHandler(RealtimeBaseHandler):
+    """Owns conversation item injection and pipeline-to-protocol translation."""
+
+    def handle_conversation_item_create(
+        self,
+        conn_id: str,
+        event: ConversationItemCreateEvent,
+    ) -> list[ServerEvent]:
+        """Inject a text message or function-call output into the LLM context.
+
+        Items are added to the LLM chat context but do NOT trigger response
+        generation on their own.  A subsequent ``response.create`` event is
+        required to trigger the model.
+
+        While a response is generating, the item is *deferred*: applying it now
+        would race the LLM handler's end-of-turn chat write-back, which runs on
+        the pipeline thread (e.g. a ``function_call_output`` arriving before its
+        ``function_call`` is recorded, or an image stripped before the next
+        turn reads it). Deferred items are flushed, in order, once the response
+        completes — see :meth:`flush_deferred_items`.
+        """
+        st = self._state(conn_id)
+        if st.in_response:
+            st.deferred_items.append(event.item)
+            logger.debug("Deferred conversation item until the active response completes")
+            return []
+        return self._apply_item(conn_id, event.item)
+
+    def _apply_item(self, conn_id: str, item: ConversationItem) -> list[ServerEvent]:
+        """Add one item to the chat and build its ``conversation.item.created``."""
+        try:
+            self._append_item(conn_id, item)
+        except ChatItemError as exc:
+            return [self.make_error(str(exc), "invalid_conversation_item")]
+
+        if not item:
+            return []
+        st = self._state(conn_id)
+        event = ConversationItemCreatedEvent(
+            type="conversation.item.created",
+            event_id=self._next_event_id(),
+            previous_item_id=st.last_item_id,
+            item=item,
+        )
+        st.last_item_id = item.id
+        return [event]
+
+    def flush_deferred_items(self, conn_id: str) -> list[ServerEvent]:
+        """Apply items buffered during a response, in arrival order.
+
+        Called at response completion (after the generation's own write-back),
+        so a ``function_call_output`` pairs with its now-recorded ``function_call``
+        and an image survives the just-finished response's ``strip_images``.
+        """
+        st = self._state(conn_id)
+        if not st.deferred_items:
+            return []
+        items = st.deferred_items
+        st.deferred_items = []
+        events: list[ServerEvent] = []
+        for item in items:
+            events.extend(self._apply_item(conn_id, item))
+        return events
+
+    def _append_item(self, conn_id: str, item: ConversationItem) -> None:
+        """Narrow ``ConversationItem`` to ``SupportedItem`` and delegate to ``Chat.add_item``.
+
+        Raises :class:`ChatItemError` on validation failure or unsupported type.
+        """
+        add_supported_item(self._state(conn_id).runtime_config.chat, item)
+
+    # ── Pipeline event handlers ────────────────────
+
+    def on_partial_transcription(self, conn_id: str, event: PartialTranscriptionEvent) -> list[ServerEvent]:
+        """Handle partial_transcription: emit transcription delta event."""
+        return [
+            ConversationItemInputAudioTranscriptionDeltaEvent(
+                type="conversation.item.input_audio_transcription.delta",
+                event_id=self._next_event_id(),
+                content_index=self._next_input_content_index(conn_id),
+                item_id=self._input_item_id(conn_id),
+                delta=event.delta,
+            )
+        ]
+
+    def on_transcription_completed(self, conn_id: str, event: TranscriptionCompletedEvent) -> list[ServerEvent]:
+        """Handle transcription_completed: accumulate duration and emit completed event."""
+        st = self._state(conn_id)
+        st.response_usage.audio_duration_s += st.input_audio_duration_s
+        return [
+            ConversationItemInputAudioTranscriptionCompletedEvent(
+                type="conversation.item.input_audio_transcription.completed",
+                event_id=self._next_event_id(),
+                content_index=0,
+                item_id=self._input_item_id(conn_id),
+                transcript=event.transcript,
+                usage=UsageTranscriptTextUsageDuration(
+                    seconds=st.input_audio_duration_s,
+                    type="duration",
+                ),
+            )
+        ]

src/speech_to_speech/api/openai_realtime/handlers/response.py

@@ -0,0 +1,348 @@
+from __future__ import annotations
+
+import logging
+from time import perf_counter
+from typing import TYPE_CHECKING, Optional
+
+from openai.types.realtime import (
+    RealtimeResponse,
+    ResponseAudioDoneEvent,
+    ResponseAudioTranscriptDoneEvent,
+    ResponseCreatedEvent,
+    ResponseCreateEvent,
+    ResponseDoneEvent,
+    ResponseFunctionCallArgumentsDoneEvent,
+    ResponseTextDeltaEvent,
+    ResponseTextDoneEvent,
+)
+from openai.types.realtime.realtime_response import Audio, AudioOutput
+from openai.types.realtime.realtime_response_status import RealtimeResponseStatus
+from openai.types.realtime.realtime_response_usage import RealtimeResponseUsage
+
+from speech_to_speech.api.openai_realtime.handlers.base import RealtimeBaseHandler
+from speech_to_speech.LLM.chat import ChatItemError
+from speech_to_speech.pipeline.events import AssistantTextEvent
+from speech_to_speech.pipeline.messages import GenerateResponseRequest
+from speech_to_speech.utils.utils import _generate_id, is_out_of_band, response_wants_audio
+
+if TYPE_CHECKING:
+    from speech_to_speech.api.openai_realtime.service import ServerEvent, _ResponseStatus, _StatusReason
+
+logger = logging.getLogger(__name__)
+
+
+class ResponseHandler(RealtimeBaseHandler):
+    """Owns the response lifecycle: create, cancel, finish, and ID management."""
+
+    # ── ID / state helpers ────────────────────────
+
+    def _ensure_response(self, conn_id: str) -> tuple[str, str]:
+        """Ensure a response and output item exist, creating them if needed."""
+        st = self._state(conn_id)
+        if st.current_response_id is None:
+            st.current_response_id = _generate_id("resp")
+            self._start_item(conn_id)
+            st.in_response = True
+        st.response_pending = False
+        return st.current_response_id, self._current_item_id(conn_id)
+
+    def _end_response(self, conn_id: str, status: _ResponseStatus = "completed") -> None:
+        st = self._state(conn_id)
+        if status == "cancelled":
+            st.response_usage.responses_cancelled += 1
+        else:
+            st.response_usage.responses_completed += 1
+        self._service.total_usage += st.response_usage
+        logger.info(
+            "Response done (status=%s) — this response: input_tokens=%d, output_tokens=%d, audio=%.2fs"
+            " | cumulative: input_tokens=%d, output_tokens=%d, audio=%.2fs",
+            status,
+            st.response_usage.input_tokens,
+            st.response_usage.output_tokens,
+            st.response_usage.audio_duration_s,
+            self._service.total_usage.input_tokens,
+            self._service.total_usage.output_tokens,
+            self._service.total_usage.audio_duration_s,
+        )
+        st.response_usage.reset()
+        st.current_response_id = None
+        st.current_item_id = None
+        st.content_index = 0
+        st.in_response = False
+        st.response_pending = False
+        st.current_response_params = None
+        st.pending_output_text_parts = []
+        st.response_audio_wait_started_at_s = None
+        st.delay_filler_sent = False
+
+    def _start_item(self, conn_id: str) -> str:
+        """Generate a new item ID, reset content index, and store it."""
+        st = self._state(conn_id)
+        item_id = _generate_id("item")
+        st.current_item_id = item_id
+        st.content_index = 0
+        st.input_audio_duration_s = 0.0
+        return item_id
+
+    def _current_item_id(self, conn_id: str) -> str:
+        return self._state(conn_id).current_item_id or self._start_item(conn_id)
+
+    def _next_content_index(self, conn_id: str) -> int:
+        """Return the current content index and advance it."""
+        st = self._state(conn_id)
+        idx = st.content_index
+        st.content_index += 1
+        return idx
+
+    def _build_response(
+        self,
+        conn_id: str,
+        status: _ResponseStatus,
+        reason: _StatusReason | None = None,
+    ) -> RealtimeResponse:
+        """Build a fully-populated RealtimeResponse from the current connection state."""
+        st = self._state(conn_id)
+        status_details = None
+        if reason or status in ("completed", "cancelled", "incomplete", "failed"):
+            status_details = RealtimeResponseStatus(type=status, reason=reason)  # type: ignore[arg-type]
+
+        rp = st.current_response_params
+        metadata = rp.metadata if rp and rp.metadata else None
+
+        voice: Optional[str] = None
+        if rp and rp.audio and rp.audio.output and rp.audio.output.voice:
+            voice = str(rp.audio.output.voice)
+        if not voice:
+            audio_cfg = st.runtime_config.session.audio
+            audio_output = audio_cfg.output if audio_cfg is not None else None
+            voice = str(audio_output.voice) if audio_output is not None and audio_output.voice else None
+
+        # Out-of-band responses are not threaded into any conversation: report a null id.
+        conversation_id = None if is_out_of_band(rp) else st.conversation_id
+
+        return RealtimeResponse(
+            id=st.current_response_id,
+            object="realtime.response",
+            status=status,
+            status_details=status_details,
+            audio=Audio(output=AudioOutput(voice=str(voice) if voice else None)),  # type: ignore[arg-type]
+            conversation_id=conversation_id,
+            metadata=metadata,
+            usage=RealtimeResponseUsage(
+                input_tokens=st.response_usage.input_tokens,
+                output_tokens=st.response_usage.output_tokens,
+                total_tokens=st.response_usage.input_tokens + st.response_usage.output_tokens,
+            ),
+        )
+
+    # ── Public handlers ───────────────────────────
+
+    def handle_response_create(self, conn_id: str, event: ResponseCreateEvent) -> ServerEvent | None:
+        """Trigger a response.
+
+        Returns a ``ResponseCreatedEvent`` on success, a ``RealtimeErrorEvent``
+        on failure, or ``None`` if there is no text_prompt_queue.
+        """
+        st = self._state(conn_id)
+        if event.response:
+            if event.response.tool_choice and not isinstance(event.response.tool_choice, str):
+                return self.make_error(
+                    message="Only string tool_choice values are supported for now (auto, required, none).",
+                    _type="tool_choice_not_supported",
+                )
+        if st.in_response:
+            return self.make_error(
+                message="Cannot create response while another response is in progress.",
+                _type="conversation_already_has_active_response",
+            )
+
+        out_of_band = is_out_of_band(event.response)
+
+        # In-band: response.input items are added to the default conversation here so
+        # they appear in history. Out-of-band: leave the default conversation untouched —
+        # the input rides along on the request and seeds a throwaway chat in the LM.
+        if not out_of_band and event.response and event.response.input:
+            for input_item in event.response.input:
+                try:
+                    self._service.conversation._append_item(conn_id, input_item)
+                except ChatItemError as exc:
+                    return self.make_error(message=str(exc), _type="invalid_input_item")
+
+        st.in_response = True
+        st.response_pending = False
+
+        st.current_response_params = event.response
+        st.current_response_id = _generate_id("resp")
+        self._start_item(conn_id)
+        if response_wants_audio(event.response):
+            st.response_audio_wait_started_at_s = perf_counter()
+            st.delay_filler_sent = False
+        else:
+            st.response_audio_wait_started_at_s = None
+            st.delay_filler_sent = False
+
+        cfg = st.runtime_config
+        queue = self._queue(conn_id)
+        if queue:
+            # Out-of-band responses carry no turn identity: a null turn_id makes every
+            # speculative-turn staleness gate treat them as always-latest, so a new user
+            # turn mid-generation can never silently drop their output.
+            queue.put(
+                GenerateResponseRequest(
+                    runtime_config=cfg,
+                    response=event.response,
+                    turn_id=None if out_of_band else st.speculative_user_turn_id,
+                    turn_revision=None if out_of_band else st.speculative_user_turn_revision,
+                    speech_stopped_at_s=None if out_of_band else st.speculative_user_speech_stopped_at_s,
+                )
+            )
+        logger.debug("response.create received, LLM generation triggered")
+        return ResponseCreatedEvent(
+            type="response.created",
+            event_id=self._next_event_id(),
+            response=self._build_response(conn_id, "in_progress"),
+        )
+
+    def handle_response_cancel(self, conn_id: str) -> list[ServerEvent]:
+        """Cancel the in-progress response and re-enable listening."""
+        events = self.finish_response(conn_id, status="cancelled", reason="client_cancelled")
+        should_listen = self._should_listen(conn_id)
+        if should_listen:
+            should_listen.set()
+        logger.info("Response cancelled, listening re-enabled")
+        return events
+
+    def finish_response(
+        self,
+        conn_id: str,
+        status: _ResponseStatus = "completed",
+        reason: _StatusReason | None = None,
+    ) -> list[ServerEvent]:
+        """Close the current response (audio/text done + response done).
+
+        Audio responses emit ``response.output_audio.done`` for any terminal
+        status. Text-only responses emit a single ``response.output_text.done``
+        carrying the full streamed text, but only on ``status="completed"`` —
+        a cancelled or failed text response sends no audio, so it just closes
+        with ``response.done``.
+        """
+        st = self._state(conn_id)
+        events: list[ServerEvent] = []
+        if st.in_response:
+            resp_id, item_id = self._ensure_response(conn_id)
+            if response_wants_audio(st.current_response_params):
+                events.append(
+                    ResponseAudioDoneEvent(
+                        type="response.output_audio.done",
+                        event_id=self._next_event_id(),
+                        content_index=0,
+                        item_id=item_id,
+                        output_index=0,
+                        response_id=resp_id,
+                    )
+                )
+            elif status == "completed" and st.pending_output_text_parts:
+                events.append(
+                    ResponseTextDoneEvent(
+                        type="response.output_text.done",
+                        event_id=self._next_event_id(),
+                        content_index=0,
+                        item_id=item_id,
+                        output_index=0,
+                        response_id=resp_id,
+                        text="".join(st.pending_output_text_parts),
+                    )
+                )
+            events.append(
+                ResponseDoneEvent(
+                    type="response.done",
+                    event_id=self._next_event_id(),
+                    response=self._build_response(conn_id, status, reason),
+                )
+            )
+            self._end_response(conn_id, status)
+        # Apply any client items that arrived mid-generation now that in_response
+        # is cleared and the generation's own write-back has landed. Done outside
+        # the in_response guard so a stray terminal call still drains the buffer.
+        events.extend(self._service.conversation.flush_deferred_items(conn_id))
+        return events
+
+    # ── Pipeline event handlers ───────────────────
+
+    def on_assistant_text(
+        self,
+        conn_id: str,
+        event: AssistantTextEvent,
+        *,
+        wait_for_pending_reopen: bool = True,
+    ) -> list[ServerEvent] | None:
+        """Handle assistant_text: emit transcript and/or tool-call events."""
+        if self._service.speculative_turns:
+            commit_result: bool | None
+            if wait_for_pending_reopen:
+                commit_result = self._service.speculative_turns.commit_if_latest_after_reopen_grace(
+                    event.turn_id,
+                    event.turn_revision,
+                )
+            else:
+                commit_result = self._service.speculative_turns.try_commit_if_latest_after_reopen_grace(
+                    event.turn_id,
+                    event.turn_revision,
+                )
+            if commit_result is None:
+                return None
+            if not commit_result:
+                logger.debug("Dropping stale assistant text for turn=%s rev=%s", event.turn_id, event.turn_revision)
+                return []
+        st = self._state(conn_id)
+        events: list[ServerEvent] = []
+        resp_id, item_id = self._ensure_response(conn_id)
+        st.last_item_id = item_id
+        output_idx = 0
+        if event.text:
+            if response_wants_audio(st.current_response_params):
+                events.append(
+                    ResponseAudioTranscriptDoneEvent(
+                        type="response.output_audio_transcript.done",
+                        event_id=self._next_event_id(),
+                        content_index=0,
+                        item_id=item_id,
+                        output_index=output_idx,
+                        response_id=resp_id,
+                        transcript=event.text,
+                    )
+                )
+            else:
+                # Stream the delta now; the matching response.output_text.done is
+                # emitted once at close in finish_response, carrying the per-chunk
+                # parts collected here, space-joined ("" when there were none).
+                st.pending_output_text_parts.append(event.text)
+                events.append(
+                    ResponseTextDeltaEvent(
+                        type="response.output_text.delta",
+                        event_id=self._next_event_id(),
+                        content_index=0,
+                        item_id=item_id,
+                        output_index=output_idx,
+                        response_id=resp_id,
+                        delta=event.text,
+                    )
+                )
+            output_idx += 1
+        if event.tools:
+            st.response_usage.tool_calls += len(event.tools)
+            for tool in event.tools:
+                events.append(
+                    ResponseFunctionCallArgumentsDoneEvent(
+                        type="response.function_call_arguments.done",
+                        event_id=self._next_event_id(),
+                        call_id=tool.call_id,
+                        name=tool.name,
+                        arguments=tool.arguments,
+                        item_id=item_id,
+                        output_index=output_idx,
+                        response_id=resp_id,
+                    )
+                )
+                output_idx += 1
+        return events

src/speech_to_speech/api/openai_realtime/handlers/session.py

@@ -0,0 +1,69 @@
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+from openai.types.realtime import (
+    RealtimeErrorEvent,
+    SessionCreatedEvent,
+    SessionUpdateEvent,
+)
+from openai.types.realtime.realtime_transcription_session_create_request import (
+    RealtimeTranscriptionSessionCreateRequest,
+)
+
+from speech_to_speech.api.openai_realtime.handlers.base import RealtimeBaseHandler
+
+logger = logging.getLogger(__name__)
+
+
+class SessionHandler(RealtimeBaseHandler):
+    """Owns session lifecycle: config updates and session.created events."""
+
+    def handle_session_update(self, conn_id: str, event: SessionUpdateEvent) -> Optional[RealtimeErrorEvent]:
+        """Apply session config changes.
+
+        Only ``RealtimeSessionCreateRequest`` sessions are accepted;
+        ``RealtimeTranscriptionSessionCreateRequest`` sessions not yet supported.
+        Incoming fields are deep-merged into the existing session so that
+        partial updates preserve previously-set values.
+        """
+        s = event.session
+	
+	# Forza il bypass della voce inviata dal browser
+        if hasattr(s, "voice") and s.voice is not None:
+            s.voice = None
+        elif isinstance(s, dict) and "voice" in s:
+            s["voice"] = None
+
+        if s is None:
+            return None
+
+        if isinstance(s, RealtimeTranscriptionSessionCreateRequest):
+            return self.make_error(
+                message="Only 'realtime' session type is supported; transcription sessions are not.",
+                _type="invalid_session_type",
+            )
+
+        model = getattr(s, "model", None)
+        if model is not None:
+            logger.info(f"Session model set to: {model}")
+
+        cfg = self._state(conn_id).runtime_config
+        current = cfg.session
+        if current is None:
+            cfg.session = s
+        else:
+            cfg.apply_session_update(s)
+        logger.info("Session configuration updated")
+        return None
+
+    def build_session_created(self, conn_id: str) -> SessionCreatedEvent:
+        """Build a SessionCreatedEvent populated with the current config."""
+        cfg = self._state(conn_id).runtime_config
+        session = cfg.session
+        return SessionCreatedEvent(
+            type="session.created",
+            event_id=self._next_event_id(),
+            session=session,
+        )

src/speech_to_speech/api/openai_realtime/pipeline_unit.py

@@ -0,0 +1,70 @@
+import asyncio
+from queue import Queue
+from threading import Event
+from typing import Any, Optional
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from speech_to_speech.api.openai_realtime.service import RealtimeService
+from speech_to_speech.pipeline.cancel_scope import CancelScope
+
+
+class SessionState(BaseModel):
+    """Per-websocket ephemeral state.
+
+    Created when the route handler claims a PipelineUnit on `ws.accept()`; dropped
+    when the websocket disconnects. Holding the websocket reference, the service
+    session id, and any send-loop scratch (pending_output_item) here ensures these
+    fields share one lifecycle — a stale value can't outlive its session.
+
+    `drained` is set by the send loop when SESSION_END travels through the handler
+    chain back to the output queue; the route handler awaits it before clearing
+    `PipelineUnit.session`, so a new client cannot claim the unit until in-flight
+    work from this session has fully reset.
+    """
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    websocket: Any
+    session_id: str = ""
+    pending_output_item: Any = None
+    drained: asyncio.Event = Field(default_factory=asyncio.Event)
+    # Wall-clock time when the ws disconnected (route handler released its claim).
+    # `None` while the client is still active. Used by /v1/pool to surface stuck
+    # units (handlers haven't finished propagating SESSION_END).
+    released_at: Optional[float] = None
+    # Wall-clock time when the drain wait gave up and quarantined the unit
+    # (SESSION_END_QUARANTINE_TIMEOUT_S elapsed). The unit stays unclaimable —
+    # its handlers may still emit this session's output — until SESSION_END
+    # actually drains. Reported as "stuck" by /v1/pool.
+    quarantined_at: Optional[float] = None
+
+
+class PipelineUnit(BaseModel):
+    """One isolated realtime pipeline.
+
+    Each unit owns its queues, events, RealtimeService, and the chain of handler
+    instances (VAD, STT, transcription notifier, LM, LM output processor, TTS).
+    Lives inside the pool managed by RealtimeServer; the websocket route handler
+    claims a free unit (`session is None`) on `accept` and releases it on disconnect
+    by setting `session` back to None.
+    """
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    index: int
+    service: RealtimeService
+    cancel_scope: CancelScope
+    should_listen: Event
+    response_playing: Event
+    input_queue: Queue
+    output_queue: Queue
+    text_output_queue: Queue
+    text_prompt_queue: Queue
+    handlers: list[Any]
+    delay_filler_threshold_s: float = 0.0
+    delay_filler_short_clips: list[bytes] = Field(default_factory=list)
+    delay_filler_full_clips: list[bytes] = Field(default_factory=list)
+    http_tts_reserved: bool = False
+
+    session: Optional[SessionState] = None

src/speech_to_speech/api/openai_realtime/runtime_config.py

@@ -0,0 +1,81 @@
+from openai.types.realtime import RealtimeSessionCreateRequest
+from openai.types.realtime.realtime_audio_config import RealtimeAudioConfig
+from openai.types.realtime.realtime_audio_config_input import RealtimeAudioConfigInput
+from openai.types.realtime.realtime_audio_config_output import RealtimeAudioConfigOutput
+from pydantic import BaseModel, ConfigDict, Field, field_validator
+
+from speech_to_speech.LLM.chat import Chat
+
+
+def _apply_update(current: BaseModel, update: BaseModel) -> None:
+    """Apply explicitly-set fields from *update* onto *current* in-place,
+    recursing into nested BaseModel children so partial nested updates
+    don't overwrite unset fields.
+
+    Only fields present in update.model_fields_set (i.e. actually
+    sent by the client) are considered.
+    """
+    for field_name in update.model_fields_set:
+        new_val = getattr(update, field_name)
+        old_val = getattr(current, field_name, None)
+        if isinstance(new_val, BaseModel) and isinstance(old_val, BaseModel):
+            _apply_update(old_val, new_val)
+        else:
+            setattr(current, field_name, new_val)
+
+
+class RuntimeConfig(BaseModel):
+    """
+    Shared mutable configuration written by the RealtimeService on
+    session.update and read by pipeline handlers (VAD, LLM, TTS) during
+    processing.  Python's GIL makes simple attribute reads/writes atomic,
+    so no explicit locking is needed for primitive values.
+
+    The canonical state lives in 'session' (a full
+    'RealtimeSessionCreateRequest').
+    """
+
+    model_config = ConfigDict(validate_assignment=True, arbitrary_types_allowed=True)
+
+    chat: Chat = Field(default_factory=lambda: Chat(10))
+    session: RealtimeSessionCreateRequest = Field(
+        default_factory=lambda: RealtimeSessionCreateRequest(type="realtime"),
+        validate_default=True,
+    )
+
+    @field_validator("session", mode="after")
+    @classmethod
+    def _ensure_audio_structure(cls, v: RealtimeSessionCreateRequest) -> RealtimeSessionCreateRequest:
+        """Guarantee 'audio.input' and 'audio.output' are never None."""
+        if v.audio is None:
+            v.audio = RealtimeAudioConfig()
+        if v.audio.input is None:
+            v.audio.input = RealtimeAudioConfigInput()
+        if v.audio.output is None:
+            v.audio.output = RealtimeAudioConfigOutput()
+        return v
+
+    @property
+    def interrupt_response_enabled(self) -> bool:
+        """Whether barge-in should cancel an active response.
+
+        Reads 'turn_detection.interrupt_response' from the session config,
+        handling both Pydantic models ('ServerVad') and plain dicts.
+        Defaults to 'True' (OpenAI API default).
+        """
+        assert self.session.audio is not None and self.session.audio.input is not None
+        td = self.session.audio.input.turn_detection
+        if td is None:
+            return True
+        if hasattr(td, "interrupt_response"):
+            val = td.interrupt_response
+        elif isinstance(td, dict):
+            val = td.get("interrupt_response", True)
+        else:
+            return True
+        return val if val is not None else True
+
+    def apply_session_update(self, update: RealtimeSessionCreateRequest) -> None:
+        """Merge non-None, explicitly-set fields from 'update' into the
+        current 'session', preserving any fields not present in the update."""
+        _apply_update(self.session, update)

src/speech_to_speech/api/openai_realtime/server.py

@@ -0,0 +1,61 @@
+import logging
+import threading
+from threading import Event
+
+import uvicorn
+
+from speech_to_speech.api.openai_realtime.pipeline_unit import PipelineUnit
+from speech_to_speech.api.openai_realtime.websocket_router import create_app
+
+logger = logging.getLogger(__name__)
+
+
+class RealtimeServer:
+    """
+    Pipeline handler for the OpenAI Realtime API mode.
+
+    Owns a pool of isolated PipelineUnits and a single uvicorn server.
+    The websocket route claims the next free unit on each accept and releases
+    it on disconnect; once all units are in use, further connections are rejected.
+    """
+
+    def __init__(
+        self,
+        stop_event: Event,
+        pool: list[PipelineUnit],
+        host: str = "0.0.0.0",
+        port: int = 8765,
+    ) -> None:
+        if not pool:
+            raise ValueError("RealtimeServer requires at least one PipelineUnit in the pool")
+        self.stop_event = stop_event
+        self.pool = pool
+        self.host = host
+        self.port = port
+
+    def run(self) -> None:
+        """Start the FastAPI/uvicorn server (called from a ThreadManager thread)."""
+        app = create_app(pool=self.pool, stop_event=self.stop_event)
+
+        logger.info(
+            f"OpenAI Realtime API starting on ws://{self.host}:{self.port}/v1/realtime (pool size {len(self.pool)})"
+        )
+
+        config = uvicorn.Config(
+            app,
+            host=self.host,
+            port=self.port,
+            log_level="info",
+        )
+        server = uvicorn.Server(config)
+
+        server.install_signal_handlers = lambda: None  # type: ignore[attr-defined]
+
+        def _watch_stop() -> None:
+            self.stop_event.wait()
+            server.should_exit = True
+
+        watcher = threading.Thread(target=_watch_stop, daemon=True)
+        watcher.start()
+
+        server.run()

src/speech_to_speech/api/openai_realtime/service.py

@@ -0,0 +1,568 @@
+import logging
+from collections.abc import Mapping
+from queue import Queue
+from threading import Event as ThreadingEvent
+from time import perf_counter
+from typing import Any, Callable, Literal, Optional, TypeVar, Union
+
+from openai.types.realtime import (
+    ConversationItem,
+    ConversationItemCreatedEvent,
+    ConversationItemCreateEvent,
+    ConversationItemInputAudioTranscriptionCompletedEvent,
+    ConversationItemInputAudioTranscriptionDeltaEvent,
+    InputAudioBufferAppendEvent,
+    InputAudioBufferSpeechStartedEvent,
+    InputAudioBufferSpeechStoppedEvent,
+    RealtimeError,
+    RealtimeErrorEvent,
+    ResponseAudioDeltaEvent,
+    ResponseAudioDoneEvent,
+    ResponseAudioTranscriptDoneEvent,
+    ResponseCancelEvent,
+    ResponseCreatedEvent,
+    ResponseCreateEvent,
+    ResponseDoneEvent,
+    ResponseFunctionCallArgumentsDoneEvent,
+    ResponseTextDeltaEvent,
+    ResponseTextDoneEvent,
+    SessionCreatedEvent,
+    SessionUpdateEvent,
+)
+from openai.types.realtime.realtime_response_create_params import RealtimeResponseCreateParams
+from pydantic import BaseModel, ConfigDict, Field, ValidationError
+
+from speech_to_speech.api.openai_realtime.handlers import (
+    AudioHandler,
+    ConversationHandler,
+    ResponseHandler,
+    SessionHandler,
+)
+from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig
+from speech_to_speech.LLM.chat import Chat, make_user_message
+from speech_to_speech.pipeline.events import (
+    AssistantTextEvent,
+    PartialTranscriptionEvent,
+    PipelineEvent,
+    ResponseFailedEvent,
+    SpeechStartedEvent,
+    SpeechStoppedEvent,
+    TokenUsageEvent,
+    TranscriptionCompletedEvent,
+)
+from speech_to_speech.pipeline.messages import GenerateResponseRequest
+from speech_to_speech.pipeline.queue_types import TextPromptItem
+from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
+from speech_to_speech.utils.utils import _generate_id
+
+logger = logging.getLogger(__name__)
+
+PIPELINE_SAMPLE_RATE = 16000
+CHUNK_SAMPLES = 512
+BYTES_PER_SAMPLE = 2
+CHUNK_SIZE_BYTES = CHUNK_SAMPLES * BYTES_PER_SAMPLE
+
+_ResponseStatus = Literal["completed", "cancelled", "failed", "incomplete", "in_progress"]
+_StatusReason = Literal["turn_detected", "client_cancelled", "max_output_tokens", "content_filter"]
+
+_EVENT_TYPE_TO_MODEL: dict[str, type[BaseModel]] = {
+    "input_audio_buffer.append": InputAudioBufferAppendEvent,
+    "session.update": SessionUpdateEvent,
+    "conversation.item.create": ConversationItemCreateEvent,
+    "response.create": ResponseCreateEvent,
+    "response.cancel": ResponseCancelEvent,
+}
+
+ClientEvent = Union[
+    InputAudioBufferAppendEvent,
+    SessionUpdateEvent,
+    ConversationItemCreateEvent,
+    ResponseCreateEvent,
+    ResponseCancelEvent,
+]
+
+ServerEvent = Union[
+    SessionCreatedEvent,
+    RealtimeErrorEvent,
+    InputAudioBufferSpeechStartedEvent,
+    InputAudioBufferSpeechStoppedEvent,
+    ConversationItemCreatedEvent,
+    ConversationItemInputAudioTranscriptionDeltaEvent,
+    ConversationItemInputAudioTranscriptionCompletedEvent,
+    ResponseCreatedEvent,
+    ResponseDoneEvent,
+    ResponseAudioDeltaEvent,
+    ResponseAudioDoneEvent,
+    ResponseAudioTranscriptDoneEvent,
+    ResponseFunctionCallArgumentsDoneEvent,
+    ResponseTextDeltaEvent,
+    ResponseTextDoneEvent,
+]
+
+RealtimeEvent = Union[ClientEvent, ServerEvent]
+
+
+_UsageMetricsT = TypeVar("_UsageMetricsT", bound="UsageMetrics")
+
+
+class UsageMetrics(BaseModel):
+    """Per-response usage counters.
+
+    Supports ``+=`` for rolling per-response metrics into a global total
+    and ``reset()`` for clearing per-response state after rollup.
+    """
+
+    input_tokens: int = 0
+    output_tokens: int = 0
+    audio_duration_s: float = 0.0
+    responses_completed: int = 0
+    responses_cancelled: int = 0
+    tool_calls: int = 0
+    turns: int = 0
+
+    def __iadd__(self: _UsageMetricsT, other: "UsageMetrics") -> _UsageMetricsT:
+        for field in UsageMetrics.model_fields:
+            setattr(self, field, getattr(self, field) + getattr(other, field))
+        return self
+
+    def reset(self) -> None:
+        for field, info in UsageMetrics.model_fields.items():
+            setattr(self, field, info.default)
+
+
+class GlobalUsageMetrics(UsageMetrics):
+    """Server-wide metrics that extend per-response counters with
+    connection and error tracking."""
+
+    connections: int = 0
+    # connection duration in seconds.
+    # latency tts, llm, vad, stt (mean, max, p90)
+    errors_by_type: dict[str, int] = Field(default_factory=dict)
+
+    def record_error(self, error_type: str) -> None:
+        self.errors_by_type[error_type] = self.errors_by_type.get(error_type, 0) + 1
+
+    @property
+    def total_errors(self) -> int:
+        return sum(self.errors_by_type.values())
+
+
+class ConnState(BaseModel):
+    """Per-connection mutable state, including all protocol-level IDs."""
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    session_id: str = Field(default_factory=lambda: _generate_id("session"))
+    conversation_id: str = Field(default_factory=lambda: _generate_id("conv"))
+    runtime_config: RuntimeConfig = Field(default_factory=RuntimeConfig)
+    in_response: bool = False
+    response_pending: bool = False
+    audio_buffer_has_data: bool = False
+    audio_remainder: bytes = b""
+    current_response_id: Optional[str] = None
+    current_item_id: Optional[str] = None
+    content_index: int = 0
+    input_content_index: int = 0
+    input_audio_duration_s: float = 0.0
+    last_item_id: Optional[str] = None
+    current_response_params: RealtimeResponseCreateParams | None = None
+    pending_output_text_parts: list[str] = Field(default_factory=list)
+    response_usage: UsageMetrics = Field(default_factory=UsageMetrics)
+    response_audio_wait_started_at_s: float | None = None
+    delay_filler_sent: bool = False
+    delay_filler_cursor: int = 0
+    delay_filler_last_kind: str | None = None
+    delay_filler_last_index: int | None = None
+    speculative_turn_id: Optional[str] = None
+    speculative_turn_revision: Optional[int] = None
+    speculative_user_turn_id: Optional[str] = None
+    speculative_user_turn_revision: Optional[int] = None
+    speculative_user_speech_stopped_at_s: Optional[float] = None
+    speculative_user_item_id: Optional[str] = None
+    speculative_input_item_id: Optional[str] = None
+    speculative_audio_duration_s: float = 0.0
+    # Client conversation.item.create items that arrived while a response was
+    # generating. Applying them mid-generation races the LLM handler's chat
+    # write-back (cross-thread), so they are buffered here and flushed in order
+    # once the response completes. See ConversationHandler.flush_deferred_items.
+    deferred_items: list[ConversationItem] = Field(default_factory=list)
+
+
+class RealtimeService:
+    """Translates between OpenAI Realtime protocol events and internal pipeline messages.
+
+    One instance is shared across all WebSocket connections.  Per-connection
+    state (response lifecycle, audio buffer) is tracked internally by
+    connection id.
+    """
+
+    def __init__(
+        self,
+        text_prompt_queue: Queue[TextPromptItem] | None = None,
+        should_listen: ThreadingEvent | None = None,
+        chat_size: int = 10,
+        speculative_turns: SpeculativeTurnTracker | None = None,
+    ) -> None:
+        self.text_prompt_queue = text_prompt_queue
+        self.should_listen = should_listen
+        self._chat_size = chat_size
+        self.speculative_turns = speculative_turns
+        self._conns: dict[str, ConnState] = {}
+        self.total_usage = GlobalUsageMetrics()
+
+        self.audio = AudioHandler(self)
+        self.session = SessionHandler(self)
+        self.response = ResponseHandler(self)
+        self.conversation = ConversationHandler(self)
+
+        self._pipeline_dispatch: dict[type[PipelineEvent], Callable[..., list[ServerEvent]]] = {
+            SpeechStartedEvent: self.audio.on_speech_started,
+            SpeechStoppedEvent: self.audio.on_speech_stopped,
+            TokenUsageEvent: self._on_token_usage,
+            PartialTranscriptionEvent: self.conversation.on_partial_transcription,
+            TranscriptionCompletedEvent: self._on_transcription_completed,
+            ResponseFailedEvent: self._on_response_failed,
+        }
+
+    # ── Connection lifecycle ─────────────────────
+
+    def register(self) -> str:
+        """Register a new connection and return its session_id."""
+        if self.speculative_turns:
+            self.speculative_turns.reset()
+        state = ConnState(runtime_config=RuntimeConfig(chat=Chat(self._chat_size)))
+        self._conns[state.session_id] = state
+        self.total_usage.connections += 1
+        return state.session_id
+
+    def unregister(self, conn_id: str) -> None:
+        st = self._conns.pop(conn_id, None)
+        if st is not None:
+            # Suppress any in-flight compaction splice so a daemon worker can't
+            # mutate a Chat tied to a closed session, and don't make further
+            # billable LLM calls on its behalf once the splice is suppressed.
+            st.runtime_config.chat.close()
+            self.total_usage += st.response_usage
+            logger.info(
+                "Session %s unregistered — cumulative: input_tokens=%d, output_tokens=%d, audio=%.2fs",
+                conn_id,
+                self.total_usage.input_tokens,
+                self.total_usage.output_tokens,
+                self.total_usage.audio_duration_s,
+            )
+
+    def _state(self, conn_id: str) -> ConnState:
+        return self._conns[conn_id]
+
+    @property
+    def connection_ids(self) -> list[str]:
+        return list(self._conns)
+
+    # ── Client event parsing ─────────────────────
+
+    @staticmethod
+    def _next_event_id() -> str:
+        return _generate_id("event")
+
+    def parse_client_event(self, raw: Mapping[str, object]) -> Optional[ClientEvent]:
+        raw_type = raw.get("type")
+        event_type: Optional[str] = raw_type if isinstance(raw_type, str) else None
+        if event_type is None:
+            logger.warning("Client event missing 'type' field")
+            return None
+        model_cls = _EVENT_TYPE_TO_MODEL.get(event_type)
+        if model_cls is None:
+            logger.warning(f"Unknown client event type: {event_type}")
+            return None
+        try:
+            return model_cls.model_validate(raw)  # type: ignore[return-value]
+        except ValidationError as e:
+            # ── Retrocompatibilità session.update ─────────────────────────────────
+            # Il formato standard OpenAI Realtime GA annida instructions/tools/tool_choice
+            # DENTRO "session": { ... }. Per tollerare anche client che li mandano
+            # alla RADICE (es. bug prima della Fase 2 MCP), ricostruiamo qui il payload
+            # corretto e riproviamo la validazione. Funziona SOLO per session.update.
+            if event_type == "session.update":
+                try:
+                    raw_dict = dict(raw) if not isinstance(raw, dict) else raw
+                    session_inner = raw_dict.get("session")
+                    if isinstance(session_inner, dict):
+                        normalized_inner = dict(session_inner)
+                    else:
+                        normalized_inner = {}
+                    SESSION_KNOWN_TOP_LEVEL = (
+                        "instructions",
+                        "tools",
+                        "tool_choice",
+                        "model",
+                        "voice",
+                        "modalities",
+                        "input_audio_format",
+                        "output_audio_format",
+                        "input_audio_transcription",
+                        "turn_detection",
+                        "input_audio_noise_reduction",
+                    )
+                    promoted = 0
+                    for key in SESSION_KNOWN_TOP_LEVEL:
+                        if key in raw_dict and key not in normalized_inner:
+                            normalized_inner[key] = raw_dict[key]
+                            promoted += 1
+                    missing_type = "type" not in normalized_inner
+                    if missing_type:
+                        normalized_inner["type"] = "realtime"
+                    needs_rebuild = (
+                        promoted > 0
+                        or not isinstance(session_inner, dict)
+                        or missing_type
+                    )
+                    if needs_rebuild:
+                        rebuilt = {"type": "session.update", "session": normalized_inner}
+                        logger.info(
+                            "session.update parser fallback: promoted=%d flat keys, injected_type=%s. Re-validating.",
+                            promoted,
+                            missing_type,
+                        )
+                        return model_cls.model_validate(rebuilt)  # type: ignore[return-value]
+                except Exception as fallback_err:
+                    logger.error(
+                        "session.update fallback validation also failed: %s",
+                        fallback_err,
+                    )
+            logger.error(f"Invalid {event_type} payload: {e}")
+            return None
+
+    # ── Client event handlers ────────────────────
+
+    def build_session_created(self, conn_id: str) -> SessionCreatedEvent:
+        return self.session.build_session_created(conn_id)
+
+    def handle_session_update(self, conn_id: str, event: SessionUpdateEvent) -> Optional[RealtimeErrorEvent]:
+        return self.session.handle_session_update(conn_id, event)
+
+    def handle_audio_append(self, conn_id: str, event: InputAudioBufferAppendEvent) -> list[bytes]:
+        return self.audio.handle_audio_append(conn_id, event)
+
+    def handle_audio_commit(self, conn_id: str) -> RealtimeErrorEvent | None:
+        return self.audio.handle_audio_commit(conn_id)
+
+    def encode_audio_chunk(self, conn_id: str, audio: bytes) -> list[ServerEvent]:
+        return self.audio.encode_audio_chunk(conn_id, audio)
+
+    def handle_response_create(self, conn_id: str, event: ResponseCreateEvent) -> ServerEvent | None:
+        return self.response.handle_response_create(conn_id, event)
+
+    def handle_response_cancel(self, conn_id: str) -> list[ServerEvent]:
+        return self.response.handle_response_cancel(conn_id)
+
+    def finish_response(
+        self,
+        conn_id: str,
+        status: _ResponseStatus = "completed",
+        reason: _StatusReason | None = None,
+    ) -> list[ServerEvent]:
+        return self.response.finish_response(conn_id, status, reason)
+
+    def handle_conversation_item_create(self, conn_id: str, event: ConversationItemCreateEvent) -> list[ServerEvent]:
+        return self.conversation.handle_conversation_item_create(conn_id, event)
+
+    def dispatch_pipeline_event(self, conn_id: str, event: PipelineEvent) -> list[ServerEvent]:
+        """Route a pipeline text_output_queue event to the appropriate handler."""
+        events = self._dispatch_pipeline_event(conn_id, event, wait_for_pending_reopen=True)
+        return [] if events is None else events
+
+    def try_dispatch_pipeline_event(self, conn_id: str, event: PipelineEvent) -> list[ServerEvent] | None:
+        """Non-blocking dispatch.
+
+        Returns ``None`` when dispatch must be retried after a speculative
+        reopen candidate resolves.
+        """
+        return self._dispatch_pipeline_event(conn_id, event, wait_for_pending_reopen=False)
+
+    def should_defer_pipeline_event(self, event: PipelineEvent) -> bool:
+        if self.speculative_turns is None or not isinstance(event, (AssistantTextEvent, TokenUsageEvent)):
+            return False
+        return self.speculative_turns.has_pending_reopen_or_grace(
+            getattr(event, "turn_id", None),
+            getattr(event, "turn_revision", None),
+        )
+
+    def _dispatch_pipeline_event(
+        self,
+        conn_id: str,
+        event: PipelineEvent,
+        *,
+        wait_for_pending_reopen: bool,
+    ) -> list[ServerEvent] | None:
+        is_stale = self._is_stale_turn_event(event, wait_for_pending_reopen=wait_for_pending_reopen)
+        if is_stale is None:
+            return None
+        if is_stale:
+            logger.info(
+                "Ignoring stale %s for turn=%s rev=%s",
+                event.type,
+                getattr(event, "turn_id", None),
+                getattr(event, "turn_revision", None),
+            )
+            return []
+
+        self._observe_turn_event(event)
+        if isinstance(event, AssistantTextEvent):
+            return self.response.on_assistant_text(
+                conn_id,
+                event,
+                wait_for_pending_reopen=wait_for_pending_reopen,
+            )
+        handler = self._pipeline_dispatch.get(type(event))
+        if handler is None:
+            logger.debug("Unhandled pipeline event type: %s", type(event).__name__)
+            return []
+        return handler(conn_id, event)
+
+    def _is_stale_turn_event(self, event: PipelineEvent, *, wait_for_pending_reopen: bool = True) -> bool | None:
+        if self.speculative_turns is None:
+            return False
+        if not isinstance(
+            event,
+            (PartialTranscriptionEvent, TranscriptionCompletedEvent, AssistantTextEvent, TokenUsageEvent),
+        ):
+            return False
+        turn_id = getattr(event, "turn_id", None)
+        turn_revision = getattr(event, "turn_revision", None)
+        if isinstance(event, (AssistantTextEvent, TokenUsageEvent)):
+            is_latest: bool | None
+            if wait_for_pending_reopen:
+                is_latest = self.speculative_turns.is_latest_after_reopen_grace(turn_id, turn_revision)
+            else:
+                is_latest = self.speculative_turns.try_is_latest_after_reopen_grace(turn_id, turn_revision)
+            if is_latest is None:
+                return None
+            return not is_latest
+        return not self.speculative_turns.is_latest(turn_id, turn_revision)
+
+    def _observe_turn_event(self, event: PipelineEvent) -> None:
+        if self.speculative_turns is None:
+            return
+        self.speculative_turns.observe(
+            getattr(event, "turn_id", None),
+            getattr(event, "turn_revision", None),
+        )
+
+    # ── STT → LM bridge ────────────────────────────
+
+    def _on_transcription_completed(self, conn_id: str, event: TranscriptionCompletedEvent) -> list[ServerEvent]:
+        """Handle a final STT transcription: emit protocol event, append to chat, trigger LM."""
+        st = self._state(conn_id)
+        same_speculative_turn = event.turn_id is not None and event.turn_id == st.speculative_user_turn_id
+        if same_speculative_turn:
+            st.response_usage.audio_duration_s -= st.speculative_audio_duration_s
+        else:
+            st.speculative_audio_duration_s = 0.0
+
+        events = self.conversation.on_transcription_completed(conn_id, event)
+        if event.turn_id is not None:
+            st.speculative_audio_duration_s = st.input_audio_duration_s
+
+        cfg = st.runtime_config
+        transcript = event.transcript
+        if transcript:
+            if same_speculative_turn and st.speculative_user_item_id:
+                replaced = cfg.chat.replace_user_message_text(st.speculative_user_item_id, transcript)
+                if not replaced:
+                    item = cfg.chat.add_item(make_user_message(transcript))
+                    st.speculative_user_item_id = item.id
+            else:
+                item = cfg.chat.add_item(make_user_message(transcript))
+                st.speculative_user_item_id = item.id
+        elif same_speculative_turn and st.speculative_user_item_id:
+            cfg.chat.remove_user_message(st.speculative_user_item_id)
+            st.speculative_user_item_id = None
+        elif event.turn_id is not None and event.turn_id != st.speculative_user_turn_id:
+            st.speculative_user_item_id = None
+
+        if event.turn_id is not None:
+            st.speculative_user_turn_id = event.turn_id
+            st.speculative_user_turn_revision = event.turn_revision
+            st.speculative_user_speech_stopped_at_s = event.speech_stopped_at_s
+
+        queue = self.text_prompt_queue
+        if queue and transcript:
+            st.response_pending = True
+            st.response_audio_wait_started_at_s = perf_counter()
+            st.delay_filler_sent = False
+            queue.put(
+                GenerateResponseRequest(
+                    runtime_config=cfg,
+                    language_code=event.language_code,
+                    turn_id=event.turn_id,
+                    turn_revision=event.turn_revision,
+                    speech_stopped_at_s=event.speech_stopped_at_s,
+                )
+            )
+
+        return events
+
+    # ── Metrics ────────────────────────────────────
+
+    def _on_token_usage(self, conn_id: str, event: TokenUsageEvent) -> list[ServerEvent]:
+        """Accumulate input/output token counts on the connection's usage metrics."""
+        if self.speculative_turns and not self.speculative_turns.is_latest(
+            event.turn_id,
+            event.turn_revision,
+        ):
+            logger.debug("Dropping stale token usage for turn=%s rev=%s", event.turn_id, event.turn_revision)
+            return []
+        st = self._state(conn_id)
+        st.response_usage.input_tokens += event.input_tokens
+        st.response_usage.output_tokens += event.output_tokens
+        logger.info(
+            "Token usage (response): input=%d, output=%d",
+            st.response_usage.input_tokens,
+            st.response_usage.output_tokens,
+        )
+        return []
+
+    def _on_response_failed(self, conn_id: str, event: ResponseFailedEvent) -> list[ServerEvent]:
+        """Surface the failure to the client and close the response as ``failed``.
+
+        Emitted when generation failed (e.g. invalid out-of-band input, or the
+        provider rejecting an empty context). A top-level ``error`` event carries
+        the human-readable reason — ``response.done.status_details.error`` only
+        has code/type, no message — then ``finish_response`` closes the slot.
+
+        Idempotent: gated on an active response, and ``finish_response`` is itself
+        a no-op once the slot is closed, so a later EndOfResponse-driven close does
+        nothing.
+        """
+        logger.info("Response failed: %s", event.message)
+        if not self._state(conn_id).in_response:
+            return []
+        events: list[ServerEvent] = [self.make_error(event.message, "response_failed")]
+        events.extend(self.response.finish_response(conn_id, status="failed"))
+        return events
+
+    def get_usage(self) -> dict[str, Any]:
+        """Return cumulative usage metrics across all completed responses."""
+        data = self.total_usage.model_dump()
+        data["total_tokens"] = data["input_tokens"] + data["output_tokens"]
+        data["total_errors"] = self.total_usage.total_errors
+        return data
+
+    # ── Error ───────────────────────────────────
+
+    def make_error(self, message: str, _type: str) -> RealtimeErrorEvent:
+        self.total_usage.record_error(_type)
+        return build_error_event(message, _type)
+
+
+def build_error_event(message: str, error_type: str) -> RealtimeErrorEvent:
+    """Construct a RealtimeErrorEvent without touching any service-instance state.
+
+    Used by the websocket route handler on pool rejection, where no unit's
+    service should be charged with the error in its usage metrics.
+    """
+    return RealtimeErrorEvent(
+        type="error",
+        error=RealtimeError(message=message, type=error_type),
+        event_id=_generate_id("event"),
+    )

src/speech_to_speech/api/openai_realtime/utils.py

@@ -0,0 +1,12 @@
+import numpy as np
+from scipy.signal import resample_poly
+
+
+def resample(audio_int16: bytes, from_rate: int, to_rate: int) -> bytes:
+    """Resample int16 PCM audio between sample rates using polyphase filtering."""
+    if from_rate == to_rate:
+        return audio_int16
+    samples = np.frombuffer(audio_int16, dtype=np.int16).astype(np.float32) / 32768.0
+    gcd = np.gcd(to_rate, from_rate)
+    resampled = resample_poly(samples, up=to_rate // gcd, down=from_rate // gcd)
+    return np.clip(resampled * 32768, -32768, 32767).astype(np.int16).tobytes()

src/speech_to_speech/api/openai_realtime/websocket_router.py

@@ -0,0 +1,942 @@
+import asyncio
+import io
+import logging
+import random
+import shutil
+import subprocess
+import time
+import uuid
+import wave
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from queue import Empty, Queue
+from threading import Event as ThreadingEvent
+from typing import Any, Callable, Literal, TypeVar
+
+import numpy as np
+from fastapi import FastAPI, HTTPException, Response, WebSocket, WebSocketDisconnect
+from openai.types.realtime import (
+    ConversationItemCreateEvent,
+    InputAudioBufferAppendEvent,
+    InputAudioBufferCommitEvent,
+    ResponseCancelEvent,
+    ResponseCreateEvent,
+    SessionUpdateEvent,
+)
+from pydantic import BaseModel, ConfigDict, Field
+from starlette.websockets import WebSocketState
+
+from speech_to_speech.api.openai_realtime.pipeline_unit import PipelineUnit, SessionState
+from speech_to_speech.api.openai_realtime.service import ServerEvent, build_error_event
+from speech_to_speech.RAG import get_global_rag
+
+# Router amministrazione RAG — importato ma montato solo se RAG è attivo.
+# Perché in un file separato? Così se `sentence-transformers` non è installato
+# il resto della pipeline (senza --rag_enabled) continua a funzionare.
+try:
+    from speech_to_speech.RAG.router import router as _rag_router
+except Exception as _rag_import_exc:  # pragma: no cover - dipendenze mancanti
+    _rag_router = None
+    _rag_import_err = _rag_import_exc
+else:
+    _rag_import_err = None
+from speech_to_speech.pipeline.control import SESSION_END, PipelineControlMessage, is_control_message
+from speech_to_speech.pipeline.events import (
+    AssistantTextEvent,
+    PartialTranscriptionEvent,
+    PipelineEvent,
+    SpeechStartedEvent,
+    SpeechStoppedEvent,
+    TokenUsageEvent,
+    TranscriptionCompletedEvent,
+)
+from speech_to_speech.pipeline.log_context import pipeline_log_ctx
+from speech_to_speech.pipeline.messages import AUDIO_RESPONSE_DONE, PIPELINE_END, AudioOutput, TTSInput
+from speech_to_speech.utils.utils import response_wants_audio
+
+logger = logging.getLogger(__name__)
+MAX_AUDIO_BATCH_BYTES = 6400
+HTTP_TTS_SAMPLE_RATE = 16000
+# How long the release path waits for SESSION_END to propagate through the
+# handler chain back to output_queue before warning that the unit is stuck.
+# Tests monkeypatch this to a small value since their fixtures usually skip
+# the real handler chain.
+SESSION_END_DRAIN_TIMEOUT_S = 10.0
+# Past this, the unit is quarantined: its service session is unregistered
+# (closing the chat so late handler output can't mutate or bill it), but the
+# unit stays unclaimable. Releasing it instead would let a new client claim a
+# unit whose handlers may still emit the previous session's output (e.g. a
+# transcript, which carries no session identity and would be appended to the
+# new session's conversation) — a cross-session leak. If SESSION_END does
+# eventually drain, the chain has proven itself clean and the unit returns to
+# the pool; a dead handler keeps it quarantined forever, visible in /v1/pool
+# as "stuck".
+SESSION_END_QUARANTINE_TIMEOUT_S = 180.0
+QItem = TypeVar("QItem")
+
+
+class SpeechSynthesisRequest(BaseModel):
+    model_config = ConfigDict(populate_by_name=True)
+
+    input: str = Field(min_length=1, description="Text to synthesize")
+    model: str | None = Field(default=None, description="Reserved for OpenAI-compatible clients")
+    voice: str | None = Field(default=None, description="Reserved for future voice override support")
+    response_format: Literal["wav", "mp3"] | None = Field(
+        default=None,
+        description="Desired output audio format",
+    )
+    format: Literal["wav", "mp3"] | None = Field(
+        default=None,
+        alias="format",
+        description="Alias for response_format",
+    )
+    language: str | None = Field(default=None, description="Optional target language hint")
+
+    def resolved_format(self) -> Literal["wav", "mp3"]:
+        return self.response_format or self.format or "wav"
+
+
+async def _send_event(ws: WebSocket, event: ServerEvent) -> None:
+    # Skip cleanly when the ws is already closing/closed — happens during Ctrl-C
+    # shutdown, where the lifespan starts closing sockets while the route handler
+    # or send loop is still in flight pushing events.
+    if ws.application_state != WebSocketState.CONNECTED:
+        return
+    try:
+        await ws.send_json(event.model_dump())
+    except WebSocketDisconnect:
+        logger.debug("Skipped event: ws disconnected mid-send")
+    except RuntimeError as e:
+        # Race: ws closed between the state check above and the send. Starlette
+        # raises a plain RuntimeError("Unexpected ASGI message 'websocket.send'
+        # after sending 'websocket.close' ...") — harmless during shutdown.
+        msg = str(e)
+        if "websocket.close" in msg or "websocket.disconnect" in msg or "response already completed" in msg:
+            logger.debug(f"Skipped event: ws already closed ({msg})")
+        else:
+            logger.error(f"Failed to send event to client: {e}")
+    except Exception as e:  # noqa: BLE001
+        logger.error(f"Failed to send event to client: {e}")
+
+
+async def _send_events(ws: WebSocket, events: list[ServerEvent]) -> None:
+    for event in events:
+        await _send_event(ws, event)
+
+
+def _keep_audio_sentinel(item: Any) -> bool:
+    # SESSION_END must survive barge-in flushes of output_queue: dropping it
+    # would leave the release path waiting forever for the drain signal.
+    return _is_audio_done(item) or is_control_message(item, SESSION_END.kind)
+
+
+def _keep_user_text_event(item: Any) -> bool:
+    return isinstance(
+        item,
+        (SpeechStoppedEvent, PartialTranscriptionEvent, TranscriptionCompletedEvent, TokenUsageEvent),
+    )
+
+
+def _audio_payload(item: Any) -> Any:
+    return item.audio if isinstance(item, AudioOutput) else item
+
+
+def _audio_generation(item: Any) -> int | None:
+    return item.cancel_generation if isinstance(item, AudioOutput) else None
+
+
+def _flush_queue(q: Queue[QItem], *, preserve: Callable[[QItem], bool] | None = None) -> None:
+    """Drain a queue, optionally preserving items matching *preserve*.
+
+    Preserved items are re-inserted at the **front** of the queue
+    (atomically under the queue's mutex) so they are processed before
+    anything a pipeline thread may have enqueued during the drain.
+    """
+    preserved: list[QItem] = []
+    while True:
+        try:
+            item = q.get_nowait()
+            if preserve and preserve(item):
+                preserved.append(item)
+        except Empty:
+            break
+    if preserved:
+        with q.mutex:
+            for item in reversed(preserved):
+                q.queue.appendleft(item)
+            q.not_empty.notify(len(preserved))
+
+
+async def _drain_pending_response_events(
+    ws: WebSocket | None,
+    unit: PipelineUnit,
+    session_id: str | None,
+) -> None:
+    if session_id is None:
+        return
+
+    preserved: list[Any] = []
+    drained_assistant = 0
+    drained_usage = 0
+    drain_assistant_events = True
+    try:
+        while True:
+            try:
+                item = unit.text_output_queue.get_nowait()
+            except Empty:
+                break
+            # Usage is accounting-only, so keep the old whole-queue drain behavior.
+            # Assistant events are client-visible response output and stop at the
+            # first non-response boundary to preserve normal text-event ordering.
+            if isinstance(item, TokenUsageEvent):
+                unit.service.dispatch_pipeline_event(session_id, item)
+                drained_usage += 1
+            elif drain_assistant_events and isinstance(item, AssistantTextEvent):
+                drained_assistant += 1
+                if _generation_is_discardable(unit, item.cancel_generation):
+                    continue
+                events = unit.service.dispatch_pipeline_event(session_id, item)
+                if ws is not None and events:
+                    await _send_events(ws, events)
+            else:
+                preserved.append(item)
+                drain_assistant_events = False
+    finally:
+        if preserved:
+            with unit.text_output_queue.mutex:
+                for item in reversed(preserved):
+                    unit.text_output_queue.queue.appendleft(item)
+                unit.text_output_queue.not_empty.notify(len(preserved))
+
+    if drained_assistant or drained_usage:
+        logger.debug(
+            "Pipeline %d: drained %d assistant event(s) and %d token usage event(s) before response completion",
+            unit.index,
+            drained_assistant,
+            drained_usage,
+        )
+
+
+def _clean_unit(unit: PipelineUnit, preserve: Callable[[Any], bool] | None = None) -> None:
+    """Cancel in-flight work and flush queues for a single pipeline unit.
+
+    All four pipeline queues are drained — input audio, transcript-to-LM,
+    LM-to-TTS output, and the text-event side channel — so pending work from
+    a released session cannot be picked up by handlers and leak into the next
+    session that claims this unit. SESSION_END is enqueued by the route
+    handler *after* this returns to serve as the soft reset signal for
+    stateful handlers.
+    """
+    unit.cancel_scope.cancel()
+    _flush_queue(unit.input_queue)
+    _flush_queue(unit.text_prompt_queue)
+    _flush_queue(unit.output_queue, preserve=preserve)
+    _flush_queue(unit.text_output_queue, preserve=preserve)
+    unit.response_playing.clear()
+    unit.cancel_scope.reset()
+    unit.should_listen.set()
+
+
+def _to_audio_bytes(chunk: Any) -> bytes:
+    chunk = _audio_payload(chunk)
+    if isinstance(chunk, PipelineControlMessage):
+        raise TypeError(f"unexpected control message on audio output queue: {chunk!r}")
+    if isinstance(chunk, np.ndarray) or hasattr(chunk, "tobytes"):
+        return chunk.tobytes()
+    return chunk
+
+
+def _is_audio_done(item: Any) -> bool:
+    payload = _audio_payload(item)
+    return isinstance(payload, bytes) and payload == AUDIO_RESPONSE_DONE
+
+
+def _is_pipeline_end(item: Any) -> bool:
+    payload = _audio_payload(item)
+    return isinstance(payload, bytes) and payload == PIPELINE_END
+
+
+def _generation_is_discardable(unit: PipelineUnit, generation: int | None) -> bool:
+    """Whether output tagged with *generation* should be dropped.
+
+    A generation is discardable if it has been superseded (``is_stale``) or if the
+    cancel scope is in its post-cancel discard window and this is not the current
+    live generation. Shared by audio and assistant-text so the two paths stay in
+    lockstep: dropping text whenever ``discarding`` is set (without this generation
+    check) silently swallows the transcript of a fresh response when ``discarding``
+    lingers — e.g. a superseded speculative turn whose TTS never emitted an
+    AUDIO_RESPONSE_DONE sentinel, so response_done() never cleared the flag.
+    """
+    if generation is not None and unit.cancel_scope.is_stale(generation):
+        return True
+    if unit.cancel_scope.discarding and generation != unit.cancel_scope.generation:
+        return True
+    return False
+
+
+def _should_discard_audio(unit: PipelineUnit, item: Any) -> bool:
+    return _generation_is_discardable(unit, _audio_generation(item))
+
+
+def _mark_audio_started(unit: PipelineUnit, session_id: str | None) -> None:
+    if session_id is None:
+        return
+    st = unit.service._state(session_id)
+    st.response_pending = False
+    st.response_audio_wait_started_at_s = None
+
+
+def _get_qwen_tts_handler(unit: PipelineUnit) -> Any | None:
+    for handler in unit.handlers:
+        if handler.__class__.__name__ == "Qwen3TTSHandler":
+            return handler
+    return None
+
+
+def _encode_wav_bytes(pcm_bytes: bytes, sample_rate: int = HTTP_TTS_SAMPLE_RATE) -> bytes:
+    buffer = io.BytesIO()
+    with wave.open(buffer, "wb") as wav_file:
+        wav_file.setnchannels(1)
+        wav_file.setsampwidth(2)
+        wav_file.setframerate(sample_rate)
+        wav_file.writeframes(pcm_bytes)
+    return buffer.getvalue()
+
+
+def _encode_mp3_bytes(pcm_bytes: bytes, sample_rate: int = HTTP_TTS_SAMPLE_RATE) -> bytes:
+    ffmpeg_path = shutil.which("ffmpeg")
+    if ffmpeg_path is None:
+        raise RuntimeError("ffmpeg is required to return MP3 audio")
+
+    result = subprocess.run(
+        [
+            ffmpeg_path,
+            "-hide_banner",
+            "-loglevel",
+            "error",
+            "-f",
+            "s16le",
+            "-ar",
+            str(sample_rate),
+            "-ac",
+            "1",
+            "-i",
+            "pipe:0",
+            "-f",
+            "mp3",
+            "-codec:a",
+            "libmp3lame",
+            "pipe:1",
+        ],
+        input=pcm_bytes,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        check=False,
+    )
+    if result.returncode != 0:
+        stderr = result.stderr.decode("utf-8", errors="ignore").strip()
+        raise RuntimeError(f"ffmpeg mp3 encoding failed: {stderr or 'unknown error'}")
+    return result.stdout
+
+
+def _collect_http_tts_pcm(unit: PipelineUnit, request: SpeechSynthesisRequest) -> bytes:
+    handler = _get_qwen_tts_handler(unit)
+    if handler is None:
+        raise RuntimeError("Qwen3-TTS handler not configured on the reserved pipeline unit")
+
+    _clean_unit(unit)
+    tts_input = TTSInput(
+        text=request.input.strip(),
+        language_code=request.language,
+        turn_id=f"http-tts-{uuid.uuid4().hex}",
+        turn_revision=0,
+    )
+
+    chunks: list[np.ndarray] = []
+    for chunk in handler.process(tts_input):
+        payload = chunk.audio if isinstance(chunk, AudioOutput) else chunk
+        if isinstance(payload, bytes):
+            if payload in (AUDIO_RESPONSE_DONE, PIPELINE_END):
+                continue
+            chunks.append(np.frombuffer(payload, dtype=np.int16).copy())
+            continue
+
+        array = np.asarray(payload, dtype=np.int16).reshape(-1)
+        if array.size:
+            chunks.append(array.copy())
+
+    _clean_unit(unit)
+    if not chunks:
+        raise RuntimeError("Qwen3-TTS produced no audio")
+    return np.concatenate(chunks).astype(np.int16, copy=False).tobytes()
+
+
+async def _maybe_send_delay_filler(ws: WebSocket | None, unit: PipelineUnit, session_id: str | None) -> bool:
+    if (
+        ws is None
+        or session_id is None
+        or unit.delay_filler_threshold_s <= 0
+        or not (unit.delay_filler_short_clips or unit.delay_filler_full_clips)
+    ):
+        return False
+    st = unit.service._state(session_id)
+    started_at_s = st.response_audio_wait_started_at_s
+    if started_at_s is None or st.delay_filler_sent or unit.response_playing.is_set():
+        return False
+    if not (st.response_pending or st.in_response):
+        return False
+    if not response_wants_audio(st.current_response_params):
+        return False
+    if time.perf_counter() - started_at_s < unit.delay_filler_threshold_s:
+        return False
+
+    filler_groups: list[tuple[str, list[bytes], float]] = []
+    if unit.delay_filler_short_clips:
+        filler_groups.append(("short", unit.delay_filler_short_clips, 0.65))
+    if unit.delay_filler_full_clips:
+        filler_groups.append(("full", unit.delay_filler_full_clips, 0.35))
+
+    if len(filler_groups) == 1:
+        filler_kind, filler_clips, _ = filler_groups[0]
+    else:
+        filler_kind, filler_clips, _ = random.choices(
+            filler_groups,
+            weights=[group[2] for group in filler_groups],
+            k=1,
+        )[0]
+
+    if len(filler_clips) == 1:
+        filler_index = 0
+    else:
+        candidate_indexes = list(range(len(filler_clips)))
+        if st.delay_filler_last_kind == filler_kind and st.delay_filler_last_index in candidate_indexes:
+            candidate_indexes.remove(st.delay_filler_last_index)
+        filler_index = random.choice(candidate_indexes)
+
+    clip = filler_clips[filler_index]
+    st.delay_filler_cursor += 1
+    st.delay_filler_last_kind = filler_kind
+    st.delay_filler_last_index = filler_index
+    st.delay_filler_sent = True
+    _mark_audio_started(unit, session_id)
+
+    for offset in range(0, len(clip), MAX_AUDIO_BATCH_BYTES):
+        batch = clip[offset : offset + MAX_AUDIO_BATCH_BYTES]
+        if batch:
+            await _send_events(ws, unit.service.encode_audio_chunk(session_id, batch))
+
+    unit.response_playing.set()
+    unit.should_listen.set()
+    logger.info("Pipeline %d: sent delay filler clip", unit.index)
+    return True
+
+
+def _safe_unregister(unit: PipelineUnit, session_id: str) -> None:
+    try:
+        unit.service.unregister(session_id)
+    except Exception:
+        logger.exception(f"Pipeline {unit.index}: unregister failed for session {session_id}")
+
+
+async def _release_unit_after_drain(unit: PipelineUnit, session: Any, session_id: str) -> None:
+    """Wait for SESSION_END to propagate, then release the unit.
+
+    Runs in its own asyncio task so the route handler's finally block can return
+    immediately. The unit stays unavailable for new claims (unit.session != None)
+    until SESSION_END travels all the way through the handler chain back to
+    output_queue — observed by the send loop, which sets session.drained.
+
+    Past SESSION_END_QUARANTINE_TIMEOUT_S (a wedged or dead handler thread) the
+    unit is quarantined, NOT released: still-running handlers could emit the old
+    session's output (transcripts carry no session identity) into whichever
+    session claimed the unit next, and a dead handler would make the unit accept
+    clients it can never serve. The session is unregistered right away so late
+    output can't mutate or bill the closed conversation; the unit itself only
+    returns to the pool if SESSION_END eventually drains, proving the chain is
+    clean. Operators can spot quarantined units in `/v1/pool` (state "stuck").
+    """
+    elapsed = 0.0
+    warned = False
+    try:
+        while not session.drained.is_set():
+            await asyncio.sleep(0.05)
+            elapsed += 0.05
+            if not warned and elapsed >= SESSION_END_DRAIN_TIMEOUT_S:
+                logger.warning(
+                    f"Pipeline {unit.index}: SESSION_END not drained after {elapsed:.1f}s — "
+                    f"unit will remain unavailable until handlers finish (session {session_id})"
+                )
+                warned = True
+            if session.quarantined_at is None and elapsed >= SESSION_END_QUARANTINE_TIMEOUT_S:
+                session.quarantined_at = time.monotonic()
+                _safe_unregister(unit, session_id)
+                logger.error(
+                    f"Pipeline {unit.index}: SESSION_END still not drained after {elapsed:.0f}s — "
+                    f"quarantining unit until the handler chain drains (session {session_id})"
+                )
+    finally:
+        # Runs when the drain completed (chain proven clean) or the task is
+        # cancelled at shutdown. Release unconditionally: even if unregister
+        # raises, the unit must not stay claimed forever.
+        try:
+            _safe_unregister(unit, session_id)
+        finally:
+            unit.session = None
+        recovered = " after quarantine" if session.quarantined_at is not None else ""
+        logger.info(f"Pipeline {unit.index} released{recovered} (session {session_id} ended)")
+
+
+def create_app(pool: list[PipelineUnit], stop_event: ThreadingEvent) -> FastAPI:
+    @asynccontextmanager
+    async def lifespan(app: FastAPI) -> AsyncIterator[None]:
+        # One send loop per pipeline unit; each polls its own queues and forwards
+        # to the websocket currently attached via unit.session.
+        send_tasks = [asyncio.create_task(_send_loop_for(unit)) for unit in pool]
+        yield
+        for task in send_tasks:
+            task.cancel()
+        for task in send_tasks:
+            try:
+                await task
+            except asyncio.CancelledError:
+                pass
+        for unit in pool:
+            sess = unit.session
+            if sess is not None:
+                try:
+                    await sess.websocket.close()
+                except Exception:
+                    pass
+
+    app = FastAPI(lifespan=lifespan)
+
+    # ── Montaggio router RAG (solo se attivo) ────────────────────────────────
+    if get_global_rag() is not None and _rag_router is not None:
+        app.include_router(_rag_router)
+        logger.info(
+            "RAG HTTP API montata su %s",
+            getattr(_rag_router, "prefix", "/v1/rag"),
+        )
+    elif _rag_router is None and _rag_import_err is not None:
+        logger.warning(
+            "Router RAG non disponibile — import fallito: %s",
+            _rag_import_err,
+        )
+    else:
+        logger.debug("Router RAG omesso (RAG non attivo in questa istanza).")
+
+    # Strong references to in-flight drain-and-release tasks (asyncio only
+    # holds tasks weakly); each task removes itself on completion.
+    release_tasks: set[asyncio.Task[None]] = set()
+
+    def _claim_unit(ws: WebSocket) -> PipelineUnit | None:
+        """Atomically (between asyncio yield points) reserve the first idle unit.
+
+        Creates a placeholder SessionState that the caller fills in with the
+        session_id after RealtimeService.register().
+        """
+        for unit in pool:
+            if unit.session is None and not unit.http_tts_reserved:
+                unit.session = SessionState(websocket=ws)
+                return unit
+        return None
+
+    def _reserve_http_tts_unit() -> PipelineUnit | None:
+        for unit in pool:
+            if unit.session is None and not unit.http_tts_reserved and _get_qwen_tts_handler(unit) is not None:
+                unit.http_tts_reserved = True
+                return unit
+        return None
+
+    def _has_qwen_tts_capacity() -> bool:
+        return any(_get_qwen_tts_handler(unit) is not None for unit in pool)
+
+    @app.post("/v1/audio/speech")
+    async def speech_endpoint(request: SpeechSynthesisRequest) -> Response:
+        if not _has_qwen_tts_capacity():
+            raise HTTPException(
+                status_code=501,
+                detail="Qwen TTS is not configured on this server instance.",
+            )
+
+        unit = _reserve_http_tts_unit()
+        if unit is None:
+            raise HTTPException(
+                status_code=503,
+                detail="All Qwen TTS pipeline slots are currently busy.",
+            )
+
+        try:
+            pcm_bytes = await asyncio.to_thread(_collect_http_tts_pcm, unit, request)
+            fmt = request.resolved_format()
+            if fmt == "mp3":
+                payload = await asyncio.to_thread(_encode_mp3_bytes, pcm_bytes, HTTP_TTS_SAMPLE_RATE)
+                media_type = "audio/mpeg"
+                extension = "mp3"
+            else:
+                payload = _encode_wav_bytes(pcm_bytes, HTTP_TTS_SAMPLE_RATE)
+                media_type = "audio/wav"
+                extension = "wav"
+        except HTTPException:
+            raise
+        except Exception as e:
+            logger.error("HTTP speech synthesis failed on pipeline %d: %s", unit.index, e, exc_info=True)
+            raise HTTPException(status_code=500, detail=f"TTS synthesis failed: {e}") from e
+        finally:
+            unit.http_tts_reserved = False
+
+        return Response(
+            content=payload,
+            media_type=media_type,
+            headers={
+                "Content-Disposition": f'inline; filename="speech.{extension}"',
+                "X-Audio-Sample-Rate": str(HTTP_TTS_SAMPLE_RATE),
+            },
+        )
+
+    @app.websocket("/v1/realtime")
+    async def realtime_endpoint(ws: WebSocket) -> None:
+        await ws.accept()
+
+        unit = _claim_unit(ws)
+        if unit is None:
+            logger.warning(f"Rejected connection: all {len(pool)} pipeline slots in use")
+            # Stateless error event — rejection is not chargeable to any unit's usage metrics.
+            await _send_event(
+                ws,
+                build_error_event(
+                    f"All {len(pool)} session slots are in use. Disconnect an existing client first.",
+                    error_type="session_limit_reached",
+                ),
+            )
+            await ws.close(code=1008, reason="All session slots are in use")
+            return
+
+        pipeline_log_ctx.set(unit.index)
+        # _claim_unit guarantees unit.session is not None for the returned unit.
+        assert unit.session is not None
+        # Everything after the claim runs inside try so the finally below always
+        # releases the unit, even if session setup fails.
+        session_id = ""
+        try:
+            session_id = unit.service.register()
+            unit.session.session_id = session_id
+            logger.info(f"Client connected to pipeline {unit.index} (session {session_id})")
+
+            # Defensive: drain edge queues and reset events so stale data from a
+            # previous session that survived SESSION_END propagation doesn't leak.
+            _clean_unit(unit)
+
+            await _send_event(ws, unit.service.build_session_created(session_id))
+
+            while not stop_event.is_set():
+                try:
+                    raw = await asyncio.wait_for(ws.receive_json(), timeout=0.1)
+                except asyncio.TimeoutError:
+                    continue
+
+                event = unit.service.parse_client_event(raw)
+                if event is None:
+                    await _send_event(
+                        ws,
+                        unit.service.make_error(
+                            f"Unknown or invalid event: {raw.get('type')}", "unknown_or_invalid_event"
+                        ),
+                    )
+                    continue
+
+                if isinstance(event, InputAudioBufferAppendEvent):
+                    chunks = unit.service.handle_audio_append(session_id, event)
+                    rt_cfg = unit.service._state(session_id).runtime_config
+                    for chunk in chunks:
+                        unit.input_queue.put((chunk, rt_cfg))
+
+                elif isinstance(event, InputAudioBufferCommitEvent):
+                    err = unit.service.handle_audio_commit(session_id)
+                    if err:
+                        await _send_event(ws, err)
+
+                elif isinstance(event, SessionUpdateEvent):
+                    err = unit.service.handle_session_update(session_id, event)
+                    if err:
+                        await _send_event(ws, err)
+
+                elif isinstance(event, ConversationItemCreateEvent):
+                    events = unit.service.handle_conversation_item_create(session_id, event)
+                    if events:
+                        await _send_events(ws, events)
+
+                elif isinstance(event, ResponseCreateEvent):
+                    result = unit.service.handle_response_create(session_id, event)
+                    if result:
+                        if result.type != "error":
+                            unit.cancel_scope.new_response()
+                        await _send_event(ws, result)
+
+                elif isinstance(event, ResponseCancelEvent):
+                    was_active = unit.service._state(session_id).in_response
+                    if was_active:
+                        unit.cancel_scope.cancel()
+                    _flush_queue(unit.output_queue, preserve=_keep_audio_sentinel)
+                    _flush_queue(unit.text_output_queue, preserve=_keep_user_text_event)
+                    events = unit.service.handle_response_cancel(session_id)
+                    if events:
+                        await _send_events(ws, events)
+                    unit.response_playing.clear()
+
+        except WebSocketDisconnect:
+            logger.info(f"Client {session_id} disconnected from pipeline {unit.index}")
+        except Exception as e:
+            logger.error(f"Client {session_id} on pipeline {unit.index} error: {type(e).__name__}: {e}", exc_info=True)
+        finally:
+            # Hold the session reference: the send loop's snapshot will still resolve
+            # to this object until we clear unit.session, so any handler output that
+            # arrives during the drain window is sent to the now-closed ws (silently
+            # dropped) instead of leaking to whichever client claims this unit next.
+            old_session = unit.session
+            if old_session is not None:
+                old_session.released_at = time.monotonic()
+            _clean_unit(unit)
+            # Tag SESSION_END with this session's id so that, after a force
+            # release, a late arrival can't satisfy the next session's drain.
+            unit.input_queue.put(PipelineControlMessage(SESSION_END.kind, session_id=session_id))
+            # Spawn the drain-and-release as a separate task so the route handler's
+            # finally returns immediately. Awaiting here is unreliable: after
+            # WebSocketDisconnect propagates, subsequent awaits in the same task
+            # can be skipped/cancelled by Starlette's runner and never resume.
+            # asyncio holds tasks weakly — keep a reference until it completes.
+            task = asyncio.create_task(_release_unit_after_drain(unit, old_session, session_id))
+            release_tasks.add(task)
+            task.add_done_callback(release_tasks.discard)
+
+    @app.get("/v1/usage")
+    async def usage_endpoint() -> dict[str, Any]:
+        # Aggregate usage across the pool. Numeric fields sum; dict fields (e.g.
+        # errors_by_type) merge with numeric leaves summed too, so per-unit error
+        # counts don't get dropped by the first-unit's value.
+        def _merge(into: dict[str, Any], src: dict[str, Any]) -> None:
+            for k, v in src.items():
+                if isinstance(v, (int, float)):
+                    into[k] = into.get(k, 0) + v
+                elif isinstance(v, dict):
+                    sub = into.setdefault(k, {})
+                    if isinstance(sub, dict):
+                        _merge(sub, v)
+                else:
+                    into.setdefault(k, v)
+
+        total: dict[str, Any] = {}
+        for unit in pool:
+            _merge(total, unit.service.get_usage())
+        return total
+
+    @app.get("/v1/pool")
+    async def pool_endpoint() -> dict[str, Any]:
+        now = time.monotonic()
+
+        def _state(u: PipelineUnit) -> dict[str, Any]:
+            s = u.session
+            if s is None and u.http_tts_reserved:
+                return {"index": u.index, "state": "http_tts", "session_id": None}
+            if s is None:
+                return {"index": u.index, "state": "idle", "session_id": None}
+            if s.released_at is None:
+                return {"index": u.index, "state": "active", "session_id": s.session_id}
+            # Drain wait gave up (quarantine timeout): the unit stays occupied
+            # until SESSION_END actually drains — possibly forever if a handler
+            # thread died. Surfaced distinctly so operators can act on it.
+            if s.quarantined_at is not None:
+                return {
+                    "index": u.index,
+                    "state": "stuck",
+                    "session_id": s.session_id,
+                    "draining_for_s": round(now - s.released_at, 2),
+                    "stuck_for_s": round(now - s.quarantined_at, 2),
+                }
+            # released by client but SESSION_END hasn't drained yet → unit
+            # is still occupied; surface elapsed time so operators can spot
+            # stuck handlers.
+            return {
+                "index": u.index,
+                "state": "draining",
+                "session_id": s.session_id,
+                "draining_for_s": round(now - s.released_at, 2),
+            }
+
+        return {
+            "size": len(pool),
+            "in_use": sum(1 for u in pool if u.session is not None),
+            "units": [_state(u) for u in pool],
+        }
+
+    async def _send_loop_for(unit: PipelineUnit) -> None:
+        """Per-pipeline send loop. Polls this unit's output queues and forwards
+        to the websocket currently attached via unit.session.
+
+        Per-session scratch (pending_output_item) lives on SessionState, so it
+        disappears together with the websocket when the session is released —
+        no stale sentinel can leak into the next claim.
+        """
+        pipeline_log_ctx.set(unit.index)
+        while not stop_event.is_set():
+            try:
+                # Snapshot the session once per iteration; if the route releases the
+                # unit mid-iteration, we continue against the prior snapshot which is
+                # consistent (its websocket is still valid until ws.close() returns).
+                session = unit.session
+                ws = session.websocket if session is not None else None
+                session_id = session.session_id if session is not None else None
+
+                # Text events first (speech_started cancels active response).
+                try:
+                    text_msg = unit.text_output_queue.get_nowait()
+                    is_speech_start = isinstance(text_msg, SpeechStartedEvent)
+
+                    was_in_response = False
+                    was_response_pending = False
+                    if is_speech_start and session_id:
+                        st = unit.service._state(session_id)
+                        was_in_response = st.in_response
+                        was_response_pending = st.response_pending
+
+                    if isinstance(text_msg, AssistantTextEvent) and _generation_is_discardable(
+                        unit, text_msg.cancel_generation
+                    ):
+                        pass
+                    elif ws is not None and isinstance(text_msg, PipelineEvent) and session_id:
+                        events = unit.service.dispatch_pipeline_event(session_id, text_msg)
+                        if events:
+                            await _send_events(ws, events)
+
+                    if is_speech_start and (was_in_response or was_response_pending):
+                        active_cfg = unit.service._state(session_id).runtime_config if session_id else None
+                        if text_msg.interrupt_response and (
+                            active_cfg is None or active_cfg.interrupt_response_enabled
+                        ):
+                            unit.cancel_scope.cancel()
+                            if session_id:
+                                unit.service._state(session_id).response_pending = False
+                            _flush_queue(unit.output_queue, preserve=_keep_audio_sentinel)
+                            _flush_queue(unit.text_output_queue, preserve=_keep_user_text_event)
+                            if unit.response_playing.is_set():
+                                unit.response_playing.clear()
+                            logger.info(
+                                "Pipeline %d: speech during %s: cancelled, queue flushed",
+                                unit.index,
+                                "response" if was_in_response else "pending response",
+                            )
+                        else:
+                            logger.info(
+                                f"Pipeline {unit.index}: speech during response: interrupt_response disabled, ignoring"
+                            )
+                except Empty:
+                    pass
+
+                if (
+                    session is not None
+                    and session.pending_output_item is None
+                    and unit.output_queue.empty()
+                    and await _maybe_send_delay_filler(ws, unit, session_id)
+                ):
+                    await asyncio.sleep(0.01)
+                    continue
+
+                try:
+                    if session is not None and session.pending_output_item is not None:
+                        audio_chunk = session.pending_output_item
+                        session.pending_output_item = None
+                    else:
+                        audio_chunk = unit.output_queue.get_nowait()
+
+                    if _is_pipeline_end(audio_chunk):
+                        await _drain_pending_response_events(ws, unit, session_id)
+                        if ws is not None and session_id:
+                            await _send_events(ws, unit.service.finish_response(session_id))
+                        break
+
+                    if _is_audio_done(audio_chunk):
+                        audio_generation = _audio_generation(audio_chunk)
+                        if audio_generation is not None and unit.cancel_scope.is_stale(audio_generation):
+                            if session_id:
+                                unit.service._state(session_id).response_pending = False
+                            unit.cancel_scope.response_done(audio_generation)
+                            unit.should_listen.set()
+                            logger.info(f"Pipeline {unit.index}: stale response complete, listening re-enabled")
+                            continue
+                        await _drain_pending_response_events(ws, unit, session_id)
+                        if ws is not None and session_id:
+                            await _send_events(ws, unit.service.finish_response(session_id))
+                        if session_id:
+                            unit.service._state(session_id).response_pending = False
+                        unit.response_playing.clear()
+                        unit.cancel_scope.response_done(audio_generation)
+                        unit.should_listen.set()
+                        logger.info(f"Pipeline {unit.index}: response complete, listening re-enabled")
+                        continue
+
+                    # SESSION_END travels from input_queue through every handler to
+                    # output_queue. Observing it here means the chain has fully reset;
+                    # signal the release path so it can clear unit.session. A tag from
+                    # another session means the emitting session was force-released —
+                    # its late SESSION_END must not stand in for this session's drain.
+                    if is_control_message(audio_chunk, SESSION_END.kind):
+                        chunk_session_id = getattr(audio_chunk, "session_id", None)
+                        if session is not None and chunk_session_id in (None, session.session_id):
+                            session.drained.set()
+                            logger.debug(f"Pipeline {unit.index}: SESSION_END drained")
+                        continue
+
+                    if is_control_message(audio_chunk):
+                        continue
+
+                    if _should_discard_audio(unit, audio_chunk):
+                        continue
+
+                    audio_chunk = _to_audio_bytes(audio_chunk)
+
+                    audio_batch = bytearray(audio_chunk)
+                    while len(audio_batch) < MAX_AUDIO_BATCH_BYTES:
+                        try:
+                            next_chunk = unit.output_queue.get_nowait()
+                        except Empty:
+                            break
+
+                        if (
+                            _is_pipeline_end(next_chunk)
+                            or _is_audio_done(next_chunk)
+                            or is_control_message(next_chunk, SESSION_END.kind)
+                        ):
+                            # Only stash if we still have a session; otherwise drop it.
+                            if session is not None:
+                                session.pending_output_item = next_chunk
+                            break
+
+                        if _should_discard_audio(unit, next_chunk):
+                            continue
+
+                        next_audio = _to_audio_bytes(next_chunk)
+                        if len(audio_batch) + len(next_audio) > MAX_AUDIO_BATCH_BYTES:
+                            if session is not None:
+                                session.pending_output_item = next_chunk
+                            break
+                        audio_batch.extend(next_audio)
+
+                    if not unit.response_playing.is_set():
+                        unit.response_playing.set()
+                        unit.should_listen.set()
+                    _mark_audio_started(unit, session_id)
+
+                    if ws is not None and session_id:
+                        await _send_events(ws, unit.service.encode_audio_chunk(session_id, bytes(audio_batch)))
+                except Empty:
+                    pass
+
+                await asyncio.sleep(0.01)
+
+            except asyncio.CancelledError:
+                break
+            except Exception as e:
+                logger.error(f"Pipeline {unit.index} send loop error: {e}")
+                await asyncio.sleep(0.1)
+
+    return app