first

speech_to_speech/LLM/README.md

@@ -0,0 +1,139 @@
+# LLM Summary
+
+## Available LLM backends (`--llm_backend`)
+
+Runtime-supported values in `s2s_pipeline.py`:
+
+- `transformers` → `language_model.py` (Transformers backend)
+- `mlx-lm` → `language_model.py` (MLX backend)
+- `responses-api` → `responses_api_language_model.py`
+
+## Usage
+
+### 1) Transformers (`--llm_backend transformers`)
+
+- Handler: `LanguageModelHandler`
+- Typical use: local GPU/CPU inference using Hugging Face Transformers
+- Backend-specific args prefix: `--llm_*`
+- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
+
+```bash
+python s2s_pipeline.py \
+  --llm_backend transformers \
+  --model_name Qwen/Qwen3-4B-Instruct-2507 \
+  --llm_device cuda \
+  --llm_torch_dtype float16 \
+  --llm_gen_max_new_tokens 128
+```
+
+Common options:
+- `--llm_gen_min_new_tokens`
+- `--llm_gen_temperature`
+- `--llm_gen_do_sample`
+- `--chat_size`
+- `--init_chat_prompt`
+
+### 2) MLX-LM (`--llm_backend mlx-lm`)
+
+- Handler: `LanguageModelHandler`
+- Typical use: Apple Silicon local inference
+- Backend-specific args prefix: same as Transformers (`--llm_*`)
+
+```bash
+python s2s_pipeline.py \
+  --llm_backend mlx-lm \
+  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
+  --llm_device mps \
+  --llm_gen_max_new_tokens 128
+```
+
+Common options:
+- `--llm_gen_temperature`
+- `--llm_gen_do_sample`
+- `--chat_size`
+- `--init_chat_prompt`
+
+### 3) OpenAI-compatible API (`--llm_backend responses-api`)
+
+- Handler: `ResponsesApiModelHandler`
+- Typical use: remote model serving via OpenAI-compatible endpoints
+- Backend-specific args prefix: `--responses_api_*`
+- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
+
+```bash
+python s2s_pipeline.py \
+  --llm_backend responses-api \
+  --model_name gpt-5.4-mini \
+  --responses_api_api_key YOUR_API_KEY \
+  --responses_api_base_url https://api.example.com/v1 \
+  --responses_api_stream true
+```
+
+Common options:
+- `--chat_size`
+- `--init_chat_prompt`
+- `--user_role`
+
+## LLM Behavior
+
+When STT is set to language auto-detection (`--language auto`), LLM handlers can receive `(text, language_code)` and prepend a language control instruction like:
+
+- `Please reply to my message in <language>.`
+
+This helps the assistant respond in the detected language. The behavior is opt-in via `--enable_lang_prompt` (shared across all backends); it defaults to `False`.
+
+## Setup
+
+### CUDA setup
+
+```bash
+python s2s_pipeline.py \
+  --llm_backend transformers \
+  --model_name microsoft/Phi-3-mini-4k-instruct
+```
+
+### Local Mac setup
+
+```bash
+python s2s_pipeline.py \
+  --local_mac_optimal_settings \
+  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16
+```
+
+`--local_mac_optimal_settings` already sets `--llm_backend mlx-lm` and will default the model to `mlx-community/Qwen3-4B-Instruct-2507-bf16` if not overridden.
+
+### Realtime (OpenAI-compatible) setup
+
+Run the server in realtime mode, then connect with the realtime client:
+
+```bash
+# 1. Start the pipeline in realtime mode
+python s2s_pipeline.py \
+  --mode realtime \
+  --llm_backend mlx-lm \
+  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
+  --ws_host 0.0.0.0 \
+  --ws_port 8765
+
+# 2. Connect with the realtime client
+python listen_and_play_realtime.py --host 127.0.0.1 --port 8765
+```
+
+Or with `--local_mac_optimal_settings` on Apple Silicon:
+
+```bash
+python s2s_pipeline.py \
+  --local_mac_optimal_settings \
+  --mode realtime \
+  --ws_host 0.0.0.0 \
+  --ws_port 8765
+```
+
+### Remote API setup
+
+```bash
+python s2s_pipeline.py \
+  --llm_backend responses-api \
+  --model_name gpt-5.4-mini \
+  --responses_api_api_key YOUR_API_KEY
+```

speech_to_speech/LLM/base_openai_compatible_language_model.py

@@ -0,0 +1,588 @@
+from __future__ import annotations
+
+import logging
+from abc import ABC, abstractmethod
+from collections.abc import Iterator
+from typing import Any, Optional
+
+import httpx
+from nltk import sent_tokenize
+from openai import OpenAI
+from openai.types.realtime.conversation_item import (
+    RealtimeConversationItemAssistantMessage,
+    RealtimeConversationItemFunctionCall,
+)
+from openai.types.realtime.realtime_conversation_item_assistant_message import (
+    Content as AssistantContent,
+)
+from openai.types.responses import ResponseFunctionToolCall
+from pydantic import BaseModel, ConfigDict, Field
+
+from speech_to_speech.baseHandler import BaseHandler
+from speech_to_speech.LLM.chat import (
+    Chat,
+    ChatItemError,
+    SupportedItem,
+    build_active_chat,
+    make_system_message,
+    make_user_message,
+)
+from speech_to_speech.LLM.compaction_prompt import CompactGenerateFn, build_compactor
+from speech_to_speech.LLM.text_prompt import build_text_system_prompt
+from speech_to_speech.LLM.utils import remove_unspeechable, resolve_auto_language
+from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt
+from speech_to_speech.pipeline.cancel_scope import CancelScope
+from speech_to_speech.pipeline.handler_types import LLMIn, LLMOut
+from speech_to_speech.pipeline.messages import (
+    EndOfResponse,
+    LLMResponseChunk,
+    TokenUsage,
+)
+from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
+from speech_to_speech.utils.utils import is_out_of_band, response_wants_audio
+
+logger = logging.getLogger(__name__)
+
+# About 18–24 seconds of default SDK backoff before warmup fails.
+WARMUP_MAX_RETRIES = 6
+
+
+# ── Normalised provider events ────────────────────────────────────────────────
+# Each backend's stream/response is mapped to this small vocabulary so the shared
+# speech-pipeline logic (sentence batching, cancellation, history, token usage)
+# lives in one place. Subclasses differ only in how they produce these events.
+
+
+class TextDelta(BaseModel):
+    """Incremental assistant text. Always RAW (unfiltered); the base applies
+    ``remove_unspeechable`` for the audio path."""
+
+    text: str
+
+
+class AssistantMessage(BaseModel):
+    """A complete assistant turn to write back to history."""
+
+    content: list[AssistantContent]
+
+
+class ToolCall(BaseModel):
+    """A complete function tool call (``call_id`` / ``id`` already regenerated)."""
+
+    item: ResponseFunctionToolCall
+
+
+class Usage(BaseModel):
+    """Token accounting for the turn."""
+
+    input_tokens: int
+    output_tokens: int
+
+
+ProviderEvent = TextDelta | AssistantMessage | ToolCall | Usage
+
+
+class _Turn(BaseModel):
+    """Per-request context threaded through generation (immutable for the turn)."""
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    language_code: Optional[str]
+    gen: int | None
+    runtime_config: Any
+    response: Any
+    turn_id: str | None
+    turn_revision: int | None
+    speech_stopped_at_s: float | None
+    wants_audio: bool
+
+
+class _GenState(BaseModel):
+    """Mutable accumulators collected while consuming a turn's events."""
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    tools: list[ResponseFunctionToolCall] = Field(default_factory=list)
+    pending: list[SupportedItem] = Field(default_factory=list)
+    clean_text: str = ""  # filtered text, kept only for the debug log
+    input_tokens: int = 0
+    output_tokens: int = 0
+
+
+class BaseOpenAICompatibleHandler(BaseHandler[LLMIn, LLMOut], ABC):
+    """Shared lifecycle for OpenAI-compatible LLM backends (Responses & Chat
+    Completions).
+
+    Subclasses implement four hooks — :meth:`warmup`,
+    :meth:`_build_compaction_generate_fn`, :meth:`_serialize`, :meth:`_request`,
+    :meth:`_iter_events` and :meth:`_build_optional_kwargs` — and inherit the
+    request/response orchestration: speculative-turn gating, cancellation,
+    sentence batching, text-only vs audio handling, history write-back, token
+    usage, out-of-band handling and error termination.
+    """
+
+    # ── setup ─────────────────────────────────────────────────────────────────
+
+    def setup(
+        self,
+        model_name: str = "gpt-5.4-mini",
+        device: str = "cuda",
+        gen_kwargs: dict[str, Any] = {},
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
+        stream: bool = True,
+        user_role: str = "user",
+        cancel_scope: CancelScope | None = None,
+        speculative_turns: SpeculativeTurnTracker | None = None,
+        disable_thinking: bool = True,
+        reasoning_effort: Optional[str] = None,
+        request_timeout_s: float = 20.0,
+        stream_batch_sentences: int = 3,
+        enable_lang_prompt: bool = False,
+        compact_history: bool = False,
+        **_kwargs: Any,
+    ) -> None:
+        self.cancel_scope = cancel_scope
+        self.speculative_turns = speculative_turns
+        self.model_name = model_name
+        self.stream = stream
+        self.stream_batch_sentences = max(1, stream_batch_sentences)
+        self.enable_lang_prompt = enable_lang_prompt
+        self.gen_kwargs = dict(gen_kwargs)
+        self.request_timeout_s = float(request_timeout_s)
+        self.request_timeout = httpx.Timeout(
+            self.request_timeout_s,
+            connect=min(10.0, self.request_timeout_s),
+        )
+
+        self.user_role = user_role
+        self.client = OpenAI(api_key=api_key, base_url=base_url)
+        self._extra_body = self._build_extra_body(base_url, disable_thinking, reasoning_effort)
+        self.compactor = build_compactor(self._build_compaction_generate_fn()) if compact_history else None
+        self.warmup()
+
+    @staticmethod
+    def _is_official_openai(base_url: Optional[str]) -> bool:
+        """Whether ``base_url`` points at the official OpenAI server.
+
+        Normalises a trailing slash so ``https://api.openai.com/v1/`` is also
+        recognised; the official server rejects the provider-specific extra_body
+        keys we send to vLLM / the HF router.
+        """
+        if base_url is None:
+            return False
+        return base_url.rstrip("/") == "https://api.openai.com/v1"
+
+    @classmethod
+    def _build_extra_body(
+        cls,
+        base_url: Optional[str],
+        disable_thinking: bool,
+        reasoning_effort: Optional[str],
+    ) -> Optional[dict[str, Any]]:
+        """Build the provider-specific ``extra_body`` used to disable reasoning.
+
+        Providers differ in how reasoning is turned off: vLLM/Qwen honour
+        ``chat_template_kwargs.enable_thinking=false``, while others (e.g. GLM via
+        the HF router) ignore that and require ``reasoning_effort='none'``. A
+        non-empty ``reasoning_effort`` therefore takes precedence; otherwise we fall
+        back to the chat-template flag. None of this applies to the official
+        OpenAI server, which rejects unknown extra_body keys.
+        """
+        if base_url is None or cls._is_official_openai(base_url):
+            return None
+        if reasoning_effort:
+            return {"reasoning_effort": reasoning_effort}
+        if disable_thinking:
+            return {"chat_template_kwargs": {"enable_thinking": False}}
+        return None
+
+    # ── subclass hooks ──────────────────────────────────────────────────────--
+
+    @abstractmethod
+    def warmup(self) -> None:
+        """Issue a cheap request so the model/connection is ready before serving."""
+        ...
+
+    @abstractmethod
+    def _build_compaction_generate_fn(self) -> CompactGenerateFn:
+        """Return a ``(system, user) -> text`` fn used to compact long histories."""
+        ...
+
+    @abstractmethod
+    def _serialize(self, active_chat: Chat) -> Any:
+        """Serialise the chat to the backend's request payload (input/messages)."""
+        ...
+
+    @abstractmethod
+    def _request(self, api_input: Any, optional_kwargs: dict[str, Any]) -> Any:
+        """Issue the create() call and return the response or stream."""
+        ...
+
+    @abstractmethod
+    def _iter_stream_events(self, api_response: Any) -> Iterator[ProviderEvent]:
+        """Map a streaming response to normalised :data:`ProviderEvent`s."""
+        ...
+
+    @abstractmethod
+    def _iter_response_events(self, api_response: Any) -> Iterator[ProviderEvent]:
+        """Map a non-streaming response to normalised :data:`ProviderEvent`s."""
+        ...
+
+    def _iter_events(self, api_response: Any) -> Iterator[ProviderEvent]:
+        """Dispatch to the stream/non-stream mapper. ``self.stream`` is the single
+        source of truth (it set the request's ``stream=`` flag), so the response
+        type always matches it."""
+        if self.stream:
+            yield from self._iter_stream_events(api_response)
+        else:
+            yield from self._iter_response_events(api_response)
+
+    @abstractmethod
+    def _build_optional_kwargs(self, req_tools: Any, req_tool_choice: Any) -> dict[str, Any]:
+        """Build the per-request tools/tool_choice kwargs in the backend's shape."""
+        ...
+
+    # ── speculative-turn / cancellation gating ─────────────────────────────────
+
+    def _turn_is_latest(self, turn_id: str | None, turn_revision: int | None) -> bool:
+        return self.speculative_turns is None or self.speculative_turns.is_latest(turn_id, turn_revision)
+
+    def _generation_is_stale(self, gen: int | None) -> bool:
+        return gen is not None and self.cancel_scope is not None and self.cancel_scope.is_stale(gen)
+
+    def _turn_output_allowed(self, turn_id: str | None, turn_revision: int | None) -> bool:
+        if self.speculative_turns is None:
+            return True
+        return self.speculative_turns.is_latest_after_reopen_grace(turn_id, turn_revision)
+
+    def _apply_config(
+        self,
+        chat: Chat,
+        instructions: Optional[str],
+        wants_audio: bool = True,
+    ) -> None:
+        if instructions:
+            builder = build_voice_system_prompt if wants_audio else build_text_system_prompt
+            full_instructions = builder(instructions)
+            chat.add_item(make_system_message(full_instructions))
+
+    # ── output helpers ──────────────────────────────────────────────────────--
+
+    def _chunk(
+        self,
+        turn: _Turn,
+        *,
+        text: str = "",
+        tools: list[ResponseFunctionToolCall] | None = None,
+        language_code: Optional[str] = None,
+    ) -> LLMResponseChunk:
+        return LLMResponseChunk(
+            text=text,
+            language_code=language_code if language_code is not None else turn.language_code,
+            tools=tools or [],
+            runtime_config=turn.runtime_config,
+            response=turn.response,
+            turn_id=turn.turn_id,
+            turn_revision=turn.turn_revision,
+            speech_stopped_at_s=turn.speech_stopped_at_s,
+            cancel_generation=turn.gen,
+        )
+
+    def _record_tool_call(self, state: _GenState, turn: _Turn, item: ResponseFunctionToolCall) -> Iterator[LLMOut]:
+        """Emit a tool call, persisting it (and any assistant text seen so far)
+        to history *before* it is forwarded to the client.
+
+        The function_call must already exist in the conversation by the time the
+        client returns its ``function_call_output``; otherwise a fast client
+        races ahead of the deferred end-of-turn write-back and the output is
+        rejected ("No function_call with call_id ... found"), which makes the
+        model re-issue the same tool call. The call lands in ``_pending_tool_calls``
+        (not serialized until its output pairs it), so eager recording is safe.
+
+        Out-of-band turns never touch the default conversation, and a stale turn
+        records nothing (it is not forwarded to the client either)."""
+        state.tools.append(item)
+        fc_item = RealtimeConversationItemFunctionCall(
+            type="function_call",
+            name=item.name,
+            arguments=item.arguments,
+            call_id=item.call_id,
+            id=item.id,
+            status=item.status,
+        )
+        if self._generation_is_stale(turn.gen) or not self._turn_output_allowed(turn.turn_id, turn.turn_revision):
+            logger.info("LLM generation cancelled (stale speculative turn)")
+            return
+        if not is_out_of_band(turn.response):
+            # Flush assistant text accumulated before this call first (so history
+            # order matches what the client received), then persist the call —
+            # all before the chunk leaves for the client.
+            chat = turn.runtime_config.chat
+            for pending_item in state.pending:
+                chat.add_item(pending_item)
+            state.pending.clear()
+            chat.add_item(fc_item)
+        yield self._chunk(turn, tools=[item])
+
+    # ── consumption ─────────────────────────────────────────────────────────--
+
+    def _consume_streaming(self, events: Iterator[ProviderEvent], state: _GenState, turn: _Turn) -> Iterator[LLMOut]:
+        cancelled = False
+        printable_text = ""
+        sentence_batch: list[str] = []
+
+        def _flush(batch: list[str]) -> Iterator[LLMOut]:
+            if not batch:
+                return
+            if not self._turn_output_allowed(turn.turn_id, turn.turn_revision):
+                logger.info("LLM generation cancelled (stale speculative turn)")
+                return
+            yield self._chunk(turn, text=" ".join(batch))
+
+        for event in events:
+            if self._generation_is_stale(turn.gen) or not self._turn_is_latest(turn.turn_id, turn.turn_revision):
+                logger.info("LLM generation cancelled (interruption)")
+                cancelled = True
+                break
+
+            if isinstance(event, Usage):
+                state.input_tokens = event.input_tokens
+                state.output_tokens = event.output_tokens
+            elif isinstance(event, AssistantMessage):
+                state.pending.append(
+                    RealtimeConversationItemAssistantMessage(type="message", role="assistant", content=event.content)
+                )
+            elif isinstance(event, ToolCall):
+                # Flush any pending spoken text before emitting the tool call.
+                if printable_text.strip():
+                    sentence_batch.append(printable_text.strip())
+                    printable_text = ""
+                if sentence_batch:
+                    if not self._turn_output_allowed(turn.turn_id, turn.turn_revision):
+                        logger.info("LLM generation cancelled (stale speculative turn)")
+                        cancelled = True
+                        break
+                    yield from _flush(sentence_batch)
+                    sentence_batch = []
+                yield from self._record_tool_call(state, turn, event.item)
+            elif isinstance(event, TextDelta):
+                if not turn.wants_audio:
+                    # Text-only: forward verbatim. Keep every character (no
+                    # remove_unspeechable, which strips TTS-unfriendly symbols) and
+                    # don't sentence-split (sent_tokenize collapses newlines/markdown).
+                    state.clean_text += event.text
+                    if event.text:
+                        if not self._turn_output_allowed(turn.turn_id, turn.turn_revision):
+                            logger.info("LLM generation cancelled (stale speculative turn)")
+                            cancelled = True
+                            break
+                        yield self._chunk(turn, text=event.text)
+                    continue
+                new_text = remove_unspeechable(event.text)
+                state.clean_text += new_text
+                printable_text += new_text
+                sentences = sent_tokenize(printable_text)
+                if len(sentences) > 1:
+                    for s in sentences[:-1]:
+                        sentence_batch.append(s)
+                        if len(sentence_batch) >= self.stream_batch_sentences:
+                            if not self._turn_output_allowed(turn.turn_id, turn.turn_revision):
+                                logger.info("LLM generation cancelled (stale speculative turn)")
+                                cancelled = True
+                                break
+                            yield from _flush(sentence_batch)
+                            sentence_batch = []
+                    if cancelled:
+                        break
+                    printable_text = sentences[-1]
+
+        if not cancelled:
+            if printable_text.strip():
+                sentence_batch.append(printable_text.strip())
+            if sentence_batch:
+                if self._generation_is_stale(turn.gen):
+                    logger.info("LLM generation cancelled (interruption)")
+                else:
+                    logger.debug(f"Clean text: {state.clean_text}")
+                    yield from _flush(sentence_batch)
+            logger.info(f"Tools: {state.tools}")
+
+    def _consume_nonstreaming(self, events: Iterator[ProviderEvent], state: _GenState, turn: _Turn) -> Iterator[LLMOut]:
+        if self._generation_is_stale(turn.gen) or not self._turn_is_latest(turn.turn_id, turn.turn_revision):
+            logger.info("LLM generation cancelled (interruption)")
+            return
+        for event in events:
+            if isinstance(event, Usage):
+                state.input_tokens = event.input_tokens
+                state.output_tokens = event.output_tokens
+            elif isinstance(event, AssistantMessage):
+                state.pending.append(
+                    RealtimeConversationItemAssistantMessage(type="message", role="assistant", content=event.content)
+                )
+            elif isinstance(event, ToolCall):
+                yield from self._record_tool_call(state, turn, event.item)
+            elif isinstance(event, TextDelta):
+                # Text-only keeps every character verbatim; audio strips
+                # TTS-unfriendly symbols via remove_unspeechable.
+                spoken = event.text if not turn.wants_audio else remove_unspeechable(event.text)
+                state.clean_text += spoken
+                out = spoken if not turn.wants_audio else spoken.strip()
+                if (
+                    out
+                    and not self._generation_is_stale(turn.gen)
+                    and self._turn_output_allowed(turn.turn_id, turn.turn_revision)
+                ):
+                    yield self._chunk(turn, text=out)
+        logger.debug(f"Clean text: {state.clean_text}")
+        logger.info(f"Tools: {state.tools}")
+
+    # ── orchestration ─────────────────────────────────────────────────────────
+
+    def _generate(
+        self,
+        active_chat: Chat,
+        original_chat: Chat,
+        turn: _Turn,
+        optional_kwargs: dict[str, Any],
+    ) -> Iterator[LLMOut]:
+        api_response: Any = None
+        state = _GenState()
+        error_message: str | None = None
+        api_input = self._serialize(active_chat)
+        # Images the model actually sees this turn; only these are stripped on
+        # write-back, so an image a fast client injects mid-generation for the
+        # next turn survives (it is not in this serialized snapshot).
+        consumed_image_ids = active_chat.image_message_ids()
+        if not api_input:
+            # Nothing to send: empty `instructions` and no `input` (in the response,
+            # the default conversation, or the out-of-band context). The provider
+            # would reject this; fail with a clear message instead of an opaque error.
+            error_message = "Cannot generate a response: no instructions and no input were provided."
+
+        try:
+            if error_message is None:
+                api_response = self._request(api_input, optional_kwargs)
+            if api_response is not None:
+                events = self._iter_events(api_response)
+                if self.stream:
+                    yield from self._consume_streaming(events, state, turn)
+                else:
+                    yield from self._consume_nonstreaming(events, state, turn)
+        except httpx.ReadTimeout:
+            logger.warning(
+                "OpenAI API read timed out after %.1fs; ending the current response",
+                self.request_timeout_s,
+            )
+            if not self._generation_is_stale(turn.gen) and self._turn_output_allowed(turn.turn_id, turn.turn_revision):
+                # Canned apology carries no language_code (mirrors the prior handlers).
+                yield LLMResponseChunk(
+                    text="Wow I'm a bit slow today, could you repeat that?",
+                    runtime_config=turn.runtime_config,
+                    response=turn.response,
+                    turn_id=turn.turn_id,
+                    turn_revision=turn.turn_revision,
+                    speech_stopped_at_s=turn.speech_stopped_at_s,
+                    cancel_generation=turn.gen,
+                )
+        except Exception as exc:
+            # Any other generation failure must still terminate the response: record
+            # the error and fall through to the EndOfResponse below. Without this the
+            # exception would escape process() and no EndOfResponse would be emitted,
+            # leaving st.in_response stuck and locking every subsequent response.
+            logger.exception("LLM generation failed; ending the current response")
+            if error_message is None:
+                error_message = f"Language model generation failed: {exc}"
+        finally:
+            if api_response is not None and hasattr(api_response, "close"):
+                try:
+                    api_response.close()
+                except Exception:
+                    pass
+
+        if (
+            error_message is None
+            and not self._generation_is_stale(turn.gen)
+            and self._turn_output_allowed(turn.turn_id, turn.turn_revision)
+        ):
+            # Out-of-band responses emit output and usage but never write back to the
+            # default conversation (their context was a throwaway chat).
+            if not is_out_of_band(turn.response):
+                # Tool calls (and any assistant text preceding them) were already
+                # written eagerly in _record_tool_call; only trailing items remain.
+                for item in state.pending:
+                    original_chat.add_item(item)
+                original_chat.strip_images(consumed_image_ids)
+                original_chat.trim_if_needed(self.compactor)
+            if state.input_tokens or state.output_tokens:
+                yield TokenUsage(
+                    input_tokens=state.input_tokens,
+                    output_tokens=state.output_tokens,
+                    turn_id=turn.turn_id,
+                    turn_revision=turn.turn_revision,
+                )
+        yield EndOfResponse(
+            turn_id=turn.turn_id, turn_revision=turn.turn_revision, cancel_generation=turn.gen, error=error_message
+        )
+
+    def process(self, request: LLMIn) -> Iterator[LLMOut]:
+        """Process a language model request and yield LLMResponseChunks."""
+        runtime_config = request.runtime_config
+        response = request.response
+        turn_id = request.turn_id
+        turn_revision = request.turn_revision
+        speech_stopped_at_s = request.speech_stopped_at_s
+        if not self._turn_is_latest(turn_id, turn_revision):
+            logger.info("Skipping stale LLM request for turn=%s rev=%s", turn_id, turn_revision)
+            yield EndOfResponse(turn_id=turn_id, turn_revision=turn_revision)
+            return
+
+        original_chat = runtime_config.chat
+        if is_out_of_band(response):
+            try:
+                active_chat = build_active_chat(original_chat, response)
+            except ChatItemError as exc:
+                logger.info("Out-of-band response rejected: %s", exc)
+                yield EndOfResponse(turn_id=turn_id, turn_revision=turn_revision, error=str(exc))
+                return
+        else:
+            active_chat = original_chat.copy()
+        language_code = request.language_code
+        instructions = (
+            response.instructions if response and response.instructions else runtime_config.session.instructions
+        ) or ""
+        req_tools = response.tools if response and response.tools else runtime_config.session.tools
+        req_tool_choice = (
+            response.tool_choice if response and response.tool_choice else runtime_config.session.tool_choice
+        )
+        wants_audio = response_wants_audio(response)
+        self._apply_config(active_chat, instructions, wants_audio)
+        language_code, lang_name = resolve_auto_language(language_code)
+        if lang_name and self.enable_lang_prompt:
+            active_chat.add_item(make_user_message(f"Please reply to my message in {lang_name}."))
+
+        optional_kwargs = self._build_optional_kwargs(req_tools, req_tool_choice)
+
+        # CancelScope.is_stale(gen) is checked when the stream iterator advances; a
+        # blocked read inside httpx cannot be aborted by cancel_scope.cancel() from
+        # the websocket router. Mitigations: request_timeout_s / ReadTimeout.
+        gen = self.cancel_scope.generation if self.cancel_scope else None
+
+        turn = _Turn(
+            language_code=language_code,
+            gen=gen,
+            runtime_config=runtime_config,
+            response=response,
+            turn_id=turn_id,
+            turn_revision=turn_revision,
+            speech_stopped_at_s=speech_stopped_at_s,
+            wants_audio=wants_audio,
+        )
+        yield from self._generate(active_chat, original_chat, turn, optional_kwargs)
+
+    @property
+    def timing_log_level(self) -> int:
+        return logging.INFO
+
+    def should_log_timing(self, output: LLMOut) -> bool:
+        return isinstance(output, LLMResponseChunk) and self.last_time > self.min_time_to_debug

speech_to_speech/LLM/chat.py

@@ -0,0 +1,760 @@
+from __future__ import annotations
+
+import json
+import logging
+import threading
+from collections.abc import Callable
+from typing import Any, Literal, Union
+
+from openai.types.realtime import ConversationItem
+from openai.types.realtime.conversation_item import (
+    RealtimeConversationItemAssistantMessage,
+    RealtimeConversationItemFunctionCall,
+    RealtimeConversationItemFunctionCallOutput,
+    RealtimeConversationItemSystemMessage,
+    RealtimeConversationItemUserMessage,
+)
+from openai.types.realtime.realtime_conversation_item_assistant_message import (
+    Content as AssistantContent,
+)
+from openai.types.realtime.realtime_conversation_item_system_message import Content as SystemContent
+from openai.types.realtime.realtime_conversation_item_user_message import Content as UserContent
+from openai.types.realtime.realtime_response_create_params import RealtimeResponseCreateParams
+from openai.types.responses.response_input_image_param import ResponseInputImageParam
+from openai.types.responses.response_input_message_content_list_param import (
+    ResponseInputMessageContentListParam,
+)
+from openai.types.responses.response_input_param import (
+    FunctionCallOutput,
+    ResponseFunctionToolCallParam,
+    ResponseInputItemParam,
+    ResponseInputParam,
+    ResponseOutputMessageParam,
+)
+from openai.types.responses.response_input_param import (
+    Message as ResponseMessage,
+)
+from openai.types.responses.response_input_text_param import ResponseInputTextParam
+from openai.types.responses.response_output_text_param import ResponseOutputTextParam
+from pydantic import BaseModel
+
+from speech_to_speech.utils.utils import _generate_id
+
+logger = logging.getLogger(__name__)
+
+
+class ChatItemError(Exception):
+    """Raised when a conversation item fails validation in :meth:`Chat.add_item`."""
+
+
+class CompactionResult(BaseModel):
+    """Output of a :data:`CompactFn` summarization run."""
+
+    user_summary: str
+    assistant_summary: str
+
+
+def _ensure_id(value: str | None, prefix: str) -> str:
+    if value is None:
+        return _generate_id(prefix)
+    if not value.startswith(f"{prefix}_"):
+        raise ChatItemError(f"ID must start with '{prefix}_', got {value!r}")
+    return value
+
+
+SupportedItem = Union[
+    RealtimeConversationItemSystemMessage,
+    RealtimeConversationItemUserMessage,
+    RealtimeConversationItemAssistantMessage,
+    RealtimeConversationItemFunctionCall,
+    RealtimeConversationItemFunctionCallOutput,
+]
+
+
+CompactFn = Callable[[ResponseInputParam], CompactionResult]
+
+
+class Chat:
+    """Manages conversation history with bounded size to avoid OOM issues.
+
+    The buffer stores ``ConversationItem`` objects (user messages, assistant
+    messages, function calls, function call outputs).  System messages are
+    stored separately in ``init_chat_message`` and never placed in the buffer.
+
+    History bounding is decided per ``add_item`` call via the ``compactor``
+    argument:
+
+    - ``compactor=None``: when the user-turn count exceeds ``size`` the oldest
+      complete turn is evicted in place. Synchronous, lossy, no LLM involvement.
+    - ``compactor=<fn>``: when ``size`` is exceeded, ``fn`` is invoked in a
+      background thread to summarize older turns into a single user/assistant
+      pair (with pending function calls preserved). Single-flight: while a
+      compaction is running, additional triggers are silently bypassed.
+    """
+
+    def __init__(self, size: int) -> None:
+        self.size = size
+        self.init_chat_message: RealtimeConversationItemSystemMessage | None = None
+        # ``size`` is the number of user turns to keep.  When exceeded the
+        # oldest complete turn (everything up to the next user message)
+        # is evicted -- or, with a compactor, summarized in the background.
+        self.buffer: list[SupportedItem] = []
+        self._pending_tool_calls: dict[str, RealtimeConversationItemFunctionCall] = {}
+        self._user_turn_count: int = 0
+
+        # All state mutations and serializations go through _lock. Public methods
+        # acquire it once; internal callers that already hold it use the
+        # ``_locked`` helpers, so no reentry is needed (regular Lock is safe).
+        self._lock = threading.Lock()
+        self._compact_in_flight: bool = False
+        self._compact_thread: threading.Thread | None = None
+        self._shutdown = threading.Event()
+        self._gen_counter = 0
+
+    # ── Internal mutators (caller holds _lock) ─────────────────
+
+    def _evict_oldest_turn(self) -> None:
+        """Remove items from the front until the next user message boundary."""
+        if not self.buffer:
+            return
+        first = self.buffer.pop(0)
+        if isinstance(first, RealtimeConversationItemUserMessage):
+            self._user_turn_count -= 1
+        while self.buffer and not isinstance(self.buffer[0], RealtimeConversationItemUserMessage):
+            self.buffer.pop(0)
+
+    def _has_call_id_in_buffer(self, call_id: str) -> bool:
+        for entry in self.buffer:
+            if isinstance(entry, RealtimeConversationItemFunctionCall) and entry.call_id == call_id:
+                return True
+        return False
+
+    def _mark_call_completed(
+        self, call_id: str, status: Literal["completed", "incomplete", "in_progress"] | None = None
+    ) -> None:
+        """Set ``status`` to ``"completed"`` on the matching function_call."""
+        for entry in self.buffer:
+            if isinstance(entry, RealtimeConversationItemFunctionCall) and entry.call_id == call_id:
+                entry.status = "completed" if status is None else status
+                return
+
+    def append_tool_output(self, call_id: str, output_item: RealtimeConversationItemFunctionCallOutput) -> None:
+        """Append a ``function_call_output``, re-injecting its ``function_call`` if evicted.
+
+        Also marks the paired ``function_call`` as ``"completed"`` if its
+        status was ``None``.
+
+        Raises :class:`ChatItemError` if *call_id* is unknown.
+        """
+        with self._lock:
+            self._append_tool_output_locked(call_id, output_item)
+
+    def _append_tool_output_locked(self, call_id: str, output_item: RealtimeConversationItemFunctionCallOutput) -> None:
+        """Body of :meth:`append_tool_output`. Caller must hold ``_lock``."""
+        if self._has_call_id_in_buffer(call_id):
+            self._pending_tool_calls.pop(call_id, None)
+            self._mark_call_completed(call_id, output_item.status)
+            self.buffer.append(output_item)
+            return
+
+        if call_id in self._pending_tool_calls:
+            logger.info("Re-injecting evicted function_call for call_id=%s", call_id)
+            fc = self._pending_tool_calls.pop(call_id)
+            fc.status = "completed" if output_item.status is None else output_item.status
+            self.buffer.append(fc)
+            self.buffer.append(output_item)
+            return
+
+        raise ChatItemError(f"No function_call with call_id '{call_id}' found in conversation history.")
+
+    def init_chat(self, message: RealtimeConversationItemSystemMessage) -> None:
+        with self._lock:
+            self.init_chat_message = message
+
+    def add_item(self, item: SupportedItem) -> SupportedItem:
+        """Validate and route a conversation item into the chat buffer.
+
+        Does not enforce the soft size limit — call :meth:`trim_if_needed`
+        explicitly after each successful generation to evict or compact old
+        turns. A hard upper bound at ``2 * size`` is enforced inline as a
+        runaway-client safety net: if the user-turn count exceeds it, the
+        oldest complete turn is evicted (lossy, no compaction).
+
+        Raises :class:`ChatItemError` if the item fails validation.
+        """
+        with self._lock:
+            if isinstance(item, RealtimeConversationItemSystemMessage):
+                item.id = _ensure_id(item.id, "sys")
+                self.init_chat_message = item
+                logger.debug("Set system message via conversation item")
+
+            elif isinstance(item, RealtimeConversationItemUserMessage):
+                item.id = _ensure_id(item.id, "msg")
+                item.content = [
+                    p
+                    for p in item.content
+                    if (p.type == "input_text" and p.text) or (p.type == "input_image" and p.image_url)
+                ]
+                if not item.content:
+                    raise ChatItemError(
+                        "Message has no supported content. Supported modalities: input_text, input_image."
+                    )
+                self.buffer.append(item)
+                self._user_turn_count += 1
+                logger.debug("Added user message to chat (%d parts)", len(item.content))
+
+            elif isinstance(item, RealtimeConversationItemAssistantMessage):
+                item.id = _ensure_id(item.id, "msg")
+                item.content = [p for p in item.content if p.type == "output_text" and p.text]
+                if not item.content:
+                    return item
+                self.buffer.append(item)
+                logger.debug("Added assistant message to chat (%d parts)", len(item.content))
+
+            elif isinstance(item, RealtimeConversationItemFunctionCall):
+                item.id = _ensure_id(item.id, "fc")
+                item.call_id = _ensure_id(item.call_id, "call")
+                self._pending_tool_calls[item.call_id] = item
+                logger.debug("Added function_call to chat (call_id=%s)", item.call_id)
+
+            elif isinstance(item, RealtimeConversationItemFunctionCallOutput):
+                item.id = _ensure_id(item.id, "fco")
+                self._append_tool_output_locked(item.call_id, item)
+                logger.debug("Added function_call_output to chat (call_id=%s)", item.call_id)
+
+            else:
+                raise ChatItemError(f"Unsupported item type: {getattr(item, 'type', None)}")
+
+            if self.size > 0 and self._user_turn_count > 2 * self.size:
+                logger.warning(
+                    "Chat buffer exceeded hard cap (%d > 2 * size=%d); evicting oldest turn",
+                    self._user_turn_count,
+                    self.size,
+                )
+                while self._user_turn_count > 2 * self.size:
+                    self._evict_oldest_turn()
+
+            return item
+
+    def trim_if_needed(self, compactor: CompactFn | None = None) -> None:
+        """Enforce the size limit after a generation completes. Fires when
+        ``user_turn_count > size``.
+
+        - ``compactor=None``: synchronous eviction of the oldest complete turn.
+        - ``compactor=<fn>``: launch a background compaction (single-flight).
+
+        Call once after each successful generation, not inside :meth:`add_item`.
+        """
+        with self._lock:
+            if self._user_turn_count <= self.size:
+                return
+            if compactor is not None:
+                self._maybe_trigger_compaction(compactor)
+            else:
+                while self._user_turn_count > self.size:
+                    self._evict_oldest_turn()
+
+    def replace_user_message_text(self, item_id: str, text: str) -> bool:
+        """Replace the text content of an existing user message.
+
+        Used by speculative turn revisions: the conversation turn remains the
+        same, but the STT transcript is superseded by a transcription of a
+        longer raw-audio buffer.
+        """
+
+        with self._lock:
+            for item in self.buffer:
+                if not isinstance(item, RealtimeConversationItemUserMessage) or item.id != item_id:
+                    continue
+                item.content = [UserContent(type="input_text", text=text)]
+                logger.debug("Replaced speculative user message %s", item_id)
+                return True
+        return False
+
+    def remove_user_message(self, item_id: str) -> bool:
+        """Remove an existing user message from the bounded chat buffer."""
+
+        with self._lock:
+            for index, item in enumerate(self.buffer):
+                if not isinstance(item, RealtimeConversationItemUserMessage) or item.id != item_id:
+                    continue
+                del self.buffer[index]
+                self._user_turn_count -= 1
+                logger.debug("Removed speculative user message %s", item_id)
+                return True
+        return False
+
+    def to_responses_api_chat(self, items: list[SupportedItem] | None = None) -> ResponseInputParam:
+        """Serialize the chat (system prompt + buffer) for the OpenAI Responses API.
+
+        If *items* is provided, serialize that slice instead of the live buffer
+        (used by the compaction snapshot).
+        """
+        with self._lock:
+            return self._to_responses_api_chat_locked(items if items is not None else self.buffer)
+
+    def _to_responses_api_chat_locked(self, items: list[SupportedItem]) -> ResponseInputParam:
+        """Body of :meth:`to_responses_api_chat`. Caller must hold ``_lock``."""
+        buffer_items = list(items)
+        result: list[ResponseInputItemParam] = []
+        if self.init_chat_message:
+            result.append(
+                ResponseMessage(
+                    content=[
+                        ResponseInputTextParam(text=p.text or "A helpful AI assistant.", type="input_text")
+                        for p in self.init_chat_message.content
+                    ],
+                    role="system",
+                    type="message",
+                )
+            )
+        for item in buffer_items:
+            assert item.id is not None and item.id != "", f"item.id is {item.id}"
+            if isinstance(item, RealtimeConversationItemUserMessage):
+                content: ResponseInputMessageContentListParam = []
+                for user_part in item.content:
+                    if user_part.type == "input_text" and user_part.text is not None:
+                        content.append(ResponseInputTextParam(text=user_part.text or "", type="input_text"))
+                    elif user_part.type == "input_image" and user_part.image_url is not None:
+                        img = ResponseInputImageParam(type="input_image", detail=user_part.detail or "auto")
+                        if user_part.image_url is not None:
+                            img["image_url"] = user_part.image_url
+                        content.append(img)
+                if content:
+                    result.append(ResponseMessage(content=content, role="user", type="message"))
+            elif isinstance(item, RealtimeConversationItemAssistantMessage):
+                assistant_content: list[ResponseOutputTextParam] = []
+                for assistant_part in item.content:
+                    if assistant_part.type == "output_text" and assistant_part.text is not None:
+                        assistant_content.append(
+                            ResponseOutputTextParam(text=assistant_part.text, type="output_text", annotations=[])
+                        )
+                if assistant_content:
+                    result.append(
+                        ResponseOutputMessageParam(
+                            id=item.id,
+                            content=assistant_content,
+                            role="assistant",
+                            status=item.status or "completed",
+                            type="message",
+                        )
+                    )
+            elif isinstance(item, RealtimeConversationItemFunctionCall) and item.call_id is not None:
+                assert item.call_id is not None and item.call_id != ""
+                function_call = ResponseFunctionToolCallParam(
+                    arguments=item.arguments,
+                    call_id=item.call_id,
+                    name=item.name,
+                    type="function_call",
+                    id=item.id,
+                )
+                if item.id is not None:
+                    function_call["id"] = item.id
+                if item.status is not None:
+                    function_call["status"] = item.status
+                result.append(function_call)
+            elif isinstance(item, RealtimeConversationItemFunctionCallOutput):
+                function_call_output = FunctionCallOutput(
+                    call_id=item.call_id,
+                    output=item.output,
+                    type="function_call_output",
+                )
+                if item.id is not None:
+                    function_call_output["id"] = item.id
+                if item.status is not None:
+                    function_call_output["status"] = item.status
+                result.append(function_call_output)
+        return result
+
+    def to_transformers_chat(self) -> list[dict[str, Any]]:
+        """Serialize the full chat for HuggingFace transformers ``apply_chat_template``.
+
+        User messages with only text produce a plain string ``content`` value.
+        User messages containing images keep ``content`` as a list of dicts so
+        VLM pipelines can process them.
+        """
+        with self._lock:
+            messages: list[TransformersChatMessage] = []
+            if self.init_chat_message:
+                text = " ".join(p.text for p in self.init_chat_message.content if p.text)
+                messages.append(TransformersSystemMessage(content=text))
+            for item in self.buffer:
+                if isinstance(item, RealtimeConversationItemUserMessage):
+                    has_images = any(p.type == "input_image" for p in item.content)
+                    if has_images:
+                        messages.append(
+                            TransformersUserMessage(content=[p.model_dump(exclude_none=True) for p in item.content])
+                        )
+                    else:
+                        text = " ".join(p.text for p in item.content if p.type == "input_text" and p.text)
+                        messages.append(TransformersUserMessage(content=text))
+                elif isinstance(item, RealtimeConversationItemAssistantMessage):
+                    text = " ".join(p.text for p in item.content if p.text)
+                    messages.append(TransformersAssistantMessage(content=text))
+                elif isinstance(item, RealtimeConversationItemFunctionCall):
+                    assert item.call_id is not None and item.call_id != ""
+                    args: Any = item.arguments
+                    try:
+                        args = json.loads(args) if isinstance(args, str) else args
+                    except (json.JSONDecodeError, TypeError):
+                        args = {}
+                    messages.append(
+                        TransformersFunctionCallMessage(
+                            tool_calls=[
+                                TransformersToolCall(
+                                    id=item.call_id,
+                                    function=TransformersToolCallFunction(name=item.name, arguments=args),
+                                )
+                            ]
+                        )
+                    )
+                elif isinstance(item, RealtimeConversationItemFunctionCallOutput):
+                    name = ""
+                    for prev in reversed(messages):
+                        if isinstance(prev, TransformersFunctionCallMessage):
+                            for tc in prev.tool_calls:
+                                if tc.id == item.call_id:
+                                    name = tc.function.name
+                                    break
+                            if name:
+                                break
+                    messages.append(
+                        TransformersToolMessage(
+                            tool_call_id=item.call_id,
+                            name=name,
+                            content=item.output,
+                        )
+                    )
+            return [m.model_dump() for m in messages]
+
+    def copy(self) -> Chat:
+        """Return a shallow snapshot safe for concurrent read access."""
+        with self._lock:
+            clone = Chat(self.size)
+            clone.init_chat_message = self.init_chat_message
+            clone.buffer = list(self.buffer)
+            clone._pending_tool_calls = dict(self._pending_tool_calls)
+            clone._user_turn_count = self._user_turn_count
+            return clone
+
+    def reset(self) -> None:
+        """Clear all conversation state. Cancels any in-flight compaction splice."""
+        with self._lock:
+            self._gen_counter += 1
+            self._compact_in_flight = False
+            self.buffer = []
+            self.init_chat_message = None
+            self._pending_tool_calls = {}
+            self._user_turn_count = 0
+
+    def close(self) -> None:
+        """Permanently shut down the chat. In-flight compaction splice is suppressed.
+
+        The compaction worker (a daemon thread) is not joined: it may be blocked
+        in an LLM call. Process exit reaps it.
+        """
+        self._shutdown.set()
+        with self._lock:
+            self._gen_counter += 1
+            self._compact_in_flight = False
+
+    def image_message_ids(self) -> set[str]:
+        """IDs of user messages currently carrying ``input_image`` content."""
+        with self._lock:
+            return {
+                item.id
+                for item in self.buffer
+                if isinstance(item, RealtimeConversationItemUserMessage)
+                and item.id is not None
+                and any(p.type == "input_image" for p in item.content)
+            }
+
+    def strip_images(self, only_ids: set[str] | None = None) -> None:
+        """Remove image content parts from user messages in the buffer.
+
+        Called after appending the assistant response so images don't persist
+        across turns. With *only_ids*, strip only those message IDs — the images
+        the just-completed response actually consumed (captured before the
+        request was sent). This leaves intact an image a fast client injected
+        mid-generation for the *next* turn, which the current response never saw.
+        Without *only_ids*, every image is stripped.
+        """
+        with self._lock:
+            for item in self.buffer:
+                if isinstance(item, RealtimeConversationItemUserMessage):
+                    if only_ids is not None and item.id not in only_ids:
+                        continue
+                    item.content = [p for p in item.content if p.type != "input_image"]
+
+    # ── Compaction internals ──────────────────────────────────
+
+    def _snapshot_for_compaction(
+        self,
+    ) -> tuple[ResponseInputParam, set[str], int]:
+        """Compute the snapshot of items eligible for compaction.
+
+        Caller must hold ``_lock``. Returns
+        ``(serialized_snapshot, marker_ids, n_turns)``. ``marker_ids``
+        identifies the buffer items that may be removed when the splice runs.
+        Always leaves the most recent user turn untouched (it may be in-flight).
+        Returns an empty result if there are fewer than 2 compactable turns.
+        """
+        n_turns = max(0, self._user_turn_count - 1)
+        if n_turns < 2:
+            return [], set(), n_turns
+
+        # Slice up to (but not including) the (n_turns + 1)-th user message.
+        user_seen = 0
+        end_idx = len(self.buffer)
+        for i, entry in enumerate(self.buffer):
+            if isinstance(entry, RealtimeConversationItemUserMessage):
+                user_seen += 1
+                if user_seen == n_turns + 1:
+                    end_idx = i
+                    break
+
+        items_to_compact = self.buffer[:end_idx]
+        marker_ids = {entry.id for entry in items_to_compact if entry.id is not None}
+        snapshot = self._to_responses_api_chat_locked(items=items_to_compact)
+        # Strip image parts so the summarizer doesn't have to handle them.
+        for raw in snapshot:
+            if not isinstance(raw, dict) or raw.get("role") != "user":
+                continue
+            msg: dict[str, Any] = raw  # type: ignore[assignment]
+            content = msg.get("content")
+            if isinstance(content, list):
+                msg["content"] = [c for c in content if not (isinstance(c, dict) and c.get("type") == "input_image")]
+        return snapshot, marker_ids, n_turns
+
+    def _maybe_trigger_compaction(self, compactor: CompactFn) -> None:
+        """Start a background compaction worker. Bypass silently if one is running.
+
+        Caller must hold ``_lock``.
+        """
+        if self._shutdown.is_set() or self._compact_in_flight:
+            return
+        snapshot, marker_ids, n_turns = self._snapshot_for_compaction()
+        if n_turns < 2 or not marker_ids:
+            return
+        gen = self._gen_counter
+        self._compact_in_flight = True
+        thread = threading.Thread(
+            target=self._compact_worker,
+            args=(compactor, snapshot, marker_ids, gen),
+            daemon=True,
+            name="chat-compact",
+        )
+        self._compact_thread = thread
+        logger.info(
+            "Chat compaction triggered: compacting %d turn(s) (%d item(s)), buffer size=%d",
+            n_turns,
+            len(marker_ids),
+            len(self.buffer),
+        )
+        thread.start()
+
+    def _compact_worker(
+        self,
+        compactor: CompactFn,
+        snapshot: ResponseInputParam,
+        marker_ids: set[str],
+        gen: int,
+    ) -> None:
+        """Worker thread entry point."""
+        try:
+            if self._shutdown.is_set() or self._gen_counter != gen:
+                return
+            try:
+                result = compactor(snapshot)
+            except Exception:
+                logger.exception("Chat compaction failed; chat unchanged")
+                return
+            if not isinstance(result, CompactionResult):
+                logger.error("Compactor must return a CompactionResult, got %r", type(result).__name__)
+                return
+            if self._shutdown.is_set() or self._gen_counter != gen:
+                return
+            self._apply_compaction(result, marker_ids, gen)
+        finally:
+            # Don't clobber the flag if reset/close has advanced the gen.
+            with self._lock:
+                if self._gen_counter == gen:
+                    self._compact_in_flight = False
+
+    def _apply_compaction(
+        self,
+        result: CompactionResult,
+        marker_ids: set[str],
+        gen: int,
+    ) -> None:
+        """Splice the summary in front of items not consumed by compaction.
+
+        FC/FCO pairing is left entirely to :meth:`add_item` / :meth:`append_tool_output`.
+        Compaction only drops items; it never inserts an FC into the buffer.
+        Pending FCs (no FCO yet) stay in ``_pending_tool_calls`` and will be
+        appended adjacent to their FCO when it arrives.
+        """
+        with self._lock:
+            if self._shutdown.is_set() or self._gen_counter != gen:
+                return
+            # Keep FC if its FCO is outside the compacted range -- otherwise
+            # the FCO in `remaining` would be orphaned.
+            fco_call_ids_in_range = {
+                x.call_id
+                for x in self.buffer
+                if isinstance(x, RealtimeConversationItemFunctionCallOutput) and x.id in marker_ids
+            }
+            fc_ids_to_keep = {
+                x.id
+                for x in self.buffer
+                if x.id in marker_ids
+                and isinstance(x, RealtimeConversationItemFunctionCall)
+                and x.call_id not in fco_call_ids_in_range
+            }
+            drop_ids = marker_ids - fc_ids_to_keep
+            remaining = [x for x in self.buffer if x.id not in drop_ids]
+
+            user_msg = make_user_message(result.user_summary)
+            user_msg.id = _generate_id("msg")
+            asst_msg = make_assistant_message(result.assistant_summary)
+            asst_msg.id = _generate_id("msg")
+
+            self.buffer = [user_msg, asst_msg, *remaining]
+            self._user_turn_count = sum(1 for x in self.buffer if isinstance(x, RealtimeConversationItemUserMessage))
+            logger.info(
+                "Chat compaction applied: buffer now %d item(s), %d user turn(s)",
+                len(self.buffer),
+                self._user_turn_count,
+            )
+
+
+# ---------------------------------------------------------------------------
+# Transformers chat message models
+# ---------------------------------------------------------------------------
+
+
+class TransformersToolCallFunction(BaseModel):
+    name: str
+    arguments: dict[str, Any]
+
+
+class TransformersToolCall(BaseModel):
+    type: Literal["function"] = "function"
+    id: str
+    function: TransformersToolCallFunction
+
+
+class TransformersSystemMessage(BaseModel):
+    role: Literal["system"] = "system"
+    content: str
+
+
+class TransformersUserMessage(BaseModel):
+    role: Literal["user"] = "user"
+    content: str | list[dict[str, Any]]
+
+
+class TransformersAssistantMessage(BaseModel):
+    role: Literal["assistant"] = "assistant"
+    content: str
+
+
+class TransformersFunctionCallMessage(BaseModel):
+    role: Literal["assistant"] = "assistant"
+    tool_calls: list[TransformersToolCall]
+
+
+class TransformersToolMessage(BaseModel):
+    role: Literal["tool"] = "tool"
+    tool_call_id: str
+    name: str
+    content: str
+
+
+TransformersChatMessage = Union[
+    TransformersSystemMessage,
+    TransformersUserMessage,
+    TransformersAssistantMessage,
+    TransformersFunctionCallMessage,
+    TransformersToolMessage,
+]
+
+
+# ---------------------------------------------------------------------------
+# Factory helpers -- hide verbose constructors behind simple calls
+# ---------------------------------------------------------------------------
+
+
+def make_user_message(text: str) -> RealtimeConversationItemUserMessage:
+    return RealtimeConversationItemUserMessage(
+        type="message",
+        role="user",
+        content=[UserContent(type="input_text", text=text)],
+    )
+
+
+def make_assistant_message(text: str) -> RealtimeConversationItemAssistantMessage:
+    return RealtimeConversationItemAssistantMessage(
+        type="message",
+        role="assistant",
+        content=[AssistantContent(type="output_text", text=text)],
+    )
+
+
+def make_system_message(text: str) -> RealtimeConversationItemSystemMessage:
+    return RealtimeConversationItemSystemMessage(
+        type="message",
+        role="system",
+        content=[SystemContent(type="input_text", text=text)],
+    )
+
+
+def add_supported_item(chat: Chat, item: ConversationItem) -> None:
+    """Narrow a protocol ``ConversationItem`` to a :data:`SupportedItem` and add it to *chat*.
+
+    Raises :class:`ChatItemError` on validation failure or unsupported type. Shared
+    by the conversation handler (in-band item injection) and the language-model
+    handlers (seeding an out-of-band response's throwaway chat from ``response.input``).
+    """
+    # call_id on function_call items must be client-supplied: it is referenced later by
+    # function_call_output items, so we cannot silently generate one here.
+    if isinstance(item, RealtimeConversationItemFunctionCall) and (
+        item.call_id is None or not item.call_id.startswith("call_")
+    ):
+        raise ChatItemError("function_call item is missing a call_id. The call_id should start with 'call_'.")
+
+    if isinstance(
+        item,
+        (
+            RealtimeConversationItemSystemMessage,
+            RealtimeConversationItemUserMessage,
+            RealtimeConversationItemAssistantMessage,
+            RealtimeConversationItemFunctionCall,
+            RealtimeConversationItemFunctionCallOutput,
+        ),
+    ):
+        chat.add_item(item)
+        return
+
+    raise ChatItemError(f"Unsupported item type: {getattr(item, 'type', None)}")
+
+
+def build_active_chat(original_chat: Chat, response: RealtimeResponseCreateParams | None) -> Chat:
+    """Build the chat an *out-of-band* response generates against (caller ensures out-of-band).
+
+    Mirrors the OpenAI realtime semantics for ``input``:
+
+    - ``input is None`` -> a read-only **copy of the default conversation** (the
+      out-of-band response reads history but never commits back).
+    - ``input == []`` -> a **fresh, empty chat** (context cleared; only the
+      system prompt, added later by the handler, will be present).
+    - ``input == [...]`` -> a **fresh chat seeded** with those items.
+
+    Raises :class:`ChatItemError` if an ``input`` item fails validation.
+    """
+    if response is not None and response.input is not None:
+        fresh = Chat(original_chat.size)
+        for item in response.input:
+            add_supported_item(fresh, item)
+        return fresh
+    return original_chat.copy()

speech_to_speech/LLM/chat_completions_language_model.py

@@ -0,0 +1,311 @@
+from __future__ import annotations
+
+import json
+import logging
+import time
+from collections.abc import Iterator
+from typing import Any, cast
+
+from openai import Stream
+from openai.types.chat import (
+    ChatCompletionChunk,
+    ChatCompletionContentPartImageParam,
+    ChatCompletionContentPartParam,
+    ChatCompletionContentPartTextParam,
+    ChatCompletionNamedToolChoiceParam,
+    ChatCompletionToolChoiceOptionParam,
+    ChatCompletionToolParam,
+)
+from openai.types.chat.chat_completion_content_part_image_param import ImageURL
+from openai.types.chat.chat_completion_named_tool_choice_param import Function as NamedToolChoiceFunction
+from openai.types.realtime.realtime_conversation_item_assistant_message import (
+    Content as AssistantContent,
+)
+from openai.types.responses import ResponseFunctionToolCall
+from openai.types.shared_params import FunctionDefinition
+
+from speech_to_speech.LLM.base_openai_compatible_language_model import (
+    WARMUP_MAX_RETRIES,
+    AssistantMessage,
+    BaseOpenAICompatibleHandler,
+    ProviderEvent,
+    TextDelta,
+    ToolCall,
+    Usage,
+)
+from speech_to_speech.LLM.chat import Chat
+from speech_to_speech.LLM.compaction_prompt import CompactGenerateFn
+from speech_to_speech.utils.utils import _generate_id
+
+logger = logging.getLogger(__name__)
+
+
+def _to_chat_tools(req_tools: Any) -> list[ChatCompletionToolParam] | None:
+    """Convert Responses-API function tools to Chat-Completions tool format.
+
+    Responses tools are flat ``{type:"function", name, description, parameters}``;
+    Chat Completions nests them under a ``function`` key. Items already in the
+    nested form (or non-function tools) are passed through untouched.
+    """
+    if not req_tools:
+        return None
+    chat_tools: list[ChatCompletionToolParam] = []
+    for t in req_tools:
+        d = t if isinstance(t, dict) else t.model_dump(exclude_none=True)
+        if d.get("type") == "function" and "function" not in d:
+            fn = FunctionDefinition(name=d["name"])
+            if d.get("description") is not None:
+                fn["description"] = d["description"]
+            if d.get("parameters") is not None:
+                fn["parameters"] = d["parameters"]
+            chat_tools.append(ChatCompletionToolParam(type="function", function=fn))
+        else:
+            chat_tools.append(cast("ChatCompletionToolParam", d))
+    return chat_tools
+
+
+def _to_chat_tool_choice(tool_choice: Any) -> ChatCompletionToolChoiceOptionParam:
+    """Convert a Responses-API tool_choice to Chat-Completions form.
+
+    The string forms ("auto"/"required"/"none") are identical across both APIs;
+    only the forced-function object differs (flat ``name`` vs nested ``function``).
+    """
+    if isinstance(tool_choice, dict) and tool_choice.get("type") == "function" and "name" in tool_choice:
+        return ChatCompletionNamedToolChoiceParam(
+            type="function", function=NamedToolChoiceFunction(name=tool_choice["name"])
+        )
+    if tool_choice is not None and not isinstance(tool_choice, (str, dict)):
+        d = tool_choice.model_dump(exclude_none=True)
+        if d.get("type") == "function" and "name" in d:
+            return ChatCompletionNamedToolChoiceParam(type="function", function=NamedToolChoiceFunction(name=d["name"]))
+        return cast("ChatCompletionToolChoiceOptionParam", d)
+    return cast("ChatCompletionToolChoiceOptionParam", tool_choice)
+
+
+class ChatCompletionsApiModelHandler(BaseOpenAICompatibleHandler):
+    """LLM handler that talks to an OpenAI-compatible ``/v1/chat/completions`` server.
+
+    Functionally mirrors :class:`ResponsesApiModelHandler` but uses the mature
+    Chat Completions streaming tool-call protocol (``choices[].delta.tool_calls``)
+    instead of ``/v1/responses``. This is the robust path for vLLM + Qwen tool
+    calling. The conversation is serialised with :meth:`Chat.to_transformers_chat`,
+    which already emits OpenAI chat messages including ``tool_calls``/``tool`` roles.
+    """
+
+    def warmup(self) -> None:
+        logger.info(f"Warming up {self.__class__.__name__}")
+        start = time.time()
+        self.client.with_options(max_retries=WARMUP_MAX_RETRIES).chat.completions.create(
+            model=self.model_name,
+            messages=[
+                {"role": "system", "content": "You are a helpful assistant"},
+                {"role": "user", "content": "Hello"},
+            ],
+            extra_body=self._extra_body,
+            timeout=self.request_timeout,
+        )
+        end = time.time()
+        logger.info(f"{self.__class__.__name__}:  warmed up! time: {(end - start):.3f} s")
+
+    def _build_compaction_generate_fn(self) -> CompactGenerateFn:
+        """Return a generate fn that calls Chat Completions for compaction."""
+        client = self.client
+        model_name = self.model_name
+        timeout = self.request_timeout
+        extra_body = self._extra_body
+
+        def generate(system: str, user: str) -> str:
+            response = client.chat.completions.create(
+                model=model_name,
+                messages=[
+                    {"role": "system", "content": system},
+                    {"role": "user", "content": user},
+                ],
+                extra_body=extra_body,
+                timeout=timeout,
+            )
+            return response.choices[0].message.content or ""
+
+        return generate
+
+    @staticmethod
+    def _to_chat_content_part(part: dict[str, Any]) -> ChatCompletionContentPartParam:
+        """Convert one transformers content part to Chat-Completions shape.
+
+        ``to_transformers_chat`` keeps Realtime-style parts (``input_text`` /
+        ``input_image`` with a bare-string ``image_url``). The Chat Completions
+        HTTP API instead wants ``{type:"text", text}`` and
+        ``{type:"image_url", image_url:{url, detail}}``. Unknown parts pass through.
+        """
+        ptype = part.get("type")
+        if ptype == "input_text":
+            return ChatCompletionContentPartTextParam(type="text", text=part.get("text") or "")
+        if ptype == "input_image":
+            raw_url: Any = part.get("image_url")
+            if isinstance(raw_url, dict):
+                image_url = cast("ImageURL", raw_url)
+            else:
+                image_url = ImageURL(url=raw_url)
+                detail = part.get("detail")
+                if detail is not None:
+                    image_url["detail"] = detail
+            return ChatCompletionContentPartImageParam(type="image_url", image_url=image_url)
+        return cast("ChatCompletionContentPartParam", part)
+
+    @classmethod
+    def _chat_messages(cls, chat: Chat) -> list[dict[str, Any]]:
+        """Serialise the chat for the Chat Completions API.
+
+        ``Chat.to_transformers_chat`` targets HuggingFace ``apply_chat_template``,
+        so two shapes need fixing up for the OpenAI Chat Completions HTTP API:
+        tool-call ``arguments`` must be a JSON *string* (not a parsed object), and
+        multimodal ``content`` parts must use the Chat Completions ``text`` /
+        ``image_url`` shape rather than the Realtime ``input_text`` /
+        ``input_image`` shape.
+        """
+        messages = chat.to_transformers_chat()
+        for message in messages:
+            for tool_call in message.get("tool_calls") or []:
+                fn = tool_call.get("function")
+                if fn is not None and not isinstance(fn.get("arguments"), str):
+                    fn["arguments"] = json.dumps(fn.get("arguments") or {}, ensure_ascii=False)
+            content = message.get("content")
+            if isinstance(content, list):
+                message["content"] = [cls._to_chat_content_part(p) for p in content]
+            if message.get("role") == "tool":
+                message.pop("name", None)
+        return messages
+
+    # ── base hooks ──────────────────────────────────────────────────────────--
+
+    def _serialize(self, active_chat: Chat) -> list[dict[str, Any]]:
+        return self._chat_messages(active_chat)
+
+    def _build_optional_kwargs(self, req_tools: Any, req_tool_choice: Any) -> dict[str, Any]:
+        optional_kwargs: dict[str, Any] = {}
+        chat_tools = _to_chat_tools(req_tools)
+        if chat_tools is not None:
+            optional_kwargs["tools"] = chat_tools
+        if req_tool_choice is not None:
+            optional_kwargs["tool_choice"] = _to_chat_tool_choice(req_tool_choice)
+        return optional_kwargs
+
+
+    def _request(self, api_input: list[dict[str, Any]], optional_kwargs: dict[str, Any]) -> Any:
+        create_kwargs: dict[str, Any] = dict(optional_kwargs)
+        if self.stream:
+            create_kwargs["stream_options"] = {"include_usage": True}
+        
+        # Svuota i parametri dei tool per evitare il blocco 400 di NVIDIA NIM
+        create_kwargs.pop("tools", None)
+        create_kwargs.pop("tool_choice", None)
+        
+        return self.client.chat.completions.create(
+            model=self.model_name,
+            messages=api_input,  # type: ignore[arg-type]  # runtime dicts match the Chat Completions message shape
+            stream=self.stream,
+            extra_body=self._extra_body,
+            timeout=self.request_timeout,
+            **create_kwargs,
+        )
+
+#    def _request(self, api_input: list[dict[str, Any]], optional_kwargs: dict[str, Any]) -> Any:
+#        create_kwargs: dict[str, Any] = dict(optional_kwargs)
+#        if self.stream:
+#            create_kwargs["stream_options"] = {"include_usage": True}
+#        return self.client.chat.completions.create(
+#            model=self.model_name,
+#            messages=api_input,  # type: ignore[arg-type]  # runtime dicts match the Chat Completions message shape
+#            stream=self.stream,
+#            extra_body=self._extra_body,
+#            timeout=self.request_timeout,
+#            **create_kwargs,
+#        )
+
+    def _iter_stream_events(self, api_response: Stream[ChatCompletionChunk]) -> Iterator[ProviderEvent]:
+        # Accumulate streamed tool-call deltas, keyed by their stream index, and the
+        # raw assistant text, then emit assistant message + tool calls + usage once
+        # the stream is exhausted.
+        tool_accum: dict[int, dict[str, str]] = {}
+        usage: Usage | None = None
+        raw_text = ""
+        for chunk in api_response:
+            # Usage-only trailing chunk (choices == []) when include_usage is set.
+            if chunk.usage is not None:
+                usage = Usage(
+                    input_tokens=chunk.usage.prompt_tokens or 0, output_tokens=chunk.usage.completion_tokens or 0
+                )
+            if not chunk.choices:
+                continue
+            delta = chunk.choices[0].delta
+            if delta.tool_calls:
+                for tc in delta.tool_calls:
+                    entry = tool_accum.setdefault(tc.index, {"name": "", "args": "", "id": ""})
+                    if tc.id:
+                        entry["id"] = tc.id
+                    if tc.function is not None:
+                        if tc.function.name:
+                            entry["name"] = tc.function.name
+                        if tc.function.arguments:
+                            entry["args"] += tc.function.arguments
+            # A refusal streams as `delta.refusal` with `delta.content` None;
+            # surface it as assistant text so it is spoken and stored.
+            text_piece = delta.content or getattr(delta, "refusal", None)
+            if text_piece:
+                raw_text += text_piece
+                yield TextDelta(text=text_piece)
+
+        if raw_text.strip():
+            yield AssistantMessage(content=[AssistantContent(type="output_text", text=raw_text)])
+        yield from self._tool_calls_from_accum(tool_accum)
+        if usage is not None:
+            yield usage
+
+    def _iter_response_events(self, api_response: Any) -> Iterator[ProviderEvent]:
+        usage = api_response.usage
+        if usage:
+            yield Usage(input_tokens=usage.prompt_tokens or 0, output_tokens=usage.completion_tokens or 0)
+        # A valid-but-empty response (e.g. content filter) returns no choices;
+        # complete cleanly with no assistant text rather than raising IndexError.
+        message = api_response.choices[0].message if api_response.choices else None
+        if message is None:
+            return
+        # A refusal arrives as `message.refusal` with `message.content` None; treat
+        # it as assistant text so it is spoken and stored.
+        raw_content = message.content or getattr(message, "refusal", None)
+        if raw_content:
+            yield AssistantMessage(content=[AssistantContent(type="output_text", text=raw_content)])
+            yield TextDelta(text=raw_content)
+        tool_accum: dict[int, dict[str, str]] = {}
+        for tc in message.tool_calls or []:
+            tool_accum[len(tool_accum)] = {
+                "name": tc.function.name or "",
+                "args": tc.function.arguments or "",
+                "id": tc.id or "",
+            }
+        yield from self._tool_calls_from_accum(tool_accum)
+
+    @staticmethod
+    def _tool_calls_from_accum(tool_accum: dict[int, dict[str, str]]) -> Iterator[ToolCall]:
+        """Turn accumulated tool-call deltas into ToolCall events.
+
+        IDs are regenerated (mirroring the Responses handler) so the rest of the
+        pipeline pairs each call_id with its function_call_output consistently.
+        """
+        for index in sorted(tool_accum):
+            entry = tool_accum[index]
+            if not entry["name"]:
+                continue
+            yield ToolCall(
+                item=ResponseFunctionToolCall(
+                    type="function_call",
+                    name=entry["name"],
+                    arguments=entry["args"] or "{}",
+                    call_id=_generate_id("call"),
+                    id=_generate_id("fc"),
+                    status="completed",
+                )
+            )
+
+    def on_session_end(self) -> None:
+        logger.debug("Chat Completions API language model session state reset")

speech_to_speech/LLM/compaction_prompt.py

@@ -0,0 +1,181 @@
+"""Prompt template and factory for the conversation compaction (history summarization) function.
+
+Compaction reduces an unbounded conversation history to a tight user/assistant
+summary pair, letting the pipeline continue indefinitely without running out of
+context window.  The factory :func:`build_compactor` returns a :data:`CompactFn`
+compatible with :meth:`~speech_to_speech.LLM.chat.Chat.trim_if_needed`.
+
+:data:`CompactGenerateFn` is the backend-agnostic generation interface:
+``(system_prompt: str, user_prompt: str) -> response_text: str``.
+Each handler wraps its own model into this interface and passes it to
+:func:`build_compactor`.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import re
+from collections.abc import Callable
+from typing import Any
+
+from speech_to_speech.LLM.chat import CompactFn, CompactionResult
+
+logger = logging.getLogger(__name__)
+
+# Callable[[system_prompt, user_prompt], response_text]
+CompactGenerateFn = Callable[[str, str], str]
+
+# ---------------------------------------------------------------------------
+# Prompts
+# ---------------------------------------------------------------------------
+
+COMPACTION_SYSTEM_PROMPT = """\
+You are a conversation memory compressor for a real-time voice AI assistant.
+
+Your task: read a multi-turn conversation and produce a dense summary so the
+assistant can continue naturally, as if it remembers everything that was said.
+
+Output a single JSON object with exactly two string fields:
+  "user_summary"      — 1–5 sentences capturing what the user has been asking
+                        about, any preferences or constraints they have stated,
+                        and where the conversation stands from their perspective.
+  "assistant_summary" — 1–5 sentences capturing what the assistant has
+                        explained, decided, or done (including tool calls and
+                        their results), plus any open questions or commitments.
+
+Rules:
+- Be information-dense: preserve names, numbers, file paths, error messages, and
+  other specifics that would be needed to continue the conversation correctly.
+- Omit small-talk and filler that carries no forward context.
+- Write in third person, past tense
+  (e.g. "The user asked about…", "The assistant provided…").
+- Emit only the JSON object — no markdown, no code fences, no extra keys.\
+"""
+
+COMPACTION_USER_TEMPLATE = """\
+Summarize the following conversation.  Return only the JSON object.
+
+--- CONVERSATION START ---
+{conversation}
+--- CONVERSATION END ---\
+"""
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _render_transcript(snapshot: list[Any]) -> str:
+    """Render a ResponseInputParam snapshot as a readable plain-text transcript."""
+    lines: list[str] = []
+    for item in snapshot:
+        if not isinstance(item, dict):
+            continue
+        item_type = item.get("type", "message")
+        role: str = item.get("role", "")
+
+        if role == "system":
+            continue
+
+        if item_type == "function_call":
+            name = item.get("name", "")
+            args = item.get("arguments", "")
+            lines.append(f"[Tool call: {name}({args})]")
+            continue
+
+        if item_type == "function_call_output":
+            out = item.get("output", "")
+            lines.append(f"[Tool result: {out}]")
+            continue
+
+        # Regular user / assistant message
+        content = item.get("content", "")
+        if isinstance(content, list):
+            text = " ".join(
+                c.get("text", "")
+                for c in content
+                if isinstance(c, dict) and c.get("type") in ("input_text", "output_text")
+            ).strip()
+        elif isinstance(content, str):
+            text = content.strip()
+        else:
+            continue
+
+        if text:
+            label = role.capitalize() if role else "Unknown"
+            lines.append(f"{label}: {text}")
+
+    return "\n\n".join(lines)
+
+
+_JSON_BLOCK_RE = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL)
+
+
+def _extract_json(text: str) -> dict[str, Any]:
+    """Extract the first JSON object from *text*, stripping markdown code fences."""
+    try:
+        return json.loads(text)
+    except json.JSONDecodeError:
+        pass
+
+    m = _JSON_BLOCK_RE.search(text)
+    if m:
+        return json.loads(m.group(1))
+
+    start = text.find("{")
+    end = text.rfind("}")
+    if start != -1 and end > start:
+        return json.loads(text[start : end + 1])
+
+    raise ValueError(f"No JSON object found in compaction response: {text!r}")
+
+
+# ---------------------------------------------------------------------------
+# Factory
+# ---------------------------------------------------------------------------
+
+
+def build_compactor(generate_fn: CompactGenerateFn) -> CompactFn:
+    """Return a :data:`CompactFn` that summarizes history using *generate_fn*.
+
+    *generate_fn* is the only model-specific dependency: it receives
+    ``(system_prompt, user_prompt)`` and returns the model's text response.
+    Both :class:`~speech_to_speech.LLM.openai_api_language_model.OpenApiModelHandler`
+    and :class:`~speech_to_speech.LLM.language_model.BaseLanguageModelHandler`
+    subclasses expose a ``_build_compaction_generate_fn()`` method that wraps
+    their respective backend into this interface.
+
+    The returned callable is safe to call from a background thread.
+
+    Args:
+        generate_fn: Backend-agnostic text generation callable.
+
+    Returns:
+        A callable ``(snapshot: ResponseInputParam) -> CompactionResult``.
+    """
+
+    def compact(snapshot: list[Any]) -> CompactionResult:
+        transcript = _render_transcript(snapshot)
+        if not transcript.strip():
+            logger.warning("Compaction called with an empty transcript; returning empty summaries")
+            return CompactionResult(user_summary="", assistant_summary="")
+
+        user_content = COMPACTION_USER_TEMPLATE.format(conversation=transcript)
+        raw_text = generate_fn(COMPACTION_SYSTEM_PROMPT, user_content)
+
+        data = _extract_json(raw_text)
+        user_summary = str(data.get("user_summary", "")).strip()
+        assistant_summary = str(data.get("assistant_summary", "")).strip()
+
+        if not user_summary or not assistant_summary:
+            raise ValueError(f"Compaction response missing required fields. Got: {data!r}")
+
+        logger.debug(
+            "Compaction complete. user=%d chars  assistant=%d chars",
+            len(user_summary),
+            len(assistant_summary),
+        )
+        return CompactionResult(user_summary=user_summary, assistant_summary=assistant_summary)
+
+    return compact

speech_to_speech/LLM/lm_output_processor.py

@@ -0,0 +1,148 @@
+"""
+LLM Output Processor
+
+Intercepts LLM output to:
+1. Extract tool calls and send them via text_output_queue
+2. Forward clean text to TTS pipeline
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Iterator
+from queue import Queue
+
+from speech_to_speech.baseHandler import BaseHandler
+from speech_to_speech.pipeline.events import AssistantTextEvent, ResponseFailedEvent, TokenUsageEvent
+from speech_to_speech.pipeline.handler_types import LLMOut, TTSIn
+from speech_to_speech.pipeline.messages import EndOfResponse, LLMResponseChunk, TokenUsage, TTSInput
+from speech_to_speech.pipeline.queue_types import TextEventItem
+from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
+from speech_to_speech.utils.utils import response_wants_audio
+
+logger = logging.getLogger(__name__)
+
+
+class LMOutputProcessor(BaseHandler[LLMOut, TTSIn]):
+    """
+    Processes LLM output to extract tool calls and forward clean text to TTS.
+
+    Input: :class:`LLMResponseChunk`, :class:`TokenUsage`, or :class:`EndOfResponse` from LLM
+    Output: :class:`TTSInput` or :class:`EndOfResponse` to TTS
+    Side effect: Sends :class:`AssistantTextEvent` / :class:`TokenUsageEvent` to text_output_queue
+    """
+
+    def setup(
+        self,
+        text_output_queue: Queue[TextEventItem] | None = None,
+        speculative_turns: SpeculativeTurnTracker | None = None,
+    ) -> None:
+        """
+        Initialize the processor.
+
+        Args:
+            text_output_queue: Queue to send text messages and tool calls
+        """
+        self.text_output_queue = text_output_queue
+        self.speculative_turns = speculative_turns
+
+    def _turn_output_allowed(self, turn_id: str | None, turn_revision: int | None) -> bool:
+        if self.speculative_turns is None:
+            return True
+        return self.speculative_turns.is_latest_after_reopen_grace(turn_id, turn_revision)
+
+    def process(self, lm_output: LLMOut) -> Iterator[TTSIn]:
+        """
+        Process LLM output: send text/tools to WebSocket, forward clean text to TTS.
+
+        Yields:
+            :class:`TTSInput` or :class:`EndOfResponse` for TTS
+        """
+        if isinstance(lm_output, TokenUsage):
+            if not self._turn_output_allowed(
+                lm_output.turn_id,
+                lm_output.turn_revision,
+            ):
+                logger.debug(
+                    "Dropping stale token usage for turn=%s rev=%s", lm_output.turn_id, lm_output.turn_revision
+                )
+                return
+            if self.text_output_queue is not None:
+                self.text_output_queue.put(
+                    TokenUsageEvent(
+                        input_tokens=lm_output.input_tokens or 0,
+                        output_tokens=lm_output.output_tokens or 0,
+                        turn_id=lm_output.turn_id,
+                        turn_revision=lm_output.turn_revision,
+                    )
+                )
+            return
+
+        if isinstance(lm_output, EndOfResponse):
+            if not self._turn_output_allowed(
+                lm_output.turn_id,
+                lm_output.turn_revision,
+            ):
+                logger.debug(
+                    "Dropping stale end-of-response for turn=%s rev=%s",
+                    lm_output.turn_id,
+                    lm_output.turn_revision,
+                )
+                return
+            # A failed generation (e.g. invalid out-of-band input) closes the response as
+            # "failed" via the text side-channel, then falls through to emit the normal
+            # EndOfResponse so the audio path still re-enables listening / releases the slot.
+            if lm_output.error and self.text_output_queue is not None:
+                self.text_output_queue.put(
+                    ResponseFailedEvent(
+                        message=lm_output.error,
+                        turn_id=lm_output.turn_id,
+                        turn_revision=lm_output.turn_revision,
+                    )
+                )
+            yield EndOfResponse(
+                turn_id=lm_output.turn_id,
+                turn_revision=lm_output.turn_revision,
+                cancel_generation=lm_output.cancel_generation,
+            )
+            return
+
+        if not isinstance(lm_output, LLMResponseChunk):
+            logger.warning("LMOutputProcessor received unexpected type: %s", type(lm_output))
+            return
+
+        if not self._turn_output_allowed(
+            lm_output.turn_id,
+            lm_output.turn_revision,
+        ):
+            logger.debug("Dropping stale LLM chunk for turn=%s rev=%s", lm_output.turn_id, lm_output.turn_revision)
+            return
+
+        logger.debug(f"LM processor: text='{lm_output.text}', tools={lm_output.tools}")
+
+        if self.text_output_queue is not None:
+            event = AssistantTextEvent(
+                text=lm_output.text,
+                turn_id=lm_output.turn_id,
+                turn_revision=lm_output.turn_revision,
+                cancel_generation=lm_output.cancel_generation,
+            )
+            if lm_output.tools:
+                event.tools = lm_output.tools
+                logger.info(f"Sending to clients: text='{lm_output.text}', tools={[t.name for t in lm_output.tools]}")
+            else:
+                logger.debug(f"Sending to clients: text='{lm_output.text}' (no tools)")
+            self.text_output_queue.put(event)
+
+        if lm_output.text and response_wants_audio(lm_output.response):
+            logger.debug(f"Forwarding to TTS: '{lm_output.text}'")
+            yield TTSInput(
+                text=lm_output.text,
+                language_code=lm_output.language_code,
+                runtime_config=lm_output.runtime_config,
+                response=lm_output.response,
+                turn_id=lm_output.turn_id,
+                turn_revision=lm_output.turn_revision,
+                speech_stopped_at_s=lm_output.speech_stopped_at_s,
+                cancel_generation=lm_output.cancel_generation,
+            )

speech_to_speech/LLM/responses_api_language_model.py

@@ -0,0 +1,149 @@
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Iterator
+from typing import Any
+
+from openai import Stream
+from openai.types.realtime.realtime_conversation_item_assistant_message import (
+    Content as AssistantContent,
+)
+from openai.types.responses import (
+    ResponseCompletedEvent,
+    ResponseFunctionToolCall,
+    ResponseOutputItemDoneEvent,
+    ResponseOutputMessage,
+    ResponseTextDeltaEvent,
+)
+
+from speech_to_speech.LLM.base_openai_compatible_language_model import (
+    WARMUP_MAX_RETRIES,
+    AssistantMessage,
+    BaseOpenAICompatibleHandler,
+    ProviderEvent,
+    TextDelta,
+    ToolCall,
+    Usage,
+)
+from speech_to_speech.LLM.chat import Chat
+from speech_to_speech.LLM.compaction_prompt import CompactGenerateFn
+from speech_to_speech.utils.utils import _generate_id
+
+logger = logging.getLogger(__name__)
+
+
+class ResponsesApiModelHandler(BaseOpenAICompatibleHandler):
+    """LLM handler that talks to an OpenAI ``/v1/responses`` server."""
+
+    def warmup(self) -> None:
+        logger.info(f"Warming up {self.__class__.__name__}")
+        start = time.time()
+        self.client.with_options(max_retries=WARMUP_MAX_RETRIES).responses.create(
+            model=self.model_name,
+            input=[
+                {
+                    "type": "message",
+                    "role": "system",
+                    "content": [{"type": "input_text", "text": "You are a helpful assistant"}],
+                },
+                {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Hello"}]},
+            ],
+            timeout=self.request_timeout,
+        )
+        end = time.time()
+        logger.info(f"{self.__class__.__name__}:  warmed up! time: {(end - start):.3f} s")
+
+    def _build_compaction_generate_fn(self) -> CompactGenerateFn:
+        """Return a generate fn that calls the Responses API for compaction."""
+        client = self.client
+        model_name = self.model_name
+        timeout = self.request_timeout
+
+        def generate(system: str, user: str) -> str:
+            response = client.responses.create(
+                model=model_name,
+                input=[
+                    {
+                        "type": "message",
+                        "role": "system",
+                        "content": [{"type": "input_text", "text": system}],
+                    },
+                    {
+                        "type": "message",
+                        "role": "user",
+                        "content": [{"type": "input_text", "text": user}],
+                    },
+                ],
+                timeout=timeout,
+            )
+            return response.output_text
+
+        return generate
+
+    # ── base hooks ──────────────────────────────────────────────────────────--
+
+    def _serialize(self, active_chat: Chat) -> Any:
+        return active_chat.to_responses_api_chat()
+
+    def _build_optional_kwargs(self, req_tools: Any, req_tool_choice: Any) -> dict[str, Any]:
+        optional_kwargs: dict[str, Any] = {}
+        if req_tools is not None:
+            optional_kwargs["tools"] = req_tools
+        if req_tool_choice is not None:
+            optional_kwargs["tool_choice"] = req_tool_choice
+        return optional_kwargs
+
+    def _request(self, api_input: Any, optional_kwargs: dict[str, Any]) -> Any:
+        return self.client.responses.create(
+            model=self.model_name,
+            input=api_input,
+            stream=self.stream,
+            extra_body=self._extra_body,
+            timeout=self.request_timeout,
+            **optional_kwargs,
+        )
+
+    @staticmethod
+    def _assistant_content(content: Any) -> list[AssistantContent]:
+        return [
+            AssistantContent(type="output_text", text=c.text if c.type == "output_text" else c.refusal) for c in content
+        ]
+
+    def _iter_stream_events(self, api_response: Stream) -> Iterator[ProviderEvent]:
+        for raw_event in api_response:
+            if isinstance(raw_event, ResponseTextDeltaEvent):
+                yield TextDelta(text=raw_event.delta)
+            elif isinstance(raw_event, ResponseOutputItemDoneEvent):
+                item = raw_event.item
+                if isinstance(item, ResponseFunctionToolCall):
+                    item.call_id = _generate_id("call")
+                    item.id = _generate_id("fc")
+                    yield ToolCall(item=item)
+                elif isinstance(item, ResponseOutputMessage):
+                    yield AssistantMessage(content=self._assistant_content(item.content))
+            elif isinstance(raw_event, ResponseCompletedEvent):
+                usage = getattr(raw_event.response, "usage", None)
+                if usage:
+                    yield Usage(input_tokens=usage.input_tokens or 0, output_tokens=usage.output_tokens or 0)
+
+    def _iter_response_events(self, api_response: Any) -> Iterator[ProviderEvent]:
+        usage = api_response.usage
+        if usage:
+            yield Usage(input_tokens=usage.input_tokens or 0, output_tokens=usage.output_tokens or 0)
+        for message in api_response.output:
+            if isinstance(message, ResponseFunctionToolCall):
+                message.call_id = _generate_id("call")
+                message.id = _generate_id("fc")
+                yield ToolCall(item=message)
+            elif isinstance(message, ResponseOutputMessage):
+                yield AssistantMessage(content=self._assistant_content(message.content))
+                # Text-only keeps every character; the base applies remove_unspeechable
+                # for audio. Only output_text parts are spoken (refusals are stored).
+                raw = "".join(c.text for c in message.content if c.type == "output_text")
+                yield TextDelta(text=raw)
+            else:
+                logger.warning(f"Not supported message type: {message.type}")
+
+    def on_session_end(self) -> None:
+        logger.debug("OpenAI API language model session state reset")

speech_to_speech/LLM/text_prompt.py

@@ -0,0 +1,44 @@
+"""Text-channel system prompt: lead + session prompt + tail (strongest constraints last)."""
+
+TEXT_SYSTEM_PROMPT_LEAD = """\
+You are a helpful assistant in a text conversation.
+"""
+
+TEXT_SYSTEM_PROMPT_TAIL = """\
+## Text Rules
+- Write clearly and directly. Match length to the request: concise for simple questions, fuller when the task genuinely needs it.
+- Use markdown when it helps (lists, code blocks, tables, emphasis); don't over-format simple answers.
+- This is a written channel: no spoken-style filler and no action/emote text like *laughs*.
+- Use tools when they help fulfill the request. No preamble sentence is required before a tool call.
+- For slow or external tools, just call the tool and use the result; you don't need to announce it.
+- If unsure whether a tool is needed, just answer directly.
+"""
+
+# Skeleton for the assembled system message (placeholders filled in build_text_system_prompt).
+_TEXT_SYSTEM_PROMPT_FULL = """\
+{lead}
+
+Session Prompt:
+{session_prompt}{optional_tools}
+
+{tail}
+"""
+
+
+def build_text_system_prompt(session_prompt: str, *, tool_section: str = "") -> str:
+    """Context → session prompt → optional tool block → strongest text rules last."""
+    tools = tool_section.strip()
+    optional_tools = f"\n\n{tools}" if tools else ""
+    return _TEXT_SYSTEM_PROMPT_FULL.format(
+        lead=TEXT_SYSTEM_PROMPT_LEAD.rstrip(),
+        session_prompt=session_prompt.strip(),
+        optional_tools=optional_tools,
+        tail=TEXT_SYSTEM_PROMPT_TAIL.rstrip(),
+    )
+
+
+# Full text instructions without a separate session block (legacy / rare direct use).
+TEXT_SYSTEM_PROMPT = "{lead}\n\n{tail}".format(
+    lead=TEXT_SYSTEM_PROMPT_LEAD.rstrip(),
+    tail=TEXT_SYSTEM_PROMPT_TAIL.rstrip(),
+)

speech_to_speech/LLM/tool_call/function_call.py

@@ -0,0 +1,321 @@
+#!/usr/bin/env python3
+"""
+Function parser for extracting function names, parameter names, and values from string function calls.
+
+Uses Python's ``tokenize`` and ``ast`` modules so that nested parentheses,
+strings containing ')' characters, tuples, dicts, etc. are handled correctly.
+"""
+
+import ast
+import io
+import json
+import logging
+import re
+import tokenize
+from collections import OrderedDict
+from typing import Any, Dict, List, Tuple
+
+from openai.types.responses import ResponseFunctionToolCall
+from pydantic import BaseModel
+
+from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
+from speech_to_speech.utils.utils import _generate_id
+
+logger = logging.getLogger(__name__)
+
+_POSITIONAL_RE = re.compile(r"^__arg_\d+__$")
+_LENIENT_CALL_RE = re.compile(
+    r"\b[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)*\s*"
+    r"\((?:[^()\"']+|\"(?:\\.|[^\"])*\"|'(?:\\.|[^'])*')*\)"
+)
+
+
+# ── AST / tokenize helpers ───────────────────────────────────────────
+
+
+def _split_top_level_calls(source: str) -> List[str]:
+    """Split *source* into individual ``name(...)`` expression strings.
+
+    Uses the tokenizer to walk tokens and track parenthesis depth so that
+    nested parens, strings with ')' chars, etc. are handled correctly.
+    """
+    tokens = list(tokenize.generate_tokens(io.StringIO(source).readline))
+    calls: List[str] = []
+    i = 0
+
+    while i < len(tokens):
+        tok = tokens[i]
+        if tok.type != tokenize.NAME:
+            i += 1
+            continue
+
+        start = i
+        j = i + 1
+
+        # Walk past dotted attribute access (e.g. ``mobile.click``)
+        while j + 1 < len(tokens) and tokens[j].string == "." and tokens[j + 1].type == tokenize.NAME:
+            j += 2
+
+        if j >= len(tokens) or tokens[j].string != "(":
+            i += 1
+            continue
+
+        # Track balanced parens
+        depth = 0
+        end = None
+        k = j
+        while k < len(tokens):
+            t = tokens[k]
+            if t.type == tokenize.OP and t.string == "(":
+                depth += 1
+            elif t.type == tokenize.OP and t.string == ")":
+                depth -= 1
+                if depth == 0:
+                    end = k
+                    break
+            k += 1
+
+        if end is None:
+            i += 1
+            continue
+
+        calls.append(tokenize.untokenize(tokens[start : end + 1]).strip())
+        i = end + 1
+
+    return calls
+
+
+def _split_simple_calls_with_regex(source: str) -> List[str]:
+    """Extract complete simple ``name(args)`` spans from malformed model output.
+
+    This fallback can recover well-formed siblings before a tokenizer error,
+    but not the incomplete call that caused the tokenizer error.
+    """
+    return [match.group(0).strip() for match in _LENIENT_CALL_RE.finditer(source)]
+
+
+def _parse_function_exprs(
+    expressions: List[str],
+    pattern_to_match: list[str],
+    *,
+    skip_invalid: bool = False,
+) -> List["FunctionToolCall"]:
+    results: List[FunctionToolCall] = []
+    for expr in expressions:
+        try:
+            call = _parse_call_expr(expr)
+        except Exception:
+            if skip_invalid:
+                continue
+            raise
+        if pattern_to_match and all(pattern not in call.function_name for pattern in pattern_to_match):
+            continue
+        results.append(call)
+    return results
+
+
+def _extract_function_name(node: ast.expr) -> str:
+    """Return the dotted function name from a Call node's ``func`` attribute."""
+    if isinstance(node, ast.Name):
+        return node.id
+    if isinstance(node, ast.Attribute):
+        base = _extract_function_name(node.value)
+        return f"{base}.{node.attr}" if base else node.attr
+    raise ValueError(f"Unsupported function target: {ast.dump(node)}")
+
+
+def _literal_from_ast(node: ast.AST) -> Any:
+    """Convert an AST node to a Python literal value."""
+    if isinstance(node, ast.Constant):
+        return node.value
+
+    if isinstance(node, ast.Name):
+        return node.id
+
+    if isinstance(node, ast.List):
+        return [_literal_from_ast(elt) for elt in node.elts]
+
+    if isinstance(node, ast.Tuple):
+        return [_literal_from_ast(elt) for elt in node.elts]
+
+    if isinstance(node, ast.Dict):
+        return {
+            _literal_from_ast(key): _literal_from_ast(value)
+            for key, value in zip(node.keys, node.values)
+            if key is not None
+        }
+
+    if isinstance(node, ast.UnaryOp) and isinstance(node.op, (ast.USub, ast.UAdd)):
+        value = _literal_from_ast(node.operand)
+        if not isinstance(value, (int, float)):
+            raise ValueError(f"Unsupported unary literal: {ast.dump(node)}")
+        return -value if isinstance(node.op, ast.USub) else value
+
+    raise ValueError(f"Unsupported literal: {ast.dump(node)}")
+
+
+def _parse_call_expr(expr: str) -> "FunctionToolCall":
+    """Parse a single ``name(args...)`` expression string into a FunctionToolCall."""
+    parsed = ast.parse(expr, mode="eval").body
+    if not isinstance(parsed, ast.Call):
+        raise ValueError(f"Expression is not a function call: {expr!r}")
+
+    parameters: "OrderedDict[str, Any]" = OrderedDict()
+
+    for idx, arg in enumerate(parsed.args):
+        parameters[f"__arg_{idx}__"] = _literal_from_ast(arg)
+
+    for kw in parsed.keywords:
+        if kw.arg is None:
+            raise ValueError("**kwargs are not supported")
+        parameters[kw.arg] = _literal_from_ast(kw.value)
+
+    return FunctionToolCall(
+        function_name=_extract_function_name(parsed.func),
+        parameters=parameters,
+        original_string=expr,
+    )
+
+
+# ── Data model ───────────────────────────────────────────────────────
+
+
+class FunctionToolCall(BaseModel):
+    """Represents a parsed function call with its parameters."""
+
+    function_name: str
+    parameters: Dict[str, Any]
+    original_string: str
+    description: str = ""
+
+    def to_realtime_function_tool_call(
+        self,
+        function_tools: list[FunctionTool] | None = None,
+    ) -> ResponseFunctionToolCall:
+        positional = {k for k in self.parameters if _POSITIONAL_RE.match(k)}
+        if positional:
+            logger.warning(
+                "Dropping positional arguments for '%s': %s",
+                self.function_name,
+                positional,
+            )
+        arguments = {k: v for k, v in self.parameters.items() if not _POSITIONAL_RE.match(k)}
+
+        if function_tools is not None:
+            tool = next(
+                (t for t in function_tools if t.name == self.function_name),
+                None,
+            )
+            if tool is None:
+                available = [t.name for t in function_tools]
+                raise ValueError(f"Function '{self.function_name}' not found in available tools: {available}")
+
+            schema = tool.parameters if isinstance(tool.parameters, dict) else {}
+            properties = schema.get("properties", {})
+            required = set(schema.get("required", []))
+
+            undeclared = {k for k in arguments if k not in properties}
+            if undeclared:
+                logger.warning(
+                    "Dropping undeclared parameters for '%s': %s",
+                    self.function_name,
+                    undeclared,
+                )
+                arguments = {k: v for k, v in arguments.items() if k in properties}
+
+            missing = required - set(arguments.keys())
+            if missing:
+                raise ValueError(f"Missing required parameters for '{self.function_name}': {missing}")
+
+        return ResponseFunctionToolCall(
+            name=self.function_name,
+            arguments=json.dumps(arguments),
+            call_id=_generate_id("call"),
+            type="function_call",
+            id=_generate_id("fc"),
+            status="in_progress",
+        )
+
+
+# ── Public API ───────────────────────────────────────────────────────
+
+
+def parse_function_call(function_string: str, pattern_to_match: list[str] = []) -> List[FunctionToolCall]:
+    """Parse a function call string and extract all function calls found.
+
+    Args:
+        function_string: String representation of function calls.
+        pattern_to_match: If non-empty, only calls whose function name
+            contains at least one of these substrings are returned.
+
+    Returns:
+        List of FunctionToolCall objects with parsed information.
+    """
+    function_string = function_string.strip()
+    if not function_string:
+        return []
+
+    try:
+        expressions = _split_top_level_calls(function_string)
+    except tokenize.TokenError:
+        return _parse_function_exprs(
+            _split_simple_calls_with_regex(function_string),
+            pattern_to_match,
+            skip_invalid=True,
+        )
+
+    return _parse_function_exprs(expressions, pattern_to_match)
+
+
+def parse_multiple_functions(function_strings: List[str]) -> List[FunctionToolCall]:
+    """Parse multiple function call strings.
+
+    Args:
+        function_strings: List of function call strings.
+
+    Returns:
+        List of FunctionToolCall objects.
+    """
+    results: List[FunctionToolCall] = []
+    for func_str in function_strings:
+        try:
+            results.extend(parse_function_call(func_str))
+        except Exception:
+            continue
+    return results
+
+
+def extract_function_calls_from_text(text: str, block_regex: str = ".*") -> Tuple[str, List[FunctionToolCall]]:
+    """Extract function calls from delimited code blocks inside *text*.
+
+    The LLM is prompted to wrap tool calls inside code-block delimiters
+    (e.g. ``<code>func(x=1)</code>``).  This function finds those blocks,
+    parses the function calls within them, and returns the remaining text
+    (with blocks stripped) alongside the parsed calls.
+
+    Args:
+        text: Full model output potentially containing code blocks.
+        block_regex: Regex matching the code-block delimiters **and** their
+            content (e.g. ``r"<code>.*?</code>"``).  Only text **inside**
+            matched blocks is scanned for function calls.
+
+    Returns:
+        ``(outside_text, function_calls)`` -- the text with blocks stripped
+        and the parsed function calls found inside the blocks.
+    """
+    if not block_regex:
+        return text, []
+
+    matches = list(re.finditer(block_regex, text, flags=re.DOTALL))
+    if not matches:
+        return text, []
+
+    outside = re.sub(block_regex, "", text, flags=re.DOTALL)
+    inside = " ".join(match.group(0) for match in matches).strip()
+    if not inside:
+        return outside, []
+
+    try:
+        return outside, parse_function_call(inside)
+    except Exception:
+        return outside, []

speech_to_speech/LLM/tool_call/function_tool.py

@@ -0,0 +1,34 @@
+import textwrap
+
+from openai.types.realtime import RealtimeFunctionTool
+
+from speech_to_speech.LLM.tool_call.signature_from_schema import signature_from_schema
+
+
+class FunctionTool(RealtimeFunctionTool):
+    def to_code_prompt(self, include_args_doc: bool = True) -> str:
+        """Generate a code-style prompt string for this function tool.
+
+        Args:
+            include_args_doc: If True, include argument descriptions in the docstring.
+                ⚠️ This lets the model see each argument's purpose but significantly increases
+                token usage (e.g. 906 tokens without vs 3434 with for the default Reachy Mini
+                tool profile). Enable depending on the model's capabilities and context limit.
+        """
+        signature = signature_from_schema(self.parameters)
+
+        tool_doc = self.description or ""
+
+        if isinstance(self.parameters, dict) and include_args_doc:
+            props = self.parameters.get("properties", {})
+            if props:
+                arg_lines = []
+                for arg_name, arg_schema in props.items():
+                    desc = arg_schema.get("description", "") if isinstance(arg_schema, dict) else ""
+                    arg_lines.append(f"{arg_name}: {desc}")
+                args_doc = f"Args:\n{textwrap.indent(chr(10).join(arg_lines), '    ')}"
+                tool_doc += f"\n\n{args_doc}"
+
+        tool_doc = f'"""{tool_doc}\n"""'
+
+        return f"def {self.name}{signature}:\n{textwrap.indent(tool_doc, '    ')}"

speech_to_speech/LLM/tool_call/signature_from_schema.py

@@ -0,0 +1,108 @@
+import inspect
+from typing import Any, Literal, Union
+
+JSON_TYPE_TO_PYTHON_TYPE = {
+    "string": str,
+    "number": float,
+    "boolean": bool,
+    "integer": int,
+    "object": dict,
+    "array": list,
+    "null": type(None),
+}
+
+
+def _dedupe_types(types: list[Any]) -> list[Any]:
+    seen = []
+    for t in types:
+        if t not in seen:
+            seen.append(t)
+    return seen
+
+
+def _annotation_from_spec(spec: dict[str, Any]) -> Any:
+    if not spec or not isinstance(spec, dict):
+        return Any
+
+    # const → Literal[value]
+    if "const" in spec:
+        return Literal[spec["const"]]
+
+    # enum → Literal[val1, val2, ...]
+    if "enum" in spec:
+        values = spec["enum"]
+        if not values:
+            return Any
+        return Literal[tuple(values)]
+
+    # anyOf / oneOf → Union[Type1, Type2, ...]
+    for key in ("anyOf", "oneOf"):
+        if key in spec:
+            variants = [_annotation_from_spec(s) for s in spec[key]]
+            unique = _dedupe_types(variants)
+            if len(unique) == 0:
+                return Any
+            if len(unique) == 1:
+                return unique[0]
+            return Union[tuple(unique)]
+
+    # allOf → merge sub-schemas then resolve
+    if "allOf" in spec:
+        merged = {}
+        for sub in spec["allOf"]:
+            merged.update(sub)
+        return _annotation_from_spec(merged)
+
+    json_type = spec.get("type")
+
+    if json_type is None:
+        return Any
+
+    # type as list, e.g. ["string", "null"] → Union[str, None] (Optional)
+    if isinstance(json_type, list):
+        types = [JSON_TYPE_TO_PYTHON_TYPE.get(t, Any) for t in json_type]
+        unique = _dedupe_types(types)
+        if len(unique) == 0:
+            return Any
+        if len(unique) == 1:
+            return unique[0]
+        return Union[tuple(unique)]
+
+    # array with items → list[ItemType]
+    if json_type == "array" and "items" in spec:
+        item_type = _annotation_from_spec(spec["items"])
+        return list[item_type]  # type: ignore[valid-type]
+
+    return JSON_TYPE_TO_PYTHON_TYPE.get(json_type, Any)
+
+
+def signature_from_schema(schema: object | None) -> inspect.Signature:
+    if not schema or not isinstance(schema, dict):
+        return inspect.Signature()
+
+    props = schema.get("properties", {})
+    required = set(schema.get("required", []))
+    params = []
+
+    for name, spec in props.items():
+        annotation = _annotation_from_spec(spec)
+
+        has_schema_default = "default" in spec if isinstance(spec, dict) else False
+
+        if name in required and not has_schema_default:
+            default = inspect.Parameter.empty
+        elif has_schema_default:
+            default = spec["default"]
+        else:
+            default = None
+
+        params.append(
+            inspect.Parameter(
+                name,
+                inspect.Parameter.POSITIONAL_OR_KEYWORD,
+                default=default,
+                annotation=annotation,
+            )
+        )
+
+    return inspect.Signature(params)

speech_to_speech/LLM/tool_call/tool_prompt.py

@@ -0,0 +1,111 @@
+"""
+Optional system-prompt builder that instructs a local LLM to output tool calls
+inside delimited code blocks (e.g. ``<code>func(arg='val')</code>``).
+
+The prompt is rendered from a Jinja2 template and relies on
+``FunctionTool.to_code_prompt()`` to expose each tool's Python-style signature.
+"""
+
+import re
+
+from jinja2 import Template
+
+from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
+
+# ---------------------------------------------------------------------------
+# Default delimiters
+# ---------------------------------------------------------------------------
+ENTER_CODE = "<code>"
+END_CODE = "</code>"
+
+# ---------------------------------------------------------------------------
+# Jinja2 template
+# ---------------------------------------------------------------------------
+# ``enter_code`` / ``end_code`` are the block delimiters the model must wrap
+# every tool call in.  ``tools`` is a list of FunctionTool instances whose
+# ``.to_code_prompt()`` is called inside the template.
+# ---------------------------------------------------------------------------
+
+TOOL_PROMPT_TEMPLATE = Template(
+    """\
+Available tools:
+
+{% for tool in tools %}
+{{ tool.to_code_prompt() }}
+
+{% endfor %}
+To call a tool, put exactly one named-argument function call inside {{ enter_code }}...{{ end_code }}:
+{{ enter_code }}function_name(required_arg='value'){{ end_code }}
+
+Rules:
+- You may say one brief natural sentence before the tool call; for slow information tools, briefly say that you will check.
+- For expression/background tools, always speak first. For requested expressions, use a short pattern like "Sure, here's my best <emotion>."; otherwise use a fitting empathetic sentence.
+- Do not mention tags, functions, or tools. Keep prose outside tags brief, and do not claim tool results before a tool result is available.
+- Use named arguments only; quote strings. Omit optional args instead of placeholder values like "random", "none", "", or null.
+- Only one tool call may appear in a response.\
+""",
+    keep_trailing_newline=True,
+)
+
+# Text-channel variant: same call format and structural rules, without the
+# voice-specific "speak first" prose.
+TEXT_TOOL_PROMPT_TEMPLATE = Template(
+    """\
+Available tools:
+
+{% for tool in tools %}
+{{ tool.to_code_prompt() }}
+
+{% endfor %}
+To call a tool, put exactly one named-argument function call inside {{ enter_code }}...{{ end_code }}:
+{{ enter_code }}function_name(required_arg='value'){{ end_code }}
+
+Rules:
+- Call a tool directly when it helps fulfill the request; no preamble sentence is required.
+- Do not mention tags, functions, or tools in your prose, and do not claim tool results before a tool result is available.
+- Use named arguments only; quote strings. Omit optional args instead of placeholder values like "random", "none", "", or null.
+- Only one tool call may appear in a response.\
+""",
+    keep_trailing_newline=True,
+)
+
+
+# ---------------------------------------------------------------------------
+# Public helpers
+# ---------------------------------------------------------------------------
+
+
+def build_tool_system_prompt(
+    tools: list[FunctionTool],
+    enter_code: str = ENTER_CODE,
+    end_code: str = END_CODE,
+    *,
+    text_only: bool = False,
+) -> str:
+    """Render the tool-calling system-prompt section.
+
+    Returns an empty string when *tools* is empty so it can be
+    unconditionally appended to a base system prompt.  When *text_only* is set,
+    the written-channel variant is used (no voice "speak first" prose).
+    """
+    if not tools:
+        return ""
+
+    template = TEXT_TOOL_PROMPT_TEMPLATE if text_only else TOOL_PROMPT_TEMPLATE
+    return template.render(
+        tools=tools,
+        enter_code=enter_code,
+        end_code=end_code,
+    )
+
+
+def build_block_regex(
+    enter_code: str = ENTER_CODE,
+    end_code: str = END_CODE,
+) -> str:
+    """Build a regex that matches a single code block (non-greedy).
+
+    >>> build_block_regex("<code>", "</code>")
+    '<code>.*?</code>'
+    """
+    return f"{re.escape(enter_code)}.*?{re.escape(end_code)}"

speech_to_speech/LLM/utils.py

@@ -0,0 +1,75 @@
+import base64
+import io
+import re
+from typing import Optional
+
+import requests  # type: ignore[import-untyped]
+from PIL import Image
+
+SMART_PUNCT_TRANSLATION = str.maketrans(
+    {
+        "\u2018": "'",
+        "\u2019": "'",
+        "\u201c": '"',
+        "\u201d": '"',
+    }
+)
+
+SPEECHABLE_PATTERN = re.compile(
+    r"[^\w\s.,!?;:'\"\-()\/\\@#%&*+=$€£¥₹₽¢\[\]{}<>~`^|…—–\n\r\t]",
+    flags=re.UNICODE,
+)
+
+
+def remove_unspeechable(text: str) -> str:
+    """Keep only speechable characters: letters, digits, punctuation, whitespace.
+    support unicode characters (english, arabic, chinese, japanese, korean, etc.)
+    """
+    text = text.translate(SMART_PUNCT_TRANSLATION)
+    return SPEECHABLE_PATTERN.sub("", text)
+
+
+WHISPER_LANGUAGE_TO_LLM_LANGUAGE = {
+    "en": "english",
+    "fr": "french",
+    "es": "spanish",
+    "zh": "chinese",
+    "ja": "japanese",
+    "ko": "korean",
+    "hi": "hindi",
+    "de": "german",
+    "pt": "portuguese",
+    "pl": "polish",
+    "it": "italian",
+    "nl": "dutch",
+}
+
+
+def resolve_auto_language(language_code: Optional[str]) -> tuple[Optional[str], Optional[str]]:
+    """Strip the ``-auto`` suffix and resolve the human-readable language name.
+
+    Returns ``(clean_code, language_name)``.  ``language_name`` is non-None
+    when the code (with or without ``-auto``) maps to a known language.
+    """
+    if not language_code:
+        return language_code, None
+    if language_code.endswith("-auto"):
+        language_code = language_code[:-5]
+    if language_code not in WHISPER_LANGUAGE_TO_LLM_LANGUAGE:
+        return language_code, None
+    return language_code, WHISPER_LANGUAGE_TO_LLM_LANGUAGE.get(language_code)
+
+
+def image_url_to_pil(image_url: str) -> Image.Image:
+    """Convert an image URL or base64 data URI to a PIL Image.
+
+    Accepts:
+    - 'data:image/...;base64,<b64>' data URIs
+    - 'https://...`` or ``http://...' URLs (fetched with a 10s timeout)
+    """
+    if image_url.startswith("data:"):
+        _, b64_data = image_url.split(",", 1)
+        return Image.open(io.BytesIO(base64.b64decode(b64_data)))
+    resp = requests.get(image_url, timeout=10)
+    resp.raise_for_status()
+    return Image.open(io.BytesIO(resp.content))

speech_to_speech/LLM/voice_prompt.py

@@ -0,0 +1,48 @@
+"""Voice-channel system prompt: lead + session prompt + tail (strongest constraints last)."""
+
+VOICE_SYSTEM_PROMPT_LEAD = """\
+You are in a spoken conversation. The user speaks and hears you.
+The session prompt defines persona, facts, goals, and tool descriptions. These channel rules only control spoken output and tool-use behavior.
+"""
+
+VOICE_SYSTEM_PROMPT_TAIL = """\
+## Voice Rules
+- Keep replies brief by default: usually one spoken sentence, two if needed. Go longer only when asked.
+- Speak naturally. No markdown, bullets, headings, visual formatting, or action/emote text like *laughs*.
+- Treat transcripts as noisy. Correct likely mishearings only if asked or meaning depends on it.
+- Speech is the default. Use at most one tool when it helps fulfill the request or clearly fits the moment.
+- Before a tool call, use a brief natural utterance unless the user asked for silence or tool-only output. For slow information tools, briefly say that you will check.
+- For expression/background tools, speak first. If asked to show an expression, use a short pattern like "Sure, here's my best <emotion>." Otherwise use a fitting empathetic sentence. Never mention tools.
+- After completed expression/background/physical-action tools, do not add a second spoken comment unless the result has user-facing information.
+- Use motion, dance, emotion, and similar tools sparingly when they add empathy, celebration, playfulness, or a requested physical action.
+- If unsure whether a tool is needed, just speak.
+"""
+
+# Skeleton for the assembled system message (placeholders filled in build_voice_system_prompt).
+_VOICE_SYSTEM_PROMPT_FULL = """\
+{lead}
+
+Session Prompt:
+{session_prompt}{optional_tools}
+
+{tail}
+"""
+
+
+def build_voice_system_prompt(session_prompt: str, *, tool_section: str = "") -> str:
+    """Context → session prompt → optional tool block → strongest voice rules last."""
+    tools = tool_section.strip()
+    optional_tools = f"\n\n{tools}" if tools else ""
+    return _VOICE_SYSTEM_PROMPT_FULL.format(
+        lead=VOICE_SYSTEM_PROMPT_LEAD.rstrip(),
+        session_prompt=session_prompt.strip(),
+        optional_tools=optional_tools,
+        tail=VOICE_SYSTEM_PROMPT_TAIL.rstrip(),
+    )
+
+
+# Full voice instructions without a separate session block (legacy / rare direct use).
+VOICE_SYSTEM_PROMPT = "{lead}\n\n{tail}".format(
+    lead=VOICE_SYSTEM_PROMPT_LEAD.rstrip(),
+    tail=VOICE_SYSTEM_PROMPT_TAIL.rstrip(),
+)