last

docs/PROJECT_OVERVIEW.md

@@ -42,6 +42,8 @@ Dynamic knowledge base updates are exposed over the same HTTP server via a 11-en
 | **Automatic language detection** | Via `lingua-language-detector` for the assistant reply language. |
 | **Live partial transcription** | Parakeet-TDT emits live partial transcriptions every 500 ms so clients can display "user is speaking …" text. |
 | **Full tool-calling support** | With `responses-api` backend: streamed function argument delivery, parallel tool calls, custom voices over TTS — everything compliant with the OpenAI Responses API surface. |
+| **Always-fresh dynamic tool calls** | Dual highest-priority system prompt rule (voice & text channels) guarantees that any dynamic/live information tool such as `get_user_briefing`, weather, calendar, status, search, and database lookups is **re-invoked on every user request** — never answered from chat history. The rule is injected twice (prompt head + prompt tail) using 🔴 MANDATORY blocks to maximise compliance with modern instruction-tuned LLMs. |
+| **Internal-marker guard on TTS path** | Two-layer filter (in `LMOutputProcessor` + per-TTS handler) automatically suppresses utterances that contain internal control markers such as `<<<APIPILOT_CALL:...>>>`, so the assistant never literally speaks tool-call directives while still routing them over the text channel. |
 | **Optional RAG server-side injection** | Transparent retrieval hook on every LLM turn. Sentence-Transformers embeddings, NPZ-index persistence, configurable top-k / cosine threshold, system or user-message injection. Multilingual by default (Spanish / Italian / English out of the box). |
 | **Dynamic Knowledge Base REST API** | 11 endpoints to list, search, upsert, update, remove, and reload KB contents at runtime — including cross-restart persistence through `kb/_dynamic.jsonl`. |
 | **Structured CLI / JSON configuration** | Every parameter is a `@dataclass` HfArgumentParser argument with sane defaults, or a full JSON config file you can pass in as single argument. |
@@ -93,6 +95,7 @@ Key invariants:
 - **The LLM stage never sees raw audio**. The pipeline only forwards text buffers to the LLM handler, so model vendors and backend switches remain transparent.
 - **The RAG stage is a pure side-effect on the text buffer**: it reads the most recent user text, runs an embedding + cosine top-k + threshold filter, and prepends matching knowledge as an extra system (or user) message — with a clear log line so you can always see exactly what was injected.
 - **TTS streams** (all modern backends): audio deltas are emitted as they are synthesised, so the first byte of a reply reaches the speaker well before the LLM has finished generating text.
+- **Internal markers never reach the speaker**: `LMOutputProcessor` (LLM→TTS handover) **and** each individual TTS handler run `contains_internal_markers()` from [`utils.py`](file:///home/azurian/speech-to-speech/src/speech_to_speech/utils/utils.py#L60-L86) and short-circuit any chunk that contains tool-call payloads such as `APIPILOT_CALL:...`. Markers still flow through the text side-channel so clients can react to tool invocations.
 
 ---
 
@@ -144,6 +147,8 @@ Configurable via [ModuleArguments.mode](file:///home/azurian/speech-to-speech/sr
 
 All TTS backends are reachable through a common streaming interface so the rest of the pipeline is backend-agnostic.
 
+Every TTS handler, together with the upstream [`lm_output_processor.py`](file:///home/azurian/speech-to-speech/src/speech_to_speech/LLM/lm_output_processor.py), enforces the **internal-marker invariant**: any utterance containing tool-call control payloads such as `<<<APIPILOT_CALL:NAME=...,ARGUMENTS=...>>>` is filtered out **before** audio synthesis begins. The marker patterns live in a single place: [`utils/utils.py`](file:///home/azurian/speech-to-speech/src/speech_to_speech/utils/utils.py#L60-L86) (see `contains_internal_markers`).
+
 ---
 
 ## 6. RAG Server-Side (optional add-on)
@@ -272,6 +277,7 @@ speech-to-speech/
 │   ├── s2s_pipeline.py                                ← entry point (main), pipeline builder, realtime pool
 │   ├── baseHandler.py
 │   ├── chat.py
+│   ├── utils/utils.py                                 ← shared helpers: `response_wants_audio`, `contains_internal_markers` (anti-APIPILOT filter)
 │   ├── pipeline/                                      ← queue types, CancelScope, handler types
 │   │
 │   ├── arguments_classes/                             ← @dataclass HfArgumentParser args (one per handler)