# Audio Agent Engine — Updates

This file documents all notable changes to the audio-agent-engine project.
Entries are written in English and ordered newest-first.

---

## 2026-10-21 — Feat: demo routes audio tool results to the external source

### Changed

- **Cue asset layout** — filler, start-filler and backchannel banks now live
  under `<base>/<lang>/<voice_profile>/` instead of `<base>/<lang>/`, so
  multiple TTS voice configurations can coexist on disk without overwriting one
  another.
- **[`src/config.rs`](src/config.rs)** — added a resolved cue-asset voice
  profile (`Config::cue_voice_profile()`), with optional explicit override via
  `ASSET_VOICE_PROFILE`. When unset, the engine derives the profile from the
  active voice setup (prefer `TTS_VOICE` when it is not `default`, otherwise
  fall back to the reference-audio file stem, then to `default`).
- **[`src/assets.rs`](src/assets.rs)** / **[`src/main.rs`](src/main.rs)** /
  **[`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)** /
  **[`src/ws/session.rs`](src/ws/session.rs)** — cue-bank bootstrap, runtime
  language switching and cue metadata now all resolve the same
  `<lang>/<voice_profile>` directory layout.
- **Legacy cue-bank migration** — when old single-voice banks still exist
  directly under `<base>/<lang>/`, startup now copies their flat files
  (`*.raw`, `texts.txt`, `.voice_signature`) into the active
  `<base>/<lang>/<voice_profile>/` directory if the target files are missing.
  The legacy files are kept in place.
- **[`run/refresh_voice_assets.sh`](run/refresh_voice_assets.sh)** — the voice
  asset refresh script now targets the current cue voice profile and refreshes
  directories under `<base>/<lang>/<voice_profile>/`, keeping other voice
  profiles intact.
- **[`README.md`](README.md)** — documented the new cue-bank directory layout
  and the optional `ASSET_VOICE_PROFILE` override.
- **[`sdk/demo.html`](sdk/demo.html)** — a tool definition may now carry an
  extra `audio: true` flag (ignored by the backend, which only reads
  `name`/`description`/`parameters`/`url`/`method`/`headers`). When the model
  calls such a tool, the demo rebuilds the call's effective URL (arguments of
  read-style verbs are appended as query parameters, mirroring the backend's
  argument placement), writes it into the external audio source field and
  starts playback on the engine's dedicated channel. `get_today_podcast_mp3` is
  flagged this way, so a podcast request is played back instead of only being
  returned to the model. The manual Play button now shares the same
  `playExternalSource()` helper.
- **[`sdk/demo.html`](sdk/demo.html)** — the default session prompt and the
  built-in tool descriptions now explicitly tell the model to call
  `get_today_podcast_mp3` on the first relevant podcast/audio request and
  `get_public_day_report` on the first relevant report/alerts request, to
  reduce cases where the user has to repeat the same intent before tool calling
  is triggered.
- **Markdown document mode** — when the user explicitly asks for a Markdown
  document, the demo prompt now instructs the model to keep a very short spoken
  preface outside the block and place the full document inside
  `<chat_markdown>...</chat_markdown>`.
- **[`sdk/demo.html`](sdk/demo.html)** — audio tool results are queued on
  `tool_call` and autoplay after a fixed post-`assistant_end` delay, rather
  than waiting for the SDK player to become idle. This avoids false timeouts in
  deployments where the engine keeps sending mixed 20 ms PCM frames
  continuously, even during silence, while keeping the post-reply timing
  deterministic.
- **[`src/tools.rs`](src/tools.rs)** / **[`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)** —
  the engine now strips `<chat_markdown>...</chat_markdown>` blocks out of the
  spoken stream, emits them separately as `assistant_markdown` events, and
  keeps the document text in the assistant history for follow-up turns.
- **[`sdk/demo.html`](sdk/demo.html)** — the SDK demo renders
  `assistant_markdown` payloads as formatted Markdown bubbles in chat instead
  of showing raw source text.
- **[`sdk/audio-agent-client.js`](sdk/audio-agent-client.js)** /
  **[`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts)** — the SDK now
  exposes the `assistant_markdown` event type for clients that want chat-only,
  non-TTS Markdown documents.
- **[`sdk/audio-agent-client.js`](sdk/audio-agent-client.js)** /
  **[`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts)** — `PcmPlayer`
  now exposes `isIdle()` and emits a local `idle` event when its scheduled
  audio has fully drained, giving browser clients a reliable post-playback
  signal.

---

## 2026-10-21 — Chore: voice-asset refresh helper script

### Added

- **[`run/refresh_voice_assets.sh`](run/refresh_voice_assets.sh)** — bash helper
  that reads the current `.env`, analyzes the active TTS voice / reference /
  sampling / cue-text settings, computes a voice signature, inspects every cue
  bank (`fillers`, `start_filler`, `backchannel`) across the discovered
  languages, and decides which generated assets are stale.
- The script stores a per-directory `.voice_signature`, supports `--dry-run`,
  `--force`, `--lang <code>`, `--base-lang-only` and `--no-restart`, deletes the
  generated `*.raw` clips (plus `texts.txt` for text-driven banks) when a
  refresh is needed, restarts `pm2-piperust`, and writes the updated signature
  markers back to the asset directories.

---

## 2026-10-21 — Chore: SDK demo installs a default day-report tool

### Changed

- **[`sdk/demo.html`](sdk/demo.html)** — the demo now installs a default
  runtime tool set through `POST /api/tools` right after connect:
  `get_public_day_report`, mapped to
  `GET https://aquachile.azurianlab.com/api/v1/day-report` with query
  parameters `limit` and `date`, plus `get_today_podcast_mp3`, mapped to
  `GET https://aquachile.azurianlab.com/api/v1/podcast/today/mp3` with query
  parameter `download`. This lets the demo exercise real tool calling without
  relying on an external `LLM_TOOLS_FILE`.

---

## 2026-10-21 — Feat: phone-login proxy (`POST /api/auth/phone-login`)

### Added

- **`POST /api/auth/phone-login`** ([`src/auth_api.rs`](src/auth_api.rs)) — a
  server-side proxy that forwards `{ "phoneNumber": "…" }` to the AquaChile auth
  API (`AUTH_API_BASE` + `AUTH_PHONE_LOGIN_PATH`) and returns its status and body
  unchanged (on success: `sessionToken` + `user`). It exists because that API
  rejects browser calls from the client origin (CORS preflight `400 Disallowed
  CORS origin`, no `Access-Control-Allow-Origin`), so a direct `fetch` from the
  demo is blocked; going through the engine makes the call same-origin. When the
  request also carries a live `session_id`, the engine stores a sanitized user
  profile (`name`, `email`, `businessRoles`, `permissionSet`, `centers`,
  `permissions`) as hidden context for that session's LLM. Errors: `400` for an
  empty/invalid body or missing `phoneNumber`, `404` for an unknown
  `session_id`, `502` when the upstream is unreachable.
- Config `AUTH_API_BASE` (default `https://aquachile.azurianlab.com`) and
  `AUTH_PHONE_LOGIN_PATH` (default `/api/v1/auth/phone-login`).
- Swagger UI (`/docs`) documents the endpoint (new `auth` tag,
  `PhoneLoginRequest` schema).

### Changed

- **[JavaScript SDK](sdk/audio-agent-client.js)** — added
  `phoneLogin(phoneNumber, sessionId?)` to `AudioAgentRest`; types updated in
  [`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts) and the reference
  in [`sdk/README.md`](sdk/README.md).
- **[`sdk/demo.html`](sdk/demo.html)** — at session start it reads `phoneNumber`
  from the page URL (`?phoneNumber=+44…`) and authenticates through the engine
  proxy, showing the resolved user in the header, a chat chip and the event log.
  (Replaces the earlier direct cross-origin `fetch`, which the browser blocked.)
- **Hidden LLM user context** — the session user block now instructs the model
  to treat `permissions`, `permissionSet`, `businessRoles` and `centers` as
  access/domain scope, avoid fabricating unavailable access, constrain KPI /
  report / alert interpretation to the verified user scope, and suggest next
  steps aligned with the user's role.

---

## 2026-10-21 — Feat: per-session system prompt (`GET` / `POST /api/prompt`)

### Added

- **Server-assigned session id** — every WebSocket session is given an id
  (`s-<epoch_ms>-<seq>`, the same stem used for its audit log) by a new
  per-session prompt registry ([`src/prompt.rs`](src/prompt.rs)) and the id is
  pushed to the client right after connect, before `cues`, as
  `{"type":"session","id":"…"}` ([`src/ws/session.rs`](src/ws/session.rs)).
  `SessionAudit::new` now receives that id instead of generating its own, so the
  audit file and the live session share one identifier.
- **`GET` / `POST /api/prompt`** ([`src/prompt_api.rs`](src/prompt_api.rs)) — read
  or override the LLM system prompt at runtime, **per session** (with a
  `session_id`) or process-wide (without one). `GET` returns the effective prompt
  plus metadata (`source`: `session` / `global` / `env` / `default` / `none`,
  `has_override`, `direct_answers_applied`); `POST` installs an override
  (`prompt: null` / `""` clears it). Overrides live **in memory only** — a
  restart restores the configured default and a session override is dropped on
  disconnect; an unknown session id returns `404`.
- The LLM turn now resolves the system prompt through the registry
  ([`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)):
  session override → process-wide override → `LLM_SYSTEM_PROMPT` → built-in
  default. With `/api/prompt` never called the behaviour is unchanged.
- `Config::llm_system_prompt_from_env` records whether the configured prompt came
  from `LLM_SYSTEM_PROMPT` (reported as `source: "env"`).
- Swagger UI (`/docs`) documents the new endpoint (new `prompt` tag, schemas
  `PromptResponse` / `SetPromptRequest`).

### Changed

- **[JavaScript SDK](sdk/audio-agent-client.js)** — added the `session` event to
  `CLIENT_EVENTS` and `getPrompt(sessionId?)` / `setPrompt(prompt, sessionId?)`
  to `AudioAgentRest`; types updated in
  [`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts) and the reference
  in [`sdk/README.md`](sdk/README.md).
- **[`sdk/demo.html`](sdk/demo.html)** — captures the session id from the
  `session` event and adds a **System prompt** panel to reload / override / clear
  the prompt of the current session over REST. It also applies a built-in
  `SESSION_PROMPT` (the Vocero / AquaChile assistant instructions) automatically
  at the start of every session, right after the `session` event.

---

## 2026-10-21 — Chore: English-only code and documentation convention

### Changed

- **All code is now English-only.** Every comment, doc-comment, error/log string
  and test description across `src/` and `sdk/` was reviewed and the remaining
  Italian was translated — e.g. [`src/stt/tokenizer.rs`](src/stt/tokenizer.rs).
  Per-language *content* is intentionally left as-is: the voice-cloning reading
  phrases in [`src/customvoice.rs`](src/customvoice.rs) and the clip phrase
  tables in [`src/lang.rs`](src/lang.rs) must stay in their own language.
- **Documentation is now English-only.** [`TOOL_CALLING.md`](TOOL_CALLING.md) and
  [`sdk/README.md`](sdk/README.md) dropped their `Italiano` / `Español`
  sections and [`README.md`](README.md) examples switched to English sample text.
  This supersedes the previous IT/EN/ES documentation convention.

---

## 2026-10-21 — Feat: JavaScript client SDK + opt-in CORS

### Added

- **`sdk/audio-agent-client.js`** — a dependency-free, single-file ESM client
  SDK (browser + Node) with types in
  [`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts) and a
  reference in [`sdk/README.md`](sdk/README.md). It exists so the
  client can be **decoupled** from the engine and hosted on another server:
  - `AudioAgentClient` — the realtime `/ws` session: `sendAudio()` (accepts
    `Int16Array` / `Float32Array` / `ArrayBuffer` / view / `number[]`, normalized
    to 16-bit LE mono 44.1 kHz), typed controls (`sendUserText`, `interrupt`,
    `playAudioSource`, `stopAudioSource`, `setAudioSourceVolume`,
    `setReference`, `clearReference`, `speak`), optional auto-reconnect with
    exponential backoff, and one event per server message (`cues`, `user`,
    `assistant_delta`, `assistant_end`, `cue`, `barge_in`, `tool_call`,
    `audio_source`, `error`) plus a local `audio` event with the decoded frames.
  - `AudioAgentRest` — typed REST wrapper: `health`, `getTools` / `setTools`,
    `getCustomVoicePhrases` / `setCustomVoice` / `clearCustomVoice`,
    `transcribe`, `openapi`; failures throw an error carrying `status`/`body`.
  - Browser audio helpers `MicCapture` (mic → 882-sample `Int16Array` frames via
    an inline AudioWorklet), `PcmPlayer` (jitter-buffered playback with
    `flush()` for barge-in) and `AudioSession` (wires client + mic + player).
  - PCM utilities `int16ArrayFromBytes`, `float32ToInt16`, `int16ToFloat32`,
    `toInt16` and the `AUDIO` constants.
- **`GET /sdk/audio-agent-client.js`** ([`src/main.rs`](src/main.rs),
  [`src/web/mod.rs`](src/web/mod.rs)) — serves the SDK (embedded at compile
  time, like the HTML clients) so a client hosted elsewhere can fetch it.
- **`sdk/demo.html`** + **`GET /sdk/demo.html`** — a complete demo client built
  on the SDK (English UI, like all the code in this project): connect/disconnect,
  live session state, streaming
  agent replies with word-level *karaoke* highlighting, user/agent chat bubbles,
  inline chips for `cue` / `tool_call` / `audio_source` events, a barge-in
  counter, a microphone toggle with a live level meter, a text-turn input, an
  external audio source (play / stop / volume) and a raw event log. It imports
  `./audio-agent-client.js`, so it works both served by the engine and copied to
  another origin; it warns when opened outside a secure context (no mic on
  `file://`).
- **Opt-in CORS** — `CORS_ALLOW_ORIGIN` (comma-separated origins, or a single
  `*`) adds a [`tower-http`](https://docs.rs/tower-http) CORS layer
  ([`src/main.rs`](src/main.rs), [`src/config.rs`](src/config.rs)) for browser
  clients on another origin. Empty (the default) adds **no** layer, so the
  engine behaves exactly as before. The WebSocket `/ws` is unaffected (a
  handshake is not a CORS request).

### Notes

- No build step for the SDK: copy the file, or import it from the engine origin.
- `tower-http` was already a transitive dependency; the `cors` feature adds no
  new crates.
- Docs updated: [README.md](README.md#client-sdk-javascript) (new section, route
  table, env table, feature bullet), [`.env.example`](.env.example), and the new
  [`sdk/README.md`](sdk/README.md) (IT/EN/ES).

---

## 2026-10-21 — Feat: every HTTP verb for tools (list / get / add / update / delete / …)

### Changed

- **`method` now accepts any HTTP verb** ([`src/tools.rs`](src/tools.rs)),
  case-insensitive: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`,
  `TRACE` and custom verbs (e.g. `PURGE`). Before this change anything other than
  `GET` fell through to `POST`, so `PUT`/`PATCH`/`DELETE` tools silently became
  POSTs.
- **Argument placement follows the verb.** The arguments left after the URL
  placeholders go to the **query string** for the read-style verbs
  (`GET`, `HEAD`, `OPTIONS`, `TRACE`, `DELETE`) and to the **JSON body** for
  every other verb (`POST`, `PUT`, `PATCH`, custom); body-style verbs always send
  a body (`{}` when nothing is left). This makes a CRUD surface expressible:
  `GET /items?limit=10`, `GET /items/{id}`, `POST /items`,
  `PATCH|PUT /items/{id}`, `DELETE /items/{id}`, `HEAD /items/{id}`.
- **New helpers** `http_method()`, `args_in_body()` and `place_args()` in
  [`src/tools.rs`](src/tools.rs); `ToolRegistry::execute` now builds the request
  through `Client::request(method, url)` instead of separate `get`/`post`
  branches.
- An invalid verb is **warned at load** and, at execution time, returned to the
  model as `Error: invalid HTTP method '<verb>' for tool '<name>'` (never a
  panic, never a blocking failure).

### Notes

- `DELETE` is **query-style**: put the id in the URL (`/items/{id}`) and any
  extra argument ends up as a query parameter.
- Tests: two new unit tests (`every_http_verb_is_accepted`,
  `args_placement_follows_the_verb`) — `cargo test --bin audio-agent-engine --
  tools::` → 9 passed.
- Docs updated: [TOOL_CALLING.md](TOOL_CALLING.md) (IT/EN/ES, per-verb argument
  table + full CRUD example set) and [README.md](README.md#tool-calling).

---

## 2026-10-21 — Feat: runtime tool definitions — `GET` / `POST /api/tools`

### Added

- **New REST endpoint `POST /api/tools`** ([`src/tools_api.rs`](src/tools_api.rs)).
  The body is the **same JSON array** used by `LLM_TOOLS_FILE`; it installs a new
  set of tool definitions at runtime, **without a restart** (until now the file
  was only read once at startup). The definitions apply from the **next turn**
  on.
- **`GET /api/tools`** reports the active definitions (in the OpenAI `tools`
  shape they are advertised to the model with) plus their origin: `source` is
  `file`, `request` or `none`.
- **Errors:** `400` empty body / non-UTF-8 / invalid JSON / JSON array with no
  tool definition, `503` LLM disabled. Responses use the shared
  `{ok, …}` / `{ok:false,error}` shapes.
- **OpenAPI/Swagger UI:** both handlers are documented (tag `tools`, schema
  `ToolsResponse`) in [`src/apidoc.rs`](src/apidoc.rs) and appear at `/docs`.

### Changed

- **`ToolRegistry` can be built from a string** — `ToolRegistry::from_json`
  ([`src/tools.rs`](src/tools.rs)) is the new core builder; `load` now reads the
  file and delegates to it, so a body accepted by the endpoint is also valid as
  an `LLM_TOOLS_FILE` (and vice versa). Added `len()` / `names()`.
- **The LLM client's registry is swappable** — `LlmClient::tools` moved from
  `Option<Arc<ToolRegistry>>` to an `Arc<RwLock<…>>` with `set_tools(…)` /
  `tools_from_request()` ([`src/llm.rs`](src/llm.rs)). The registry is re-read on
  **every** request and at the start of every turn
  ([`turn_manager.rs`](src/orchestrator/turn_manager.rs)), so an override takes
  effect on the next turn while a session in flight is unaffected.
- Docs: new [TOOL_CALLING.md](TOOL_CALLING.md) (IT/EN/ES operating guide) and a
  "Runtime definitions" subsection in [README.md](README.md#tool-calling).

### Notes

- The override lives **in memory only**: it never writes the tools file, and a
  restart restores the `LLM_TOOLS_FILE` definitions. When the endpoint is never
  called, the file definitions stay in use — that remains the default.
- Verified on the running service: `GET` before any POST →
  `{"count":2,"source":"file"}`; after a POST → `{"count":1,"source":"request"}`;
  invalid JSON and `[]` → `400`; restart → back to
  `{"count":2,"source":"file"}`.

---

## 2026-10-17 — Feat: `POST /api/transcribe` — Parakeet transcription of an uploaded audio file

### Added

- **New REST endpoint `POST /api/transcribe`** ([`src/transcribe.rs`](src/transcribe.rs)).
  The body is the raw audio file (any container `ffmpeg` can read: wav, mp3,
  m4a, ogg, flac…). The file is decoded to **mono 16-bit PCM at 16 kHz**
  (`ffmpeg -ac 1 -ar 16000 -f s16le pipe:1`) and the **already-loaded** Parakeet
  TDT model from [`AppState::stt`](src/ws/handler.rs) transcribes the whole take
  in a single inference on a blocking thread (the inference is CPU-bound and
  holds the ONNX session lock). Response:
  `{"ok": true, "text": …, "duration_ms": …, "tokens": …}`.
- **Errors:** `400` empty body, `503` STT model not loaded, `502` ffmpeg decode
  failed, `500` temp-file / inference failure. Uploads capped at **32 MB** (same
  as `/api/customvoice`).
- **OpenAPI/Swagger UI:** the endpoint is documented (tag `transcribe`, schema
  `TranscribeResponse`) in [`src/apidoc.rs`](src/apidoc.rs) and appears at
  `/docs`.

### Notes

- **Whole-file, single transcript** by design (no VAD segmentation) and **no
  `lang` parameter**: Parakeet TDT v3 is multilingual, the language is
  auto-detected.
- No second ONNX environment is created — the model is shared with the live
  session (respects the one-environment-per-process constraint). The upload is
  spilled to a temp file (ffmpeg needs a seekable input) and removed afterwards.
- Verified end-to-end: `voice_profiles/solis.wav` (14.9 s, Spanish) →
  `duration_ms=14918`, `tokens=83`, correct transcript; empty body → `400`.

---

## 2026-10-17 — Feat: OpenAPI document + Swagger UI at `/docs`

### Added

- **Swagger UI for the REST surface.** The engine now serves an interactive
  Swagger UI at **`GET /docs`** backed by an OpenAPI 3 document at
  **`GET /api-docs/openapi.json`**. The document is derived from the handlers
  with [`utoipa`](https://docs.rs/utoipa) (`utoipa = "5"`, `axum_extras`) and
  rendered by [`utoipa-swagger-ui`](https://docs.rs/utoipa-swagger-ui) v8
  (axum 0.7). New module [`src/apidoc.rs`](src/apidoc.rs) holds the
  `#[derive(OpenApi)]` document (`ApiDoc`) and the response schemas
  (`PhrasesResponse`, `PhraseEntry`, `SetVoiceResponse`, `ClearResponse`,
  `ErrorResponse`); the endpoints carry `#[utoipa::path]` annotations.
- **Documented endpoints:** `GET /health`, `GET /api/customvoice/phrases`,
  `POST /api/customvoice` (query params `lang` / `text` via
  `#[derive(IntoParams)]` on `SetParams`; raw `application/octet-stream` body),
  `POST /api/customvoice/clear`. WebSocket (`/ws`) and the HTML client pages are
  intentionally excluded (streaming / static assets).

### Notes

- The `zip` crate is pinned to the **2.2.x** line in `Cargo.toml`: the
  `utoipa-swagger-ui` 8.1 build script (which unpacks the embedded UI assets)
  targets the older zip 2.x API and does not compile against zip 2.3+/2.6. The
  root `axum` stays at 0.7.9 — `utoipa-swagger-ui` 8.1 resolves to it, so no
  duplicate Axum is pulled in.
- Swagger UI is only for the REST endpoints; `/docs` requires the same
  HTTP/HTTPS origin as the engine.

---

## 2026-10-17 — Feat: custom voice accepts an uploaded audio file + typed transcript

### Added

- **Upload source on the `/customvoice` page.** A **Source** toggle now offers
  two ways to build the reference: **Record now** (the existing microphone flow,
  read a predefined phrase for ≥ 10 s) or **Upload a file** — pick any audio
  file `ffmpeg` can read (`accept="audio/*"`: wav, mp3, m4a, ogg, flac…) and
  **type (or paste) its transcript**. In upload mode the transcript is required
  and gates the *Use it* button, since the typed text is what drives the clone.
  Both modes share the same *Use it* / *Reset* actions and preview player.
- **`text` query parameter on `POST /api/customvoice`.** The endpoint is now
  `POST /api/customvoice?lang=<code>[&text=<transcript>]`. A non-blank `text`
  overrides the language's predefined reading phrase for the transcript written
  to the sibling `.txt` and passed to `set_reference`; when absent or blank the
  predefined phrase is used as before (so a plain recording is unchanged).
  Backend: `SetParams` gained `text: Option<String>` and `set_custom_voice`
  resolves the transcript from it (`src/customvoice.rs`).

### Notes

- No format constraints are imposed on the upload: the same ffmpeg probe used
  for recordings handles whatever container is sent. The 32 MB body cap already
  covers uncompressed WAVs.
- README updated (feature bullet, usage comment, *Custom voice builder* flow and
  API/route tables, `text` parameter).

---

## 2026-10-17 — Fix: short utterances ("uno", "due") dropped by the STT gate

### Fixed

- **The probability average now counts voiced windows only.** The gate's
  `avg_prob` was the mean of the raw VAD probability over **every** window from
  speech start to segment close — including the `VAD_MIN_SILENCE_MS` (700 ms)
  hangover that ends the utterance. For a long phrase that silence is a small
  fraction, but for a short word it is most of the segment: "uno" (~350 ms of
  voice + ~700 ms of silence) averaged **0.065–0.199** and was discarded as
  `too_soft`, even though the user was plainly speaking. In the same session the
  long phrases passed at 0.32–0.60. The average is now taken only over windows
  whose probability is ≥ `VAD_NEG_THRESHOLD` (`src/vad/silero.rs`), so the voiced
  portion scores its true value and short real words clear the gate while noise
  still does not.

### Notes

- `VAD_MIN_AVG_PROB` stays at `0.25`; its meaning is now stricter (it no longer
  benefits from the silence dilution), but genuine short words score well above
  it. README / `.env` / `.env.example` / `src/vad/mod.rs` docs aligned.
- Tune with `VAD_MIN_AVG_PROB`: raise it if noisy segments become hallucinated
  turns, lower it if genuinely soft speech is still dropped.

---

## 2026-10-17 — Fix: too many utterances dropped by the STT gate

### Fixed

- **`VAD_MIN_AVG_PROB` lowered `0.45` → `0.25`.** Log analysis showed **62
  drops, all `too_soft`** (none for duration), still occurring after a restart.
  Accepted utterances measured `avg_prob` 0.47–0.76 while dropped ones measured
  0.08–0.45 — including plainly real speech cut by a hair (2136 ms @ 0.428,
  3576 ms @ 0.436). The gate sat almost on top of the per-frame speech threshold
  (`VAD_THRESHOLD=0.5`): a real utterance hovers around the threshold on its
  onset/release, so its *average* lands just under 0.45 and the segment is
  discarded. The `0.25` default leaves a proper margin — genuine soft speech
  (0.3–0.45) now passes, while clearly noisy segments (below 0.25) are still
  filtered. Update `VAD_MIN_AVG_PROB` in `.env` (default, `.env.example`,
  `src/vad/mod.rs` and README aligned).

### Notes

- Tune with `VAD_MIN_AVG_PROB`: lower it further if genuine speech is still
  dropped, raise it if noisy segments are turning into hallucinated turns.
  `0.0` disables the probability gate.

---

## 2026-10-17 — Fix: custom voice still louder than the default voice

### Fixed

- **Custom voice level pushed below the reference.** Matching the shipped
  reference at the *integrated* level was not enough: a measured take lands at
  ≈`-23.25 LUFS` against `solis.wav`'s ≈`-23.44 LUFS`, i.e. the same loudness,
  yet it still sounded louder. The difference is **dynamic range and peaks** —
  the user take measures ≈`6.2 LRA` / `-2.2 dBTP` versus `1.6 LRA` / `-7.8 dBTP`
  for the mastered reference, so its peaks sit ~5.5 dB hotter and read as
  "louder". The normalization target is now `-26 LUFS` with a `-5 dBTP`
  ceiling, which drops a take ~3 dB and pulls its peaks down to ≈`-5 dBTP` so a
  custom voice sits just under the default instead of on top of it.
- Defaults changed: `CUSTOMVOICE_TARGET_LUFS` `-23` → **`-26`**,
  `CUSTOMVOICE_TRUE_PEAK` `-1.5` → **`-5`** (code, `.env`, `.env.example`,
  README).

### Notes

- Lower `CUSTOMVOICE_TARGET_LUFS` further for a quieter custom voice; the value
  is applied at record time, so a take must be re-recorded (or the endpoint
  re-posted) for a change to take effect.
- The custom voice lives only in memory (`TtsEngine::set_reference`): a restart
  reverts the reference to the configured `TTS_REFERENCE_AUDIO` voice.

---

## 2026-10-17 — Fix: custom voice louder than the default voice

### Fixed

- **Loudness target aligned with the shipped reference.** The custom-voice
  chain normalized to `-18 LUFS` while `voice_profiles/solis.wav` measures
  ≈`-23.4 LUFS`, so a custom voice came out ~5.4 dB louder than the voice it
  replaced. The default is now **`-23`** (`CUSTOMVOICE_TARGET_LUFS`), matching
  the reference — swapping voices no longer changes how loud the agent sounds.
- **Mono downmix moved inside the filter chain.** `-ac 1` used to be applied at
  the output stage, i.e. *after* loudnorm: a **stereo** upload was measured as
  stereo and then folded to mono, landing the written file ~3 dB below the
  target. `aformat=channel_layouts=mono` now runs first, so loudnorm measures
  the signal that is actually written.

### Notes

- Verified: `solis.wav` (stereo, -23.4 LUFS) and a quiet mono take (-48.6 LUFS)
  both render to exactly **-23.00 LUFS**, mono 44.1 kHz. `cargo test` → 25
  passed.

---

## 2026-10-17 — Post-processing for the custom voice (level match + cleanup)

### Added

- **Post-processing chain** for a recorded custom voice
  (`src/customvoice.rs`), applied before the reference is installed:
  `highpass=f=80` (rumble/DC), optional `afftdn` denoise + `deesser`,
  leading/trailing silence trim (`silenceremove` + `areverse`) and a
  **two-pass EBU R128 `loudnorm` with `linear=true`**.
- **New env knobs**: `CUSTOMVOICE_POSTPROCESS` (default `true`),
  `CUSTOMVOICE_DENOISE` (default `true`), `CUSTOMVOICE_TARGET_LUFS`
  (default `-18`), `CUSTOMVOICE_TRUE_PEAK` (default `-1.5`).

### Notes

- The loudness stage measures the cleaned signal first, then applies the result
  as a **single constant gain** — a per-frame (single-pass) norm is what makes
  takes jump in volume, which is the problem this fixes. Verified: a take at
  **-48.6 LUFS** and one at **-17.5 LUFS** both render to **≈ -18 LUFS** with
  TP at **-1.5 dBTP**.
- Any stage failing (unreadable take, all-silence clip, filter quirk) degrades
  to the plain transcode with a warning — a take is never lost.
- Error handling verified: garbage input → `502` with a clean message, unknown
  language → `400`. `cargo test` → 25 passed.

---

## 2026-10-17 — Custom voice builder (`/customvoice`) + Mivocero side menu

### Added

- **Custom-voice builder page** at `GET /customvoice`
  (`src/web/customvoice.html`). It asks the user for a **language**, shows a
  predefined phrase to read (long enough for cloning, ~10 s), records it with
  `MediaRecorder` and installs the take as the active voice. The **Stop**
  button is disabled until **10 s** have elapsed (timer + progress bar), the
  take auto-stops at 20 s, and it can be played back and discarded before use.
- **Backend endpoints** (`src/customvoice.rs`): `GET /api/customvoice/phrases`
  (reading phrases per language + the configured default), `POST
  /api/customvoice?lang=<code>` (raw recording body), `POST
  /api/customvoice/clear`. The body cap of the upload route is raised to 32 MB.
- **Side menu** on `/mivocero` (☰ drawer) linking to the chat and to the
  custom-voice page; the same drawer is on `/customvoice`.

### Notes

- Phrases live server-side, so the reference transcript always matches the text
  the user read. `POST /api/customvoice` transcodes the upload to mono 16-bit
  PCM WAV at `OUTPUT_SAMPLE_RATE` (44.1 kHz) with `ffmpeg`, writes it next to
  `TTS_REFERENCE_AUDIO` (the directory the Fish Audio backend can read) with a
  sibling `<name>.txt`, and installs it via `TtsEngine::set_reference` — live
  for new replies, no restart.
- `POST /api/customvoice/clear` restores the reference configured at startup
  (`TTS_REFERENCE_AUDIO`) rather than dropping it entirely.
- Verified: uploading an audio file returns `{"ok":true,…}` and produces a
  44.1 kHz/mono `pcm_s16le` WAV with its `.txt`; `clear` logs the restore of
  `solis.wav`. Tests: `cargo test` → 25 passed.

---

## 2026-10-17 — Tool calling survives a mis-matched server-side tool parser

### Fixed

- **Tool calls dropped on the streaming path are recovered.** The production
  vLLM server runs `--tool-call-parser hermes`, which expects the
  `<tool_call>{"name":…,"arguments":{…}}</tool_call>` JSON shape; the Qwen3
  model emits the XML shape instead, so the parser throws and the call is
  discarded. In streaming the turn then ended with an **empty** reply
  (`finish_reason: "stop"`) and no `tool_call` ever fired — the assistant either
  stayed silent or narrated a call it never made.
- **Non-streaming fallback** in the turn loop
  (`src/orchestrator/turn_manager.rs`): a round that comes back empty while
  tools are configured is re-asked once **without streaming** via the new
  `LlmClient::chat_reply` (`src/llm.rs`), which returns both the visible text
  and the parsed tool calls.
- **Structured calls are parsed too** — `tools::parse_openai_tool_calls`
  (`src/tools.rs`) reads the standard `message.tool_calls[]` array, so the
  engine keeps working when the server parser *does* match (and returns
  `finish_reason: "tool_calls"`). The native `<tool_call>` text path is
  unchanged.

### Notes

- End-to-end verified: "Che tempo fa a New York adesso?" now emits
  `tool_call {name: get_weather, arguments: {city: "New York"}}` and answers
  with live wttr.in data. A tool-free turn is unchanged (no extra request).
- The clean fix remains server-side — `--tool-call-parser qwen3_coder` matches
  the Qwen3 XML format. The fallback makes the engine correct regardless, at the
  cost of one extra request only on an otherwise-empty round.
- Tests: `cargo test` → 25 passed (2 new in `tools::tests`).

---


## 2026-10-17 — Longer end-of-utterance window (`VAD_MIN_SILENCE_MS` 300 → 700 ms)

### Changed

- **`VAD_MIN_SILENCE_MS` raised** from `300` to `700` ms in `.env` /
  `.env.example`, and the fallback default in `src/vad/mod.rs` aligned to
  `700`. This is the silence window the VAD waits for before closing an
  utterance (see `SileroVad::step`, `silence_run_ms >= min_silence_ms`).
- **README** — updated the config table and the `SpeechEnd` trigger row.

### Notes

- With `300` ms a brief thinking pause (~0,3 s) closed the turn and sent a
  partial sentence to the STT/LLM. `700` ms tolerates a natural mid-sentence
  pause before the agent answers. The trade-off is a slightly later reply (the
  agent now waits ~0,7 s of silence instead of ~0,3 s), which the filler /
  start-filler cues already mask.
- The backchannel micro-pause window is unaffected: its `pause_max_ms` (`260`)
  is still below the new cap (`min_silence_ms - 32 = 668`).

---

## 2026-10-17 — Mivocero: barge-in indicator + karaoke on the user bubbles

### Added

- **Server `{"type":"barge_in","source":"vad","prob":…}` event** in
  `src/ws/session.rs` — emitted when the server-side VAD onset interrupts the
  agent. The onset is detected in the session task, so the browser had no other
  way to know a barge-in had happened.
- **Barge-in indicator** in `src/web/mivocero.html` — a red **Barge-in** pill in
  the header that flashes and bumps a session counter. Driven by the
  `barge_in` event (VAD onset) and by the local **Stop** button; the counter
  resets on each connect.

### Changed

- **Karaoke on the user bubbles** in `src/web/mivocero.html` — the user's own
  messages (STT transcript and typed text) are now rendered word by word like
  the assistant's, with colours tuned for the blue bubble. Because the STT
  returns the whole transcript only once the utterance has ended, the words are
  lit by a short left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word)
  instead of following the audio playhead.
- **README** — documented the `barge_in` event in both the server→client table
  and the audit `out` row, and updated the Mivocero section (karaoke, barge-in
  indicator).

---

## 2026-10-17 — VAD barge-in gated on audible output (no more silent cancelled replies)

### Added

- **`AudioMixer::is_audible()`** in `src/audio/mixer.rs` — reports whether the
  agent has *audible* output: the reply is playing out (`tts_primed`) or a
  latency-mask cue (filler / start filler) is queued. Unlike `is_active()` it
  is `false` while the reply is still being synthesised or sits in the TTS
  pre-buffer — nothing has reached the user's ears yet.
- **`TurnManager::is_audible()`** in `src/orchestrator/turn_manager.rs` —
  forwards the mixer method for the barge-in gate.

### Changed

- **VAD-driven barge-in gate** in `src/ws/session.rs` — a `SpeechOnset` now
  interrupts only when `TurnManager::is_audible()` is true, instead of
  `is_speaking()` (`is_active()`). `is_speaking()` is retained for the
  backchannel gate, where cueing over any in-flight TTS would still be wrong.

### Notes

- Root cause: `is_speaking()` is already `true` as soon as the TTS request is
  in flight (`tts_stream_active`), so a VAD onset landing during the
  synthesis / pre-buffer window (~1.2–1.6 s) cancelled the reply before a
  single sample was played. On the analysed session
  (`s-1792251858371-0001`, ~210 s, 21 turns) three replies streamed their text
  to the client but produced **zero playback**, and three more were truncated
  after 0.1–0.7 s; several of the onsets were false triggers (the utterance
  was later dropped by the gate with `avg_prob` 0.28 / 0.21).
- A reply cancelled this way is not orphaned: the next turn's `on_eou` already
  cancels the previous in-flight synthesis and restarts the stream, so the
  audio follows the newest turn instead of being lost.

---

## 2026-10-17 — Filler fade-out on reply start (no more overlap with the answer)

### Changed

- **Mixer filler fade-out** in `src/audio/mixer.rs` — when the reply's playback
  starts (TTS pre-buffer reached) while an end-of-turn filler or a start filler
  is still queued, the cue is now **faded out** over `FILLER_FADEOUT_FRAMES`
  (4 frames = 80 ms) and then cut, instead of being ducked to `filler_gain_duck`
  and left playing underneath the whole answer. The filler is only a latency
  mask, so it now hands off cleanly to the reply.
  - New `AudioMixer::filler_fade_frames` counter, armed on the priming
    transition and reset by `mark_stream_start` / `flush`.
- **`START_FILLER_GO_DELAY` raised** from `1200` to `1800` ms in `.env` /
  `.env.example` — the cue now only fires when the first audio chunk is
  genuinely late, instead of on the boundary where it used to overlap the
  beginning of the reply.

### Notes

- Root cause: with `START_FILLER_GO_DELAY=1200` the first TTS audio arrived
  0.75–1.4 s after the cue, while the `sf03` clip is ~2.83 s long — so the cue
  played for its first second at full gain and its tail then overlapped the
  start of the answer. On the analysed session this happened on 5 of 6 start
  fillers (all `sf03`).

---

## 2026-10-17 — Utterance gate: filter spurious VAD turns before STT

### Added

- **Utterance gate** in `src/ws/session.rs` — a VAD segment is now dropped
  **before** transcription when it is too short or too "soft", so the STT can no
  longer turn a VAD false-trigger (breath, click, acoustic echo, noise) into a
  hallucinated text turn that would each otherwise cost a full LLM turn.
  Two independent thresholds, both env-driven:
  - `VAD_MIN_UTTERANCE_MS` (`VadSettings::min_utterance_ms`, default `300`,
    `0` disables) — drop segments shorter than this.
  - `VAD_MIN_AVG_PROB` (`VadSettings::min_avg_prob`, default `0.45`, `0.0`
    disables) — drop segments whose average speech probability is below this.
  A dropped segment is logged and recorded in the audit as `utterance_dropped`
  (`duration_ms`, `avg_prob`, `too_short`, `too_soft`).
- **Empty-transcript guard** in `src/orchestrator/turn_manager.rs::on_eou` — a
  transcript that decodes to an empty string (or has no tokens at all) no
  longer opens an LLM turn.

### Configuration

- New `.env` keys next to the other `VAD_*` options:

  ```
  VAD_MIN_UTTERANCE_MS=300
  VAD_MIN_AVG_PROB=0.45
  ```

### Notes

- The gate runs only when `VAD_SEGMENT_STT=true`. Raise the probability to be
  stricter; lower the duration to keep very short backchannels ("sì", "ok").
- Documentation updated: README (VAD configuration table, VAD wiring section,
  audit-event table) and project memory.

---

## 2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)

### Added

- **`cues` WebSocket event** — sent once at session start by
  `src/ws/session.rs`, it reports the **effective policy** of the three cues for
  the active language: the filler (`delay_ms`, `gain_idle`, `gain_duck`, clip
  count, bank dir), the start filler (`go_delay_ms`, clip count, bank dir) and
  the backchannel (gain, min-speech, interval, jitter, pause window,
  probability, volume jitter, per-turn cap, categories, clip count). The
  backchannel `pause_max_ms` is capped to the VAD EOU window exactly like
  `BackchannelScheduler::pause_max`; the bank paths are the language
  subdirectories (`assets/fillers/<lang>`). Purely informational.
- **Cue methods panel** in `src/web/mivocero.html` — a collapsible
  `<details>` section under the cue tiles with one card per cue: the **method**
  used (when it fires, how a clip is chosen) and the **live parameters** from
  the `cues` event, so the panel always matches the running configuration
  instead of hard-coded values.

### Notes

- The panel is display-only; it changes no behaviour. Values reflect the
  process configuration at session start (env-derived), so a `.env` change is
  visible after the next session.

---

## 2026-10-17 — Per-session audit log (structured JSONL)

### Added

- **`src/audit.rs`** — a best-effort per-session audit sink. Each WebSocket
  session is recorded to its own JSONL file
  (`SESSION_AUDIT_DIR/<id>.jsonl`, id `s-<epoch_ms>-<seq>`), one record per line:
  `{"ts":…,"elapsed_ms":…,"event":…,"data":…}`. Records are flushed as they are
  written, and any failure (directory/file creation, write) only disables the
  sink for that session — the session itself is never affected.
- **Event coverage (complete)**:
  - `session_start` — full non-secret configuration snapshot plus the active
    runtime capabilities (STT / VAD / tokenizer / LLM + model);
  - `control` — inbound `user_text` turns;
  - `out` — **every** event sent to the client, tapped at the single sender
    point in `src/ws/session.rs` so nothing is missed (`user`,
    `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`,
    `audio_source` state, …);
  - `interrupt` — client barge-in (JSON or legacy text form);
  - `barge_in` — VAD-driven barge-in (with the onset probability);
  - `vad_speech_start` / `vad_speech_end` — utterance boundaries;
  - `media` / `reference` / `speak` — external-audio control, voice-reference
    changes and direct TTS requests;
  - `language_switch` — detected language change (`from`, `to`, `generated`) —
    `src/orchestrator/turn_manager.rs`;
  - `error` — transport/receiver errors;
  - `session_end` — reason, duration, session id, file and per-event counts.
- **Config** — `SESSION_AUDIT_ENABLED` (default `true`) and
  `SESSION_AUDIT_DIR` (default `logs/sessions`); documented in `.env`,
  `.env.example` and the README. `logs/` is git-ignored.

### Changed

- `TurnManager::new` takes an `Arc<SessionAudit>` (used for the language-switch
  record); every other event reaches the log through the sender tap.

### Notes

- The audit path is off the audio hot path (the sink only touches the outbound
  text events, not the 20 ms frames) and is disabled cleanly when the LLM/VAD/…
  are off. Intended for post-mortem analysis and for tuning the latency knobs.

---

## 2026-10-17 — Cue emission indicators in the Mivocero demo

### Added

- **`{"type":"cue"}` WebSocket event** — the server now tells the client each
  time a latency-mask / backchannel cue is **actually emitted** into the mixer:
  - **filler** (end-of-turn) — `src/orchestrator/turn_manager.rs`
    (`schedule_filler`),
  - **start_filler** (turn start) — `src/orchestrator/turn_manager.rs`
    (`schedule_start_filler`),
  - **backchannel** (during user speech) — `src/ws/session.rs` (mic task).
  The payload is `{"type":"cue","kind":…,"name":…}` and, for backchannel, also
  `"category":…`. A cue that is skipped (audio already arrived / playing, or a
  barge-in) sends nothing.
- **`BackchannelScheduler::maybe_emit`** now returns `(name, category)` instead
  of just the name, so the backchannel cue carries its category.
- **Mivocero cue indicators** — `src/web/mivocero.html` gained a row of three
  tiles (**Filler**, **Start filler**, **Backchannel**) that flash (colour-coded
  dot + border glow) and bump a per-session counter on every `cue` event, with a
  detail line showing the backchannel category (or the clip name). Counters reset
  on each connect.

### Notes

- Only real emissions are reported, so the panel is a faithful view of what the
  user actually hears — useful for tuning `FILLER_DELAY_MS` /
  `START_FILLER_GO_DELAY` and the backchannel cadence.

---

## 2026-10-17 — Per-language subdirectory for every clip bank (base included)

### Changed

- **`src/main.rs`** — the startup bootstrap now generates and loads the three
  base banks (filler, start-filler, backchannel) from
  `<dir>/<LANG_DEFAULT>/` via [`assets::lang_dir`](src/assets.rs) instead of the
  top-level directory. Every language — **including the default one** — now
  lives in its own subdirectory, so the filler, start-filler and backchannel
  share the exact same layout for every language and no language is special-cased
  to the top level.
- **`src/assets.rs`** — [`lang_dir`](src/assets.rs) doc updated accordingly.

### Notes

- On the first start after this change the base clips are re-synthesized into
  `assets/<bank>/<lang>/`. To avoid the re-synthesis, move the existing
  top-level `*.raw` files (and `texts.txt`) into the matching language
  subdirectory by hand — this was done for the `es` deployment.
- No other language's clips are touched: the per-language subdirectory remains
  what isolates each language's refresh (`remove_stale_clips`).

---

## 2026-10-17 — Multilingual clip banks (backchannel, filler & start filler)

### Added

- **`src/lang.rs`** — new multilingual module. It holds built-in phrase tables
  for **backchannel** (category + phrase), **filler** (interjections) and
  **start filler** (turn-start acknowledgments, with Drama-3 style tags) in
  Spanish, English, Italian, French, German and Portuguese, exposes accessors
  for them, and implements the auxiliary LLM calls: `detect_language` (returns
  only the ISO 639-1 code, or `und`) and `generate_backchannel_texts` /
  `generate_filler_texts` / `generate_start_filler_texts` (LLM fallback for
  languages absent from the tables).
  It also defines `LangBanks` (a language's loaded clip banks, including its
  start-filler bank) and `LanguageState` (the session's active language plus a
  generation counter, kept in a parking-lot `RwLock`).
- **On-demand per-language generation** — when the user's language differs from
  the active one, that language's backchannel/filler/start-filler clips are
  synthesized into a **per-language subdirectory** (`assets/backchannel/<lang>/`,
  `assets/fillers/<lang>/`, `assets/start_filler/<lang>/`) and the session
  switches to them. Because the refresh pass only prunes clips inside the
  directory it is given, **no other language's clips are ever removed**; the
  base language keeps living in the top-level directories, so no migration is
  needed.
- **New env vars** — `LANG_DEFAULT` (default `es`), `LANG_DETECT_ENABLED`
  (default `true`), `LANG_MIN_CHARS` (default `12`), `LANG_REDETECT_EVERY`
  (default `5`) and `LANG_LLM_FALLBACK_ENABLED` (default `true`).

### Changed

- **`src/llm.rs`** — new `LlmClient::simple_chat` for short, non-streaming
  auxiliary requests (explicit system + user message, **no** tools), used by the
  language detection and phrase-generation calls.
- **`src/config.rs`** — five new `LANG_*` fields, parsed in `from_env`.
- **`src/assets.rs`** — `lang_dir` helper plus `ensure_lang_backchannel_clips` /
  `ensure_lang_filler_clips` / `ensure_lang_start_filler_clips`, which scope the
  text-bank refresh to `<base>/<lang>/` so each language is refreshed
  independently.
- **`src/orchestrator/backchannel.rs`** — the clip/category mapping is extracted
  into `map_clips`, and `BackchannelScheduler::retarget` re-points a live
  scheduler to a new bank when the language changes. `DEFAULT_CATEGORY` is now
  `pub(crate)` so `lang.rs` can reuse it.
- **`src/orchestrator/turn_manager.rs`** — `TurnManager` now holds an
  `Arc<LanguageState>` instead of a single filler bank; `next_filler` and the
  start filler read from the active language (`schedule_start_filler` picks from
  `langs.current().start_filler`). A new `maybe_detect_language` runs on each
  user turn (first turn + every `LANG_REDETECT_EVERY`) and spawns
  `switch_language_if_needed`, which detects, generates/loads the target
  language's banks (start filler optional: a missing table just leaves that cue
  silent) and swaps the active state — in the background, so the reply is never
  blocked.
- **`src/ws/session.rs`** — builds the shared `LanguageState`; the mic task polls
  the language generation counter each frame and calls `retarget` on the
  backchannel scheduler only when it changes.
- **`.env` / `.env.example`** — new "Multilingual clip banks" section;
  **`README.md`** — new "Multilingual clip banks" section.

### Why

- The banks were pre-synthesized in a single language, so a user speaking any
  other language heard Spanish cues — including the start filler. Generating the
  matching language on demand keeps the acknowledgment natural in a multilingual
  deployment, while the per-language directory layout guarantees that adding a
  language never destroys the clips already generated for the others.

### Notes

- Detection needs the LLM (`LANG_DETECT_ENABLED` is a no-op without it); the LLM
  phrase fallback is used only for languages outside the built-in tables.
- Generating a new language's bank calls the TTS backend once per phrase, which
  can take a few seconds on the first switch; it is idempotent (an
  already-generated language is only reloaded).

---

## 2026-10-16 — Mivocero: auto-disconnect an idle session after 2 minutes

### Added

- **Idle guard in the Mivocero client** ([`src/web/mivocero.html`](src/web/mivocero.html)) —
  the session closes itself after `IDLE_TIMEOUT_MS` (2 minutes) without any
  activity. `touchActivity()` re-arms the timer and is called on every server
  event, on every received audio frame and on every user action sent
  (`sendJson`), so an ongoing reply or a playing external source keeps the
  session alive. On expiry it logs the reason, appends an explanatory message to
  the chat and calls `disconnect()`, which closes the WebSocket and releases the
  microphone.

### Changed

- **Reconnect after close** — `ws.onclose` now clears the `ws` reference.
  Without it the Connect button stayed bound to the stale socket, so a session
  closed by the idle guard (or by any disconnect) could not be reopened without
  reloading the page.

### Why

- A microphone-backed session left open keeps the device busy and the server
  streaming silent 20 ms frames forever. An inactivity timeout releases both
  without user action, while still allowing a long reply or a background radio
  to run uninterrupted.

### Notes

- The timer counts *any* traffic in either direction: TTS audio is sent as
  binary frames, so speaking does not trip the timeout.
- The timeout is a client-side constant (`IDLE_TIMEOUT_MS` in the HTML); change
  it there and rebuild (the page is embedded with `include_str!`).

---

## 2026-10-16 — Tool calling: configurable HTTP tools for the LLM

### Added

- **`src/tools.rs`** — new tool-calling module. It loads tool definitions from a
  JSON file and provides: a `ToolRegistry` (builds the OpenAI `tools` array,
  executes a call over HTTP), a client-side parser for the model's **native**
  `<tool_call>` format, and a `ToolCallStripper` that removes tool-call blocks
  from the streamed reply — holding back a block that is still incomplete, so it
  is never spoken by the TTS nor shown in the chat box even when split across
  deltas (same hold-back pattern as the chain-of-thought stripper).
- **`tools.example.json`** — ready-to-use definitions: `get_weather` (GET with a
  `{city}` URL placeholder, wttr.in) and `send_webhook` (POST with a custom
  header), covering both request shapes.
- **New env vars** — `LLM_TOOLS_ENABLED` (auto: on when `LLM_TOOLS_FILE` is set),
  `LLM_TOOLS_FILE`, `LLM_TOOL_MAX_ITERATIONS` (default `4`),
  `LLM_TOOL_TIMEOUT_SECS` (default `15`) and `LLM_TOOL_RESULT_MAX_CHARS`
  (default `2000`).
- **`ChatMessage::tool`** — `role: "tool"` message used to hand a tool result
  back to the model.
- **WebSocket protocol** — informational
  `{"type":"tool_call","name":"…","arguments":{…}}` event emitted when the model
  calls a tool.

### Changed

- **`LlmClient`** carries an optional `Arc<ToolRegistry>`
  ([`src/llm.rs`](src/llm.rs)); when present, every request advertises
  `tools` + `tool_choice:"auto"`, so the model knows which functions it may
  call. The registry is built by `load_tools(config)`; a missing or invalid file
  logs a warning and leaves tool calling disabled without breaking the pipeline.
- **[`TurnManager::run_llm_turn`](src/orchestrator/turn_manager.rs)** now runs a
  bounded tool loop: each round streams a reply through the `ToolCallStripper`;
  if a call was emitted, the assistant message (with its native call) plus one
  `tool` message per result are appended and the model is re-prompted, up to
  `LLM_TOOL_MAX_ITERATIONS`. The TTS pipeline is created once and every round's
  visible text flows into it, so the assistant can say something short before
  calling. Only the spoken text is written to the rolling history.

### Why

- The deployment's vLLM server runs **without** a tool parser
  (`--enable-auto-tool-choice --tool-call-parser …`), so it never returns
  structured `tool_calls` / `finish_reason:"tool_calls"`. Tools are still
  advertised in the request and the Qwen model emits the call as text in its
  native format; parsing it client-side keeps the feature working with any
  server and requires no server-side reconfiguration.
- Tool definitions live in a JSON file rather than `.env` so the verbose JSON
  schemas stay readable and editable.

### Notes

- Tool errors (unknown tool, transport failure, non-2xx status) are returned to
  the model as text so it can recover; a tool never fails the turn.
- `GET` sends arguments not consumed by a `{arg}` placeholder as query
  parameters; `POST` sends them as a JSON body. `headers` are sent verbatim.

---

## 2026-10-16 — External audio channel: inject a URL / radio / YouTube source

### Added

- **`src/media.rs`** — new [`MediaSource`](src/media.rs) module implementing an
  external audio source for a WebSocket session. A source is decoded by an
  external process and its raw PCM is pushed into the mixer's new external ring
  buffer. `ffmpeg` decodes direct URLs / radio / HLS
  (`-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1`, with
  `-reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5`); YouTube links are
  first resolved to a direct URL with `yt-dlp` (`-f bestaudio/best -g
  --no-playlist`). Child processes are killed on stop / replacement /
  disconnection (`kill_on_drop(true)` + `CancellationToken`), and a monotonic
  generation counter prevents a stale task from clearing a newer source.
- **Mixer external channel** — `AudioMixer` gains `external_queue` and a runtime
  gain ([`src/audio/mixer.rs`](src/audio/mixer.rs)). The mixing formula becomes
  `… + (external × external_gain)`; `set_external_gain` updates the gain (an
  atomic, so it takes effect on the next frame without rebuilding the mixer).
  The external queue is **not** cleared by `flush()` and does not count towards
  `is_active()`: a barge-in stops the agent, not the background audio.
- **New env vars** — `EXTERNAL_AUDIO_GAIN` (default `0.6`),
  `MEDIA_FFMPEG_BIN` (default `ffmpeg`), `MEDIA_YTDLP_BIN` (default `yt-dlp`) and
  `MEDIA_YTDLP_ARGS` (extra `yt-dlp` args, e.g. `--cookies …`).
- **WebSocket protocol** — new `audio_source` message:
  `{"type":"audio_source","action":"play","url":"…"}`,
  `{"type":"audio_source","action":"stop"}` and
  `{"type":"audio_source","action":"volume","gain":0.6}`. The server reports
  state back with `{"type":"audio_source","state":"loading|playing|stopped|error","url":"…"}`
  ([`src/ws/session.rs`](src/ws/session.rs)).
- **Mivocero client** — footer bar with a URL input, **Play** / **Stop**
  buttons, a volume slider (0–150 %) and a live state label, driving the external
  channel ([`src/web/mivocero.html`](src/web/mivocero.html)).

### Changed

- [`TurnManager::set_external_gain`](src/orchestrator/turn_manager.rs) forwards
  the client volume to the mixer.
- New module declared in [`src/main.rs`](src/main.rs) (`mod media;`).

### Why

- Lets the agent play a music/radio/YouTube bed under the voice — useful for
  demos (radio, background music) — while keeping the channel fully independent
  of the speech pipeline so it survives barge-in.

### Notes

- Only needed when the external channel is used: `ffmpeg` for URL/radio/HLS and
  `yt-dlp` for YouTube. YouTube may require cookies in some environments; pass
  them via `MEDIA_YTDLP_ARGS=--cookies /path/cookies.txt`.
- A bounded source (VOD) is throttled to the mixer's drain rate via backpressure
  (`remaining_capacity()` + 20 ms sleeps), so a fast decoder cannot overflow the
  ring buffer.

---

## 2026-10-16 — Drama 3 de-emphasis: sampling knobs + neutral style tag

### Added

- **`TTS_TEMPERATURE`** (default `0.45`), **`TTS_TOP_P`** (default `0.75`) and
  **`TTS_TOP_K`** (unset) — optional sampling overrides forwarded in the
  `/v1/audio/speech` payload by the `fish` engine. An empty value omits the field
  so the server default applies. New `env_opt_f32` / `env_opt_u32` helpers in
  [`src/config.rs`](src/config.rs) implement the "unset → fallback, empty →
  `None`" semantics.
- **`TTS_STYLE_TAG`** (default `[conversational, casual]`) — a neutral style tag
  prepended to every synthesis request whose text does not already start with a
  `[...]` tag ([`FishAudioTts::styled`](src/tts/fish_audio.rs)). Empty disables
  the prefix.

### Changed

- [`SpeechRequest`](src/tts/fish_audio.rs) gains optional `temperature`, `top_p`
  and `top_k` fields (serialized only when set); the request body now uses the
  styled text as `input`. Startup logs `TTS de-emphasis settings …`, and the
  per-request debug log shows the sampling values and whether the text was
  prefixed (`styled=`).

### Why

- Drama 3 over-articulates: LLM replies could come back with a theatrical,
  radio-drama cadence. Exaggerated intonations live in the tails of the acoustic
  token distribution, so trimming the sampling tails (`temperature` / `top_p`)
  flattens the delivery, and a neutral style tag steers untagged (LLM) prose into
  a conversational register. Clip banks carry their own tags and are untouched.

### Notes

- `repetition_penalty` is deliberately **not** exposed: this server rejects any
  value other than `1.0` (`Drama 3 request: repetition_penalty must be 1.0`).
  `min_p`, `seed` and a nested `sampling_params` are rejected too.
- The voice-cloning reference (`solis.wav`) is left unchanged for now — its
  "locutor profesional" prosody is a further, separate lever.
- **`.env` gotcha surfaced during this change.** `dotenvy` fails on unquoted
  values containing spaces, and a single bad line makes it discard the whole
  file: the process then silently starts with built-in defaults (no LLM, plain
  HTTP) while `server.sh` still reports the scheme from its own `.env` grep. The
  style tag is therefore quoted (`TTS_STYLE_TAG="[conversational, casual]"`).

---

## 2026-10-16 — Backchannel: random reaction depth inside the micro-pause

### Changed

- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) now draws
  the depth at which it reacts **per pause**: the first time a micro-pause is
  seen, a target is picked uniformly in `[BACKCHANNEL_PAUSE_MIN_MS,
  BACKCHANNEL_PAUSE_MAX_MS]` and the cue fires only once `silence_ms()` reaches
  it (new `pause_target_ms` field, cleared when the pause ends or the turn
  resets). Previously the cue fired at the *first* analysis window inside the
  window, so with a 32 ms VAD step every cue landed at exactly
  `BACKCHANNEL_PAUSE_MIN_MS`.
- A pause that grows past the upper bound (on its way to end-of-utterance) is
  abandoned without firing.

### Why

- Verification on a live session showed every cue at `pause_ms=96`, i.e. all cues
  reacting at exactly the minimum — a residual regularity in an otherwise
  naturalized feature. Randomizing the depth removes it: cues now land anywhere
  in the window (observed `pause_ms` of `96`, `128`, `192`, …).

### Notes

- No new environment variables: the behaviour is driven by the existing
  `BACKCHANNEL_PAUSE_MIN_MS` / `BACKCHANNEL_PAUSE_MAX_MS` window.
- Because the VAD advances in 32 ms windows, the emitted `pause_ms` is a multiple
  of 32 ms; the randomly drawn target is what varies.

---

## 2026-10-16 — Backchannel sounds more natural (micro-pause cues, jittered timing, categories)

### Added

- **`BACKCHANNEL_JITTER_MS`** (default `1200`), **`BACKCHANNEL_PAUSE_MIN_MS`**
  (default `96`), **`BACKCHANNEL_PAUSE_MAX_MS`** (default `260`),
  **`BACKCHANNEL_PROB`** (default `0.65`) and **`BACKCHANNEL_VOLUME_JITTER`**
  (default `0.2`) — new `.env` knobs (`.env`, `.env.example`,
  [`BackchannelSettings`](src/orchestrator/backchannel.rs)) that shape the cue
  timing, spacing, hit rate and loudness drift.
- **Category prefix** in `BACKCHANNEL_TEXTS`: a line may be written
  `category::phrase` (e.g. `ack::[calm] Mm-hmm...`, `agree::[agreeing] Claro...`);
  [`parse_texts`](src/orchestrator/backchannel.rs) stores the category alongside
  the phrase and defaults to `ack` when the prefix is absent.
- [`VadDetector::silence_ms`](src/vad/silero.rs) — exposes the running silence
  inside the current utterance (multiples of the 32 ms analysis window), i.e. the
  length of the micro-pause the speaker is currently in.
- [`FillerBank::names`](src/orchestrator/turn_manager.rs) — read accessor for the
  clip names, used to map the bank order onto the phrase/category list.

### Changed

- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) now takes
  the VAD micro-pause length and fires a cue **only inside a
  `PAUSE_MIN_MS..PAUSE_MAX_MS` window** (the upper bound is capped just below
  `VAD_MIN_SILENCE_MS` so it never collides with end-of-utterance), **at most once
  per pause** (latched with `pause_settled`).
- Cue timing is **randomized**: the first cue of a turn waits
  `MIN_SPEECH_MS + rand(0..JITTER_MS)`, and the gap to the next cue is
  `INTERVAL_MS ± JITTER_MS/2` — instead of a fixed interval.
- Each qualifying pause passes a **probabilistic gate** (`PROB`); a miss does not
  retry on the same pause.
- The clip is chosen with an **anti-repetition** rule (skips the last two clips)
  and its samples are scaled by **±`VOLUME_JITTER`** before being pushed to
  `backchannel_queue`.
- Selection follows a **category progression** through the turn — it prefers a
  category other than the last one used and favours categories not yet used this
  turn (e.g. `ack` then `agree`), falling back to a random clip.
- `BACKCHANNEL_MAX_PER_TURN` default lowered **8 → 3**, so the agent emits a
  couple of cues per turn rather than chattering.

### Why

- A fixed interval with a fixed clip made the backchannel feel mechanical and
  detached from the speaker. Real listeners react to *pauses*, not to a metronome:
  cueing inside the speaker's own micro-pauses, with jittered spacing, an
  occasional deliberate skip, mild volume drift and a little progression
  (acknowledge → agree) reads as attentive listening instead of a loop.

### Notes

- The naturalness knobs are all optional; omitting them keeps the previous
  behaviour except for the `MAX_PER_TURN` default (now `3`).
- Nested clamps in `from_env`: `interval_ms >= 500`, `pause_max >= pause_min`,
  and `gain`/`prob`/`volume_jitter` are clamped to sane ranges.
- A debug line is logged only when a cue is actually emitted:
  `backchannel emitted during user speech … pause_ms=…` (enable
  `audio_agent_engine::orchestrator=debug` in `RUST_LOG`).

---

## 2026-10-16 — Backchannel becomes a random phrase bank driven by `BACKCHANNEL_TEXTS`

### Added

- **`BACKCHANNEL_TEXTS`** (`.env`, `.env.example`,
  [`BackchannelSettings`](src/orchestrator/backchannel.rs)): the backchannel bank
  is now a list of phrases, **one per line** (a `|` separator is also accepted),
  with Drama-3 style tags passed straight through to the TTS. `.env` ships 8
  Spanish phrases (`[calm] Mm-hmm...`, `[attentive] Ajá...`, …); the Rust default
  is [`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs).
- [`assets::ensure_backchannel_clips`](src/assets.rs), implemented on top of the
  new generic [`assets::ensure_text_bank_clips`](src/assets.rs): synthesizes **one
  clip per phrase** at startup (`bc01`, `bc02`, …, `TEXT_BANK_CONCURRENCY = 4` at
  a time) and removes clips left over from a previous list. The list the current
  clips were made from is stored in `BACKCHANNEL_DIR/texts.txt`; when it is
  unchanged and every clip is present the whole synthesis is **skipped**, so a
  restart costs ~2 s instead of ~1 s per phrase. The index is only written once
  every clip succeeded, so a partial failure is retried at the next startup.

### Changed

- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) picks the
  clip **at random** ([`FillerBank::random`](src/orchestrator/turn_manager.rs),
  xorshift64 seeded from the clock) instead of round-robin, so the acknowledgment
  varies from turn to turn — the same behaviour the start filler already had. The
  `next_index` cursor was dropped.
- `assets::ensure_start_filler_clips` is now a thin wrapper over
  `ensure_text_bank_clips`, so the start filler and the backchannel share one
  implementation. Its logs gained a `label` field
  (`refreshing backchannel clips at startup …`).
- The backchannel no longer uses the fixed `assets::BACKCHANNEL_CLIPS` list
  (removed); its defaults moved to `DEFAULT_BACKCHANNEL_TEXTS`.

### Why

- A fixed 4-clip bank repeated the same "mm-hmm / ajá" over and over; a curated
  list with tags keeps the agent sounding natural and is editable straight from
  `.env`, exactly like the start filler.
- The old bootstrap (`ensure_clips`) only generated *missing* clips, so changing
  the phrases required deleting files by hand; the phrase list is now the source
  of truth.

### Notes

- Clips are named by position, so switching from the old names (`mmhmm`, `aja`,
  `claro`, `yaveo`) to `bc01`…**removes** the four legacy files on the first
  restart after this change (logged as `removed stale backchannel clip`).
- `BACKCHANNEL_GAIN`, `BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS` and
  `BACKCHANNEL_MAX_PER_TURN` are unchanged.

---

## 2026-10-16 — Fix: start filler fired on every turn (gate on chunk arrival, not playback)

### Fixed

- [`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs) gated the
  cue on `AudioMixer::tts_playback_started`, which only flips once the **TTS
  pre-buffer** (`TTS_PREBUFFER_MS`) is full — i.e. when playback starts, ~1.9 s
  after the dispatch in the reference deployment. The first response chunk
  actually arrives at ~0.7–1.0 s, so a `START_FILLER_GO_DELAY` of 1200 ms was
  almost always *shorter* than the gate's own signal and the acknowledgment was
  queued on essentially every turn.
- The gate now uses the new
  [`AudioMixer::tts_audio_arrived`](src/audio/mixer.rs), latched as soon as the
  first TTS samples are queued for the turn (checked on the 20 ms ticker),
  **before** the pre-buffer is reached. Both `tts_audio_arrived` and
  `tts_playback_started` are reset by `mark_stream_start` / `flush`.
  A fast reply that produces audio within `START_FILLER_GO_DELAY` therefore
  suppresses the cue, as intended.

### Changed

- The skip log line is now `first response audio already arrived — start filler
  skipped` (was `response audio already playing — start filler skipped`).
- The end-of-turn filler is unchanged: it still keys off
  `tts_playback_started` (its 2500 ms delay is long enough that playback start is
  the right signal).

### Notes

- No configuration change; `START_FILLER_GO_DELAY=1200` now means what the `.env`
  comment always said ("first audio chunk of the reply"). Verified in the log: a
  text turn in which the first chunk reached the mixer ~0.4 s after the TTS
  request logged `first response audio already arrived — start filler skipped`
  with no `queued start filler` for the turn.

---

## 2026-10-16 — Start filler becomes conditional: `START_FILLER_GO_DELAY` (1200 ms)

### Changed

- The start filler is no longer played on **every** turn: it is now a *latency
  mask*, scheduled exactly like the end-of-turn filler but on its own channel and
  with its own, shorter delay.
  [`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)
  (replacing `play_start_filler`) sleeps `START_FILLER_GO_DELAY` after the turn is
  dispatched and queues a random phrase **only if the first response audio has not
  started playing** (`AudioMixer::tts_playback_started` still `false`); otherwise
  the cue is dropped. *(The gate was later changed to `tts_audio_arrived` — see
  the entry above, which fixes this cue firing on every turn.)* A pending cue is
  cancelled on barge-in / new turn through the turn's `CancellationToken`, and
  `mixer.flush` clears it once queued.
- The scheduling now happens in `run_llm_turn` / `dispatch_tts` next to
  `schedule_filler` (where the turn's cancellation token and `mark_stream_start`
  live) instead of at the top of `handle_user_turn`, because the delay must be
  measured from the dispatch of the reply and be cancellable with it.

### Added

- **`START_FILLER_GO_DELAY`** (`.env`, `.env.example`, default `1200` ms): how long
  to wait for the first response audio before falling back to the start filler.

### Why

- The cue is only useful when the reply is *late*; a fast reply (LLM + TTS first
  chunk under 1200 ms) is now answered directly, without an unnecessary
  "Un momento…" in front of it.

### Notes

- The 22-clip bank, the random pick and the startup refresh are unchanged.

---

## 2026-10-16 — Start filler: random phrase bank driven by `START_FILLER_TEXTS`

### Added

- **`START_FILLER_TEXTS`** (`.env`, `.env.example`,
  [`src/config.rs`](src/config.rs)) replaces the single `START_FILLER_TEXT`: a
  list of phrases, **one per line** (a `|` separator is also accepted), e.g. the
  Spanish set `"[calm] Un momento..."`, `"[thoughtful] A ver, veamos..."`, … with
  Drama-3 style tags passed straight through to the TTS. `.env` ships 22 phrases.
- [`assets::ensure_start_filler_clips`](src/assets.rs): synthesizes **one clip per
  phrase** at startup (`sf01`, `sf02`, …, generated
  `START_FILLER_CONCURRENCY = 4` at a time) and removes clips left over from a
  previous list. The list the current clips were made from is stored in
  `START_FILLER_DIR/texts.txt`; when it is unchanged and every clip is present the
  whole synthesis is **skipped**, so a restart costs ~2 s instead of ~1 s per
  phrase. The index is only written once every clip succeeded, so a partial
  failure is retried at the next startup.
- [`FillerBank::random`](src/orchestrator/turn_manager.rs): uniform pick over the
  bank using a per-bank xorshift64 state seeded from the clock;
  [`TurnManager::play_start_filler`](src/orchestrator/turn_manager.rs) now uses it,
  so the acknowledgment differs from turn to turn instead of always repeating the
  same sentence. *(Caller renamed to `schedule_start_filler` by the entry above,
  which also made the cue conditional.)*

### Why

- A single fixed acknowledgment ("Un momento…") becomes annoying across a long
  conversation; a random phrase from a curated list keeps the agent sounding
  natural, and the phrase list is editable straight from `.env`.
- Re-synthesizing 22 clips on *every* startup added ~21 s to the boot time
  (measured), so the bootstrap now tracks the phrase list and only re-synthesizes
  when it changed (or a clip is missing).

### Notes

- `force` was dropped from [`assets::ensure_clips`](src/assets.rs): the filler /
  backchannel banks keep the "generate only if missing" behaviour, and the start
  filler has its own list-driven bootstrap.
- [`server.sh`](server.sh) startup wait raised from ~20 s to ~60 s: the first
  startup after a list change synthesizes the whole bank before the socket is
  bound.
- A failed synthesis keeps the previous clip of that position; a barge-in still
  clears the start-filler channel (`AudioMixer::flush`), and
  `START_FILLER_ENABLED=false` disables it entirely.

---

## 2026-10-16 — Start filler: immediate turn-start acknowledgment on its own mixer channel

### Added

- **`START_FILLER_ENABLED` / `START_FILLER_TEXT` / `START_FILLER_DIR`** (`.env`,
  `.env.example`, [`src/config.rs`](src/config.rs)). The *start filler* is a
  pre-generated clip queued the moment a user turn begins — no delay — on a
  **dedicated mixer channel**, so it acknowledges the turn while the LLM/TTS are
  still working, independently of the delayed latency filler. *(Superseded by the
  entry above: `START_FILLER_TEXT` became the `START_FILLER_TEXTS` list and the
  clip is now picked at random.)*
- [`AudioMixer`](src/audio/mixer.rs): new `start_filler_queue` channel, mixed as
  `mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel ×
  BACKCHANNEL_GAIN)`, so the start filler shares the filler gain (idle `1.0`,
  ducked `0.15` under TTS). Included in `is_active()` and cleared by `flush()`.
- [`TurnManager::play_start_filler`](src/orchestrator/turn_manager.rs), called
  from `handle_user_turn` right after the in-flight reply is cancelled.
- Startup bootstrap synthesizes `START_FILLER_DIR/start_filler.raw` with the
  configured voice if it is missing ([`src/assets.rs`](src/assets.rs)); the clip
  is preloaded into its own [`FillerBank`](src/orchestrator/turn_manager.rs),
  kept out of the round-robin `FILLERS_DIR` bank.

### Why

- The delayed latency filler only fires after `FILLER_DELAY_MS` (and only if the
  reply is still silent), so with the current fast TTFB (~`0.73 s`) the agent can
  stay silent for a moment after the user stops speaking. The start filler
  provides an immediate acknowledgment instead, without touching the existing
  delayed filler.

### Notes

- *(Superseded — see the entry above.)* The clip was re-synthesized on every
  startup (`ensure_clips(..., force = true)`), so editing `START_FILLER_TEXT` took
  effect at the next restart with no manual deletion. A new clip was written only
  on success, so a failed synthesis left the previous clip in place; the filler /
  backchannel banks kept the "generate only if missing" behaviour.
- A barge-in clears the start filler along with every other channel
  (`AudioMixer::flush`); `START_FILLER_ENABLED=false` disables it entirely.

---

## 2026-10-16 — Fixed micro-gaps at the start of each sentence → `TTS_PREBUFFER_MS` 170 → 550 ms, `LLM_STREAM_CHUNK_CHARS` 120 → 240

### Changed

- **`TTS_PREBUFFER_MS` raised `170` → `550`** (`.env`, `.env.example`). With the
  current vocoder windows (first `185.8` ms, steady `371.6` ms) any pre-buffer in
  `(186, 557]` ms primes playback on the *same* second window, so `550 ms` takes
  the entire ~557 ms cushion **at no extra TTFA** (~`0.73 s`).
- **`LLM_STREAM_CHUNK_CHARS` raised `120` → `240`** (`.env`, `.env.example`). The
  first-chunk threshold stays at `20` (early flush preserved), but every following
  chunk is now twice as long → **fewer TTS requests, hence fewer chunk seams**.
- **README** (`Mixer & jitter buffering`, the `TTS_PREBUFFER_MS` /
  `LLM_STREAM_CHUNK_CHARS` tables and the TTFA note) updated to the new defaults
  and rationale. No Rust code changed — both values are read from `.env` at
  runtime.

### Why

- After the seam-quality retune left the **audible micro-interruptions at the
  start of sentences**. Diagnosis from `run/server.log`: the defect is a **TTS
  queue underrun at the chunk seams**, not a timbre artifact. Each sentence /
  fragment is a *separate* HTTP request whose PCM is concatenated with no
  crossfade; the relay advances chunk-by-chunk, so the first dip in production
  empties the queue and the mixer logs
  `TTS queue underrun — pausing playback until audio resumes` (94 occurrences in
  the sampled log; 0 `running low` warnings), then restarts with `need = 1` → an
  audible micro-gap at the start of the next sentence.
- **Why the queue empties:** with `TTS_PIPELINE_DEPTH = 3` on a single GPU the
  measured per-window cadence under concurrency is **552–613 ms (~1.5×)** versus
  **~350 ms (~0.94×)** for a lone request. A `170 ms` cushion has no margin
  against that; `550 ms` does, and it is latency-free because the pre-buffer
  already lands on the second window.
- Measured in the log at the seam (12:56:36–38): chunks of `16384` samples
  (`queue_len=16384`, `prebuffer=7497` = 170 ms) with two concurrent requests
  (`text_len=119` and `19`), each followed by an underrun-pause line.

### Notes

- Both levers are pure configuration; the binary is unchanged, so `./server.sh
  restart` (no rebuild) is enough to apply them.
- The `550 ms` cushion is the ~557 ms gap to the third window start, so raising
  it further only adds latency (it would wait for the *next* window).

---

## 2026-10-16 — Drama-3 vocoder retuned for seam quality → stride `4` / followup `8` / overlap `16`; `TTS_PREBUFFER_MS` 450 → 170 ms

### Changed

- **The Drama-3 launcher (`/root/drama3/start-drama3.sh`, pm2 `start-drama3`) now
  carries more decoder context per window.** The previous config (stride `4`,
  flat ramp `"[4,4,4]"`, overlap `8`, crossfade `256`) was replaced by:
  ```
  --vocoder.factory.stream_stride 4
  --vocoder.factory.stream_followup_stride_ramp "[8,8,8]"
  --vocoder.factory.stream_followup_stride 8
  --vocoder.factory.stream_overlap_tokens 16
  --vocoder.factory.stream_crossfade_samples 512
  --tts_engine.factory.enable_async_decode true
  ```
- **`TTS_PREBUFFER_MS` lowered `450` → `170`** (`.env`, `.env.example`,
  [`src/config.rs`](src/config.rs)). With stride `4` the *first* vocoder window is
  4 codec frames = **0.186 s**, so `170 ms` starts playback on that first window
  and **TTFA collapses to the server's own TTFB (~0.38 s)**.
- **README:** the *"Starting the Drama 3 backend (SGLang-Omni)"* subsection and
  the *"Mixer & jitter buffering"* pre-buffer note were rewritten around the new
  stride/overlap values and the ~0.186 s first / ~0.372 s steady window cadence.

### Why

- The previous latency-first config (stride `4`, overlap `8`) produced **audible
  micro-gaps and a slight detune / falsetto on long replies**. Root cause: with
  only `8` tokens of decoder warm-up context *per window*, and one TTS request
  per reply chunk (PCM concatenated without crossfade at the request seams), the
  streaming decoder's state was too shallow and its timbre/pitch wobbled at each
  window and request boundary.
- **Seam context restored without breaking real-time:** raising the overlap to
  `16` at stride `4` pushes decode to 5× real time, which the box **cannot**
  sustain (ratio rose to 1.18–1.98×). Raising *strides and overlap together* —
  stride `4` (first window) / followup `8` (steady) with overlap `16` — keeps the
  decode load at a sustainable **`(16+8)/8 = 3×`** (~0.95× ratio) while giving
  each steady window twice the context, and `crossfade 512` smooths the seams
  further.
- A/B measurements on the live endpoint (`measure_tts.py`, warm server, voice
  cloning, single request vs sentence-chunked like the engine) on a long
  multi-sentence reply: **the amplitude jump at the seam boundaries dropped from
  the 99.5+ percentile of the signal to 96–98**, i.e. the seams no longer stand
  out. TTFB stays **~0.38 s** (the first window is unchanged) and the real-time
  ratio stays **~0.94–0.98×**.

### Notes

- The first window is unchanged (4 frames), so the residual latency is still
  dominated by the server's **fixed ~0.38 s TTFB**; the steady window grew from
  0.186 s to **0.372 s** (8 frames) and arrives ~350 ms apart, so the pre-buffer
  of `170 ms` still starts playback on the very first window. Further latency
  gains would need a faster first token (model / preprocessing), not a smaller
  stride.
- `measure_tts.py` was added to the repo: it streams `/v1/audio/speech` and prints
  the per-window arrival schedule, TTFB and real-time ratio (plus a seam-jump
  summary), so future A/B runs are one command.

---

## 2026-10-16 — Mivocero: karaoke word highlighting on the assistant reply

### Changed

- **The `/mivocero` client now reveals the assistant reply in karaoke style**
  ([`src/web/mivocero.html`](src/web/mivocero.html)) instead of appending plain
  text. Each word is a `<span>`; words light up as they are *spoken*. The
  highlight is driven by the **audio playhead**: the playback AudioWorklet now
  counts the samples it has actually emitted and reports them (`played`) on its
  existing per-block message. The client maps that count onto a character offset
  with an estimated speaking rate (`KARAOKE_CPS`, ~15 chars/s) and marks each
  word `done` / `now` / pending.
- The old blinking-caret "pending" style was removed; the whole reply is shown at
  once (dim) and progressively illuminated.
- A safety net finishes the highlight: once `assistant_end` has been received and
  playback has been idle for 600 ms *after it had actually started*, any word
  still pending is marked done, so a reply always ends fully lit. New turns and
  interrupts cancel the loop.

### Notes

- The mapping from audio to text is inherently approximate (the server streams
  text and PCM on the same socket without per-word timing), so `KARAOKE_CPS` is a
  tunable estimate for the visual effect only — it never touches the audio.
- `played` counts only real samples, so the playback jitter buffer's silence and
  any underrun pauses do not advance the highlight.

---

## 2026-10-16 — End-to-end validation → `TTS_PREBUFFER_MS` 700 → 450 ms; Drama-3 startup documented

### Changed

- **`TTS_PREBUFFER_MS` lowered `700` → `450`** (`.env`, `.env.example`,
  [`src/config.rs`](src/config.rs)). With the vocoder ramp `"[10,12,16]"` the
  *first* window is 10 codec frames = 0.464 s, so `450 ms` sits just below it and
  playback now starts on that **first** window instead of waiting for the second
  one that `700 ms` required.
- **README:** new *"Starting the Drama 3 backend (SGLang-Omni)"* subsection under
  the `fish` plugin, with the exact `sgl-omni serve` command, the **pm2** workflow,
  the meaning of the two `--vocoder.factory.*` flags and the two CLI gotchas (no
  `stages.` prefix; the ramp tuple passed as a JSON string).

### Why

- An end-to-end browser test (`RUST_LOG=…,audio_agent_engine::tts=debug,…`) on the
  live engine confirmed the narrower windows: bursts of 10/12/16 frames
  (0.464 / 0.557 / 0.743 s) instead of a single 2.97 s block, `prebuffer=30870`
  samples (= 700 ms) and time-to-first-audio ≈ 1.1 s from dispatch (~0.64 s of
  which is the server's TTFB). Barge-in fired correctly on three interrupts.
- Three **tail underruns** were still observed at the end of short replies: e.g.
  2.46 s of audio produced in ~2.6 s (≈1.07× real time including the fixed
  ~0.64 s TTFB), so the queue drained just before the last words. The mixer
  recovered via resilient resume, but the gap is audible. `450 ms` targets the
  first 0.464 s window and trades a little more tail risk for a markedly earlier
  start.

### Notes

- The residual deficit is dominated by the server's **fixed ~0.64 s
  time-to-first-byte**, which no pre-buffer value can remove. Going lower would
  need a faster first token (model / preprocessing), not a smaller stride.

---

## 2026-10-16 — Finer TTS delivery granularity (Drama-3 vocoder window) → `TTS_PREBUFFER_MS` 700 ms

### Changed

- **The SGLang Drama-3 server now emits audio in ~0.74 s windows instead of
  ~2.97 s bursts.** Two flags were added to the server launcher
  (`/root/drama3/start-drama3.sh`, managed by `pm2` as `start-drama3`):
  `--vocoder.factory.stream_followup_stride 16` and
  `--vocoder.factory.stream_followup_stride_ramp "[10,12,16]"` — upstream
  defaults were `64` and `(10,12,32)`.
- **`TTS_PREBUFFER_MS` default lowered `900` → `700`** (`.env`, `.env.example`,
  [`src/config.rs`](src/config.rs)). 700 ms is deliberately just under one 0.743 s
  window, so playback starts on the *first* burst rather than waiting for a
  second one.

### Why

- Drama-3's streaming granularity is a **vocoder** setting, not an engine one:
  [`Drama3VocoderFactoryArgs`](../drama3/sglang-omni/sglang_omni/models/fishaudio_drama3/config.py)
  buffers `stream_followup_stride` codec frames before decoding and emitting, and
  `frame_length = hop_length * 4 = 2048` samples (46.4 ms at 44.1 kHz). The
  upstream `64` therefore means a 131 072-sample burst every ~2.7 s.
- Measured against the live endpoint (single, uncontended request, PCM stream):

  | | burst | period | ratio | first audio |
  |---|---|---|---|---|
  | before (`stride` 64) | 131 072 samples / 2.97 s | 2.70–2.78 s | 1.08× | 0.96 s |
  | after (`stride` 16) | 32 768 samples / 0.743 s | 0.69 s (σ ≈ 0.01 s) | 1.04× | 1.34 s |

- The finer window is what makes the small pre-buffer viable: the producer still
  runs slightly faster than real time (1.04×), but it now tops the queue up every
  0.69 s instead of every ~2.7 s, so arrival jitter is bounded by one window and
  no longer forces a multi-second buffer.
- Prefix caching (RadixAttention) was evaluated and **rejected**: the client
  already sends a constant prefix (`model` / `voice` / `references`), yet an A/B
  on the endpoint showed no TTFB difference (ref 0.62–0.64 s, no-ref 0.52 s) —
  that start-up is decode/vocoder, not prompt prefill.

### Notes

- Each window re-decodes `stream_overlap_tokens` (32) frames of context, so a
  smaller stride costs more vocoder compute: 96 frames decoded per 64 emitted
  becomes 48 per 16. The measured ratio moved 1.08× → 1.04×, still above real
  time; `stride` 8 (0.371 s windows, 40 frames decoded per 8 emitted) is the next
  step down and should only be tried with the ratio re-measured.
- The window lengths feed the vocoder's CUDA-graph capture set, so changing them
  requires a server restart.
- `stream_stride` (first window) and `stream_followup_stride_ramp` (the windows
  after it) were left near their defaults — only the steady-state value drives
  the granularity of a long reply.
- (Superseded: the `TTS_PREBUFFER_MS` default is now `450`, targeting the ramp's
  0.464 s *first* window — see the entry above.)

---

## 2026-10-16 — Resilient mixer resume + `TTS_PREBUFFER_MS` default 900 ms

### Changed

- **The mixer no longer re-fills the whole jitter buffer after a mid-reply
  underrun.** Previously, when the TTS queue emptied for more than the 200 ms
  drain grace, [`pull_frame`](src/audio/mixer.rs) cleared `tts_primed` *and*
  `tts_stream_end`, then required a full `tts_prebuffer_samples` refill before
  playback resumed — adding a second, self-inflicted pause on top of the real
  underrun. It also left the mixer "primed" between turns because
  `mark_stream_start` never re-armed `tts_primed`.
  Now:
  - the full pre-buffer is awaited **only at the first playback of a turn**; once
    playback has started, an underrun pauses and resumes as soon as *any* audio
    is queued again (`need = 1`);
  - `tts_stream_end` is no longer cleared on drain, so a genuine end-of-stream is
    still honoured;
  - `mark_stream_start` re-arms `tts_primed`, `tts_playback_started` and
    `tts_drain_frames` at the start of every turn.
- **`TTS_PREBUFFER_MS` default raised `400` → `900`** (`.env`, `.env.example`,
  [`src/config.rs`](src/config.rs)). With `400` the reply was still prone to
  drop-outs on long answers.

### Why

- Direct measurement of the SGLang Drama-3 endpoint (a single, uncontended
  request) showed it delivers **~1.08× real time, in bursts every ~2.7 s**
  (≈131 072 samples / 2.97 s per burst). Because the producer barely outruns the
  consumer and delivers in bursts, gapless playback requires a jitter buffer of
  roughly the burst period (~3 s); smaller buffers show a short real drop-out
  (pure silence) on long replies.
- The mixer fix removes the *avoidable* part of the pause (the full refill after
  an underrun) and recovers at the first byte, which is what makes the remaining
  gap equal to the backend stall rather than stall + refill.
- Default `900` is a deliberate latency-first choice: it keeps time-to-first-audio
  low (~1.8 s) and accepts a rare short drop-out on very long replies. Raising
  `TTS_PREBUFFER_MS` towards ~3000 trades that latency back for gaplessness.
  (Superseded: the default is now `700`, once the server's vocoder window was
  narrowed to ~0.74 s — see the entry above.)

### Notes

- The residual drop-out is a property of the TTS backend (~1× real time), not of
  the mixer: no buffering smaller than the backend's burst period can bridge a
  delivery stall.
- Relay diagnostics at `debug` level in [`run_llm_turn`](src/orchestrator/turn_manager.rs)
  log per-chunk relay timing (`TTS pipeline ...`) to help localise seams.

---

## 2026-10-16 — Pipelined TTS: chunks are synthesized ahead of playback

### Changed

- **The TTS sink no longer waits for one chunk's stream to finish before
  requesting the next.** Previously [`run_llm_turn`](src/orchestrator/turn_manager.rs)
  looped `while let Some(chunk) = rx.recv() { tts.synthesize(chunk).await }`, so
  a chunk's synthesis only started after the previous chunk had finished
  streaming end-to-end. Because the TTS engine needs ~0.6 s to emit its first
  audio byte, every chunk boundary left a gap roughly that long, which the old
  1.5 s pre-buffer used to hide.
- The sink is now split into two pieces:
  - a **collector** that spawns one synthesis task per chunk as soon as the LLM
    produces it (bounded by a semaphore), each streaming into its own staging
    buffer;
  - a **relay** that forwards those staging buffers into the mixer queue
    **strictly in submission order**, so playback stays in order and gapless.
  A chunk's synthesis therefore overlaps the previous chunk's playback, hiding
  the per-request start-up latency.

### Added

- **New `TTS_PIPELINE_DEPTH` configuration (default `3`)** in
  [`src/config.rs`](src/config.rs): how many TTS requests may be in flight at
  once. `1` restores the previous strictly serial behaviour.
- `SampleRingBuffer::remaining_capacity()` ([`src/audio/ring_buffer.rs`](src/audio/ring_buffer.rs))
  so the relay never pushes past the mixer queue's free space — queued (i.e.
  currently playing) audio is never dropped to make room.

### Why

- Direct measurement against the SGLang Drama-3 endpoint showed a **fixed
  ~0.62 s start-up latency** per request (request → first audio byte), with
  audio then streamed at roughly real time. Two concurrent requests are
  **served in parallel** (start-up rises only to ~0.84–1.0 s), so overlapping
  requests genuinely removes the inter-chunk gaps instead of just moving them.
- The gaps were independent of chunk size (120 → 60 chars left the same ~0.8 s
  hole), which confirmed the cause was serialization, not chunk length.

### Notes

- Staging buffers are sized at 30 s of audio per in-flight request, which is
  comfortably more than any single chunk.
- With the pipeline in place the pre-buffer only has to cover arrival jitter
  rather than the TTS engine's per-request start-up, so it can be kept small.
  (Superseded: the default was later raised to `900` — see the entry above.)

---

## 2026-10-16 — Mixer TTS pre-buffer is now configurable (default 400 ms)

### Changed

- **The TTS jitter buffer moved from a hardcoded 1.5 s constant to the
  `TTS_PREBUFFER_MS` setting (default `400`).** `AudioMixer` (in
  [`src/audio/mixer.rs`](src/audio/mixer.rs)) now takes `tts_prebuffer_samples`
  as a constructor argument instead of reading the removed
  `TTS_PREBUFFER_SAMPLES` constant, and [`run_session`](src/ws/session.rs)
  derives it from the config.

### Why

- Log analysis of a voice turn (`"Conta da dieci a uno."`) showed the first TTS
  request left ~1.2 s after the EOU and the first audio bytes arrived ~1.9 s,
  but playback only started at **+4.2 s**: the mixer had 1.486 s (65 536
  samples) queued at +2.8 s, **14 ms short of the 1.5 s threshold**, so it
  waited for the next Fish chunk, which landed 1.39 s later. The pre-buffer,
  not the text chunker, dominated the perceived latency.
- Fish Audio now delivers near real time, so 1.5 s was far more buffer than
  needed. 400 ms removes ~1.1 s of time-to-first-audio while still absorbing
  normal inter-chunk jitter (the 200 ms drain grace is unchanged).

### Notes

- Tune from `.env`: raise `TTS_PREBUFFER_MS` if you hear drop-outs, lower it for
  an even snappier start.
- The low-buffer warning threshold now scales with the configured value
  (`prebuffer / 4`).

---

## 2026-10-16 — TTS starts on the first words, not on the first sentence

### Changed

- **The streaming reply is now chunked on a word boundary instead of waiting
  for sentence punctuation.** `SentenceSplitter` (in
  [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)) keeps
  the sentence-terminator rule (`.`, `!`, `?`, `;`, newline) but adds an early
  flush: the **first** chunk is cut as soon as
  `LLM_STREAM_FIRST_CHUNK_CHARS` characters are buffered, and every following
  chunk at `LLM_STREAM_CHUNK_CHARS`. The cut lands on the first whitespace at or
  past the threshold, so a chunk always holds at least that many characters and
  never splits a word. Previously the first chunk only left at a `.`/`!`/`?`,
  so a long opening sentence delayed the first spoken word.

### Added

- **New `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`) and
  `LLM_STREAM_CHUNK_CHARS` (default `120`) configuration** in
  [`src/config.rs`](src/config.rs). A small first threshold makes the agent
  start speaking on the first few words; the larger follow-up threshold keeps
  prosody good once the audio is flowing. Setting either to `0` disables that
  flush (chunks then end on sentence punctuation only).

### Notes

- Verified on the running server: for a chat-box turn the first TTS request is
  dispatched ~0.5 s after the user text, synthesizing `"Non ho occhi fisici né"`
  (23 chars), well before the rest of the sentence
  (`"accesso a telecamere o immagini, quindi non posso vedere nulla …"`) was
  generated. Before the change the whole opening sentence had to complete first.
- The remaining time-to-first-audio is dominated by the mixer's 1.5 s TTS
  pre-buffer plus the TTS engine's first-chunk latency, not by the chunker.

---

## 2026-10-15 — Filler is now a latency guard: switchable and delay-gated

### Changed

- **The end-of-turn filler is no longer played unconditionally.** It exists to
  mask the wait for the reply, so playing it when the answer is already speaking
  only gets in the way of the first words. `TurnManager::schedule_filler`
  (in [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs))
  starts a `FILLER_DELAY_MS` timer when the turn is dispatched (both the LLM and
  the direct-TTS path) and queues the clip **only if the response audio has not
  started by then**. At most one filler per turn.
- **New `FILLER_ENABLED` / `FILLER_DELAY_MS` configuration** in
  [`src/config.rs`](src/config.rs). `FILLER_ENABLED=false` disables the filler
  entirely (no clip is ever queued); `FILLER_DELAY_MS` (default `2500`) is the
  latency guard, tunable from `.env`.
- The immediate filler pushes in `handle_user_turn` and `speak` were removed in
  favour of the single scheduled path, so the playground's direct-TTS path
  follows the same rule.

### Added

- **`AudioMixer::tts_playback_started()`** in
  [`src/audio/mixer.rs`](src/audio/mixer.rs) — set when the TTS pre-buffer is
  reached (i.e. the response audio actually starts), reset by
  `mark_stream_start` and `flush`. This is what the filler decision reads.

### Notes

- The pending filler timer is bound to the turn's `CancellationToken`, so a
  barge-in (`interrupt`, VAD onset, a new turn, or disconnect) drops it.
- Verified from `.env` on the running server: `FILLER_DELAY_MS=100` →
  `pushed filler (delayed)`; `FILLER_DELAY_MS=8000` → `response audio already
  playing — filler skipped`; `FILLER_ENABLED=false` → no filler activity at all.
- With the current `LLM_ENABLE_THINKING=false` + Fish Audio setup, a short reply
  starts playing within 2.5 s, so the filler correctly stays silent; raise
  `FILLER_DELAY_MS` (e.g. 1000) if you want it to fire more often.

---

## 2026-10-15 — LLM reasoning stage + `/mivocero` speech-to-speech chat client

### Added

- **OpenAI-compatible LLM client** — [`src/llm.rs`](src/llm.rs). New
  `LlmClient` posting to `{LLM_BASE_URL}/chat/completions`, with an
  `Authorization: Bearer {LLM_API_KEY}` header and a bounded TCP connect
  timeout (no global request timeout, so SSE streams can stay open). Two
  entry points, selected at runtime via `LLM_STREAM`:
  - `chat_stream` — SSE token streaming (`stream: true`), handling `data:`
    lines and `data: [DONE]`;
  - `chat` — a single blocking request.
- **Chain-of-thought stripping** — reasoning models (Qwen3, …) may emit their
  plan in a separate `reasoning_content` delta (vLLM) or inline between
  think markers (llama.cpp and others). `reasoning_content` is ignored and
  inline think blocks are removed by an incremental `ThinkStripper` that is
  stable across delta boundaries (it withholds a tail the size of the opening
  marker so a marker split between two deltas is still detected). Applied to
  both the chat box text and the synthesized audio.
- **LLM stage in [`TurnManager`](src/orchestrator/turn_manager.rs)** — the user
  turn (STT transcript or chat-box text) is sent to the model and the reply is
  spoken by the TTS. A rolling history window (`LLM_HISTORY_MESSAGES`) is kept
  per connection and prefixed with `LLM_SYSTEM_PROMPT`. A `SentenceSplitter`
  chunks the streaming reply on `.`, `!`, `?`, `;` and newlines so the TTS
  starts on the first sentence (`SENTENCE_SOFT_MAX` = 220 chars forces a split
  on whitespace for punctuation-free runs).
- **Barge-in covers the LLM request** — an interrupt (client JSON, VAD onset or
  disconnect) cancels the in-flight HTTP request through a
  `CancellationToken` in the `reqwest` `tokio::select!`, cancels the TTS tasks
  and flushes the mixer.
- **Bidirectional text protocol on `/ws`** — client → server
  `{"type":"user_text","text":…}` and `{"type":"interrupt"}`; server → client
  `{"type":"user","text":…}`, `{"type":"assistant_delta","text":…}`,
  `{"type":"assistant_end"}` and `{"type":"error","text":…}`. Legacy text
  commands (`INTERRUPT`, `SET_REFERENCE:`, `CLEAR_REFERENCE`, raw text → direct
  TTS) are unchanged, so the playground keeps working.
- **`/mivocero` client** — [`src/web/mivocero.html`](src/web/mivocero.html),
  embedded via [`src/web/mod.rs`](src/web/mod.rs) and served by
  [`src/main.rs`](src/main.rs). A speech-to-speech example agent with a
  microphone (same AudioWorklet capture/playback path as the playground), a
  chat box that renders the user transcript and the **streaming** assistant
  reply, a text composer and an interrupt button.
- **`LLM_*` configuration** — `Config` fields and `read_llm_env()` in
  [`src/config.rs`](src/config.rs): `LLM_ENABLED`, `LLM_BASE_URL`,
  `LLM_API_KEY`, `LLM_MODEL`, `LLM_MAX_TOKENS`, `LLM_TEMPERATURE`,
  `LLM_STREAM`, `LLM_HISTORY_MESSAGES`, `LLM_ENABLE_THINKING`,
  `LLM_STRIP_REASONING`, `LLM_DIRECT_ANSWERS`, `LLM_SYSTEM_PROMPT`.
  `LLM_ENABLED` defaults to `true` when both `LLM_BASE_URL` and `LLM_MODEL`
  are set.
- **Three independent chain-of-thought controls** (each switchable from `.env`)
  — a single mechanism does not cover every deployment:
  1. `LLM_ENABLE_THINKING` — `LlmClient` sends
     `chat_template_kwargs: {"enable_thinking": <bool>}` when the variable is
     set, turning Qwen3-style thinking off/on server-side (vLLM/SGLang). Unset
     omits the vendor-specific field so strict OpenAI-compatible servers keep
     accepting the body.
  2. `LLM_STRIP_REASONING` (default `true`) — `reasoning_content` deltas are
     dropped and inline think blocks removed; `false` passes the raw output
     (both fields) through.
  3. `LLM_DIRECT_ANSWERS` (default `true`) — appends
     [`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt. Needed
     when the server runs **no** reasoning parser: the chain-of-thought is then
     written straight into `content` as ordinary prose and is neither
     `reasoning_content` nor tagged, so filtering cannot catch it.

### Fixed

- **Chat-box turns blocked the receive loop** — `handle_control_message` in
  [`src/ws/session.rs`](src/ws/session.rs) awaited `TurnManager::on_user_text`
  inline, so while a reply was being generated the session could not read any
  further client message: an `interrupt` sent during generation was only
  honoured once the turn had finished. Text turns now run in their own task,
  and the interrupt cancels the reply immediately (verified: 0 deltas emitted
  after the interrupt, session still usable for the next turn).
- **SSE cancellation latency** — `LlmClient::chat_stream` now races the next
  stream chunk against the `CancellationToken`, so a barge-in drops the HTTP
  response instead of waiting for the model's next token.

### Notes

- Without an LLM the pipeline degrades to a plain STT → TTS echo
  (`LLM_ENABLED=false`).
- `src/orchestrator/turn_manager.rs` gained unit tests for sentence splitting
  (terminators, split across deltas, long punctuation-free runs) and
  `src/llm.rs` for think-block stripping and for the opt-in thinking field —
  `cargo test` passes (13 passed, 1 ignored).
- Verified against `Qwen/Qwen3.6-35B-A3B` on a local vLLM
  (`http://10.8.0.4:8000/v1`): without `LLM_ENABLE_THINKING=false` the model
  narrates "Here's a thinking process: 1. **Analyze User Input** …" directly in
  `content`, which the TTS reads aloud; with it, the reply is the answer only.

---

## 2026-10-15 — Startup clip bootstrap (auto-generate filler / backchannel)

### Added

- **Startup asset bootstrap** — [`src/assets.rs`](src/assets.rs). On boot the
  server checks every expected filler / backchannel clip and, if missing,
  **synthesizes it with the configured TTS engine** and writes it as
  `*.raw` (16-bit LE mono PCM at 44.1 kHz) into the target directory (created
  if needed). Existing clips are never overwritten, so a fresh checkout works
  with no manual generation step and a customized/partial bank is preserved.
- Default clip definitions:
  [`assets::FILLER_CLIPS`](src/assets.rs) (`uhm`, `eh`, `mmm`, `aver`, `bueno`,
  `pues`, `vaya`) and [`assets::BACKCHANNEL_CLIPS`](src/assets.rs) (`mmhmm`,
  `aja`, `claro`, `yaveo`). Backchannel clips are generated only when
  `BACKCHANNEL_ENABLED=true`.
- Wired in [`src/main.rs`](src/main.rs) **before** the banks are loaded; the TTS
  engine is now created earlier so it can be reused for generation.

### Notes

- Generation is best-effort and never blocks startup: a synthesis failure logs a
  warning per clip and the server still starts (that bank stays silent).
- The scratch capture buffer during generation is a 30 s
  [`SampleRingBuffer`](src/audio/ring_buffer.rs) drained after synthesis.

---

## 2026-10-15 — Filler bank (round-robin end-of-utterance fillers)

### Added

- **Filler bank** — `assets/fillers/*.raw`. The end-of-utterance filler is no
  longer a single hard-coded clip; [`TurnManager`](src/orchestrator/turn_manager.rs)
  now selects clips **round-robin** from the bank via
  [`FillerBank::nth`](src/orchestrator/turn_manager.rs) (a per-turn cursor,
  `TurnManager::next_filler`). With a single clip the behaviour is unchanged.
- Clips synthesized with the cloned reference voice (Spanish, 44.1 kHz, 16-bit
  LE mono PCM): `uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya` (~0.1–0.9 s).

### Notes

- `FillerBank::get` is kept for reference/debugging but is no longer on the
  hot path (`#[allow(dead_code)]`).

---

## 2026-10-15 — During-utterance backchannel

### Added

- **Backchannel subsystem** — [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs).
  Emits short acknowledgments ("mm-hmm", "ajá", "claro") *while the user is
  speaking*, at a low fixed gain, to signal that the agent is listening without
  taking the turn. Distinct from the end-of-utterance **filler** injected by the
  [`TurnManager`](src/orchestrator/turn_manager.rs).
- **`BackchannelScheduler`** — per-session decision logic fed by the VAD:
  `on_speech_start` on `VadEvent::SpeechStart`, `on_speech_end` on `SpeechEnd`,
  and `maybe_emit()` on every frame while speaking. A cue is emitted only when
  the user has spoken for at least `BACKCHANNEL_MIN_SPEECH_MS`, at least
  `BACKCHANNEL_INTERVAL_MS` elapsed since the last cue, fewer than
  `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn, and the agent is
  **not** speaking (`!TurnManager::is_speaking()`).
- **Preloaded clip bank** — `assets/backchannel/*.raw` (16-bit LE mono PCM at
  44.1 kHz), synthesized with the cloned reference voice, loaded into a shared
  [`FillerBank`](src/orchestrator/turn_manager.rs) and played round-robin.
- **Third mixer channel** — [`AudioMixer`](src/audio/mixer.rs) gained a dedicated
  `backchannel_queue` and a fixed `backchannel_gain`. The mix is now
  `tts + (filler × filler_gain) + (backchannel × backchannel_gain)`; the queue is
  cleared by `flush()` (barge-in).
- **`FillerBank` generalization** — one type now serves both the filler and the
  backchannel banks (`load(dir, label)`, round-robin `nth(index)`, `is_empty()`);
  the default filler lookup ("uhm") is unchanged.
- **`VadDetector::in_speech()`** — exposes whether the detector is inside a
  confirmed utterance, used to gate backchannel emission.
- **New `.env` knobs** (documented in [`.env.example`](.env.example)):
  `BACKCHANNEL_ENABLED`, `BACKCHANNEL_DIR`, `BACKCHANNEL_GAIN`,
  `BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS`,
  `BACKCHANNEL_MAX_PER_TURN`.

### Notes

- `BACKCHANNEL_GAIN` is clamped to `0..1` and `BACKCHANNEL_INTERVAL_MS` is
  clamped to `>= BACKCHANNEL_MIN_SPEECH_MS` at load time.
- Wiring lives in [`src/ws/session.rs`](src/ws/session.rs) (mic/VAD task); the
  bank is created once in [`src/main.rs`](src/main.rs) and shared via `AppState`.

---

## 2026-10-15 — Server-side VAD (Silero V5, ONNX)

### Added

- **VAD subsystem** — [`src/vad/mod.rs`](src/vad/mod.rs) +
  [`src/vad/silero.rs`](src/vad/silero.rs). Runs **Silero VAD v5** via ONNX
  Runtime on the same 16 kHz mono stream that feeds the STT engine. The model
  consumes 32 ms windows (512 samples) with a 64-sample context and a recurrent
  `state [2,1,128]`; the graph shapes and the `sr` input rank are introspected at
  load time.
- **Onset/offset state machine** with hysteresis (`VAD_THRESHOLD` /
  `VAD_NEG_THRESHOLD`) emitting three events: `SpeechOnset` (fast, for
  barge-in), `SpeechStart`, and `SpeechEnd` (carrying the captured PCM plus a
  pre-roll pad so the first phoneme is not clipped).
- **Fast server-side barge-in** — a `SpeechOnset` interrupts the agent when it is
  currently speaking (`TurnManager::is_speaking`), cancelling the in-flight TTS
  and flushing the mixer.
- **VAD-driven STT segmentation** — `SttWorker::push_utterance` receives whole
  utterances instead of raw frames, so the STT no longer needs its own RMS
  segmentation (`VAD_SEGMENT_STT=true`, default).
- **Shared ONNX environment** — [`src/onnx.rs`](src/onnx.rs). ONNX Runtime allows
  a single environment per process, so the STT and VAD pipelines now share a
  lazily created `Environment` (cheap `Arc` clones) instead of each building one.
- **New `.env` knobs** (all documented in [`.env.example`](.env.example)):
  `VAD_ENABLED`, `VAD_MODEL_PATH`, `VAD_THRESHOLD`, `VAD_NEG_THRESHOLD`,
  `VAD_MIN_SPEECH_MS`, `VAD_MIN_SILENCE_MS`, `VAD_SPEECH_PAD_MS`,
  `VAD_BARGE_IN_MIN_SPEECH_MS`, `VAD_SEGMENT_STT`, `VAD_BARGE_IN`, `VAD_DEBUG`.
- **`AudioMixer::is_active`** / `tts_stream_active` flag tracking whether a TTS
  stream is in flight, independent of the queue length, so barge-in can gate on
  "the agent is speaking".

### Fixed

- Duplicate analysis window appended to the utterance at speech start (the
  pre-roll ring already contained it) — the utterance is now seeded from the
  pre-roll ring only.
- `server.sh` now also appends `audio_agent_engine::vad=debug` to `RUST_LOG`
  (mirroring the earlier TTS fix), so VAD debug logs are never filtered out.

### Verified

`models/silero_vad.onnx` is loaded at startup (`state_shape=[2,1,128]`,
`sr_rank=0`). A live WSS session streaming 44.1 kHz speech produced, in
`run/server.log`:

```text
INFO  ws::session: VAD active for this session threshold=0.5 min_speech_ms=250 min_silence_ms=300 …
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
DEBUG vad::silero: VAD speech start speech_ms=256 prob="1.000" preroll_samples=4992
DEBUG vad::silero: VAD speech end duration_ms=1144 samples=18304 avg_prob="0.609"
INFO  ws::session: VAD utterance → STT duration_ms=1144 samples=18304
INFO  orchestrator::turn_manager: EOU detected tokens=5 text=Me escucha.
```

Talking over the agent (after a text TTS prompt) produced:

```text
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
INFO  ws::session: VAD barge-in — interrupting agent prob="0.999"
INFO  orchestrator::turn_manager: interrupt received, flushing audio
```

---

## 2026-10-15 — TTS DEBUG logs actually reach `run/server.log`

### Problem

The `debug!` diagnostics added to the TTS subsystem never appeared in the log,
even though the code and the `DEFAULT_LOG_FILTER` fallback were in place. Root
cause: the parent shell (IDE) already exports `RUST_LOG=info`, so
`EnvFilter::try_from_default_env()` succeeded and the
`info,audio_agent_engine::tts=debug` fallback was never used. In `server.sh`
the same happened — `${RUST_LOG:-info,audio_agent_engine::tts=debug}` never
expanded the default because `RUST_LOG` was already set.

### Fixed

- [`server.sh`](server.sh) now **appends** the
  `audio_agent_engine::tts=debug` directive to whatever `RUST_LOG` it
  inherits (unless that target is already mentioned), so the TTS debug logs
  cannot be silently filtered out. `start` logs the effective filter.

### Verified

A live WSS session (`Hola, …`) that is kept open for the whole synthesis now
emits, in `run/server.log`:

```text
DEBUG audio_agent_engine::tts::fish_audio: TTS request dispatched endpoint=… has_reference=true text=…
DEBUG audio_agent_engine::tts::fish_audio: TTS time-to-first-byte ttfb_ms=0 bytes=8186
DEBUG audio_agent_engine::tts::fish_audio: TTS chunk received chunk=1 bytes=8186 samples=4093 queue_len=4093
DEBUG audio_agent_engine::tts::fish_audio: TTS stream summary chunk_count=9 total_samples=180224 audio_s="4.09" elapsed_ms=3347
```

> Note: closing the WebSocket mid-stream triggers `turn_manager.interrupt()`
> (barge-in), which cancels the TTS stream before any chunk arrives — so only
> `TTS request dispatched` is logged in that case.

---

## 2026-10-15 — Optional HTTPS/WSS + clearer microphone diagnostics

### Problem

Clicking "Connect & Start Mic" in the playground failed with a generic
`mic error`, with no WebSocket connection ever reaching the server. Browsers
only expose `navigator.mediaDevices.getUserMedia` in a **secure context**
(HTTPS or `localhost`); over plain HTTP on any other origin the API is simply
absent, so `startAudio()` threw before the socket was created.

### Added

- **Optional TLS termination** — [`src/tls.rs`](src/tls.rs). Set both `TLS_CERT`
  and `TLS_KEY` (PEM) in `.env` and the server serves the same Axum router over
  HTTPS/WSS via `tokio-rustls` + `hyper` (HTTP/1 upgrades enabled, so WebSocket
  handshakes work). Without them it stays on plain HTTP. The startup log now
  says `listening (HTTPS)` / `listening (HTTP)`.
- **`./server.sh cert [--force]`** — generates a self-signed certificate under
  `certs/` whose SAN covers `localhost`, `127.0.0.1`, `::1` and the host's IPs
  (`EXTRA_SAN` to append more). `start`/`restart` auto-generate it if TLS is
  enabled but the files are missing; `status` shows the scheme and cert state.

### Changed

- **Playground diagnostics** — [`src/web/playground.html`](src/web/playground.html)
  now checks for a secure context *before* requesting the mic and reports the
  precise cause (`insecure origin` / `NotAllowedError` / `NotFoundError`) with a
  remediation hint, instead of a bare `mic error`.
- New config fields `Config::tls_cert` / `Config::tls_key` (+ `tls_enabled()`),
  read from `TLS_CERT` / `TLS_KEY`.

### Notes

- No new crates are downloaded: `hyper`, `hyper-util`, `tokio-rustls`, `rustls`
  and `rustls-pki-types` were already present as transitive deps of `reqwest`
  (`ring` provider), so they are declared with `default-features = false`.

---

## 2026-10-15 — `server.sh` control script (start / stop / restart)

Added [`server.sh`](server.sh), a small bash helper so the server does not have
to be managed by hand.

### Usage

```bash
./server.sh start [--build]     # start (auto-builds if binary missing)
./server.sh stop                # graceful stop (SIGKILL after 10 s)
./server.sh restart [--build]   # stop + start
./server.sh status              # process / port / HTTP status
./server.sh logs [N]            # tail run/server.log (default 50, follow)
./server.sh build               # build the release binary only
```

- PID is tracked in `run/server.pid`; logs are appended to `run/server.log`.
- `start` waits until the port is accepting connections (the model load takes
  ~1.5 s) before reporting success.
- Overridable env vars: `PROFILE` (`release`|`debug`), `RUST_LOG`, `PORT`.
- The script `cd`s to its own directory first, so `.env` and the relative
  `models/parakeet` path resolve correctly regardless of the caller's cwd.

Documented in the README (Quick Start).

---

## 2026-10-15 — Parakeet TDT STT reimplemented in pure Rust (ONNX)

The previous STT was a placeholder that ran `argmax` over a single `.onnx`
file — it could not actually transcribe. It has been replaced with a complete,
dependency-free (no Python/NeMo) implementation of the NVIDIA **Parakeet TDT
0.6B v3** transducer pipeline in [`src/stt/parakeet.rs`](src/stt/parakeet.rs),
mirroring the reference in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr).

### Change

- **3-stage ONNX pipeline**, all sessions kept alive by the `ort` `Environment`:
  1. `nemo128.onnx` — log-mel features `waveforms`/`waveforms_lens` → `features`/`features_lens`.
  2. `encoder-model.int8.onnx` (preferred, auto-detected) or `encoder-model.onnx` —
     `audio_signal`/`length` → `outputs`[1,1024,T']/`encoded_lengths`.
  3. `decoder_joint-model.onnx` — greedy **TDT** decoding.
- **Runtime graph introspection**: `d_model`, `num_durations`, `state_layers` and
  `state_hidden` are read from the ONNX I/O shapes instead of being hardcoded.
- **Greedy TDT decode loop** (token head + duration head; blank `<blk>` = 8192;
  `max_tokens_per_step` from `config.json`, default 10). Encoder output is
  re-packed from `[1, d_model, T']` to contiguous `[T', d_model]` frames.
- **Internal text decoding**: `vocab.txt` (`<piece> <id>`, SentencePiece `▁` →
  space) plus the reference spacing normalizer (`\A\s|\s\B|(\s)\b`). The
  `Transcript` struct now carries a `text: String`.
- **Config**: `STT_MODEL_PATH` now points to a **directory** (`models/parakeet`
  by default, was `models/parakeet-tdt.onnx`). `STT_TOKENIZER_PATH` defaults to
  empty. Updated `.env`, `.env.example` and the README.
- **TurnManager**: [`on_eou`](src/orchestrator/turn_manager.rs) now uses
  `transcript.text` directly, falling back to the external tokenizer only when
  no text was produced.

### Notes / fixes found during integration

- The `decoder_joint` graph expects `targets` and `target_length` as **int32**
  (not int64); `encoder_outputs` and both state tensors are float.
- The int8 encoder (~650 MB, self-contained) is preferred over the fp32 encoder
  (42 MB + a 2.4 GB `.onnx.data` sidecar) for speed and simpler deployment.

### Verification

Opt-in end-to-end test decodes a real 16 kHz mono WAV through the model:

```
$ PARAKEET_TEST_WAV=/tmp/stt_test/solis16k.wav \
    cargo test --release -- --ignored test_infer_real_audio --nocapture
tokens: 67 | text: "Bienvenidos a esta fascinante exploración del universo.
  Como locutor profesional, les explicaré hoy cómo las ondas electromagnéticas
  viajan por el espacio, transformando nuestra comprensión del cosmos."
test stt::parakeet::tests::test_infer_real_audio ... ok
```

The transcription matches the reference transcript verbatim (the source clip was
trimmed to 12 s). `cargo test` (unit tests) also passes.

---

## 2026-10-15 — Fix: short utterances swallowed by the TTS pre-buffer

Short phrases produced **no audio at all**. The mixer only started playing TTS
once the queue held `TTS_PREBUFFER_SAMPLES` (1.5 s). If an utterance's total
length was below that threshold (e.g. `"Cristóbal Colón"` ≈ 1.11 s / 49152
samples), the pre-buffer was never reached, so `tts_primed` stayed `false` and
the queued samples were never drained.

### Change

- `src/audio/mixer.rs`: added an `tts_stream_end` flag with
  `mark_stream_start()` / `mark_stream_end()`. When the stream has ended, the
  mixer primes as soon as *any* sample is queued (`stream_ended && tts_len > 0`),
  bypassing the pre-buffer for short utterances. The flag is reset by
  `flush()`, `mark_stream_start()` and when the queue drains.
- `src/orchestrator/turn_manager.rs`: the spawned TTS tasks now call
  `mixer.mark_stream_start()` before synthesis and `mixer.mark_stream_end()`
  after it returns — only when the request was **not** cancelled, so barge-in
  does not falsely signal completion.

### Verification

```
TTS stream complete chunk_count=6 total_samples=49152 duration_s="0.64"
TTS pre-buffer reached — starting TTS playback tts_queue_len=49152 prebuffer=66150 stream_ended=true
```

A raw WebSocket client sending `"Cristóbal Colón"` now receives ~1.12 s of
non-silent audio (previously the same request yielded only silence frames).

---

## 2026-10-15 — Dual-stack listener (fix `localhost` / `::1` connection refused)

The HTTP server bound only to `0.0.0.0` (IPv4). On hosts where `localhost`
resolves to `::1` first (confirmed via `getent hosts localhost` → `::1 localhost`),
clients and preview/port-forwarding proxies that connect to `localhost` via IPv6
received `connection refused`, surfacing as *"service is unavailable"*.

### Change

- `src/main.rs`: the listener now binds to the IPv6 wildcard
  `SocketAddr::from((Ipv6Addr::UNSPECIFIED, port))` instead of `0.0.0.0`.
  With `net.ipv6.bindv6only = 0` (Linux default, verified on this host) this is
  dual-stack: it accepts both IPv6 (`::1` / `localhost`) and IPv4 clients.

### Verification

```
ss -ltnp | grep 8787  → LISTEN *:8787
curl http://[::1]:8787/     → 200
curl http://localhost:8787/ → 200
curl http://127.0.0.1:8787/ → 200
```

---

## 2026-10-14 — Modular TTS plugin system (engine-agnostic via `.env`)

The TTS layer was hard-coded to Fish Audio / SGLang Drama 3. To support
switching engines (e.g. Kokoro, or any future backend) without touching
orchestration code, the TTS subsystem is now a pluggable trait selected at
runtime via the `TTS_ENGINE` environment variable.

### New `TtsEngine` trait (`src/tts/mod.rs`)

```rust
#[async_trait]
pub trait TtsEngine: Send + Sync {
    fn name(&self) -> &str;
    fn sample_rate(&self) -> u32;
    async fn synthesize(&self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken) -> Result<()>;
    fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}  // default no-op
}
```

- Every plugin streams **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44.1 kHz)**
  into the shared mixer queue. If an engine's native rate differs, it resamples
  internally (the mixer runs at a fixed rate).
- `synthesize()` must honor the `CancellationToken` for barge-in.
- `set_reference()` defaults to a no-op; only engines that support voice
  cloning override it.

### Factory (`src/tts/mod.rs::create_tts_engine`)

```
TTS_ENGINE=fish      → FishAudioTts  (Fish Audio / SGLang Drama 3)
TTS_ENGINE=kokoro    → KokoroTts     (stub, implement in src/tts/kokoro.rs)
```

Unknown values panic at startup with a clear message.

### Bundled plugins

| Plugin | File | Status |
|---|---|---|
| `fish` (Fish Audio Drama 3) | `src/tts/fish_audio.rs` | Production-ready (renamed from `sglang_client.rs`) |
| `kokoro` | `src/tts/kokoro.rs` | Stub — HTTP streaming call left as `TODO` with a commented example; includes a 24 kHz → 44.1 kHz linear resampler helper |

### `.env` loading

- Added `dotenvy` to auto-load a `.env` file at startup (existing env vars take
  precedence).
- Added `.env.example` documenting all env vars including `TTS_ENGINE`,
  `KOKORO_URL`, `KOKORO_VOICE`.

### Files changed

- `src/tts/mod.rs` — new `TtsEngine` trait + `create_tts_engine` factory.
- `src/tts/fish_audio.rs` — renamed from `sglang_client.rs`; `FishAudioTts` now implements `TtsEngine`.
- `src/tts/kokoro.rs` — new Kokoro plugin stub (example for adding engines).
- `src/tts/sglang_client.rs` — deleted (moved to `fish_audio.rs`).
- `src/config.rs` — new `tts_engine` field, `TTS_ENGINE` env var (default `fish`).
- `src/orchestrator/turn_manager.rs` — `tts: SglangTtsClient` → `tts: Arc<dyn TtsEngine>`.
- `src/ws/handler.rs` — `AppState.tts` → `Arc<dyn TtsEngine>`.
- `src/main.rs` — uses `create_tts_engine(&config)`; loads `.env` via `dotenvy`.
- `Cargo.toml` — added `async-trait`, `dotenvy`.
- `.env.example` — new file documenting all configuration env vars.

### How to add a new engine

1. Create `src/tts/myengine.rs` and implement `TtsEngine`.
2. Add `pub mod myengine;` to `src/tts/mod.rs`.
3. Register it in the `create_tts_engine` match.
4. Set `TTS_ENGINE=myengine` in `.env`.

### Build / verification

- `cargo build` — 0 errors, 2 dead-code warnings (Kokoro stub fields).
- `cargo test` — 4/4 passed.
- Server starts with `TTS_ENGINE=fish` (default): log shows `selected TTS engine engine="fish"` and voice-cloning reference loaded.

---

## 2026-10-14 — Cancel TTS on WebSocket disconnect

When the client disconnected, the in-flight TTS synthesis task continued
streaming from Fish Audio even though the WebSocket session was already closed,
wasting bandwidth and server resources.

### Fix (`src/ws/session.rs`)

- Added `turn_manager.interrupt()` in the session cleanup path (after the
  receiver ends, before aborting the sender/mic tasks).
- This cancels the TTS `CancellationToken`, flushes the mixer queues, and stops the
  streaming HTTP request immediately.

---

## 2026-10-14 — Fix WebSocket frame timing (Burst ticker)

The 20 ms audio sender ticker used `MissedTickBehavior::Skip`, which dropped
tick beats whenever the Tokio event loop was busy (e.g. processing a large
Fish Audio TTS chunk). Each skipped tick meant a 20 ms gap in frame delivery
to the browser, producing audible micro-interruptions even though the server's
TTS queue was well-buffered.

### Fix (`src/ws/session.rs`)

- Changed `MissedTickBehavior::Skip` → `MissedTickBehavior::Burst`.
- Missed ticks now trigger immediate back-to-back sends so the browser
  receives all due frames. The browser's jitter buffer (500 ms) absorbs the
  burst without underrunning.

---

## 2026-10-14 — Micro-interruption fix (grace period on TTS drain)

Even with the 1.5 s server pre-buffer and 500 ms browser jitter buffer,
brief audio gaps could still occur when the TTS queue momentarily hit zero
between Fish Audio chunks. The mixer immediately reset its `tts_primed`
state, forcing a full 1.5 s re-prime on the next chunk. The browser
playback processor did the same, re-entering buffering mode on any
underrun.

### Server fix (`src/audio/mixer.rs`)

- Added `tts_drain_frames` counter (`Arc<AtomicUsize>`) and
  `TTS_DRAIN_GRACE_FRAMES = 10` (200 ms).
- When the TTS queue is empty while primed, the mixer now outputs silence
  for up to 10 consecutive frames before resetting `tts_primed`. This
  absorbs brief inter-chunk gaps without re-triggering the pre-buffer.
- `flush()` also resets `tts_drain_frames`.

### Browser fix (`src/web/playground.html`)

- `PlaybackProcessor` now tracks `underrunSamples` and only re-enters
  buffering mode after 100 ms (`sampleRate * 0.10`) of continuous
  underrun. Brief underruns produce silent output but keep playback
  running, so audio resumes immediately when the next WebSocket frame
  arrives.

---

## 2026-10-14 — Prevent overlapping TTS streams (double-Speak fix)

When the user clicked **Speak** twice in quick succession, two TTS synthesis
tasks ran concurrently and both pushed PCM into the same mixer queue. This
produced overlapping/garbled audio and roughly doubled the total duration
(one observed run produced 83 s of audio for a ~42 s text).

### Server fix (`src/orchestrator/turn_manager.rs`)

- `speak()` now calls `self.mixer.flush()` immediately after
  `cancel_current_tts()`. This clears any remaining PCM from the previous
  utterance so the new one starts from a clean queue.

### Client fix (`src/web/playground.html`)

- The **Speak** button is now disabled (shows "Speaking...") as soon as a
  request is sent.
- A 1.5 s idle timer re-enables it once the server stops sending audio
  frames, preventing accidental double submissions.

---

## 2026-10-14 — Jitter buffer fixes (no more audio drop-outs)

Fish Audio Drama 3 streams PCM in unevenly-spaced chunks, which caused brief
audible drop-outs (silence gaps) during playback. Fixed with a double jitter
buffer.

### Server-side TTS pre-buffer (`src/audio/mixer.rs`)

- Added `TTS_PREBUFFER_SAMPLES` ≈ 200 ms of audio (44.1 kHz × 0.20).
- The mixer now tracks a `tts_primed` flag (`Arc<AtomicBool>`). TTS samples are
  **not** consumed until the TTS queue holds at least the pre-buffer amount.
- During the priming window only the filler ("uhm") backchannel is emitted, so
  the user hears an immediate response while TTS audio builds up.
- When the TTS queue empties (stream exhausted), `tts_primed` resets to `false`
  for the next request. `flush()` also resets it (barge-in).
- `AudioMixer::pull_frame` now uses `tts_queue.len()` instead of `is_empty()`
  for priming decisions.

### Browser-side playback jitter buffer (`src/web/playground.html`)

- The `PlaybackProcessor` AudioWorklet now starts in a `buffering` state and
  only begins playing once the ring buffer holds ≥ 200 ms of samples.
- On underrun (buffer drained mid-playback) it returns to `buffering` to avoid
  stuttering, then resumes once enough audio has re-accumulated.
- Added `available()` helper to compute the fill level of the ring buffer.

### Build / verification

- `cargo build` — 0 warnings.
- `cargo test` — 4/4 passed.

---

## 2026-10-14 — Voice cloning + 44.1 kHz native output

### Sample rate aligned to Fish Audio native output

- Changed `OUTPUT_SAMPLE_RATE` from **24 000 Hz** to **44 100 Hz** to match Fish Audio Drama 3's native PCM output.
- Removed the TTS anti-aliasing resampler from `sglang_client.rs` — Fish Audio PCM samples are pushed directly into the mixer queue. This eliminates any resampling artifacts ("metallic" sound) and pitch/speed mismatches.
- The browser `AudioContext` now runs at 44 100 Hz, so playback is sample-accurate.
- `OUTPUT_FRAME_SAMPLES` is now 882 (44.1 kHz × 20 ms), `OUTPUT_FRAME_BYTES` = 1764.
- Mic → STT path: browser capture resamples to 44.1 kHz, server resamples 44.1 kHz → 16 kHz for Parakeet (linear interpolation, STT-only path so aliasing is not a concern).

### Voice cloning (reference audio)

Added support for Fish Audio's voice-cloning `references` field.

**Config (env vars, applied at startup):**

| Env var | Description |
|---|---|
| `TTS_REFERENCE_AUDIO` | Path to a WAV file on the Fish Audio server filesystem. |
| `TTS_REFERENCE_TEXT` | Reference transcript. If it points to an existing `.txt` file, its contents are loaded; otherwise the raw string is used. If unset, the server looks for a `<audio-stem>.txt` sibling of the WAV. |

**Default:** `solis.wav` + auto-loaded `solis.txt` from `/root/pipecat-voice/voice_profiles/`.

**Runtime override (playground → server WebSocket text messages):**

- `SET_REFERENCE:<path>|<text>` — set or change the reference. `<text>` may be empty to re-use the sibling `.txt`.
- `CLEAR_REFERENCE` — disable voice cloning (use the model's default voice).

**Files changed:**

- `src/config.rs` — new `tts_reference_audio` / `tts_reference_text` fields, `Config::tts_reference()` resolver with sibling-`.txt` fallback, new env-var parsing.
- `src/tts/sglang_client.rs` — `Reference` struct, `references` field in `SpeechRequest` (omitted when `None`), runtime `set_reference()` backed by a `parking_lot::RwLock<Option<Reference>>`.
- `src/orchestrator/turn_manager.rs` — forwards `set_reference()` to the TTS client.
- `src/ws/session.rs` — handles `SET_REFERENCE:` / `CLEAR_REFERENCE` control messages.
- `src/web/playground.html` — new "Voice Cloning" card with WAV path + transcript inputs and Set/Clear buttons; reference status indicator.

### Build / verification

- `cargo build` — 0 warnings.
- `cargo test` — 4/4 passed (tokenizer).
- Direct Fish Audio `references` call confirmed HTTP 200 + `x-sample-rate: 44100`.
- End-to-end WebSocket test: `SET_REFERENCE` followed by a text-to-speak request streams 44.1 kHz PCM back to the client.