roomote_PRE/piperust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1cafe3dd3c43b19d32125d3d2f8cfb9ce2243bb2
piperust/UPDATES.md
T
valenti 1cafe3dd3c fixes tool callong
2026-10-21 10:44:49 +01:00

2351 lines
113 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Audio Agent Engine — Updates
This file documents all notable changes to the audio-agent-engine project.
Entries are written in English and ordered newest-first.
---
## 2026-10-21 — Feat: every HTTP verb for tools (list / get / add / update / delete / …)
### Changed
- **`method` now accepts any HTTP verb** ([`src/tools.rs`](src/tools.rs)),
case-insensitive: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`,
`TRACE` and custom verbs (e.g. `PURGE`). Before this change anything other than
`GET` fell through to `POST`, so `PUT`/`PATCH`/`DELETE` tools silently became
POSTs.
- **Argument placement follows the verb.** The arguments left after the URL
placeholders go to the **query string** for the read-style verbs
(`GET`, `HEAD`, `OPTIONS`, `TRACE`, `DELETE`) and to the **JSON body** for
every other verb (`POST`, `PUT`, `PATCH`, custom); body-style verbs always send
a body (`{}` when nothing is left). This makes a CRUD surface expressible:
`GET /items?limit=10`, `GET /items/{id}`, `POST /items`,
`PATCH|PUT /items/{id}`, `DELETE /items/{id}`, `HEAD /items/{id}`.
- **New helpers** `http_method()`, `args_in_body()` and `place_args()` in
[`src/tools.rs`](src/tools.rs); `ToolRegistry::execute` now builds the request
through `Client::request(method, url)` instead of separate `get`/`post`
branches.
- An invalid verb is **warned at load** and, at execution time, returned to the
model as `Error: invalid HTTP method '<verb>' for tool '<name>'` (never a
panic, never a blocking failure).
### Notes
- `DELETE` is **query-style**: put the id in the URL (`/items/{id}`) and any
extra argument ends up as a query parameter.
- Tests: two new unit tests (`every_http_verb_is_accepted`,
`args_placement_follows_the_verb`) — `cargo test --bin audio-agent-engine --
tools::` → 9 passed.
- Docs updated: [TOOL_CALLING.md](TOOL_CALLING.md) (IT/EN/ES, per-verb argument
table + full CRUD example set) and [README.md](README.md#tool-calling).
---
## 2026-10-21 — Feat: runtime tool definitions — `GET` / `POST /api/tools`
### Added
- **New REST endpoint `POST /api/tools`** ([`src/tools_api.rs`](src/tools_api.rs)).
The body is the **same JSON array** used by `LLM_TOOLS_FILE`; it installs a new
set of tool definitions at runtime, **without a restart** (until now the file
was only read once at startup). The definitions apply from the **next turn**
on.
- **`GET /api/tools`** reports the active definitions (in the OpenAI `tools`
shape they are advertised to the model with) plus their origin: `source` is
`file`, `request` or `none`.
- **Errors:** `400` empty body / non-UTF-8 / invalid JSON / JSON array with no
tool definition, `503` LLM disabled. Responses use the shared
`{ok, …}` / `{ok:false,error}` shapes.
- **OpenAPI/Swagger UI:** both handlers are documented (tag `tools`, schema
`ToolsResponse`) in [`src/apidoc.rs`](src/apidoc.rs) and appear at `/docs`.
### Changed
- **`ToolRegistry` can be built from a string** — `ToolRegistry::from_json`
([`src/tools.rs`](src/tools.rs)) is the new core builder; `load` now reads the
file and delegates to it, so a body accepted by the endpoint is also valid as
an `LLM_TOOLS_FILE` (and vice versa). Added `len()` / `names()`.
- **The LLM client's registry is swappable** — `LlmClient::tools` moved from
`Option<Arc<ToolRegistry>>` to an `Arc<RwLock<…>>` with `set_tools(…)` /
`tools_from_request()` ([`src/llm.rs`](src/llm.rs)). The registry is re-read on
**every** request and at the start of every turn
([`turn_manager.rs`](src/orchestrator/turn_manager.rs)), so an override takes
effect on the next turn while a session in flight is unaffected.
- Docs: new [TOOL_CALLING.md](TOOL_CALLING.md) (IT/EN/ES operating guide) and a
"Runtime definitions" subsection in [README.md](README.md#tool-calling).
### Notes
- The override lives **in memory only**: it never writes the tools file, and a
restart restores the `LLM_TOOLS_FILE` definitions. When the endpoint is never
called, the file definitions stay in use — that remains the default.
- Verified on the running service: `GET` before any POST →
`{"count":2,"source":"file"}`; after a POST → `{"count":1,"source":"request"}`;
invalid JSON and `[]` → `400`; restart → back to
`{"count":2,"source":"file"}`.
---
## 2026-10-17 — Feat: `POST /api/transcribe` — Parakeet transcription of an uploaded audio file
### Added
- **New REST endpoint `POST /api/transcribe`** ([`src/transcribe.rs`](src/transcribe.rs)).
The body is the raw audio file (any container `ffmpeg` can read: wav, mp3,
m4a, ogg, flac…). The file is decoded to **mono 16-bit PCM at 16 kHz**
(`ffmpeg -ac 1 -ar 16000 -f s16le pipe:1`) and the **already-loaded** Parakeet
TDT model from [`AppState::stt`](src/ws/handler.rs) transcribes the whole take
in a single inference on a blocking thread (the inference is CPU-bound and
holds the ONNX session lock). Response:
`{"ok": true, "text": …, "duration_ms": …, "tokens": …}`.
- **Errors:** `400` empty body, `503` STT model not loaded, `502` ffmpeg decode
failed, `500` temp-file / inference failure. Uploads capped at **32 MB** (same
as `/api/customvoice`).
- **OpenAPI/Swagger UI:** the endpoint is documented (tag `transcribe`, schema
`TranscribeResponse`) in [`src/apidoc.rs`](src/apidoc.rs) and appears at
`/docs`.
### Notes
- **Whole-file, single transcript** by design (no VAD segmentation) and **no
`lang` parameter**: Parakeet TDT v3 is multilingual, the language is
auto-detected.
- No second ONNX environment is created — the model is shared with the live
session (respects the one-environment-per-process constraint). The upload is
spilled to a temp file (ffmpeg needs a seekable input) and removed afterwards.
- Verified end-to-end: `voice_profiles/solis.wav` (14.9 s, Spanish) →
`duration_ms=14918`, `tokens=83`, correct transcript; empty body → `400`.
---
## 2026-10-17 — Feat: OpenAPI document + Swagger UI at `/docs`
### Added
- **Swagger UI for the REST surface.** The engine now serves an interactive
Swagger UI at **`GET /docs`** backed by an OpenAPI 3 document at
**`GET /api-docs/openapi.json`**. The document is derived from the handlers
with [`utoipa`](https://docs.rs/utoipa) (`utoipa = "5"`, `axum_extras`) and
rendered by [`utoipa-swagger-ui`](https://docs.rs/utoipa-swagger-ui) v8
(axum 0.7). New module [`src/apidoc.rs`](src/apidoc.rs) holds the
`#[derive(OpenApi)]` document (`ApiDoc`) and the response schemas
(`PhrasesResponse`, `PhraseEntry`, `SetVoiceResponse`, `ClearResponse`,
`ErrorResponse`); the endpoints carry `#[utoipa::path]` annotations.
- **Documented endpoints:** `GET /health`, `GET /api/customvoice/phrases`,
`POST /api/customvoice` (query params `lang` / `text` via
`#[derive(IntoParams)]` on `SetParams`; raw `application/octet-stream` body),
`POST /api/customvoice/clear`. WebSocket (`/ws`) and the HTML client pages are
intentionally excluded (streaming / static assets).
### Notes
- The `zip` crate is pinned to the **2.2.x** line in `Cargo.toml`: the
`utoipa-swagger-ui` 8.1 build script (which unpacks the embedded UI assets)
targets the older zip 2.x API and does not compile against zip 2.3+/2.6. The
root `axum` stays at 0.7.9 — `utoipa-swagger-ui` 8.1 resolves to it, so no
duplicate Axum is pulled in.
- Swagger UI is only for the REST endpoints; `/docs` requires the same
HTTP/HTTPS origin as the engine.
---
## 2026-10-17 — Feat: custom voice accepts an uploaded audio file + typed transcript
### Added
- **Upload source on the `/customvoice` page.** A **Source** toggle now offers
two ways to build the reference: **Record now** (the existing microphone flow,
read a predefined phrase for ≥ 10 s) or **Upload a file** — pick any audio
file `ffmpeg` can read (`accept="audio/*"`: wav, mp3, m4a, ogg, flac…) and
**type (or paste) its transcript**. In upload mode the transcript is required
and gates the *Use it* button, since the typed text is what drives the clone.
Both modes share the same *Use it* / *Reset* actions and preview player.
- **`text` query parameter on `POST /api/customvoice`.** The endpoint is now
`POST /api/customvoice?lang=<code>[&text=<transcript>]`. A non-blank `text`
overrides the language's predefined reading phrase for the transcript written
to the sibling `.txt` and passed to `set_reference`; when absent or blank the
predefined phrase is used as before (so a plain recording is unchanged).
Backend: `SetParams` gained `text: Option<String>` and `set_custom_voice`
resolves the transcript from it (`src/customvoice.rs`).
### Notes
- No format constraints are imposed on the upload: the same ffmpeg probe used
for recordings handles whatever container is sent. The 32 MB body cap already
covers uncompressed WAVs.
- README updated (feature bullet, usage comment, *Custom voice builder* flow and
API/route tables, `text` parameter).
---
## 2026-10-17 — Fix: short utterances ("uno", "due") dropped by the STT gate
### Fixed
- **The probability average now counts voiced windows only.** The gate's
`avg_prob` was the mean of the raw VAD probability over **every** window from
speech start to segment close — including the `VAD_MIN_SILENCE_MS` (700 ms)
hangover that ends the utterance. For a long phrase that silence is a small
fraction, but for a short word it is most of the segment: "uno" (~350 ms of
voice + ~700 ms of silence) averaged **0.065–0.199** and was discarded as
`too_soft`, even though the user was plainly speaking. In the same session the
long phrases passed at 0.32–0.60. The average is now taken only over windows
whose probability is ≥ `VAD_NEG_THRESHOLD` (`src/vad/silero.rs`), so the voiced
portion scores its true value and short real words clear the gate while noise
still does not.
### Notes
- `VAD_MIN_AVG_PROB` stays at `0.25`; its meaning is now stricter (it no longer
benefits from the silence dilution), but genuine short words score well above
it. README / `.env` / `.env.example` / `src/vad/mod.rs` docs aligned.
- Tune with `VAD_MIN_AVG_PROB`: raise it if noisy segments become hallucinated
turns, lower it if genuinely soft speech is still dropped.
---
## 2026-10-17 — Fix: too many utterances dropped by the STT gate
### Fixed
- **`VAD_MIN_AVG_PROB` lowered `0.45` → `0.25`.** Log analysis showed **62
drops, all `too_soft`** (none for duration), still occurring after a restart.
Accepted utterances measured `avg_prob` 0.47–0.76 while dropped ones measured
0.08–0.45 — including plainly real speech cut by a hair (2136 ms @ 0.428,
3576 ms @ 0.436). The gate sat almost on top of the per-frame speech threshold
(`VAD_THRESHOLD=0.5`): a real utterance hovers around the threshold on its
onset/release, so its *average* lands just under 0.45 and the segment is
discarded. The `0.25` default leaves a proper margin — genuine soft speech
(0.3–0.45) now passes, while clearly noisy segments (below 0.25) are still
filtered. Update `VAD_MIN_AVG_PROB` in `.env` (default, `.env.example`,
`src/vad/mod.rs` and README aligned).
### Notes
- Tune with `VAD_MIN_AVG_PROB`: lower it further if genuine speech is still
dropped, raise it if noisy segments are turning into hallucinated turns.
`0.0` disables the probability gate.
---
## 2026-10-17 — Fix: custom voice still louder than the default voice
### Fixed
- **Custom voice level pushed below the reference.** Matching the shipped
reference at the *integrated* level was not enough: a measured take lands at
≈`-23.25 LUFS` against `solis.wav`'s ≈`-23.44 LUFS`, i.e. the same loudness,
yet it still sounded louder. The difference is **dynamic range and peaks** —
the user take measures ≈`6.2 LRA` / `-2.2 dBTP` versus `1.6 LRA` / `-7.8 dBTP`
for the mastered reference, so its peaks sit ~5.5 dB hotter and read as
"louder". The normalization target is now `-26 LUFS` with a `-5 dBTP`
ceiling, which drops a take ~3 dB and pulls its peaks down to ≈`-5 dBTP` so a
custom voice sits just under the default instead of on top of it.
- Defaults changed: `CUSTOMVOICE_TARGET_LUFS` `-23` → **`-26`**,
`CUSTOMVOICE_TRUE_PEAK` `-1.5` → **`-5`** (code, `.env`, `.env.example`,
README).
### Notes
- Lower `CUSTOMVOICE_TARGET_LUFS` further for a quieter custom voice; the value
is applied at record time, so a take must be re-recorded (or the endpoint
re-posted) for a change to take effect.
- The custom voice lives only in memory (`TtsEngine::set_reference`): a restart
reverts the reference to the configured `TTS_REFERENCE_AUDIO` voice.
---
## 2026-10-17 — Fix: custom voice louder than the default voice
### Fixed
- **Loudness target aligned with the shipped reference.** The custom-voice
chain normalized to `-18 LUFS` while `voice_profiles/solis.wav` measures
≈`-23.4 LUFS`, so a custom voice came out ~5.4 dB louder than the voice it
replaced. The default is now **`-23`** (`CUSTOMVOICE_TARGET_LUFS`), matching
the reference — swapping voices no longer changes how loud the agent sounds.
- **Mono downmix moved inside the filter chain.** `-ac 1` used to be applied at
the output stage, i.e. *after* loudnorm: a **stereo** upload was measured as
stereo and then folded to mono, landing the written file ~3 dB below the
target. `aformat=channel_layouts=mono` now runs first, so loudnorm measures
the signal that is actually written.
### Notes
- Verified: `solis.wav` (stereo, -23.4 LUFS) and a quiet mono take (-48.6 LUFS)
both render to exactly **-23.00 LUFS**, mono 44.1 kHz. `cargo test` → 25
passed.
---
## 2026-10-17 — Post-processing for the custom voice (level match + cleanup)
### Added
- **Post-processing chain** for a recorded custom voice
(`src/customvoice.rs`), applied before the reference is installed:
`highpass=f=80` (rumble/DC), optional `afftdn` denoise + `deesser`,
leading/trailing silence trim (`silenceremove` + `areverse`) and a
**two-pass EBU R128 `loudnorm` with `linear=true`**.
- **New env knobs**: `CUSTOMVOICE_POSTPROCESS` (default `true`),
`CUSTOMVOICE_DENOISE` (default `true`), `CUSTOMVOICE_TARGET_LUFS`
(default `-18`), `CUSTOMVOICE_TRUE_PEAK` (default `-1.5`).
### Notes
- The loudness stage measures the cleaned signal first, then applies the result
as a **single constant gain** — a per-frame (single-pass) norm is what makes
takes jump in volume, which is the problem this fixes. Verified: a take at
**-48.6 LUFS** and one at **-17.5 LUFS** both render to **≈ -18 LUFS** with
TP at **-1.5 dBTP**.
- Any stage failing (unreadable take, all-silence clip, filter quirk) degrades
to the plain transcode with a warning — a take is never lost.
- Error handling verified: garbage input → `502` with a clean message, unknown
language → `400`. `cargo test` → 25 passed.
---
## 2026-10-17 — Custom voice builder (`/customvoice`) + Mivocero side menu
### Added
- **Custom-voice builder page** at `GET /customvoice`
(`src/web/customvoice.html`). It asks the user for a **language**, shows a
predefined phrase to read (long enough for cloning, ~10 s), records it with
`MediaRecorder` and installs the take as the active voice. The **Stop**
button is disabled until **10 s** have elapsed (timer + progress bar), the
take auto-stops at 20 s, and it can be played back and discarded before use.
- **Backend endpoints** (`src/customvoice.rs`): `GET /api/customvoice/phrases`
(reading phrases per language + the configured default), `POST
/api/customvoice?lang=<code>` (raw recording body), `POST
/api/customvoice/clear`. The body cap of the upload route is raised to 32 MB.
- **Side menu** on `/mivocero` (☰ drawer) linking to the chat and to the
custom-voice page; the same drawer is on `/customvoice`.
### Notes
- Phrases live server-side, so the reference transcript always matches the text
the user read. `POST /api/customvoice` transcodes the upload to mono 16-bit
PCM WAV at `OUTPUT_SAMPLE_RATE` (44.1 kHz) with `ffmpeg`, writes it next to
`TTS_REFERENCE_AUDIO` (the directory the Fish Audio backend can read) with a
sibling `<name>.txt`, and installs it via `TtsEngine::set_reference` — live
for new replies, no restart.
- `POST /api/customvoice/clear` restores the reference configured at startup
(`TTS_REFERENCE_AUDIO`) rather than dropping it entirely.
- Verified: uploading an audio file returns `{"ok":true,…}` and produces a
44.1 kHz/mono `pcm_s16le` WAV with its `.txt`; `clear` logs the restore of
`solis.wav`. Tests: `cargo test` → 25 passed.
---
## 2026-10-17 — Tool calling survives a mis-matched server-side tool parser
### Fixed
- **Tool calls dropped on the streaming path are recovered.** The production
vLLM server runs `--tool-call-parser hermes`, which expects the
`<tool_call>{"name":…,"arguments":{…}}</tool_call>` JSON shape; the Qwen3
model emits the XML shape instead, so the parser throws and the call is
discarded. In streaming the turn then ended with an **empty** reply
(`finish_reason: "stop"`) and no `tool_call` ever fired — the assistant either
stayed silent or narrated a call it never made.
- **Non-streaming fallback** in the turn loop
(`src/orchestrator/turn_manager.rs`): a round that comes back empty while
tools are configured is re-asked once **without streaming** via the new
`LlmClient::chat_reply` (`src/llm.rs`), which returns both the visible text
and the parsed tool calls.
- **Structured calls are parsed too** — `tools::parse_openai_tool_calls`
(`src/tools.rs`) reads the standard `message.tool_calls[]` array, so the
engine keeps working when the server parser *does* match (and returns
`finish_reason: "tool_calls"`). The native `<tool_call>` text path is
unchanged.
### Notes
- End-to-end verified: "Che tempo fa a New York adesso?" now emits
`tool_call {name: get_weather, arguments: {city: "New York"}}` and answers
with live wttr.in data. A tool-free turn is unchanged (no extra request).
- The clean fix remains server-side — `--tool-call-parser qwen3_coder` matches
the Qwen3 XML format. The fallback makes the engine correct regardless, at the
cost of one extra request only on an otherwise-empty round.
- Tests: `cargo test` → 25 passed (2 new in `tools::tests`).
---
## 2026-10-17 — Longer end-of-utterance window (`VAD_MIN_SILENCE_MS` 300 → 700 ms)
### Changed
- **`VAD_MIN_SILENCE_MS` raised** from `300` to `700` ms in `.env` /
`.env.example`, and the fallback default in `src/vad/mod.rs` aligned to
`700`. This is the silence window the VAD waits for before closing an
utterance (see `SileroVad::step`, `silence_run_ms >= min_silence_ms`).
- **README** — updated the config table and the `SpeechEnd` trigger row.
### Notes
- With `300` ms a brief thinking pause (~0,3 s) closed the turn and sent a
partial sentence to the STT/LLM. `700` ms tolerates a natural mid-sentence
pause before the agent answers. The trade-off is a slightly later reply (the
agent now waits ~0,7 s of silence instead of ~0,3 s), which the filler /
start-filler cues already mask.
- The backchannel micro-pause window is unaffected: its `pause_max_ms` (`260`)
is still below the new cap (`min_silence_ms - 32 = 668`).
---
## 2026-10-17 — Mivocero: barge-in indicator + karaoke on the user bubbles
### Added
- **Server `{"type":"barge_in","source":"vad","prob":…}` event** in
`src/ws/session.rs` — emitted when the server-side VAD onset interrupts the
agent. The onset is detected in the session task, so the browser had no other
way to know a barge-in had happened.
- **Barge-in indicator** in `src/web/mivocero.html` — a red **Barge-in** pill in
the header that flashes and bumps a session counter. Driven by the
`barge_in` event (VAD onset) and by the local **Stop** button; the counter
resets on each connect.
### Changed
- **Karaoke on the user bubbles** in `src/web/mivocero.html` — the user's own
messages (STT transcript and typed text) are now rendered word by word like
the assistant's, with colours tuned for the blue bubble. Because the STT
returns the whole transcript only once the utterance has ended, the words are
lit by a short left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word)
instead of following the audio playhead.
- **README** — documented the `barge_in` event in both the server→client table
and the audit `out` row, and updated the Mivocero section (karaoke, barge-in
indicator).
---
## 2026-10-17 — VAD barge-in gated on audible output (no more silent cancelled replies)
### Added
- **`AudioMixer::is_audible()`** in `src/audio/mixer.rs` — reports whether the
agent has *audible* output: the reply is playing out (`tts_primed`) or a
latency-mask cue (filler / start filler) is queued. Unlike `is_active()` it
is `false` while the reply is still being synthesised or sits in the TTS
pre-buffer — nothing has reached the user's ears yet.
- **`TurnManager::is_audible()`** in `src/orchestrator/turn_manager.rs` —
forwards the mixer method for the barge-in gate.
### Changed
- **VAD-driven barge-in gate** in `src/ws/session.rs` — a `SpeechOnset` now
interrupts only when `TurnManager::is_audible()` is true, instead of
`is_speaking()` (`is_active()`). `is_speaking()` is retained for the
backchannel gate, where cueing over any in-flight TTS would still be wrong.
### Notes
- Root cause: `is_speaking()` is already `true` as soon as the TTS request is
in flight (`tts_stream_active`), so a VAD onset landing during the
synthesis / pre-buffer window (~1.2–1.6 s) cancelled the reply before a
single sample was played. On the analysed session
(`s-1792251858371-0001`, ~210 s, 21 turns) three replies streamed their text
to the client but produced **zero playback**, and three more were truncated
after 0.1–0.7 s; several of the onsets were false triggers (the utterance
was later dropped by the gate with `avg_prob` 0.28 / 0.21).
- A reply cancelled this way is not orphaned: the next turn's `on_eou` already
cancels the previous in-flight synthesis and restarts the stream, so the
audio follows the newest turn instead of being lost.
---
## 2026-10-17 — Filler fade-out on reply start (no more overlap with the answer)
### Changed
- **Mixer filler fade-out** in `src/audio/mixer.rs` — when the reply's playback
starts (TTS pre-buffer reached) while an end-of-turn filler or a start filler
is still queued, the cue is now **faded out** over `FILLER_FADEOUT_FRAMES`
(4 frames = 80 ms) and then cut, instead of being ducked to `filler_gain_duck`
and left playing underneath the whole answer. The filler is only a latency
mask, so it now hands off cleanly to the reply.
- New `AudioMixer::filler_fade_frames` counter, armed on the priming
transition and reset by `mark_stream_start` / `flush`.
- **`START_FILLER_GO_DELAY` raised** from `1200` to `1800` ms in `.env` /
`.env.example` — the cue now only fires when the first audio chunk is
genuinely late, instead of on the boundary where it used to overlap the
beginning of the reply.
### Notes
- Root cause: with `START_FILLER_GO_DELAY=1200` the first TTS audio arrived
0.75–1.4 s after the cue, while the `sf03` clip is ~2.83 s long — so the cue
played for its first second at full gain and its tail then overlapped the
start of the answer. On the analysed session this happened on 5 of 6 start
fillers (all `sf03`).
---
## 2026-10-17 — Utterance gate: filter spurious VAD turns before STT
### Added
- **Utterance gate** in `src/ws/session.rs` — a VAD segment is now dropped
**before** transcription when it is too short or too "soft", so the STT can no
longer turn a VAD false-trigger (breath, click, acoustic echo, noise) into a
hallucinated text turn that would each otherwise cost a full LLM turn.
Two independent thresholds, both env-driven:
- `VAD_MIN_UTTERANCE_MS` (`VadSettings::min_utterance_ms`, default `300`,
`0` disables) — drop segments shorter than this.
- `VAD_MIN_AVG_PROB` (`VadSettings::min_avg_prob`, default `0.45`, `0.0`
disables) — drop segments whose average speech probability is below this.
A dropped segment is logged and recorded in the audit as `utterance_dropped`
(`duration_ms`, `avg_prob`, `too_short`, `too_soft`).
- **Empty-transcript guard** in `src/orchestrator/turn_manager.rs::on_eou` — a
transcript that decodes to an empty string (or has no tokens at all) no
longer opens an LLM turn.
### Configuration
- New `.env` keys next to the other `VAD_*` options:
```
VAD_MIN_UTTERANCE_MS=300
VAD_MIN_AVG_PROB=0.45
```
### Notes
- The gate runs only when `VAD_SEGMENT_STT=true`. Raise the probability to be
stricter; lower the duration to keep very short backchannels ("sì", "ok").
- Documentation updated: README (VAD configuration table, VAD wiring section,
audit-event table) and project memory.
---
## 2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)
### Added
- **`cues` WebSocket event** — sent once at session start by
`src/ws/session.rs`, it reports the **effective policy** of the three cues for
the active language: the filler (`delay_ms`, `gain_idle`, `gain_duck`, clip
count, bank dir), the start filler (`go_delay_ms`, clip count, bank dir) and
the backchannel (gain, min-speech, interval, jitter, pause window,
probability, volume jitter, per-turn cap, categories, clip count). The
backchannel `pause_max_ms` is capped to the VAD EOU window exactly like
`BackchannelScheduler::pause_max`; the bank paths are the language
subdirectories (`assets/fillers/<lang>`). Purely informational.
- **Cue methods panel** in `src/web/mivocero.html` — a collapsible
`<details>` section under the cue tiles with one card per cue: the **method**
used (when it fires, how a clip is chosen) and the **live parameters** from
the `cues` event, so the panel always matches the running configuration
instead of hard-coded values.
### Notes
- The panel is display-only; it changes no behaviour. Values reflect the
process configuration at session start (env-derived), so a `.env` change is
visible after the next session.
---
## 2026-10-17 — Per-session audit log (structured JSONL)
### Added
- **`src/audit.rs`** — a best-effort per-session audit sink. Each WebSocket
session is recorded to its own JSONL file
(`SESSION_AUDIT_DIR/<id>.jsonl`, id `s-<epoch_ms>-<seq>`), one record per line:
`{"ts":…,"elapsed_ms":…,"event":…,"data":…}`. Records are flushed as they are
written, and any failure (directory/file creation, write) only disables the
sink for that session — the session itself is never affected.
- **Event coverage (complete)**:
- `session_start` — full non-secret configuration snapshot plus the active
runtime capabilities (STT / VAD / tokenizer / LLM + model);
- `control` — inbound `user_text` turns;
- `out` — **every** event sent to the client, tapped at the single sender
point in `src/ws/session.rs` so nothing is missed (`user`,
`assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`,
`audio_source` state, …);
- `interrupt` — client barge-in (JSON or legacy text form);
- `barge_in` — VAD-driven barge-in (with the onset probability);
- `vad_speech_start` / `vad_speech_end` — utterance boundaries;
- `media` / `reference` / `speak` — external-audio control, voice-reference
changes and direct TTS requests;
- `language_switch` — detected language change (`from`, `to`, `generated`) —
`src/orchestrator/turn_manager.rs`;
- `error` — transport/receiver errors;
- `session_end` — reason, duration, session id, file and per-event counts.
- **Config** — `SESSION_AUDIT_ENABLED` (default `true`) and
`SESSION_AUDIT_DIR` (default `logs/sessions`); documented in `.env`,
`.env.example` and the README. `logs/` is git-ignored.
### Changed
- `TurnManager::new` takes an `Arc<SessionAudit>` (used for the language-switch
record); every other event reaches the log through the sender tap.
### Notes
- The audit path is off the audio hot path (the sink only touches the outbound
text events, not the 20 ms frames) and is disabled cleanly when the LLM/VAD/…
are off. Intended for post-mortem analysis and for tuning the latency knobs.
---
## 2026-10-17 — Cue emission indicators in the Mivocero demo
### Added
- **`{"type":"cue"}` WebSocket event** — the server now tells the client each
time a latency-mask / backchannel cue is **actually emitted** into the mixer:
- **filler** (end-of-turn) — `src/orchestrator/turn_manager.rs`
(`schedule_filler`),
- **start_filler** (turn start) — `src/orchestrator/turn_manager.rs`
(`schedule_start_filler`),
- **backchannel** (during user speech) — `src/ws/session.rs` (mic task).
The payload is `{"type":"cue","kind":…,"name":…}` and, for backchannel, also
`"category":…`. A cue that is skipped (audio already arrived / playing, or a
barge-in) sends nothing.
- **`BackchannelScheduler::maybe_emit`** now returns `(name, category)` instead
of just the name, so the backchannel cue carries its category.
- **Mivocero cue indicators** — `src/web/mivocero.html` gained a row of three
tiles (**Filler**, **Start filler**, **Backchannel**) that flash (colour-coded
dot + border glow) and bump a per-session counter on every `cue` event, with a
detail line showing the backchannel category (or the clip name). Counters reset
on each connect.
### Notes
- Only real emissions are reported, so the panel is a faithful view of what the
user actually hears — useful for tuning `FILLER_DELAY_MS` /
`START_FILLER_GO_DELAY` and the backchannel cadence.
---
## 2026-10-17 — Per-language subdirectory for every clip bank (base included)
### Changed
- **`src/main.rs`** — the startup bootstrap now generates and loads the three
base banks (filler, start-filler, backchannel) from
`<dir>/<LANG_DEFAULT>/` via [`assets::lang_dir`](src/assets.rs) instead of the
top-level directory. Every language — **including the default one** — now
lives in its own subdirectory, so the filler, start-filler and backchannel
share the exact same layout for every language and no language is special-cased
to the top level.
- **`src/assets.rs`** — [`lang_dir`](src/assets.rs) doc updated accordingly.
### Notes
- On the first start after this change the base clips are re-synthesized into
`assets/<bank>/<lang>/`. To avoid the re-synthesis, move the existing
top-level `*.raw` files (and `texts.txt`) into the matching language
subdirectory by hand — this was done for the `es` deployment.
- No other language's clips are touched: the per-language subdirectory remains
what isolates each language's refresh (`remove_stale_clips`).
---
## 2026-10-17 — Multilingual clip banks (backchannel, filler & start filler)
### Added
- **`src/lang.rs`** — new multilingual module. It holds built-in phrase tables
for **backchannel** (category + phrase), **filler** (interjections) and
**start filler** (turn-start acknowledgments, with Drama-3 style tags) in
Spanish, English, Italian, French, German and Portuguese, exposes accessors
for them, and implements the auxiliary LLM calls: `detect_language` (returns
only the ISO 639-1 code, or `und`) and `generate_backchannel_texts` /
`generate_filler_texts` / `generate_start_filler_texts` (LLM fallback for
languages absent from the tables).
It also defines `LangBanks` (a language's loaded clip banks, including its
start-filler bank) and `LanguageState` (the session's active language plus a
generation counter, kept in a parking-lot `RwLock`).
- **On-demand per-language generation** — when the user's language differs from
the active one, that language's backchannel/filler/start-filler clips are
synthesized into a **per-language subdirectory** (`assets/backchannel/<lang>/`,
`assets/fillers/<lang>/`, `assets/start_filler/<lang>/`) and the session
switches to them. Because the refresh pass only prunes clips inside the
directory it is given, **no other language's clips are ever removed**; the
base language keeps living in the top-level directories, so no migration is
needed.
- **New env vars** — `LANG_DEFAULT` (default `es`), `LANG_DETECT_ENABLED`
(default `true`), `LANG_MIN_CHARS` (default `12`), `LANG_REDETECT_EVERY`
(default `5`) and `LANG_LLM_FALLBACK_ENABLED` (default `true`).
### Changed
- **`src/llm.rs`** — new `LlmClient::simple_chat` for short, non-streaming
auxiliary requests (explicit system + user message, **no** tools), used by the
language detection and phrase-generation calls.
- **`src/config.rs`** — five new `LANG_*` fields, parsed in `from_env`.
- **`src/assets.rs`** — `lang_dir` helper plus `ensure_lang_backchannel_clips` /
`ensure_lang_filler_clips` / `ensure_lang_start_filler_clips`, which scope the
text-bank refresh to `<base>/<lang>/` so each language is refreshed
independently.
- **`src/orchestrator/backchannel.rs`** — the clip/category mapping is extracted
into `map_clips`, and `BackchannelScheduler::retarget` re-points a live
scheduler to a new bank when the language changes. `DEFAULT_CATEGORY` is now
`pub(crate)` so `lang.rs` can reuse it.
- **`src/orchestrator/turn_manager.rs`** — `TurnManager` now holds an
`Arc<LanguageState>` instead of a single filler bank; `next_filler` and the
start filler read from the active language (`schedule_start_filler` picks from
`langs.current().start_filler`). A new `maybe_detect_language` runs on each
user turn (first turn + every `LANG_REDETECT_EVERY`) and spawns
`switch_language_if_needed`, which detects, generates/loads the target
language's banks (start filler optional: a missing table just leaves that cue
silent) and swaps the active state — in the background, so the reply is never
blocked.
- **`src/ws/session.rs`** — builds the shared `LanguageState`; the mic task polls
the language generation counter each frame and calls `retarget` on the
backchannel scheduler only when it changes.
- **`.env` / `.env.example`** — new "Multilingual clip banks" section;
**`README.md`** — new "Multilingual clip banks" section.
### Why
- The banks were pre-synthesized in a single language, so a user speaking any
other language heard Spanish cues — including the start filler. Generating the
matching language on demand keeps the acknowledgment natural in a multilingual
deployment, while the per-language directory layout guarantees that adding a
language never destroys the clips already generated for the others.
### Notes
- Detection needs the LLM (`LANG_DETECT_ENABLED` is a no-op without it); the LLM
phrase fallback is used only for languages outside the built-in tables.
- Generating a new language's bank calls the TTS backend once per phrase, which
can take a few seconds on the first switch; it is idempotent (an
already-generated language is only reloaded).
---
## 2026-10-16 — Mivocero: auto-disconnect an idle session after 2 minutes
### Added
- **Idle guard in the Mivocero client** ([`src/web/mivocero.html`](src/web/mivocero.html)) —
the session closes itself after `IDLE_TIMEOUT_MS` (2 minutes) without any
activity. `touchActivity()` re-arms the timer and is called on every server
event, on every received audio frame and on every user action sent
(`sendJson`), so an ongoing reply or a playing external source keeps the
session alive. On expiry it logs the reason, appends an explanatory message to
the chat and calls `disconnect()`, which closes the WebSocket and releases the
microphone.
### Changed
- **Reconnect after close** — `ws.onclose` now clears the `ws` reference.
Without it the Connect button stayed bound to the stale socket, so a session
closed by the idle guard (or by any disconnect) could not be reopened without
reloading the page.
### Why
- A microphone-backed session left open keeps the device busy and the server
streaming silent 20 ms frames forever. An inactivity timeout releases both
without user action, while still allowing a long reply or a background radio
to run uninterrupted.
### Notes
- The timer counts *any* traffic in either direction: TTS audio is sent as
binary frames, so speaking does not trip the timeout.
- The timeout is a client-side constant (`IDLE_TIMEOUT_MS` in the HTML); change
it there and rebuild (the page is embedded with `include_str!`).
---
## 2026-10-16 — Tool calling: configurable HTTP tools for the LLM
### Added
- **`src/tools.rs`** — new tool-calling module. It loads tool definitions from a
JSON file and provides: a `ToolRegistry` (builds the OpenAI `tools` array,
executes a call over HTTP), a client-side parser for the model's **native**
`<tool_call>` format, and a `ToolCallStripper` that removes tool-call blocks
from the streamed reply — holding back a block that is still incomplete, so it
is never spoken by the TTS nor shown in the chat box even when split across
deltas (same hold-back pattern as the chain-of-thought stripper).
- **`tools.example.json`** — ready-to-use definitions: `get_weather` (GET with a
`{city}` URL placeholder, wttr.in) and `send_webhook` (POST with a custom
header), covering both request shapes.
- **New env vars** — `LLM_TOOLS_ENABLED` (auto: on when `LLM_TOOLS_FILE` is set),
`LLM_TOOLS_FILE`, `LLM_TOOL_MAX_ITERATIONS` (default `4`),
`LLM_TOOL_TIMEOUT_SECS` (default `15`) and `LLM_TOOL_RESULT_MAX_CHARS`
(default `2000`).
- **`ChatMessage::tool`** — `role: "tool"` message used to hand a tool result
back to the model.
- **WebSocket protocol** — informational
`{"type":"tool_call","name":"…","arguments":{…}}` event emitted when the model
calls a tool.
### Changed
- **`LlmClient`** carries an optional `Arc<ToolRegistry>`
([`src/llm.rs`](src/llm.rs)); when present, every request advertises
`tools` + `tool_choice:"auto"`, so the model knows which functions it may
call. The registry is built by `load_tools(config)`; a missing or invalid file
logs a warning and leaves tool calling disabled without breaking the pipeline.
- **[`TurnManager::run_llm_turn`](src/orchestrator/turn_manager.rs)** now runs a
bounded tool loop: each round streams a reply through the `ToolCallStripper`;
if a call was emitted, the assistant message (with its native call) plus one
`tool` message per result are appended and the model is re-prompted, up to
`LLM_TOOL_MAX_ITERATIONS`. The TTS pipeline is created once and every round's
visible text flows into it, so the assistant can say something short before
calling. Only the spoken text is written to the rolling history.
### Why
- The deployment's vLLM server runs **without** a tool parser
(`--enable-auto-tool-choice --tool-call-parser …`), so it never returns
structured `tool_calls` / `finish_reason:"tool_calls"`. Tools are still
advertised in the request and the Qwen model emits the call as text in its
native format; parsing it client-side keeps the feature working with any
server and requires no server-side reconfiguration.
- Tool definitions live in a JSON file rather than `.env` so the verbose JSON
schemas stay readable and editable.
### Notes
- Tool errors (unknown tool, transport failure, non-2xx status) are returned to
the model as text so it can recover; a tool never fails the turn.
- `GET` sends arguments not consumed by a `{arg}` placeholder as query
parameters; `POST` sends them as a JSON body. `headers` are sent verbatim.
---
## 2026-10-16 — External audio channel: inject a URL / radio / YouTube source
### Added
- **`src/media.rs`** — new [`MediaSource`](src/media.rs) module implementing an
external audio source for a WebSocket session. A source is decoded by an
external process and its raw PCM is pushed into the mixer's new external ring
buffer. `ffmpeg` decodes direct URLs / radio / HLS
(`-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1`, with
`-reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5`); YouTube links are
first resolved to a direct URL with `yt-dlp` (`-f bestaudio/best -g
--no-playlist`). Child processes are killed on stop / replacement /
disconnection (`kill_on_drop(true)` + `CancellationToken`), and a monotonic
generation counter prevents a stale task from clearing a newer source.
- **Mixer external channel** — `AudioMixer` gains `external_queue` and a runtime
gain ([`src/audio/mixer.rs`](src/audio/mixer.rs)). The mixing formula becomes
`… + (external × external_gain)`; `set_external_gain` updates the gain (an
atomic, so it takes effect on the next frame without rebuilding the mixer).
The external queue is **not** cleared by `flush()` and does not count towards
`is_active()`: a barge-in stops the agent, not the background audio.
- **New env vars** — `EXTERNAL_AUDIO_GAIN` (default `0.6`),
`MEDIA_FFMPEG_BIN` (default `ffmpeg`), `MEDIA_YTDLP_BIN` (default `yt-dlp`) and
`MEDIA_YTDLP_ARGS` (extra `yt-dlp` args, e.g. `--cookies …`).
- **WebSocket protocol** — new `audio_source` message:
`{"type":"audio_source","action":"play","url":"…"}`,
`{"type":"audio_source","action":"stop"}` and
`{"type":"audio_source","action":"volume","gain":0.6}`. The server reports
state back with `{"type":"audio_source","state":"loading|playing|stopped|error","url":"…"}`
([`src/ws/session.rs`](src/ws/session.rs)).
- **Mivocero client** — footer bar with a URL input, **Play** / **Stop**
buttons, a volume slider (0–150 %) and a live state label, driving the external
channel ([`src/web/mivocero.html`](src/web/mivocero.html)).
### Changed
- [`TurnManager::set_external_gain`](src/orchestrator/turn_manager.rs) forwards
the client volume to the mixer.
- New module declared in [`src/main.rs`](src/main.rs) (`mod media;`).
### Why
- Lets the agent play a music/radio/YouTube bed under the voice — useful for
demos (radio, background music) — while keeping the channel fully independent
of the speech pipeline so it survives barge-in.
### Notes
- Only needed when the external channel is used: `ffmpeg` for URL/radio/HLS and
`yt-dlp` for YouTube. YouTube may require cookies in some environments; pass
them via `MEDIA_YTDLP_ARGS=--cookies /path/cookies.txt`.
- A bounded source (VOD) is throttled to the mixer's drain rate via backpressure
(`remaining_capacity()` + 20 ms sleeps), so a fast decoder cannot overflow the
ring buffer.
---
## 2026-10-16 — Drama 3 de-emphasis: sampling knobs + neutral style tag
### Added
- **`TTS_TEMPERATURE`** (default `0.45`), **`TTS_TOP_P`** (default `0.75`) and
**`TTS_TOP_K`** (unset) — optional sampling overrides forwarded in the
`/v1/audio/speech` payload by the `fish` engine. An empty value omits the field
so the server default applies. New `env_opt_f32` / `env_opt_u32` helpers in
[`src/config.rs`](src/config.rs) implement the "unset → fallback, empty →
`None`" semantics.
- **`TTS_STYLE_TAG`** (default `[conversational, casual]`) — a neutral style tag
prepended to every synthesis request whose text does not already start with a
`[...]` tag ([`FishAudioTts::styled`](src/tts/fish_audio.rs)). Empty disables
the prefix.
### Changed
- [`SpeechRequest`](src/tts/fish_audio.rs) gains optional `temperature`, `top_p`
and `top_k` fields (serialized only when set); the request body now uses the
styled text as `input`. Startup logs `TTS de-emphasis settings …`, and the
per-request debug log shows the sampling values and whether the text was
prefixed (`styled=`).
### Why
- Drama 3 over-articulates: LLM replies could come back with a theatrical,
radio-drama cadence. Exaggerated intonations live in the tails of the acoustic
token distribution, so trimming the sampling tails (`temperature` / `top_p`)
flattens the delivery, and a neutral style tag steers untagged (LLM) prose into
a conversational register. Clip banks carry their own tags and are untouched.
### Notes
- `repetition_penalty` is deliberately **not** exposed: this server rejects any
value other than `1.0` (`Drama 3 request: repetition_penalty must be 1.0`).
`min_p`, `seed` and a nested `sampling_params` are rejected too.
- The voice-cloning reference (`solis.wav`) is left unchanged for now — its
"locutor profesional" prosody is a further, separate lever.
- **`.env` gotcha surfaced during this change.** `dotenvy` fails on unquoted
values containing spaces, and a single bad line makes it discard the whole
file: the process then silently starts with built-in defaults (no LLM, plain
HTTP) while `server.sh` still reports the scheme from its own `.env` grep. The
style tag is therefore quoted (`TTS_STYLE_TAG="[conversational, casual]"`).
---
## 2026-10-16 — Backchannel: random reaction depth inside the micro-pause
### Changed
- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) now draws
the depth at which it reacts **per pause**: the first time a micro-pause is
seen, a target is picked uniformly in `[BACKCHANNEL_PAUSE_MIN_MS,
BACKCHANNEL_PAUSE_MAX_MS]` and the cue fires only once `silence_ms()` reaches
it (new `pause_target_ms` field, cleared when the pause ends or the turn
resets). Previously the cue fired at the *first* analysis window inside the
window, so with a 32 ms VAD step every cue landed at exactly
`BACKCHANNEL_PAUSE_MIN_MS`.
- A pause that grows past the upper bound (on its way to end-of-utterance) is
abandoned without firing.
### Why
- Verification on a live session showed every cue at `pause_ms=96`, i.e. all cues
reacting at exactly the minimum — a residual regularity in an otherwise
naturalized feature. Randomizing the depth removes it: cues now land anywhere
in the window (observed `pause_ms` of `96`, `128`, `192`, …).
### Notes
- No new environment variables: the behaviour is driven by the existing
`BACKCHANNEL_PAUSE_MIN_MS` / `BACKCHANNEL_PAUSE_MAX_MS` window.
- Because the VAD advances in 32 ms windows, the emitted `pause_ms` is a multiple
of 32 ms; the randomly drawn target is what varies.
---
## 2026-10-16 — Backchannel sounds more natural (micro-pause cues, jittered timing, categories)
### Added
- **`BACKCHANNEL_JITTER_MS`** (default `1200`), **`BACKCHANNEL_PAUSE_MIN_MS`**
(default `96`), **`BACKCHANNEL_PAUSE_MAX_MS`** (default `260`),
**`BACKCHANNEL_PROB`** (default `0.65`) and **`BACKCHANNEL_VOLUME_JITTER`**
(default `0.2`) — new `.env` knobs (`.env`, `.env.example`,
[`BackchannelSettings`](src/orchestrator/backchannel.rs)) that shape the cue
timing, spacing, hit rate and loudness drift.
- **Category prefix** in `BACKCHANNEL_TEXTS`: a line may be written
`category::phrase` (e.g. `ack::[calm] Mm-hmm...`, `agree::[agreeing] Claro...`);
[`parse_texts`](src/orchestrator/backchannel.rs) stores the category alongside
the phrase and defaults to `ack` when the prefix is absent.
- [`VadDetector::silence_ms`](src/vad/silero.rs) — exposes the running silence
inside the current utterance (multiples of the 32 ms analysis window), i.e. the
length of the micro-pause the speaker is currently in.
- [`FillerBank::names`](src/orchestrator/turn_manager.rs) — read accessor for the
clip names, used to map the bank order onto the phrase/category list.
### Changed
- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) now takes
the VAD micro-pause length and fires a cue **only inside a
`PAUSE_MIN_MS..PAUSE_MAX_MS` window** (the upper bound is capped just below
`VAD_MIN_SILENCE_MS` so it never collides with end-of-utterance), **at most once
per pause** (latched with `pause_settled`).
- Cue timing is **randomized**: the first cue of a turn waits
`MIN_SPEECH_MS + rand(0..JITTER_MS)`, and the gap to the next cue is
`INTERVAL_MS ± JITTER_MS/2` — instead of a fixed interval.
- Each qualifying pause passes a **probabilistic gate** (`PROB`); a miss does not
retry on the same pause.
- The clip is chosen with an **anti-repetition** rule (skips the last two clips)
and its samples are scaled by **±`VOLUME_JITTER`** before being pushed to
`backchannel_queue`.
- Selection follows a **category progression** through the turn — it prefers a
category other than the last one used and favours categories not yet used this
turn (e.g. `ack` then `agree`), falling back to a random clip.
- `BACKCHANNEL_MAX_PER_TURN` default lowered **8 → 3**, so the agent emits a
couple of cues per turn rather than chattering.
### Why
- A fixed interval with a fixed clip made the backchannel feel mechanical and
detached from the speaker. Real listeners react to *pauses*, not to a metronome:
cueing inside the speaker's own micro-pauses, with jittered spacing, an
occasional deliberate skip, mild volume drift and a little progression
(acknowledge → agree) reads as attentive listening instead of a loop.
### Notes
- The naturalness knobs are all optional; omitting them keeps the previous
behaviour except for the `MAX_PER_TURN` default (now `3`).
- Nested clamps in `from_env`: `interval_ms >= 500`, `pause_max >= pause_min`,
and `gain`/`prob`/`volume_jitter` are clamped to sane ranges.
- A debug line is logged only when a cue is actually emitted:
`backchannel emitted during user speech … pause_ms=…` (enable
`audio_agent_engine::orchestrator=debug` in `RUST_LOG`).
---
## 2026-10-16 — Backchannel becomes a random phrase bank driven by `BACKCHANNEL_TEXTS`
### Added
- **`BACKCHANNEL_TEXTS`** (`.env`, `.env.example`,
[`BackchannelSettings`](src/orchestrator/backchannel.rs)): the backchannel bank
is now a list of phrases, **one per line** (a `|` separator is also accepted),
with Drama-3 style tags passed straight through to the TTS. `.env` ships 8
Spanish phrases (`[calm] Mm-hmm...`, `[attentive] Ajá...`, …); the Rust default
is [`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs).
- [`assets::ensure_backchannel_clips`](src/assets.rs), implemented on top of the
new generic [`assets::ensure_text_bank_clips`](src/assets.rs): synthesizes **one
clip per phrase** at startup (`bc01`, `bc02`, …, `TEXT_BANK_CONCURRENCY = 4` at
a time) and removes clips left over from a previous list. The list the current
clips were made from is stored in `BACKCHANNEL_DIR/texts.txt`; when it is
unchanged and every clip is present the whole synthesis is **skipped**, so a
restart costs ~2 s instead of ~1 s per phrase. The index is only written once
every clip succeeded, so a partial failure is retried at the next startup.
### Changed
- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) picks the
clip **at random** ([`FillerBank::random`](src/orchestrator/turn_manager.rs),
xorshift64 seeded from the clock) instead of round-robin, so the acknowledgment
varies from turn to turn — the same behaviour the start filler already had. The
`next_index` cursor was dropped.
- `assets::ensure_start_filler_clips` is now a thin wrapper over
`ensure_text_bank_clips`, so the start filler and the backchannel share one
implementation. Its logs gained a `label` field
(`refreshing backchannel clips at startup …`).
- The backchannel no longer uses the fixed `assets::BACKCHANNEL_CLIPS` list
(removed); its defaults moved to `DEFAULT_BACKCHANNEL_TEXTS`.
### Why
- A fixed 4-clip bank repeated the same "mm-hmm / ajá" over and over; a curated
list with tags keeps the agent sounding natural and is editable straight from
`.env`, exactly like the start filler.
- The old bootstrap (`ensure_clips`) only generated *missing* clips, so changing
the phrases required deleting files by hand; the phrase list is now the source
of truth.
### Notes
- Clips are named by position, so switching from the old names (`mmhmm`, `aja`,
`claro`, `yaveo`) to `bc01`…**removes** the four legacy files on the first
restart after this change (logged as `removed stale backchannel clip`).
- `BACKCHANNEL_GAIN`, `BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS` and
`BACKCHANNEL_MAX_PER_TURN` are unchanged.
---
## 2026-10-16 — Fix: start filler fired on every turn (gate on chunk arrival, not playback)
### Fixed
- [`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs) gated the
cue on `AudioMixer::tts_playback_started`, which only flips once the **TTS
pre-buffer** (`TTS_PREBUFFER_MS`) is full — i.e. when playback starts, ~1.9 s
after the dispatch in the reference deployment. The first response chunk
actually arrives at ~0.7–1.0 s, so a `START_FILLER_GO_DELAY` of 1200 ms was
almost always *shorter* than the gate's own signal and the acknowledgment was
queued on essentially every turn.
- The gate now uses the new
[`AudioMixer::tts_audio_arrived`](src/audio/mixer.rs), latched as soon as the
first TTS samples are queued for the turn (checked on the 20 ms ticker),
**before** the pre-buffer is reached. Both `tts_audio_arrived` and
`tts_playback_started` are reset by `mark_stream_start` / `flush`.
A fast reply that produces audio within `START_FILLER_GO_DELAY` therefore
suppresses the cue, as intended.
### Changed
- The skip log line is now `first response audio already arrived — start filler
skipped` (was `response audio already playing — start filler skipped`).
- The end-of-turn filler is unchanged: it still keys off
`tts_playback_started` (its 2500 ms delay is long enough that playback start is
the right signal).
### Notes
- No configuration change; `START_FILLER_GO_DELAY=1200` now means what the `.env`
comment always said ("first audio chunk of the reply"). Verified in the log: a
text turn in which the first chunk reached the mixer ~0.4 s after the TTS
request logged `first response audio already arrived — start filler skipped`
with no `queued start filler` for the turn.
---
## 2026-10-16 — Start filler becomes conditional: `START_FILLER_GO_DELAY` (1200 ms)
### Changed
- The start filler is no longer played on **every** turn: it is now a *latency
mask*, scheduled exactly like the end-of-turn filler but on its own channel and
with its own, shorter delay.
[`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)
(replacing `play_start_filler`) sleeps `START_FILLER_GO_DELAY` after the turn is
dispatched and queues a random phrase **only if the first response audio has not
started playing** (`AudioMixer::tts_playback_started` still `false`); otherwise
the cue is dropped. *(The gate was later changed to `tts_audio_arrived` — see
the entry above, which fixes this cue firing on every turn.)* A pending cue is
cancelled on barge-in / new turn through the turn's `CancellationToken`, and
`mixer.flush` clears it once queued.
- The scheduling now happens in `run_llm_turn` / `dispatch_tts` next to
`schedule_filler` (where the turn's cancellation token and `mark_stream_start`
live) instead of at the top of `handle_user_turn`, because the delay must be
measured from the dispatch of the reply and be cancellable with it.
### Added
- **`START_FILLER_GO_DELAY`** (`.env`, `.env.example`, default `1200` ms): how long
to wait for the first response audio before falling back to the start filler.
### Why
- The cue is only useful when the reply is *late*; a fast reply (LLM + TTS first
chunk under 1200 ms) is now answered directly, without an unnecessary
"Un momento…" in front of it.
### Notes
- The 22-clip bank, the random pick and the startup refresh are unchanged.
---
## 2026-10-16 — Start filler: random phrase bank driven by `START_FILLER_TEXTS`
### Added
- **`START_FILLER_TEXTS`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)) replaces the single `START_FILLER_TEXT`: a
list of phrases, **one per line** (a `|` separator is also accepted), e.g. the
Spanish set `"[calm] Un momento..."`, `"[thoughtful] A ver, veamos..."`, … with
Drama-3 style tags passed straight through to the TTS. `.env` ships 22 phrases.
- [`assets::ensure_start_filler_clips`](src/assets.rs): synthesizes **one clip per
phrase** at startup (`sf01`, `sf02`, …, generated
`START_FILLER_CONCURRENCY = 4` at a time) and removes clips left over from a
previous list. The list the current clips were made from is stored in
`START_FILLER_DIR/texts.txt`; when it is unchanged and every clip is present the
whole synthesis is **skipped**, so a restart costs ~2 s instead of ~1 s per
phrase. The index is only written once every clip succeeded, so a partial
failure is retried at the next startup.
- [`FillerBank::random`](src/orchestrator/turn_manager.rs): uniform pick over the
bank using a per-bank xorshift64 state seeded from the clock;
[`TurnManager::play_start_filler`](src/orchestrator/turn_manager.rs) now uses it,
so the acknowledgment differs from turn to turn instead of always repeating the
same sentence. *(Caller renamed to `schedule_start_filler` by the entry above,
which also made the cue conditional.)*
### Why
- A single fixed acknowledgment ("Un momento…") becomes annoying across a long
conversation; a random phrase from a curated list keeps the agent sounding
natural, and the phrase list is editable straight from `.env`.
- Re-synthesizing 22 clips on *every* startup added ~21 s to the boot time
(measured), so the bootstrap now tracks the phrase list and only re-synthesizes
when it changed (or a clip is missing).
### Notes
- `force` was dropped from [`assets::ensure_clips`](src/assets.rs): the filler /
backchannel banks keep the "generate only if missing" behaviour, and the start
filler has its own list-driven bootstrap.
- [`server.sh`](server.sh) startup wait raised from ~20 s to ~60 s: the first
startup after a list change synthesizes the whole bank before the socket is
bound.
- A failed synthesis keeps the previous clip of that position; a barge-in still
clears the start-filler channel (`AudioMixer::flush`), and
`START_FILLER_ENABLED=false` disables it entirely.
---
## 2026-10-16 — Start filler: immediate turn-start acknowledgment on its own mixer channel
### Added
- **`START_FILLER_ENABLED` / `START_FILLER_TEXT` / `START_FILLER_DIR`** (`.env`,
`.env.example`, [`src/config.rs`](src/config.rs)). The *start filler* is a
pre-generated clip queued the moment a user turn begins — no delay — on a
**dedicated mixer channel**, so it acknowledges the turn while the LLM/TTS are
still working, independently of the delayed latency filler. *(Superseded by the
entry above: `START_FILLER_TEXT` became the `START_FILLER_TEXTS` list and the
clip is now picked at random.)*
- [`AudioMixer`](src/audio/mixer.rs): new `start_filler_queue` channel, mixed as
`mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel ×
BACKCHANNEL_GAIN)`, so the start filler shares the filler gain (idle `1.0`,
ducked `0.15` under TTS). Included in `is_active()` and cleared by `flush()`.
- [`TurnManager::play_start_filler`](src/orchestrator/turn_manager.rs), called
from `handle_user_turn` right after the in-flight reply is cancelled.
- Startup bootstrap synthesizes `START_FILLER_DIR/start_filler.raw` with the
configured voice if it is missing ([`src/assets.rs`](src/assets.rs)); the clip
is preloaded into its own [`FillerBank`](src/orchestrator/turn_manager.rs),
kept out of the round-robin `FILLERS_DIR` bank.
### Why
- The delayed latency filler only fires after `FILLER_DELAY_MS` (and only if the
reply is still silent), so with the current fast TTFB (~`0.73 s`) the agent can
stay silent for a moment after the user stops speaking. The start filler
provides an immediate acknowledgment instead, without touching the existing
delayed filler.
### Notes
- *(Superseded — see the entry above.)* The clip was re-synthesized on every
startup (`ensure_clips(..., force = true)`), so editing `START_FILLER_TEXT` took
effect at the next restart with no manual deletion. A new clip was written only
on success, so a failed synthesis left the previous clip in place; the filler /
backchannel banks kept the "generate only if missing" behaviour.
- A barge-in clears the start filler along with every other channel
(`AudioMixer::flush`); `START_FILLER_ENABLED=false` disables it entirely.
---
## 2026-10-16 — Fixed micro-gaps at the start of each sentence → `TTS_PREBUFFER_MS` 170 → 550 ms, `LLM_STREAM_CHUNK_CHARS` 120 → 240
### Changed
- **`TTS_PREBUFFER_MS` raised `170` → `550`** (`.env`, `.env.example`). With the
current vocoder windows (first `185.8` ms, steady `371.6` ms) any pre-buffer in
`(186, 557]` ms primes playback on the *same* second window, so `550 ms` takes
the entire ~557 ms cushion **at no extra TTFA** (~`0.73 s`).
- **`LLM_STREAM_CHUNK_CHARS` raised `120` → `240`** (`.env`, `.env.example`). The
first-chunk threshold stays at `20` (early flush preserved), but every following
chunk is now twice as long → **fewer TTS requests, hence fewer chunk seams**.
- **README** (`Mixer & jitter buffering`, the `TTS_PREBUFFER_MS` /
`LLM_STREAM_CHUNK_CHARS` tables and the TTFA note) updated to the new defaults
and rationale. No Rust code changed — both values are read from `.env` at
runtime.
### Why
- After the seam-quality retune left the **audible micro-interruptions at the
start of sentences**. Diagnosis from `run/server.log`: the defect is a **TTS
queue underrun at the chunk seams**, not a timbre artifact. Each sentence /
fragment is a *separate* HTTP request whose PCM is concatenated with no
crossfade; the relay advances chunk-by-chunk, so the first dip in production
empties the queue and the mixer logs
`TTS queue underrun — pausing playback until audio resumes` (94 occurrences in
the sampled log; 0 `running low` warnings), then restarts with `need = 1` → an
audible micro-gap at the start of the next sentence.
- **Why the queue empties:** with `TTS_PIPELINE_DEPTH = 3` on a single GPU the
measured per-window cadence under concurrency is **552–613 ms (~1.5×)** versus
**~350 ms (~0.94×)** for a lone request. A `170 ms` cushion has no margin
against that; `550 ms` does, and it is latency-free because the pre-buffer
already lands on the second window.
- Measured in the log at the seam (12:56:36–38): chunks of `16384` samples
(`queue_len=16384`, `prebuffer=7497` = 170 ms) with two concurrent requests
(`text_len=119` and `19`), each followed by an underrun-pause line.
### Notes
- Both levers are pure configuration; the binary is unchanged, so `./server.sh
restart` (no rebuild) is enough to apply them.
- The `550 ms` cushion is the ~557 ms gap to the third window start, so raising
it further only adds latency (it would wait for the *next* window).
---
## 2026-10-16 — Drama-3 vocoder retuned for seam quality → stride `4` / followup `8` / overlap `16`; `TTS_PREBUFFER_MS` 450 → 170 ms
### Changed
- **The Drama-3 launcher (`/root/drama3/start-drama3.sh`, pm2 `start-drama3`) now
carries more decoder context per window.** The previous config (stride `4`,
flat ramp `"[4,4,4]"`, overlap `8`, crossfade `256`) was replaced by:
```
--vocoder.factory.stream_stride 4
--vocoder.factory.stream_followup_stride_ramp "[8,8,8]"
--vocoder.factory.stream_followup_stride 8
--vocoder.factory.stream_overlap_tokens 16
--vocoder.factory.stream_crossfade_samples 512
--tts_engine.factory.enable_async_decode true
```
- **`TTS_PREBUFFER_MS` lowered `450` → `170`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)). With stride `4` the *first* vocoder window is
4 codec frames = **0.186 s**, so `170 ms` starts playback on that first window
and **TTFA collapses to the server's own TTFB (~0.38 s)**.
- **README:** the *"Starting the Drama 3 backend (SGLang-Omni)"* subsection and
the *"Mixer & jitter buffering"* pre-buffer note were rewritten around the new
stride/overlap values and the ~0.186 s first / ~0.372 s steady window cadence.
### Why
- The previous latency-first config (stride `4`, overlap `8`) produced **audible
micro-gaps and a slight detune / falsetto on long replies**. Root cause: with
only `8` tokens of decoder warm-up context *per window*, and one TTS request
per reply chunk (PCM concatenated without crossfade at the request seams), the
streaming decoder's state was too shallow and its timbre/pitch wobbled at each
window and request boundary.
- **Seam context restored without breaking real-time:** raising the overlap to
`16` at stride `4` pushes decode to 5× real time, which the box **cannot**
sustain (ratio rose to 1.18–1.98×). Raising *strides and overlap together* —
stride `4` (first window) / followup `8` (steady) with overlap `16` — keeps the
decode load at a sustainable **`(16+8)/8 = 3×`** (~0.95× ratio) while giving
each steady window twice the context, and `crossfade 512` smooths the seams
further.
- A/B measurements on the live endpoint (`measure_tts.py`, warm server, voice
cloning, single request vs sentence-chunked like the engine) on a long
multi-sentence reply: **the amplitude jump at the seam boundaries dropped from
the 99.5+ percentile of the signal to 96–98**, i.e. the seams no longer stand
out. TTFB stays **~0.38 s** (the first window is unchanged) and the real-time
ratio stays **~0.94–0.98×**.
### Notes
- The first window is unchanged (4 frames), so the residual latency is still
dominated by the server's **fixed ~0.38 s TTFB**; the steady window grew from
0.186 s to **0.372 s** (8 frames) and arrives ~350 ms apart, so the pre-buffer
of `170 ms` still starts playback on the very first window. Further latency
gains would need a faster first token (model / preprocessing), not a smaller
stride.
- `measure_tts.py` was added to the repo: it streams `/v1/audio/speech` and prints
the per-window arrival schedule, TTFB and real-time ratio (plus a seam-jump
summary), so future A/B runs are one command.
---
## 2026-10-16 — Mivocero: karaoke word highlighting on the assistant reply
### Changed
- **The `/mivocero` client now reveals the assistant reply in karaoke style**
([`src/web/mivocero.html`](src/web/mivocero.html)) instead of appending plain
text. Each word is a `<span>`; words light up as they are *spoken*. The
highlight is driven by the **audio playhead**: the playback AudioWorklet now
counts the samples it has actually emitted and reports them (`played`) on its
existing per-block message. The client maps that count onto a character offset
with an estimated speaking rate (`KARAOKE_CPS`, ~15 chars/s) and marks each
word `done` / `now` / pending.
- The old blinking-caret "pending" style was removed; the whole reply is shown at
once (dim) and progressively illuminated.
- A safety net finishes the highlight: once `assistant_end` has been received and
playback has been idle for 600 ms *after it had actually started*, any word
still pending is marked done, so a reply always ends fully lit. New turns and
interrupts cancel the loop.
### Notes
- The mapping from audio to text is inherently approximate (the server streams
text and PCM on the same socket without per-word timing), so `KARAOKE_CPS` is a
tunable estimate for the visual effect only — it never touches the audio.
- `played` counts only real samples, so the playback jitter buffer's silence and
any underrun pauses do not advance the highlight.
---
## 2026-10-16 — End-to-end validation → `TTS_PREBUFFER_MS` 700 → 450 ms; Drama-3 startup documented
### Changed
- **`TTS_PREBUFFER_MS` lowered `700` → `450`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)). With the vocoder ramp `"[10,12,16]"` the
*first* window is 10 codec frames = 0.464 s, so `450 ms` sits just below it and
playback now starts on that **first** window instead of waiting for the second
one that `700 ms` required.
- **README:** new *"Starting the Drama 3 backend (SGLang-Omni)"* subsection under
the `fish` plugin, with the exact `sgl-omni serve` command, the **pm2** workflow,
the meaning of the two `--vocoder.factory.*` flags and the two CLI gotchas (no
`stages.` prefix; the ramp tuple passed as a JSON string).
### Why
- An end-to-end browser test (`RUST_LOG=…,audio_agent_engine::tts=debug,…`) on the
live engine confirmed the narrower windows: bursts of 10/12/16 frames
(0.464 / 0.557 / 0.743 s) instead of a single 2.97 s block, `prebuffer=30870`
samples (= 700 ms) and time-to-first-audio ≈ 1.1 s from dispatch (~0.64 s of
which is the server's TTFB). Barge-in fired correctly on three interrupts.
- Three **tail underruns** were still observed at the end of short replies: e.g.
2.46 s of audio produced in ~2.6 s (≈1.07× real time including the fixed
~0.64 s TTFB), so the queue drained just before the last words. The mixer
recovered via resilient resume, but the gap is audible. `450 ms` targets the
first 0.464 s window and trades a little more tail risk for a markedly earlier
start.
### Notes
- The residual deficit is dominated by the server's **fixed ~0.64 s
time-to-first-byte**, which no pre-buffer value can remove. Going lower would
need a faster first token (model / preprocessing), not a smaller stride.
---
## 2026-10-16 — Finer TTS delivery granularity (Drama-3 vocoder window) → `TTS_PREBUFFER_MS` 700 ms
### Changed
- **The SGLang Drama-3 server now emits audio in ~0.74 s windows instead of
~2.97 s bursts.** Two flags were added to the server launcher
(`/root/drama3/start-drama3.sh`, managed by `pm2` as `start-drama3`):
`--vocoder.factory.stream_followup_stride 16` and
`--vocoder.factory.stream_followup_stride_ramp "[10,12,16]"` — upstream
defaults were `64` and `(10,12,32)`.
- **`TTS_PREBUFFER_MS` default lowered `900` → `700`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)). 700 ms is deliberately just under one 0.743 s
window, so playback starts on the *first* burst rather than waiting for a
second one.
### Why
- Drama-3's streaming granularity is a **vocoder** setting, not an engine one:
[`Drama3VocoderFactoryArgs`](../drama3/sglang-omni/sglang_omni/models/fishaudio_drama3/config.py)
buffers `stream_followup_stride` codec frames before decoding and emitting, and
`frame_length = hop_length * 4 = 2048` samples (46.4 ms at 44.1 kHz). The
upstream `64` therefore means a 131 072-sample burst every ~2.7 s.
- Measured against the live endpoint (single, uncontended request, PCM stream):
| | burst | period | ratio | first audio |
|---|---|---|---|---|
| before (`stride` 64) | 131 072 samples / 2.97 s | 2.70–2.78 s | 1.08× | 0.96 s |
| after (`stride` 16) | 32 768 samples / 0.743 s | 0.69 s (σ ≈ 0.01 s) | 1.04× | 1.34 s |
- The finer window is what makes the small pre-buffer viable: the producer still
runs slightly faster than real time (1.04×), but it now tops the queue up every
0.69 s instead of every ~2.7 s, so arrival jitter is bounded by one window and
no longer forces a multi-second buffer.
- Prefix caching (RadixAttention) was evaluated and **rejected**: the client
already sends a constant prefix (`model` / `voice` / `references`), yet an A/B
on the endpoint showed no TTFB difference (ref 0.62–0.64 s, no-ref 0.52 s) —
that start-up is decode/vocoder, not prompt prefill.
### Notes
- Each window re-decodes `stream_overlap_tokens` (32) frames of context, so a
smaller stride costs more vocoder compute: 96 frames decoded per 64 emitted
becomes 48 per 16. The measured ratio moved 1.08× → 1.04×, still above real
time; `stride` 8 (0.371 s windows, 40 frames decoded per 8 emitted) is the next
step down and should only be tried with the ratio re-measured.
- The window lengths feed the vocoder's CUDA-graph capture set, so changing them
requires a server restart.
- `stream_stride` (first window) and `stream_followup_stride_ramp` (the windows
after it) were left near their defaults — only the steady-state value drives
the granularity of a long reply.
- (Superseded: the `TTS_PREBUFFER_MS` default is now `450`, targeting the ramp's
0.464 s *first* window — see the entry above.)
---
## 2026-10-16 — Resilient mixer resume + `TTS_PREBUFFER_MS` default 900 ms
### Changed
- **The mixer no longer re-fills the whole jitter buffer after a mid-reply
underrun.** Previously, when the TTS queue emptied for more than the 200 ms
drain grace, [`pull_frame`](src/audio/mixer.rs) cleared `tts_primed` *and*
`tts_stream_end`, then required a full `tts_prebuffer_samples` refill before
playback resumed — adding a second, self-inflicted pause on top of the real
underrun. It also left the mixer "primed" between turns because
`mark_stream_start` never re-armed `tts_primed`.
Now:
- the full pre-buffer is awaited **only at the first playback of a turn**; once
playback has started, an underrun pauses and resumes as soon as *any* audio
is queued again (`need = 1`);
- `tts_stream_end` is no longer cleared on drain, so a genuine end-of-stream is
still honoured;
- `mark_stream_start` re-arms `tts_primed`, `tts_playback_started` and
`tts_drain_frames` at the start of every turn.
- **`TTS_PREBUFFER_MS` default raised `400` → `900`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)). With `400` the reply was still prone to
drop-outs on long answers.
### Why
- Direct measurement of the SGLang Drama-3 endpoint (a single, uncontended
request) showed it delivers **~1.08× real time, in bursts every ~2.7 s**
(≈131 072 samples / 2.97 s per burst). Because the producer barely outruns the
consumer and delivers in bursts, gapless playback requires a jitter buffer of
roughly the burst period (~3 s); smaller buffers show a short real drop-out
(pure silence) on long replies.
- The mixer fix removes the *avoidable* part of the pause (the full refill after
an underrun) and recovers at the first byte, which is what makes the remaining
gap equal to the backend stall rather than stall + refill.
- Default `900` is a deliberate latency-first choice: it keeps time-to-first-audio
low (~1.8 s) and accepts a rare short drop-out on very long replies. Raising
`TTS_PREBUFFER_MS` towards ~3000 trades that latency back for gaplessness.
(Superseded: the default is now `700`, once the server's vocoder window was
narrowed to ~0.74 s — see the entry above.)
### Notes
- The residual drop-out is a property of the TTS backend (~1× real time), not of
the mixer: no buffering smaller than the backend's burst period can bridge a
delivery stall.
- Relay diagnostics at `debug` level in [`run_llm_turn`](src/orchestrator/turn_manager.rs)
log per-chunk relay timing (`TTS pipeline ...`) to help localise seams.
---
## 2026-10-16 — Pipelined TTS: chunks are synthesized ahead of playback
### Changed
- **The TTS sink no longer waits for one chunk's stream to finish before
requesting the next.** Previously [`run_llm_turn`](src/orchestrator/turn_manager.rs)
looped `while let Some(chunk) = rx.recv() { tts.synthesize(chunk).await }`, so
a chunk's synthesis only started after the previous chunk had finished
streaming end-to-end. Because the TTS engine needs ~0.6 s to emit its first
audio byte, every chunk boundary left a gap roughly that long, which the old
1.5 s pre-buffer used to hide.
- The sink is now split into two pieces:
- a **collector** that spawns one synthesis task per chunk as soon as the LLM
produces it (bounded by a semaphore), each streaming into its own staging
buffer;
- a **relay** that forwards those staging buffers into the mixer queue
**strictly in submission order**, so playback stays in order and gapless.
A chunk's synthesis therefore overlaps the previous chunk's playback, hiding
the per-request start-up latency.
### Added
- **New `TTS_PIPELINE_DEPTH` configuration (default `3`)** in
[`src/config.rs`](src/config.rs): how many TTS requests may be in flight at
once. `1` restores the previous strictly serial behaviour.
- `SampleRingBuffer::remaining_capacity()` ([`src/audio/ring_buffer.rs`](src/audio/ring_buffer.rs))
so the relay never pushes past the mixer queue's free space — queued (i.e.
currently playing) audio is never dropped to make room.
### Why
- Direct measurement against the SGLang Drama-3 endpoint showed a **fixed
~0.62 s start-up latency** per request (request → first audio byte), with
audio then streamed at roughly real time. Two concurrent requests are
**served in parallel** (start-up rises only to ~0.84–1.0 s), so overlapping
requests genuinely removes the inter-chunk gaps instead of just moving them.
- The gaps were independent of chunk size (120 → 60 chars left the same ~0.8 s
hole), which confirmed the cause was serialization, not chunk length.
### Notes
- Staging buffers are sized at 30 s of audio per in-flight request, which is
comfortably more than any single chunk.
- With the pipeline in place the pre-buffer only has to cover arrival jitter
rather than the TTS engine's per-request start-up, so it can be kept small.
(Superseded: the default was later raised to `900` — see the entry above.)
---
## 2026-10-16 — Mixer TTS pre-buffer is now configurable (default 400 ms)
### Changed
- **The TTS jitter buffer moved from a hardcoded 1.5 s constant to the
`TTS_PREBUFFER_MS` setting (default `400`).** `AudioMixer` (in
[`src/audio/mixer.rs`](src/audio/mixer.rs)) now takes `tts_prebuffer_samples`
as a constructor argument instead of reading the removed
`TTS_PREBUFFER_SAMPLES` constant, and [`run_session`](src/ws/session.rs)
derives it from the config.
### Why
- Log analysis of a voice turn (`"Conta da dieci a uno."`) showed the first TTS
request left ~1.2 s after the EOU and the first audio bytes arrived ~1.9 s,
but playback only started at **+4.2 s**: the mixer had 1.486 s (65 536
samples) queued at +2.8 s, **14 ms short of the 1.5 s threshold**, so it
waited for the next Fish chunk, which landed 1.39 s later. The pre-buffer,
not the text chunker, dominated the perceived latency.
- Fish Audio now delivers near real time, so 1.5 s was far more buffer than
needed. 400 ms removes ~1.1 s of time-to-first-audio while still absorbing
normal inter-chunk jitter (the 200 ms drain grace is unchanged).
### Notes
- Tune from `.env`: raise `TTS_PREBUFFER_MS` if you hear drop-outs, lower it for
an even snappier start.
- The low-buffer warning threshold now scales with the configured value
(`prebuffer / 4`).
---
## 2026-10-16 — TTS starts on the first words, not on the first sentence
### Changed
- **The streaming reply is now chunked on a word boundary instead of waiting
for sentence punctuation.** `SentenceSplitter` (in
[`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)) keeps
the sentence-terminator rule (`.`, `!`, `?`, `;`, newline) but adds an early
flush: the **first** chunk is cut as soon as
`LLM_STREAM_FIRST_CHUNK_CHARS` characters are buffered, and every following
chunk at `LLM_STREAM_CHUNK_CHARS`. The cut lands on the first whitespace at or
past the threshold, so a chunk always holds at least that many characters and
never splits a word. Previously the first chunk only left at a `.`/`!`/`?`,
so a long opening sentence delayed the first spoken word.
### Added
- **New `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`) and
`LLM_STREAM_CHUNK_CHARS` (default `120`) configuration** in
[`src/config.rs`](src/config.rs). A small first threshold makes the agent
start speaking on the first few words; the larger follow-up threshold keeps
prosody good once the audio is flowing. Setting either to `0` disables that
flush (chunks then end on sentence punctuation only).
### Notes
- Verified on the running server: for a chat-box turn the first TTS request is
dispatched ~0.5 s after the user text, synthesizing `"Non ho occhi fisici né"`
(23 chars), well before the rest of the sentence
(`"accesso a telecamere o immagini, quindi non posso vedere nulla …"`) was
generated. Before the change the whole opening sentence had to complete first.
- The remaining time-to-first-audio is dominated by the mixer's 1.5 s TTS
pre-buffer plus the TTS engine's first-chunk latency, not by the chunker.
---
## 2026-10-15 — Filler is now a latency guard: switchable and delay-gated
### Changed
- **The end-of-turn filler is no longer played unconditionally.** It exists to
mask the wait for the reply, so playing it when the answer is already speaking
only gets in the way of the first words. `TurnManager::schedule_filler`
(in [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs))
starts a `FILLER_DELAY_MS` timer when the turn is dispatched (both the LLM and
the direct-TTS path) and queues the clip **only if the response audio has not
started by then**. At most one filler per turn.
- **New `FILLER_ENABLED` / `FILLER_DELAY_MS` configuration** in
[`src/config.rs`](src/config.rs). `FILLER_ENABLED=false` disables the filler
entirely (no clip is ever queued); `FILLER_DELAY_MS` (default `2500`) is the
latency guard, tunable from `.env`.
- The immediate filler pushes in `handle_user_turn` and `speak` were removed in
favour of the single scheduled path, so the playground's direct-TTS path
follows the same rule.
### Added
- **`AudioMixer::tts_playback_started()`** in
[`src/audio/mixer.rs`](src/audio/mixer.rs) — set when the TTS pre-buffer is
reached (i.e. the response audio actually starts), reset by
`mark_stream_start` and `flush`. This is what the filler decision reads.
### Notes
- The pending filler timer is bound to the turn's `CancellationToken`, so a
barge-in (`interrupt`, VAD onset, a new turn, or disconnect) drops it.
- Verified from `.env` on the running server: `FILLER_DELAY_MS=100` →
`pushed filler (delayed)`; `FILLER_DELAY_MS=8000` → `response audio already
playing — filler skipped`; `FILLER_ENABLED=false` → no filler activity at all.
- With the current `LLM_ENABLE_THINKING=false` + Fish Audio setup, a short reply
starts playing within 2.5 s, so the filler correctly stays silent; raise
`FILLER_DELAY_MS` (e.g. 1000) if you want it to fire more often.
---
## 2026-10-15 — LLM reasoning stage + `/mivocero` speech-to-speech chat client
### Added
- **OpenAI-compatible LLM client** — [`src/llm.rs`](src/llm.rs). New
`LlmClient` posting to `{LLM_BASE_URL}/chat/completions`, with an
`Authorization: Bearer {LLM_API_KEY}` header and a bounded TCP connect
timeout (no global request timeout, so SSE streams can stay open). Two
entry points, selected at runtime via `LLM_STREAM`:
- `chat_stream` — SSE token streaming (`stream: true`), handling `data:`
lines and `data: [DONE]`;
- `chat` — a single blocking request.
- **Chain-of-thought stripping** — reasoning models (Qwen3, …) may emit their
plan in a separate `reasoning_content` delta (vLLM) or inline between
think markers (llama.cpp and others). `reasoning_content` is ignored and
inline think blocks are removed by an incremental `ThinkStripper` that is
stable across delta boundaries (it withholds a tail the size of the opening
marker so a marker split between two deltas is still detected). Applied to
both the chat box text and the synthesized audio.
- **LLM stage in [`TurnManager`](src/orchestrator/turn_manager.rs)** — the user
turn (STT transcript or chat-box text) is sent to the model and the reply is
spoken by the TTS. A rolling history window (`LLM_HISTORY_MESSAGES`) is kept
per connection and prefixed with `LLM_SYSTEM_PROMPT`. A `SentenceSplitter`
chunks the streaming reply on `.`, `!`, `?`, `;` and newlines so the TTS
starts on the first sentence (`SENTENCE_SOFT_MAX` = 220 chars forces a split
on whitespace for punctuation-free runs).
- **Barge-in covers the LLM request** — an interrupt (client JSON, VAD onset or
disconnect) cancels the in-flight HTTP request through a
`CancellationToken` in the `reqwest` `tokio::select!`, cancels the TTS tasks
and flushes the mixer.
- **Bidirectional text protocol on `/ws`** — client → server
`{"type":"user_text","text":…}` and `{"type":"interrupt"}`; server → client
`{"type":"user","text":…}`, `{"type":"assistant_delta","text":…}`,
`{"type":"assistant_end"}` and `{"type":"error","text":…}`. Legacy text
commands (`INTERRUPT`, `SET_REFERENCE:`, `CLEAR_REFERENCE`, raw text → direct
TTS) are unchanged, so the playground keeps working.
- **`/mivocero` client** — [`src/web/mivocero.html`](src/web/mivocero.html),
embedded via [`src/web/mod.rs`](src/web/mod.rs) and served by
[`src/main.rs`](src/main.rs). A speech-to-speech example agent with a
microphone (same AudioWorklet capture/playback path as the playground), a
chat box that renders the user transcript and the **streaming** assistant
reply, a text composer and an interrupt button.
- **`LLM_*` configuration** — `Config` fields and `read_llm_env()` in
[`src/config.rs`](src/config.rs): `LLM_ENABLED`, `LLM_BASE_URL`,
`LLM_API_KEY`, `LLM_MODEL`, `LLM_MAX_TOKENS`, `LLM_TEMPERATURE`,
`LLM_STREAM`, `LLM_HISTORY_MESSAGES`, `LLM_ENABLE_THINKING`,
`LLM_STRIP_REASONING`, `LLM_DIRECT_ANSWERS`, `LLM_SYSTEM_PROMPT`.
`LLM_ENABLED` defaults to `true` when both `LLM_BASE_URL` and `LLM_MODEL`
are set.
- **Three independent chain-of-thought controls** (each switchable from `.env`)
— a single mechanism does not cover every deployment:
1. `LLM_ENABLE_THINKING` — `LlmClient` sends
`chat_template_kwargs: {"enable_thinking": <bool>}` when the variable is
set, turning Qwen3-style thinking off/on server-side (vLLM/SGLang). Unset
omits the vendor-specific field so strict OpenAI-compatible servers keep
accepting the body.
2. `LLM_STRIP_REASONING` (default `true`) — `reasoning_content` deltas are
dropped and inline think blocks removed; `false` passes the raw output
(both fields) through.
3. `LLM_DIRECT_ANSWERS` (default `true`) — appends
[`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt. Needed
when the server runs **no** reasoning parser: the chain-of-thought is then
written straight into `content` as ordinary prose and is neither
`reasoning_content` nor tagged, so filtering cannot catch it.
### Fixed
- **Chat-box turns blocked the receive loop** — `handle_control_message` in
[`src/ws/session.rs`](src/ws/session.rs) awaited `TurnManager::on_user_text`
inline, so while a reply was being generated the session could not read any
further client message: an `interrupt` sent during generation was only
honoured once the turn had finished. Text turns now run in their own task,
and the interrupt cancels the reply immediately (verified: 0 deltas emitted
after the interrupt, session still usable for the next turn).
- **SSE cancellation latency** — `LlmClient::chat_stream` now races the next
stream chunk against the `CancellationToken`, so a barge-in drops the HTTP
response instead of waiting for the model's next token.
### Notes
- Without an LLM the pipeline degrades to a plain STT → TTS echo
(`LLM_ENABLED=false`).
- `src/orchestrator/turn_manager.rs` gained unit tests for sentence splitting
(terminators, split across deltas, long punctuation-free runs) and
`src/llm.rs` for think-block stripping and for the opt-in thinking field —
`cargo test` passes (13 passed, 1 ignored).
- Verified against `Qwen/Qwen3.6-35B-A3B` on a local vLLM
(`http://10.8.0.4:8000/v1`): without `LLM_ENABLE_THINKING=false` the model
narrates "Here's a thinking process: 1. **Analyze User Input** …" directly in
`content`, which the TTS reads aloud; with it, the reply is the answer only.
---
## 2026-10-15 — Startup clip bootstrap (auto-generate filler / backchannel)
### Added
- **Startup asset bootstrap** — [`src/assets.rs`](src/assets.rs). On boot the
server checks every expected filler / backchannel clip and, if missing,
**synthesizes it with the configured TTS engine** and writes it as
`*.raw` (16-bit LE mono PCM at 44.1 kHz) into the target directory (created
if needed). Existing clips are never overwritten, so a fresh checkout works
with no manual generation step and a customized/partial bank is preserved.
- Default clip definitions:
[`assets::FILLER_CLIPS`](src/assets.rs) (`uhm`, `eh`, `mmm`, `aver`, `bueno`,
`pues`, `vaya`) and [`assets::BACKCHANNEL_CLIPS`](src/assets.rs) (`mmhmm`,
`aja`, `claro`, `yaveo`). Backchannel clips are generated only when
`BACKCHANNEL_ENABLED=true`.
- Wired in [`src/main.rs`](src/main.rs) **before** the banks are loaded; the TTS
engine is now created earlier so it can be reused for generation.
### Notes
- Generation is best-effort and never blocks startup: a synthesis failure logs a
warning per clip and the server still starts (that bank stays silent).
- The scratch capture buffer during generation is a 30 s
[`SampleRingBuffer`](src/audio/ring_buffer.rs) drained after synthesis.
---
## 2026-10-15 — Filler bank (round-robin end-of-utterance fillers)
### Added
- **Filler bank** — `assets/fillers/*.raw`. The end-of-utterance filler is no
longer a single hard-coded clip; [`TurnManager`](src/orchestrator/turn_manager.rs)
now selects clips **round-robin** from the bank via
[`FillerBank::nth`](src/orchestrator/turn_manager.rs) (a per-turn cursor,
`TurnManager::next_filler`). With a single clip the behaviour is unchanged.
- Clips synthesized with the cloned reference voice (Spanish, 44.1 kHz, 16-bit
LE mono PCM): `uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya` (~0.1–0.9 s).
### Notes
- `FillerBank::get` is kept for reference/debugging but is no longer on the
hot path (`#[allow(dead_code)]`).
---
## 2026-10-15 — During-utterance backchannel
### Added
- **Backchannel subsystem** — [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs).
Emits short acknowledgments ("mm-hmm", "ajá", "claro") *while the user is
speaking*, at a low fixed gain, to signal that the agent is listening without
taking the turn. Distinct from the end-of-utterance **filler** injected by the
[`TurnManager`](src/orchestrator/turn_manager.rs).
- **`BackchannelScheduler`** — per-session decision logic fed by the VAD:
`on_speech_start` on `VadEvent::SpeechStart`, `on_speech_end` on `SpeechEnd`,
and `maybe_emit()` on every frame while speaking. A cue is emitted only when
the user has spoken for at least `BACKCHANNEL_MIN_SPEECH_MS`, at least
`BACKCHANNEL_INTERVAL_MS` elapsed since the last cue, fewer than
`BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn, and the agent is
**not** speaking (`!TurnManager::is_speaking()`).
- **Preloaded clip bank** — `assets/backchannel/*.raw` (16-bit LE mono PCM at
44.1 kHz), synthesized with the cloned reference voice, loaded into a shared
[`FillerBank`](src/orchestrator/turn_manager.rs) and played round-robin.
- **Third mixer channel** — [`AudioMixer`](src/audio/mixer.rs) gained a dedicated
`backchannel_queue` and a fixed `backchannel_gain`. The mix is now
`tts + (filler × filler_gain) + (backchannel × backchannel_gain)`; the queue is
cleared by `flush()` (barge-in).
- **`FillerBank` generalization** — one type now serves both the filler and the
backchannel banks (`load(dir, label)`, round-robin `nth(index)`, `is_empty()`);
the default filler lookup ("uhm") is unchanged.
- **`VadDetector::in_speech()`** — exposes whether the detector is inside a
confirmed utterance, used to gate backchannel emission.
- **New `.env` knobs** (documented in [`.env.example`](.env.example)):
`BACKCHANNEL_ENABLED`, `BACKCHANNEL_DIR`, `BACKCHANNEL_GAIN`,
`BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS`,
`BACKCHANNEL_MAX_PER_TURN`.
### Notes
- `BACKCHANNEL_GAIN` is clamped to `0..1` and `BACKCHANNEL_INTERVAL_MS` is
clamped to `>= BACKCHANNEL_MIN_SPEECH_MS` at load time.
- Wiring lives in [`src/ws/session.rs`](src/ws/session.rs) (mic/VAD task); the
bank is created once in [`src/main.rs`](src/main.rs) and shared via `AppState`.
---
## 2026-10-15 — Server-side VAD (Silero V5, ONNX)
### Added
- **VAD subsystem** — [`src/vad/mod.rs`](src/vad/mod.rs) +
[`src/vad/silero.rs`](src/vad/silero.rs). Runs **Silero VAD v5** via ONNX
Runtime on the same 16 kHz mono stream that feeds the STT engine. The model
consumes 32 ms windows (512 samples) with a 64-sample context and a recurrent
`state [2,1,128]`; the graph shapes and the `sr` input rank are introspected at
load time.
- **Onset/offset state machine** with hysteresis (`VAD_THRESHOLD` /
`VAD_NEG_THRESHOLD`) emitting three events: `SpeechOnset` (fast, for
barge-in), `SpeechStart`, and `SpeechEnd` (carrying the captured PCM plus a
pre-roll pad so the first phoneme is not clipped).
- **Fast server-side barge-in** — a `SpeechOnset` interrupts the agent when it is
currently speaking (`TurnManager::is_speaking`), cancelling the in-flight TTS
and flushing the mixer.
- **VAD-driven STT segmentation** — `SttWorker::push_utterance` receives whole
utterances instead of raw frames, so the STT no longer needs its own RMS
segmentation (`VAD_SEGMENT_STT=true`, default).
- **Shared ONNX environment** — [`src/onnx.rs`](src/onnx.rs). ONNX Runtime allows
a single environment per process, so the STT and VAD pipelines now share a
lazily created `Environment` (cheap `Arc` clones) instead of each building one.
- **New `.env` knobs** (all documented in [`.env.example`](.env.example)):
`VAD_ENABLED`, `VAD_MODEL_PATH`, `VAD_THRESHOLD`, `VAD_NEG_THRESHOLD`,
`VAD_MIN_SPEECH_MS`, `VAD_MIN_SILENCE_MS`, `VAD_SPEECH_PAD_MS`,
`VAD_BARGE_IN_MIN_SPEECH_MS`, `VAD_SEGMENT_STT`, `VAD_BARGE_IN`, `VAD_DEBUG`.
- **`AudioMixer::is_active`** / `tts_stream_active` flag tracking whether a TTS
stream is in flight, independent of the queue length, so barge-in can gate on
"the agent is speaking".
### Fixed
- Duplicate analysis window appended to the utterance at speech start (the
pre-roll ring already contained it) — the utterance is now seeded from the
pre-roll ring only.
- `server.sh` now also appends `audio_agent_engine::vad=debug` to `RUST_LOG`
(mirroring the earlier TTS fix), so VAD debug logs are never filtered out.
### Verified
`models/silero_vad.onnx` is loaded at startup (`state_shape=[2,1,128]`,
`sr_rank=0`). A live WSS session streaming 44.1 kHz speech produced, in
`run/server.log`:
```text
INFO ws::session: VAD active for this session threshold=0.5 min_speech_ms=250 min_silence_ms=300 …
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
DEBUG vad::silero: VAD speech start speech_ms=256 prob="1.000" preroll_samples=4992
DEBUG vad::silero: VAD speech end duration_ms=1144 samples=18304 avg_prob="0.609"
INFO ws::session: VAD utterance → STT duration_ms=1144 samples=18304
INFO orchestrator::turn_manager: EOU detected tokens=5 text=Me escucha.
```
Talking over the agent (after a text TTS prompt) produced:
```text
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
INFO ws::session: VAD barge-in — interrupting agent prob="0.999"
INFO orchestrator::turn_manager: interrupt received, flushing audio
```
---
## 2026-10-15 — TTS DEBUG logs actually reach `run/server.log`
### Problem
The `debug!` diagnostics added to the TTS subsystem never appeared in the log,
even though the code and the `DEFAULT_LOG_FILTER` fallback were in place. Root
cause: the parent shell (IDE) already exports `RUST_LOG=info`, so
`EnvFilter::try_from_default_env()` succeeded and the
`info,audio_agent_engine::tts=debug` fallback was never used. In `server.sh`
the same happened — `${RUST_LOG:-info,audio_agent_engine::tts=debug}` never
expanded the default because `RUST_LOG` was already set.
### Fixed
- [`server.sh`](server.sh) now **appends** the
`audio_agent_engine::tts=debug` directive to whatever `RUST_LOG` it
inherits (unless that target is already mentioned), so the TTS debug logs
cannot be silently filtered out. `start` logs the effective filter.
### Verified
A live WSS session (`Hola, …`) that is kept open for the whole synthesis now
emits, in `run/server.log`:
```text
DEBUG audio_agent_engine::tts::fish_audio: TTS request dispatched endpoint=… has_reference=true text=…
DEBUG audio_agent_engine::tts::fish_audio: TTS time-to-first-byte ttfb_ms=0 bytes=8186
DEBUG audio_agent_engine::tts::fish_audio: TTS chunk received chunk=1 bytes=8186 samples=4093 queue_len=4093
DEBUG audio_agent_engine::tts::fish_audio: TTS stream summary chunk_count=9 total_samples=180224 audio_s="4.09" elapsed_ms=3347
```
> Note: closing the WebSocket mid-stream triggers `turn_manager.interrupt()`
> (barge-in), which cancels the TTS stream before any chunk arrives — so only
> `TTS request dispatched` is logged in that case.
---
## 2026-10-15 — Optional HTTPS/WSS + clearer microphone diagnostics
### Problem
Clicking "Connect & Start Mic" in the playground failed with a generic
`mic error`, with no WebSocket connection ever reaching the server. Browsers
only expose `navigator.mediaDevices.getUserMedia` in a **secure context**
(HTTPS or `localhost`); over plain HTTP on any other origin the API is simply
absent, so `startAudio()` threw before the socket was created.
### Added
- **Optional TLS termination** — [`src/tls.rs`](src/tls.rs). Set both `TLS_CERT`
and `TLS_KEY` (PEM) in `.env` and the server serves the same Axum router over
HTTPS/WSS via `tokio-rustls` + `hyper` (HTTP/1 upgrades enabled, so WebSocket
handshakes work). Without them it stays on plain HTTP. The startup log now
says `listening (HTTPS)` / `listening (HTTP)`.
- **`./server.sh cert [--force]`** — generates a self-signed certificate under
`certs/` whose SAN covers `localhost`, `127.0.0.1`, `::1` and the host's IPs
(`EXTRA_SAN` to append more). `start`/`restart` auto-generate it if TLS is
enabled but the files are missing; `status` shows the scheme and cert state.
### Changed
- **Playground diagnostics** — [`src/web/playground.html`](src/web/playground.html)
now checks for a secure context *before* requesting the mic and reports the
precise cause (`insecure origin` / `NotAllowedError` / `NotFoundError`) with a
remediation hint, instead of a bare `mic error`.
- New config fields `Config::tls_cert` / `Config::tls_key` (+ `tls_enabled()`),
read from `TLS_CERT` / `TLS_KEY`.
### Notes
- No new crates are downloaded: `hyper`, `hyper-util`, `tokio-rustls`, `rustls`
and `rustls-pki-types` were already present as transitive deps of `reqwest`
(`ring` provider), so they are declared with `default-features = false`.
---
## 2026-10-15 — `server.sh` control script (start / stop / restart)
Added [`server.sh`](server.sh), a small bash helper so the server does not have
to be managed by hand.
### Usage
```bash
./server.sh start [--build] # start (auto-builds if binary missing)
./server.sh stop # graceful stop (SIGKILL after 10 s)
./server.sh restart [--build] # stop + start
./server.sh status # process / port / HTTP status
./server.sh logs [N] # tail run/server.log (default 50, follow)
./server.sh build # build the release binary only
```
- PID is tracked in `run/server.pid`; logs are appended to `run/server.log`.
- `start` waits until the port is accepting connections (the model load takes
~1.5 s) before reporting success.
- Overridable env vars: `PROFILE` (`release`|`debug`), `RUST_LOG`, `PORT`.
- The script `cd`s to its own directory first, so `.env` and the relative
`models/parakeet` path resolve correctly regardless of the caller's cwd.
Documented in the README (Quick Start).
---
## 2026-10-15 — Parakeet TDT STT reimplemented in pure Rust (ONNX)
The previous STT was a placeholder that ran `argmax` over a single `.onnx`
file — it could not actually transcribe. It has been replaced with a complete,
dependency-free (no Python/NeMo) implementation of the NVIDIA **Parakeet TDT
0.6B v3** transducer pipeline in [`src/stt/parakeet.rs`](src/stt/parakeet.rs),
mirroring the reference in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr).
### Change
- **3-stage ONNX pipeline**, all sessions kept alive by the `ort` `Environment`:
1. `nemo128.onnx` — log-mel features `waveforms`/`waveforms_lens` → `features`/`features_lens`.
2. `encoder-model.int8.onnx` (preferred, auto-detected) or `encoder-model.onnx` —
`audio_signal`/`length` → `outputs`[1,1024,T']/`encoded_lengths`.
3. `decoder_joint-model.onnx` — greedy **TDT** decoding.
- **Runtime graph introspection**: `d_model`, `num_durations`, `state_layers` and
`state_hidden` are read from the ONNX I/O shapes instead of being hardcoded.
- **Greedy TDT decode loop** (token head + duration head; blank `<blk>` = 8192;
`max_tokens_per_step` from `config.json`, default 10). Encoder output is
re-packed from `[1, d_model, T']` to contiguous `[T', d_model]` frames.
- **Internal text decoding**: `vocab.txt` (`<piece> <id>`, SentencePiece `▁` →
space) plus the reference spacing normalizer (`\A\s|\s\B|(\s)\b`). The
`Transcript` struct now carries a `text: String`.
- **Config**: `STT_MODEL_PATH` now points to a **directory** (`models/parakeet`
by default, was `models/parakeet-tdt.onnx`). `STT_TOKENIZER_PATH` defaults to
empty. Updated `.env`, `.env.example` and the README.
- **TurnManager**: [`on_eou`](src/orchestrator/turn_manager.rs) now uses
`transcript.text` directly, falling back to the external tokenizer only when
no text was produced.
### Notes / fixes found during integration
- The `decoder_joint` graph expects `targets` and `target_length` as **int32**
(not int64); `encoder_outputs` and both state tensors are float.
- The int8 encoder (~650 MB, self-contained) is preferred over the fp32 encoder
(42 MB + a 2.4 GB `.onnx.data` sidecar) for speed and simpler deployment.
### Verification
Opt-in end-to-end test decodes a real 16 kHz mono WAV through the model:
```
$ PARAKEET_TEST_WAV=/tmp/stt_test/solis16k.wav \
cargo test --release -- --ignored test_infer_real_audio --nocapture
tokens: 67 | text: "Bienvenidos a esta fascinante exploración del universo.
Como locutor profesional, les explicaré hoy cómo las ondas electromagnéticas
viajan por el espacio, transformando nuestra comprensión del cosmos."
test stt::parakeet::tests::test_infer_real_audio ... ok
```
The transcription matches the reference transcript verbatim (the source clip was
trimmed to 12 s). `cargo test` (unit tests) also passes.
---
## 2026-10-15 — Fix: short utterances swallowed by the TTS pre-buffer
Short phrases produced **no audio at all**. The mixer only started playing TTS
once the queue held `TTS_PREBUFFER_SAMPLES` (1.5 s). If an utterance's total
length was below that threshold (e.g. `"Cristóbal Colón"` ≈ 1.11 s / 49152
samples), the pre-buffer was never reached, so `tts_primed` stayed `false` and
the queued samples were never drained.
### Change
- `src/audio/mixer.rs`: added an `tts_stream_end` flag with
`mark_stream_start()` / `mark_stream_end()`. When the stream has ended, the
mixer primes as soon as *any* sample is queued (`stream_ended && tts_len > 0`),
bypassing the pre-buffer for short utterances. The flag is reset by
`flush()`, `mark_stream_start()` and when the queue drains.
- `src/orchestrator/turn_manager.rs`: the spawned TTS tasks now call
`mixer.mark_stream_start()` before synthesis and `mixer.mark_stream_end()`
after it returns — only when the request was **not** cancelled, so barge-in
does not falsely signal completion.
### Verification
```
TTS stream complete chunk_count=6 total_samples=49152 duration_s="0.64"
TTS pre-buffer reached — starting TTS playback tts_queue_len=49152 prebuffer=66150 stream_ended=true
```
A raw WebSocket client sending `"Cristóbal Colón"` now receives ~1.12 s of
non-silent audio (previously the same request yielded only silence frames).
---
## 2026-10-15 — Dual-stack listener (fix `localhost` / `::1` connection refused)
The HTTP server bound only to `0.0.0.0` (IPv4). On hosts where `localhost`
resolves to `::1` first (confirmed via `getent hosts localhost` → `::1 localhost`),
clients and preview/port-forwarding proxies that connect to `localhost` via IPv6
received `connection refused`, surfacing as *"service is unavailable"*.
### Change
- `src/main.rs`: the listener now binds to the IPv6 wildcard
`SocketAddr::from((Ipv6Addr::UNSPECIFIED, port))` instead of `0.0.0.0`.
With `net.ipv6.bindv6only = 0` (Linux default, verified on this host) this is
dual-stack: it accepts both IPv6 (`::1` / `localhost`) and IPv4 clients.
### Verification
```
ss -ltnp | grep 8787 → LISTEN *:8787
curl http://[::1]:8787/ → 200
curl http://localhost:8787/ → 200
curl http://127.0.0.1:8787/ → 200
```
---
## 2026-10-14 — Modular TTS plugin system (engine-agnostic via `.env`)
The TTS layer was hard-coded to Fish Audio / SGLang Drama 3. To support
switching engines (e.g. Kokoro, or any future backend) without touching
orchestration code, the TTS subsystem is now a pluggable trait selected at
runtime via the `TTS_ENGINE` environment variable.
### New `TtsEngine` trait (`src/tts/mod.rs`)
```rust
#[async_trait]
pub trait TtsEngine: Send + Sync {
fn name(&self) -> &str;
fn sample_rate(&self) -> u32;
async fn synthesize(&self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken) -> Result<()>;
fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {} // default no-op
}
```
- Every plugin streams **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44.1 kHz)**
into the shared mixer queue. If an engine's native rate differs, it resamples
internally (the mixer runs at a fixed rate).
- `synthesize()` must honor the `CancellationToken` for barge-in.
- `set_reference()` defaults to a no-op; only engines that support voice
cloning override it.
### Factory (`src/tts/mod.rs::create_tts_engine`)
```
TTS_ENGINE=fish → FishAudioTts (Fish Audio / SGLang Drama 3)
TTS_ENGINE=kokoro → KokoroTts (stub, implement in src/tts/kokoro.rs)
```
Unknown values panic at startup with a clear message.
### Bundled plugins
| Plugin | File | Status |
|---|---|---|
| `fish` (Fish Audio Drama 3) | `src/tts/fish_audio.rs` | Production-ready (renamed from `sglang_client.rs`) |
| `kokoro` | `src/tts/kokoro.rs` | Stub — HTTP streaming call left as `TODO` with a commented example; includes a 24 kHz → 44.1 kHz linear resampler helper |
### `.env` loading
- Added `dotenvy` to auto-load a `.env` file at startup (existing env vars take
precedence).
- Added `.env.example` documenting all env vars including `TTS_ENGINE`,
`KOKORO_URL`, `KOKORO_VOICE`.
### Files changed
- `src/tts/mod.rs` — new `TtsEngine` trait + `create_tts_engine` factory.
- `src/tts/fish_audio.rs` — renamed from `sglang_client.rs`; `FishAudioTts` now implements `TtsEngine`.
- `src/tts/kokoro.rs` — new Kokoro plugin stub (example for adding engines).
- `src/tts/sglang_client.rs` — deleted (moved to `fish_audio.rs`).
- `src/config.rs` — new `tts_engine` field, `TTS_ENGINE` env var (default `fish`).
- `src/orchestrator/turn_manager.rs` — `tts: SglangTtsClient` → `tts: Arc<dyn TtsEngine>`.
- `src/ws/handler.rs` — `AppState.tts` → `Arc<dyn TtsEngine>`.
- `src/main.rs` — uses `create_tts_engine(&config)`; loads `.env` via `dotenvy`.
- `Cargo.toml` — added `async-trait`, `dotenvy`.
- `.env.example` — new file documenting all configuration env vars.
### How to add a new engine
1. Create `src/tts/myengine.rs` and implement `TtsEngine`.
2. Add `pub mod myengine;` to `src/tts/mod.rs`.
3. Register it in the `create_tts_engine` match.
4. Set `TTS_ENGINE=myengine` in `.env`.
### Build / verification
- `cargo build` — 0 errors, 2 dead-code warnings (Kokoro stub fields).
- `cargo test` — 4/4 passed.
- Server starts with `TTS_ENGINE=fish` (default): log shows `selected TTS engine engine="fish"` and voice-cloning reference loaded.
---
## 2026-10-14 — Cancel TTS on WebSocket disconnect
When the client disconnected, the in-flight TTS synthesis task continued
streaming from Fish Audio even though the WebSocket session was already closed,
wasting bandwidth and server resources.
### Fix (`src/ws/session.rs`)
- Added `turn_manager.interrupt()` in the session cleanup path (after the
receiver ends, before aborting the sender/mic tasks).
- This cancels the TTS `CancellationToken`, flushes the mixer queues, and stops the
streaming HTTP request immediately.
---
## 2026-10-14 — Fix WebSocket frame timing (Burst ticker)
The 20 ms audio sender ticker used `MissedTickBehavior::Skip`, which dropped
tick beats whenever the Tokio event loop was busy (e.g. processing a large
Fish Audio TTS chunk). Each skipped tick meant a 20 ms gap in frame delivery
to the browser, producing audible micro-interruptions even though the server's
TTS queue was well-buffered.
### Fix (`src/ws/session.rs`)
- Changed `MissedTickBehavior::Skip` → `MissedTickBehavior::Burst`.
- Missed ticks now trigger immediate back-to-back sends so the browser
receives all due frames. The browser's jitter buffer (500 ms) absorbs the
burst without underrunning.
---
## 2026-10-14 — Micro-interruption fix (grace period on TTS drain)
Even with the 1.5 s server pre-buffer and 500 ms browser jitter buffer,
brief audio gaps could still occur when the TTS queue momentarily hit zero
between Fish Audio chunks. The mixer immediately reset its `tts_primed`
state, forcing a full 1.5 s re-prime on the next chunk. The browser
playback processor did the same, re-entering buffering mode on any
underrun.
### Server fix (`src/audio/mixer.rs`)
- Added `tts_drain_frames` counter (`Arc<AtomicUsize>`) and
`TTS_DRAIN_GRACE_FRAMES = 10` (200 ms).
- When the TTS queue is empty while primed, the mixer now outputs silence
for up to 10 consecutive frames before resetting `tts_primed`. This
absorbs brief inter-chunk gaps without re-triggering the pre-buffer.
- `flush()` also resets `tts_drain_frames`.
### Browser fix (`src/web/playground.html`)
- `PlaybackProcessor` now tracks `underrunSamples` and only re-enters
buffering mode after 100 ms (`sampleRate * 0.10`) of continuous
underrun. Brief underruns produce silent output but keep playback
running, so audio resumes immediately when the next WebSocket frame
arrives.
---
## 2026-10-14 — Prevent overlapping TTS streams (double-Speak fix)
When the user clicked **Speak** twice in quick succession, two TTS synthesis
tasks ran concurrently and both pushed PCM into the same mixer queue. This
produced overlapping/garbled audio and roughly doubled the total duration
(one observed run produced 83 s of audio for a ~42 s text).
### Server fix (`src/orchestrator/turn_manager.rs`)
- `speak()` now calls `self.mixer.flush()` immediately after
`cancel_current_tts()`. This clears any remaining PCM from the previous
utterance so the new one starts from a clean queue.
### Client fix (`src/web/playground.html`)
- The **Speak** button is now disabled (shows "Speaking...") as soon as a
request is sent.
- A 1.5 s idle timer re-enables it once the server stops sending audio
frames, preventing accidental double submissions.
---
## 2026-10-14 — Jitter buffer fixes (no more audio drop-outs)
Fish Audio Drama 3 streams PCM in unevenly-spaced chunks, which caused brief
audible drop-outs (silence gaps) during playback. Fixed with a double jitter
buffer.
### Server-side TTS pre-buffer (`src/audio/mixer.rs`)
- Added `TTS_PREBUFFER_SAMPLES` ≈ 200 ms of audio (44.1 kHz × 0.20).
- The mixer now tracks a `tts_primed` flag (`Arc<AtomicBool>`). TTS samples are
**not** consumed until the TTS queue holds at least the pre-buffer amount.
- During the priming window only the filler ("uhm") backchannel is emitted, so
the user hears an immediate response while TTS audio builds up.
- When the TTS queue empties (stream exhausted), `tts_primed` resets to `false`
for the next request. `flush()` also resets it (barge-in).
- `AudioMixer::pull_frame` now uses `tts_queue.len()` instead of `is_empty()`
for priming decisions.
### Browser-side playback jitter buffer (`src/web/playground.html`)
- The `PlaybackProcessor` AudioWorklet now starts in a `buffering` state and
only begins playing once the ring buffer holds ≥ 200 ms of samples.
- On underrun (buffer drained mid-playback) it returns to `buffering` to avoid
stuttering, then resumes once enough audio has re-accumulated.
- Added `available()` helper to compute the fill level of the ring buffer.
### Build / verification
- `cargo build` — 0 warnings.
- `cargo test` — 4/4 passed.
---
## 2026-10-14 — Voice cloning + 44.1 kHz native output
### Sample rate aligned to Fish Audio native output
- Changed `OUTPUT_SAMPLE_RATE` from **24 000 Hz** to **44 100 Hz** to match Fish Audio Drama 3's native PCM output.
- Removed the TTS anti-aliasing resampler from `sglang_client.rs` — Fish Audio PCM samples are pushed directly into the mixer queue. This eliminates any resampling artifacts ("metallic" sound) and pitch/speed mismatches.
- The browser `AudioContext` now runs at 44 100 Hz, so playback is sample-accurate.
- `OUTPUT_FRAME_SAMPLES` is now 882 (44.1 kHz × 20 ms), `OUTPUT_FRAME_BYTES` = 1764.
- Mic → STT path: browser capture resamples to 44.1 kHz, server resamples 44.1 kHz → 16 kHz for Parakeet (linear interpolation, STT-only path so aliasing is not a concern).
### Voice cloning (reference audio)
Added support for Fish Audio's voice-cloning `references` field.
**Config (env vars, applied at startup):**
| Env var | Description |
|---|---|
| `TTS_REFERENCE_AUDIO` | Path to a WAV file on the Fish Audio server filesystem. |
| `TTS_REFERENCE_TEXT` | Reference transcript. If it points to an existing `.txt` file, its contents are loaded; otherwise the raw string is used. If unset, the server looks for a `<audio-stem>.txt` sibling of the WAV. |
**Default:** `solis.wav` + auto-loaded `solis.txt` from `/root/pipecat-voice/voice_profiles/`.
**Runtime override (playground → server WebSocket text messages):**
- `SET_REFERENCE:<path>|<text>` — set or change the reference. `<text>` may be empty to re-use the sibling `.txt`.
- `CLEAR_REFERENCE` — disable voice cloning (use the model's default voice).
**Files changed:**
- `src/config.rs` — new `tts_reference_audio` / `tts_reference_text` fields, `Config::tts_reference()` resolver with sibling-`.txt` fallback, new env-var parsing.
- `src/tts/sglang_client.rs` — `Reference` struct, `references` field in `SpeechRequest` (omitted when `None`), runtime `set_reference()` backed by a `parking_lot::RwLock<Option<Reference>>`.
- `src/orchestrator/turn_manager.rs` — forwards `set_reference()` to the TTS client.
- `src/ws/session.rs` — handles `SET_REFERENCE:` / `CLEAR_REFERENCE` control messages.
- `src/web/playground.html` — new "Voice Cloning" card with WAV path + transcript inputs and Set/Clear buttons; reference status indicator.
### Build / verification
- `cargo build` — 0 warnings.
- `cargo test` — 4/4 passed (tokenizer).
- Direct Fish Audio `references` call confirmed HTTP 200 + `x-sample-rate: 44100`.
- End-to-end WebSocket test: `SET_REFERENCE` followed by a text-to-speak request streams 44.1 kHz PCM back to the client.
Reference in New Issue View Git Blame Copy Permalink