roomote_PRE/piperust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

step2

Browse Source
This commit is contained in:
valenti
2026-10-17 15:49:20 +01:00
parent 430997bd2e
commit bfed42c3b3
124 changed files with 5646 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files
UPDATES.md
+807
View File
@@ -5,6 +5,813 @@ Entries are written in English and ordered newest-first.
---
## 2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)
### Added
- **`cues` WebSocket event** — sent once at session start by
`src/ws/session.rs`, it reports the **effective policy** of the three cues for
the active language: the filler (`delay_ms`, `gain_idle`, `gain_duck`, clip
count, bank dir), the start filler (`go_delay_ms`, clip count, bank dir) and
the backchannel (gain, min-speech, interval, jitter, pause window,
probability, volume jitter, per-turn cap, categories, clip count). The
backchannel `pause_max_ms` is capped to the VAD EOU window exactly like
`BackchannelScheduler::pause_max`; the bank paths are the language
subdirectories (`assets/fillers/<lang>`). Purely informational.
- **Cue methods panel** in `src/web/mivocero.html` — a collapsible
`<details>` section under the cue tiles with one card per cue: the **method**
used (when it fires, how a clip is chosen) and the **live parameters** from
the `cues` event, so the panel always matches the running configuration
instead of hard-coded values.
### Notes
- The panel is display-only; it changes no behaviour. Values reflect the
process configuration at session start (env-derived), so a `.env` change is
visible after the next session.
---
## 2026-10-17 — Per-session audit log (structured JSONL)
### Added
- **`src/audit.rs`** — a best-effort per-session audit sink. Each WebSocket
session is recorded to its own JSONL file
(`SESSION_AUDIT_DIR/<id>.jsonl`, id `s-<epoch_ms>-<seq>`), one record per line:
`{"ts":…,"elapsed_ms":…,"event":…,"data":…}`. Records are flushed as they are
written, and any failure (directory/file creation, write) only disables the
sink for that session — the session itself is never affected.
- **Event coverage (complete)**:
- `session_start` — full non-secret configuration snapshot plus the active
runtime capabilities (STT / VAD / tokenizer / LLM + model);
- `control` — inbound `user_text` turns;
- `out` — **every** event sent to the client, tapped at the single sender
point in `src/ws/session.rs` so nothing is missed (`user`,
`assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`,
`audio_source` state, …);
- `interrupt` — client barge-in (JSON or legacy text form);
- `barge_in` — VAD-driven barge-in (with the onset probability);
- `vad_speech_start` / `vad_speech_end` — utterance boundaries;
- `media` / `reference` / `speak` — external-audio control, voice-reference
changes and direct TTS requests;
- `language_switch` — detected language change (`from`, `to`, `generated`) —
`src/orchestrator/turn_manager.rs`;
- `error` — transport/receiver errors;
- `session_end` — reason, duration, session id, file and per-event counts.
- **Config** — `SESSION_AUDIT_ENABLED` (default `true`) and
`SESSION_AUDIT_DIR` (default `logs/sessions`); documented in `.env`,
`.env.example` and the README. `logs/` is git-ignored.
### Changed
- `TurnManager::new` takes an `Arc<SessionAudit>` (used for the language-switch
record); every other event reaches the log through the sender tap.
### Notes
- The audit path is off the audio hot path (the sink only touches the outbound
text events, not the 20 ms frames) and is disabled cleanly when the LLM/VAD/…
are off. Intended for post-mortem analysis and for tuning the latency knobs.
---
## 2026-10-17 — Cue emission indicators in the Mivocero demo
### Added
- **`{"type":"cue"}` WebSocket event** — the server now tells the client each
time a latency-mask / backchannel cue is **actually emitted** into the mixer:
- **filler** (end-of-turn) — `src/orchestrator/turn_manager.rs`
(`schedule_filler`),
- **start_filler** (turn start) — `src/orchestrator/turn_manager.rs`
(`schedule_start_filler`),
- **backchannel** (during user speech) — `src/ws/session.rs` (mic task).
The payload is `{"type":"cue","kind":…,"name":…}` and, for backchannel, also
`"category":…`. A cue that is skipped (audio already arrived / playing, or a
barge-in) sends nothing.
- **`BackchannelScheduler::maybe_emit`** now returns `(name, category)` instead
of just the name, so the backchannel cue carries its category.
- **Mivocero cue indicators** — `src/web/mivocero.html` gained a row of three
tiles (**Filler**, **Start filler**, **Backchannel**) that flash (colour-coded
dot + border glow) and bump a per-session counter on every `cue` event, with a
detail line showing the backchannel category (or the clip name). Counters reset
on each connect.
### Notes
- Only real emissions are reported, so the panel is a faithful view of what the
user actually hears — useful for tuning `FILLER_DELAY_MS` /
`START_FILLER_GO_DELAY` and the backchannel cadence.
---
## 2026-10-17 — Per-language subdirectory for every clip bank (base included)
### Changed
- **`src/main.rs`** — the startup bootstrap now generates and loads the three
base banks (filler, start-filler, backchannel) from
`<dir>/<LANG_DEFAULT>/` via [`assets::lang_dir`](src/assets.rs) instead of the
top-level directory. Every language — **including the default one** — now
lives in its own subdirectory, so the filler, start-filler and backchannel
share the exact same layout for every language and no language is special-cased
to the top level.
- **`src/assets.rs`** — [`lang_dir`](src/assets.rs) doc updated accordingly.
### Notes
- On the first start after this change the base clips are re-synthesized into
`assets/<bank>/<lang>/`. To avoid the re-synthesis, move the existing
top-level `*.raw` files (and `texts.txt`) into the matching language
subdirectory by hand — this was done for the `es` deployment.
- No other language's clips are touched: the per-language subdirectory remains
what isolates each language's refresh (`remove_stale_clips`).
---
## 2026-10-17 — Multilingual clip banks (backchannel, filler & start filler)
### Added
- **`src/lang.rs`** — new multilingual module. It holds built-in phrase tables
for **backchannel** (category + phrase), **filler** (interjections) and
**start filler** (turn-start acknowledgments, with Drama-3 style tags) in
Spanish, English, Italian, French, German and Portuguese, exposes accessors
for them, and implements the auxiliary LLM calls: `detect_language` (returns
only the ISO 639-1 code, or `und`) and `generate_backchannel_texts` /
`generate_filler_texts` / `generate_start_filler_texts` (LLM fallback for
languages absent from the tables).
It also defines `LangBanks` (a language's loaded clip banks, including its
start-filler bank) and `LanguageState` (the session's active language plus a
generation counter, kept in a parking-lot `RwLock`).
- **On-demand per-language generation** — when the user's language differs from
the active one, that language's backchannel/filler/start-filler clips are
synthesized into a **per-language subdirectory** (`assets/backchannel/<lang>/`,
`assets/fillers/<lang>/`, `assets/start_filler/<lang>/`) and the session
switches to them. Because the refresh pass only prunes clips inside the
directory it is given, **no other language's clips are ever removed**; the
base language keeps living in the top-level directories, so no migration is
needed.
- **New env vars** — `LANG_DEFAULT` (default `es`), `LANG_DETECT_ENABLED`
(default `true`), `LANG_MIN_CHARS` (default `12`), `LANG_REDETECT_EVERY`
(default `5`) and `LANG_LLM_FALLBACK_ENABLED` (default `true`).
### Changed
- **`src/llm.rs`** — new `LlmClient::simple_chat` for short, non-streaming
auxiliary requests (explicit system + user message, **no** tools), used by the
language detection and phrase-generation calls.
- **`src/config.rs`** — five new `LANG_*` fields, parsed in `from_env`.
- **`src/assets.rs`** — `lang_dir` helper plus `ensure_lang_backchannel_clips` /
`ensure_lang_filler_clips` / `ensure_lang_start_filler_clips`, which scope the
text-bank refresh to `<base>/<lang>/` so each language is refreshed
independently.
- **`src/orchestrator/backchannel.rs`** — the clip/category mapping is extracted
into `map_clips`, and `BackchannelScheduler::retarget` re-points a live
scheduler to a new bank when the language changes. `DEFAULT_CATEGORY` is now
`pub(crate)` so `lang.rs` can reuse it.
- **`src/orchestrator/turn_manager.rs`** — `TurnManager` now holds an
`Arc<LanguageState>` instead of a single filler bank; `next_filler` and the
start filler read from the active language (`schedule_start_filler` picks from
`langs.current().start_filler`). A new `maybe_detect_language` runs on each
user turn (first turn + every `LANG_REDETECT_EVERY`) and spawns
`switch_language_if_needed`, which detects, generates/loads the target
language's banks (start filler optional: a missing table just leaves that cue
silent) and swaps the active state — in the background, so the reply is never
blocked.
- **`src/ws/session.rs`** — builds the shared `LanguageState`; the mic task polls
the language generation counter each frame and calls `retarget` on the
backchannel scheduler only when it changes.
- **`.env` / `.env.example`** — new "Multilingual clip banks" section;
**`README.md`** — new "Multilingual clip banks" section.
### Why
- The banks were pre-synthesized in a single language, so a user speaking any
other language heard Spanish cues — including the start filler. Generating the
matching language on demand keeps the acknowledgment natural in a multilingual
deployment, while the per-language directory layout guarantees that adding a
language never destroys the clips already generated for the others.
### Notes
- Detection needs the LLM (`LANG_DETECT_ENABLED` is a no-op without it); the LLM
phrase fallback is used only for languages outside the built-in tables.
- Generating a new language's bank calls the TTS backend once per phrase, which
can take a few seconds on the first switch; it is idempotent (an
already-generated language is only reloaded).
---
## 2026-10-16 — Mivocero: auto-disconnect an idle session after 2 minutes
### Added
- **Idle guard in the Mivocero client** ([`src/web/mivocero.html`](src/web/mivocero.html)) —
the session closes itself after `IDLE_TIMEOUT_MS` (2 minutes) without any
activity. `touchActivity()` re-arms the timer and is called on every server
event, on every received audio frame and on every user action sent
(`sendJson`), so an ongoing reply or a playing external source keeps the
session alive. On expiry it logs the reason, appends an explanatory message to
the chat and calls `disconnect()`, which closes the WebSocket and releases the
microphone.
### Changed
- **Reconnect after close** — `ws.onclose` now clears the `ws` reference.
Without it the Connect button stayed bound to the stale socket, so a session
closed by the idle guard (or by any disconnect) could not be reopened without
reloading the page.
### Why
- A microphone-backed session left open keeps the device busy and the server
streaming silent 20 ms frames forever. An inactivity timeout releases both
without user action, while still allowing a long reply or a background radio
to run uninterrupted.
### Notes
- The timer counts *any* traffic in either direction: TTS audio is sent as
binary frames, so speaking does not trip the timeout.
- The timeout is a client-side constant (`IDLE_TIMEOUT_MS` in the HTML); change
it there and rebuild (the page is embedded with `include_str!`).
---
## 2026-10-16 — Tool calling: configurable HTTP tools for the LLM
### Added
- **`src/tools.rs`** — new tool-calling module. It loads tool definitions from a
JSON file and provides: a `ToolRegistry` (builds the OpenAI `tools` array,
executes a call over HTTP), a client-side parser for the model's **native**
`<tool_call>` format, and a `ToolCallStripper` that removes tool-call blocks
from the streamed reply — holding back a block that is still incomplete, so it
is never spoken by the TTS nor shown in the chat box even when split across
deltas (same hold-back pattern as the chain-of-thought stripper).
- **`tools.example.json`** — ready-to-use definitions: `get_weather` (GET with a
`{city}` URL placeholder, wttr.in) and `send_webhook` (POST with a custom
header), covering both request shapes.
- **New env vars** — `LLM_TOOLS_ENABLED` (auto: on when `LLM_TOOLS_FILE` is set),
`LLM_TOOLS_FILE`, `LLM_TOOL_MAX_ITERATIONS` (default `4`),
`LLM_TOOL_TIMEOUT_SECS` (default `15`) and `LLM_TOOL_RESULT_MAX_CHARS`
(default `2000`).
- **`ChatMessage::tool`** — `role: "tool"` message used to hand a tool result
back to the model.
- **WebSocket protocol** — informational
`{"type":"tool_call","name":"…","arguments":{…}}` event emitted when the model
calls a tool.
### Changed
- **`LlmClient`** carries an optional `Arc<ToolRegistry>`
([`src/llm.rs`](src/llm.rs)); when present, every request advertises
`tools` + `tool_choice:"auto"`, so the model knows which functions it may
call. The registry is built by `load_tools(config)`; a missing or invalid file
logs a warning and leaves tool calling disabled without breaking the pipeline.
- **[`TurnManager::run_llm_turn`](src/orchestrator/turn_manager.rs)** now runs a
bounded tool loop: each round streams a reply through the `ToolCallStripper`;
if a call was emitted, the assistant message (with its native call) plus one
`tool` message per result are appended and the model is re-prompted, up to
`LLM_TOOL_MAX_ITERATIONS`. The TTS pipeline is created once and every round's
visible text flows into it, so the assistant can say something short before
calling. Only the spoken text is written to the rolling history.
### Why
- The deployment's vLLM server runs **without** a tool parser
(`--enable-auto-tool-choice --tool-call-parser …`), so it never returns
structured `tool_calls` / `finish_reason:"tool_calls"`. Tools are still
advertised in the request and the Qwen model emits the call as text in its
native format; parsing it client-side keeps the feature working with any
server and requires no server-side reconfiguration.
- Tool definitions live in a JSON file rather than `.env` so the verbose JSON
schemas stay readable and editable.
### Notes
- Tool errors (unknown tool, transport failure, non-2xx status) are returned to
the model as text so it can recover; a tool never fails the turn.
- `GET` sends arguments not consumed by a `{arg}` placeholder as query
parameters; `POST` sends them as a JSON body. `headers` are sent verbatim.
---
## 2026-10-16 — External audio channel: inject a URL / radio / YouTube source
### Added
- **`src/media.rs`** — new [`MediaSource`](src/media.rs) module implementing an
external audio source for a WebSocket session. A source is decoded by an
external process and its raw PCM is pushed into the mixer's new external ring
buffer. `ffmpeg` decodes direct URLs / radio / HLS
(`-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1`, with
`-reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5`); YouTube links are
first resolved to a direct URL with `yt-dlp` (`-f bestaudio/best -g
--no-playlist`). Child processes are killed on stop / replacement /
disconnection (`kill_on_drop(true)` + `CancellationToken`), and a monotonic
generation counter prevents a stale task from clearing a newer source.
- **Mixer external channel** — `AudioMixer` gains `external_queue` and a runtime
gain ([`src/audio/mixer.rs`](src/audio/mixer.rs)). The mixing formula becomes
`… + (external × external_gain)`; `set_external_gain` updates the gain (an
atomic, so it takes effect on the next frame without rebuilding the mixer).
The external queue is **not** cleared by `flush()` and does not count towards
`is_active()`: a barge-in stops the agent, not the background audio.
- **New env vars** — `EXTERNAL_AUDIO_GAIN` (default `0.6`),
`MEDIA_FFMPEG_BIN` (default `ffmpeg`), `MEDIA_YTDLP_BIN` (default `yt-dlp`) and
`MEDIA_YTDLP_ARGS` (extra `yt-dlp` args, e.g. `--cookies …`).
- **WebSocket protocol** — new `audio_source` message:
`{"type":"audio_source","action":"play","url":"…"}`,
`{"type":"audio_source","action":"stop"}` and
`{"type":"audio_source","action":"volume","gain":0.6}`. The server reports
state back with `{"type":"audio_source","state":"loading|playing|stopped|error","url":"…"}`
([`src/ws/session.rs`](src/ws/session.rs)).
- **Mivocero client** — footer bar with a URL input, **Play** / **Stop**
buttons, a volume slider (0–150 %) and a live state label, driving the external
channel ([`src/web/mivocero.html`](src/web/mivocero.html)).
### Changed
- [`TurnManager::set_external_gain`](src/orchestrator/turn_manager.rs) forwards
the client volume to the mixer.
- New module declared in [`src/main.rs`](src/main.rs) (`mod media;`).
### Why
- Lets the agent play a music/radio/YouTube bed under the voice — useful for
demos (radio, background music) — while keeping the channel fully independent
of the speech pipeline so it survives barge-in.
### Notes
- Only needed when the external channel is used: `ffmpeg` for URL/radio/HLS and
`yt-dlp` for YouTube. YouTube may require cookies in some environments; pass
them via `MEDIA_YTDLP_ARGS=--cookies /path/cookies.txt`.
- A bounded source (VOD) is throttled to the mixer's drain rate via backpressure
(`remaining_capacity()` + 20 ms sleeps), so a fast decoder cannot overflow the
ring buffer.
---
## 2026-10-16 — Drama 3 de-emphasis: sampling knobs + neutral style tag
### Added
- **`TTS_TEMPERATURE`** (default `0.45`), **`TTS_TOP_P`** (default `0.75`) and
**`TTS_TOP_K`** (unset) — optional sampling overrides forwarded in the
`/v1/audio/speech` payload by the `fish` engine. An empty value omits the field
so the server default applies. New `env_opt_f32` / `env_opt_u32` helpers in
[`src/config.rs`](src/config.rs) implement the "unset → fallback, empty →
`None`" semantics.
- **`TTS_STYLE_TAG`** (default `[conversational, casual]`) — a neutral style tag
prepended to every synthesis request whose text does not already start with a
`[...]` tag ([`FishAudioTts::styled`](src/tts/fish_audio.rs)). Empty disables
the prefix.
### Changed
- [`SpeechRequest`](src/tts/fish_audio.rs) gains optional `temperature`, `top_p`
and `top_k` fields (serialized only when set); the request body now uses the
styled text as `input`. Startup logs `TTS de-emphasis settings …`, and the
per-request debug log shows the sampling values and whether the text was
prefixed (`styled=`).
### Why
- Drama 3 over-articulates: LLM replies could come back with a theatrical,
radio-drama cadence. Exaggerated intonations live in the tails of the acoustic
token distribution, so trimming the sampling tails (`temperature` / `top_p`)
flattens the delivery, and a neutral style tag steers untagged (LLM) prose into
a conversational register. Clip banks carry their own tags and are untouched.
### Notes
- `repetition_penalty` is deliberately **not** exposed: this server rejects any
value other than `1.0` (`Drama 3 request: repetition_penalty must be 1.0`).
`min_p`, `seed` and a nested `sampling_params` are rejected too.
- The voice-cloning reference (`solis.wav`) is left unchanged for now — its
"locutor profesional" prosody is a further, separate lever.
- **`.env` gotcha surfaced during this change.** `dotenvy` fails on unquoted
values containing spaces, and a single bad line makes it discard the whole
file: the process then silently starts with built-in defaults (no LLM, plain
HTTP) while `server.sh` still reports the scheme from its own `.env` grep. The
style tag is therefore quoted (`TTS_STYLE_TAG="[conversational, casual]"`).
---
## 2026-10-16 — Backchannel: random reaction depth inside the micro-pause
### Changed
- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) now draws
the depth at which it reacts **per pause**: the first time a micro-pause is
seen, a target is picked uniformly in `[BACKCHANNEL_PAUSE_MIN_MS,
BACKCHANNEL_PAUSE_MAX_MS]` and the cue fires only once `silence_ms()` reaches
it (new `pause_target_ms` field, cleared when the pause ends or the turn
resets). Previously the cue fired at the *first* analysis window inside the
window, so with a 32 ms VAD step every cue landed at exactly
`BACKCHANNEL_PAUSE_MIN_MS`.
- A pause that grows past the upper bound (on its way to end-of-utterance) is
abandoned without firing.
### Why
- Verification on a live session showed every cue at `pause_ms=96`, i.e. all cues
reacting at exactly the minimum — a residual regularity in an otherwise
naturalized feature. Randomizing the depth removes it: cues now land anywhere
in the window (observed `pause_ms` of `96`, `128`, `192`, …).
### Notes
- No new environment variables: the behaviour is driven by the existing
`BACKCHANNEL_PAUSE_MIN_MS` / `BACKCHANNEL_PAUSE_MAX_MS` window.
- Because the VAD advances in 32 ms windows, the emitted `pause_ms` is a multiple
of 32 ms; the randomly drawn target is what varies.
---
## 2026-10-16 — Backchannel sounds more natural (micro-pause cues, jittered timing, categories)
### Added
- **`BACKCHANNEL_JITTER_MS`** (default `1200`), **`BACKCHANNEL_PAUSE_MIN_MS`**
(default `96`), **`BACKCHANNEL_PAUSE_MAX_MS`** (default `260`),
**`BACKCHANNEL_PROB`** (default `0.65`) and **`BACKCHANNEL_VOLUME_JITTER`**
(default `0.2`) — new `.env` knobs (`.env`, `.env.example`,
[`BackchannelSettings`](src/orchestrator/backchannel.rs)) that shape the cue
timing, spacing, hit rate and loudness drift.
- **Category prefix** in `BACKCHANNEL_TEXTS`: a line may be written
`category::phrase` (e.g. `ack::[calm] Mm-hmm...`, `agree::[agreeing] Claro...`);
[`parse_texts`](src/orchestrator/backchannel.rs) stores the category alongside
the phrase and defaults to `ack` when the prefix is absent.
- [`VadDetector::silence_ms`](src/vad/silero.rs) — exposes the running silence
inside the current utterance (multiples of the 32 ms analysis window), i.e. the
length of the micro-pause the speaker is currently in.
- [`FillerBank::names`](src/orchestrator/turn_manager.rs) — read accessor for the
clip names, used to map the bank order onto the phrase/category list.
### Changed
- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) now takes
the VAD micro-pause length and fires a cue **only inside a
`PAUSE_MIN_MS..PAUSE_MAX_MS` window** (the upper bound is capped just below
`VAD_MIN_SILENCE_MS` so it never collides with end-of-utterance), **at most once
per pause** (latched with `pause_settled`).
- Cue timing is **randomized**: the first cue of a turn waits
`MIN_SPEECH_MS + rand(0..JITTER_MS)`, and the gap to the next cue is
`INTERVAL_MS ± JITTER_MS/2` — instead of a fixed interval.
- Each qualifying pause passes a **probabilistic gate** (`PROB`); a miss does not
retry on the same pause.
- The clip is chosen with an **anti-repetition** rule (skips the last two clips)
and its samples are scaled by **±`VOLUME_JITTER`** before being pushed to
`backchannel_queue`.
- Selection follows a **category progression** through the turn — it prefers a
category other than the last one used and favours categories not yet used this
turn (e.g. `ack` then `agree`), falling back to a random clip.
- `BACKCHANNEL_MAX_PER_TURN` default lowered **8 → 3**, so the agent emits a
couple of cues per turn rather than chattering.
### Why
- A fixed interval with a fixed clip made the backchannel feel mechanical and
detached from the speaker. Real listeners react to *pauses*, not to a metronome:
cueing inside the speaker's own micro-pauses, with jittered spacing, an
occasional deliberate skip, mild volume drift and a little progression
(acknowledge → agree) reads as attentive listening instead of a loop.
### Notes
- The naturalness knobs are all optional; omitting them keeps the previous
behaviour except for the `MAX_PER_TURN` default (now `3`).
- Nested clamps in `from_env`: `interval_ms >= 500`, `pause_max >= pause_min`,
and `gain`/`prob`/`volume_jitter` are clamped to sane ranges.
- A debug line is logged only when a cue is actually emitted:
`backchannel emitted during user speech … pause_ms=…` (enable
`audio_agent_engine::orchestrator=debug` in `RUST_LOG`).
---
## 2026-10-16 — Backchannel becomes a random phrase bank driven by `BACKCHANNEL_TEXTS`
### Added
- **`BACKCHANNEL_TEXTS`** (`.env`, `.env.example`,
[`BackchannelSettings`](src/orchestrator/backchannel.rs)): the backchannel bank
is now a list of phrases, **one per line** (a `|` separator is also accepted),
with Drama-3 style tags passed straight through to the TTS. `.env` ships 8
Spanish phrases (`[calm] Mm-hmm...`, `[attentive] Ajá...`, …); the Rust default
is [`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs).
- [`assets::ensure_backchannel_clips`](src/assets.rs), implemented on top of the
new generic [`assets::ensure_text_bank_clips`](src/assets.rs): synthesizes **one
clip per phrase** at startup (`bc01`, `bc02`, …, `TEXT_BANK_CONCURRENCY = 4` at
a time) and removes clips left over from a previous list. The list the current
clips were made from is stored in `BACKCHANNEL_DIR/texts.txt`; when it is
unchanged and every clip is present the whole synthesis is **skipped**, so a
restart costs ~2 s instead of ~1 s per phrase. The index is only written once
every clip succeeded, so a partial failure is retried at the next startup.
### Changed
- [`BackchannelScheduler::maybe_emit`](src/orchestrator/backchannel.rs) picks the
clip **at random** ([`FillerBank::random`](src/orchestrator/turn_manager.rs),
xorshift64 seeded from the clock) instead of round-robin, so the acknowledgment
varies from turn to turn — the same behaviour the start filler already had. The
`next_index` cursor was dropped.
- `assets::ensure_start_filler_clips` is now a thin wrapper over
`ensure_text_bank_clips`, so the start filler and the backchannel share one
implementation. Its logs gained a `label` field
(`refreshing backchannel clips at startup …`).
- The backchannel no longer uses the fixed `assets::BACKCHANNEL_CLIPS` list
(removed); its defaults moved to `DEFAULT_BACKCHANNEL_TEXTS`.
### Why
- A fixed 4-clip bank repeated the same "mm-hmm / ajá" over and over; a curated
list with tags keeps the agent sounding natural and is editable straight from
`.env`, exactly like the start filler.
- The old bootstrap (`ensure_clips`) only generated *missing* clips, so changing
the phrases required deleting files by hand; the phrase list is now the source
of truth.
### Notes
- Clips are named by position, so switching from the old names (`mmhmm`, `aja`,
`claro`, `yaveo`) to `bc01`…**removes** the four legacy files on the first
restart after this change (logged as `removed stale backchannel clip`).
- `BACKCHANNEL_GAIN`, `BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS` and
`BACKCHANNEL_MAX_PER_TURN` are unchanged.
---
## 2026-10-16 — Fix: start filler fired on every turn (gate on chunk arrival, not playback)
### Fixed
- [`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs) gated the
cue on `AudioMixer::tts_playback_started`, which only flips once the **TTS
pre-buffer** (`TTS_PREBUFFER_MS`) is full — i.e. when playback starts, ~1.9 s
after the dispatch in the reference deployment. The first response chunk
actually arrives at ~0.7–1.0 s, so a `START_FILLER_GO_DELAY` of 1200 ms was
almost always *shorter* than the gate's own signal and the acknowledgment was
queued on essentially every turn.
- The gate now uses the new
[`AudioMixer::tts_audio_arrived`](src/audio/mixer.rs), latched as soon as the
first TTS samples are queued for the turn (checked on the 20 ms ticker),
**before** the pre-buffer is reached. Both `tts_audio_arrived` and
`tts_playback_started` are reset by `mark_stream_start` / `flush`.
A fast reply that produces audio within `START_FILLER_GO_DELAY` therefore
suppresses the cue, as intended.
### Changed
- The skip log line is now `first response audio already arrived — start filler
skipped` (was `response audio already playing — start filler skipped`).
- The end-of-turn filler is unchanged: it still keys off
`tts_playback_started` (its 2500 ms delay is long enough that playback start is
the right signal).
### Notes
- No configuration change; `START_FILLER_GO_DELAY=1200` now means what the `.env`
comment always said ("first audio chunk of the reply"). Verified in the log: a
text turn in which the first chunk reached the mixer ~0.4 s after the TTS
request logged `first response audio already arrived — start filler skipped`
with no `queued start filler` for the turn.
---
## 2026-10-16 — Start filler becomes conditional: `START_FILLER_GO_DELAY` (1200 ms)
### Changed
- The start filler is no longer played on **every** turn: it is now a *latency
mask*, scheduled exactly like the end-of-turn filler but on its own channel and
with its own, shorter delay.
[`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)
(replacing `play_start_filler`) sleeps `START_FILLER_GO_DELAY` after the turn is
dispatched and queues a random phrase **only if the first response audio has not
started playing** (`AudioMixer::tts_playback_started` still `false`); otherwise
the cue is dropped. *(The gate was later changed to `tts_audio_arrived` — see
the entry above, which fixes this cue firing on every turn.)* A pending cue is
cancelled on barge-in / new turn through the turn's `CancellationToken`, and
`mixer.flush` clears it once queued.
- The scheduling now happens in `run_llm_turn` / `dispatch_tts` next to
`schedule_filler` (where the turn's cancellation token and `mark_stream_start`
live) instead of at the top of `handle_user_turn`, because the delay must be
measured from the dispatch of the reply and be cancellable with it.
### Added
- **`START_FILLER_GO_DELAY`** (`.env`, `.env.example`, default `1200` ms): how long
to wait for the first response audio before falling back to the start filler.
### Why
- The cue is only useful when the reply is *late*; a fast reply (LLM + TTS first
chunk under 1200 ms) is now answered directly, without an unnecessary
"Un momento…" in front of it.
### Notes
- The 22-clip bank, the random pick and the startup refresh are unchanged.
---
## 2026-10-16 — Start filler: random phrase bank driven by `START_FILLER_TEXTS`
### Added
- **`START_FILLER_TEXTS`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)) replaces the single `START_FILLER_TEXT`: a
list of phrases, **one per line** (a `|` separator is also accepted), e.g. the
Spanish set `"[calm] Un momento..."`, `"[thoughtful] A ver, veamos..."`, … with
Drama-3 style tags passed straight through to the TTS. `.env` ships 22 phrases.
- [`assets::ensure_start_filler_clips`](src/assets.rs): synthesizes **one clip per
phrase** at startup (`sf01`, `sf02`, …, generated
`START_FILLER_CONCURRENCY = 4` at a time) and removes clips left over from a
previous list. The list the current clips were made from is stored in
`START_FILLER_DIR/texts.txt`; when it is unchanged and every clip is present the
whole synthesis is **skipped**, so a restart costs ~2 s instead of ~1 s per
phrase. The index is only written once every clip succeeded, so a partial
failure is retried at the next startup.
- [`FillerBank::random`](src/orchestrator/turn_manager.rs): uniform pick over the
bank using a per-bank xorshift64 state seeded from the clock;
[`TurnManager::play_start_filler`](src/orchestrator/turn_manager.rs) now uses it,
so the acknowledgment differs from turn to turn instead of always repeating the
same sentence. *(Caller renamed to `schedule_start_filler` by the entry above,
which also made the cue conditional.)*
### Why
- A single fixed acknowledgment ("Un momento…") becomes annoying across a long
conversation; a random phrase from a curated list keeps the agent sounding
natural, and the phrase list is editable straight from `.env`.
- Re-synthesizing 22 clips on *every* startup added ~21 s to the boot time
(measured), so the bootstrap now tracks the phrase list and only re-synthesizes
when it changed (or a clip is missing).
### Notes
- `force` was dropped from [`assets::ensure_clips`](src/assets.rs): the filler /
backchannel banks keep the "generate only if missing" behaviour, and the start
filler has its own list-driven bootstrap.
- [`server.sh`](server.sh) startup wait raised from ~20 s to ~60 s: the first
startup after a list change synthesizes the whole bank before the socket is
bound.
- A failed synthesis keeps the previous clip of that position; a barge-in still
clears the start-filler channel (`AudioMixer::flush`), and
`START_FILLER_ENABLED=false` disables it entirely.
---
## 2026-10-16 — Start filler: immediate turn-start acknowledgment on its own mixer channel
### Added
- **`START_FILLER_ENABLED` / `START_FILLER_TEXT` / `START_FILLER_DIR`** (`.env`,
`.env.example`, [`src/config.rs`](src/config.rs)). The *start filler* is a
pre-generated clip queued the moment a user turn begins — no delay — on a
**dedicated mixer channel**, so it acknowledges the turn while the LLM/TTS are
still working, independently of the delayed latency filler. *(Superseded by the
entry above: `START_FILLER_TEXT` became the `START_FILLER_TEXTS` list and the
clip is now picked at random.)*
- [`AudioMixer`](src/audio/mixer.rs): new `start_filler_queue` channel, mixed as
`mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel ×
BACKCHANNEL_GAIN)`, so the start filler shares the filler gain (idle `1.0`,
ducked `0.15` under TTS). Included in `is_active()` and cleared by `flush()`.
- [`TurnManager::play_start_filler`](src/orchestrator/turn_manager.rs), called
from `handle_user_turn` right after the in-flight reply is cancelled.
- Startup bootstrap synthesizes `START_FILLER_DIR/start_filler.raw` with the
configured voice if it is missing ([`src/assets.rs`](src/assets.rs)); the clip
is preloaded into its own [`FillerBank`](src/orchestrator/turn_manager.rs),
kept out of the round-robin `FILLERS_DIR` bank.
### Why
- The delayed latency filler only fires after `FILLER_DELAY_MS` (and only if the
reply is still silent), so with the current fast TTFB (~`0.73 s`) the agent can
stay silent for a moment after the user stops speaking. The start filler
provides an immediate acknowledgment instead, without touching the existing
delayed filler.
### Notes
- *(Superseded — see the entry above.)* The clip was re-synthesized on every
startup (`ensure_clips(..., force = true)`), so editing `START_FILLER_TEXT` took
effect at the next restart with no manual deletion. A new clip was written only
on success, so a failed synthesis left the previous clip in place; the filler /
backchannel banks kept the "generate only if missing" behaviour.
- A barge-in clears the start filler along with every other channel
(`AudioMixer::flush`); `START_FILLER_ENABLED=false` disables it entirely.
---
## 2026-10-16 — Fixed micro-gaps at the start of each sentence → `TTS_PREBUFFER_MS` 170 → 550 ms, `LLM_STREAM_CHUNK_CHARS` 120 → 240
### Changed
- **`TTS_PREBUFFER_MS` raised `170` → `550`** (`.env`, `.env.example`). With the
current vocoder windows (first `185.8` ms, steady `371.6` ms) any pre-buffer in
`(186, 557]` ms primes playback on the *same* second window, so `550 ms` takes
the entire ~557 ms cushion **at no extra TTFA** (~`0.73 s`).
- **`LLM_STREAM_CHUNK_CHARS` raised `120` → `240`** (`.env`, `.env.example`). The
first-chunk threshold stays at `20` (early flush preserved), but every following
chunk is now twice as long → **fewer TTS requests, hence fewer chunk seams**.
- **README** (`Mixer & jitter buffering`, the `TTS_PREBUFFER_MS` /
`LLM_STREAM_CHUNK_CHARS` tables and the TTFA note) updated to the new defaults
and rationale. No Rust code changed — both values are read from `.env` at
runtime.
### Why
- After the seam-quality retune left the **audible micro-interruptions at the
start of sentences**. Diagnosis from `run/server.log`: the defect is a **TTS
queue underrun at the chunk seams**, not a timbre artifact. Each sentence /
fragment is a *separate* HTTP request whose PCM is concatenated with no
crossfade; the relay advances chunk-by-chunk, so the first dip in production
empties the queue and the mixer logs
`TTS queue underrun — pausing playback until audio resumes` (94 occurrences in
the sampled log; 0 `running low` warnings), then restarts with `need = 1` → an
audible micro-gap at the start of the next sentence.
- **Why the queue empties:** with `TTS_PIPELINE_DEPTH = 3` on a single GPU the
measured per-window cadence under concurrency is **552–613 ms (~1.5×)** versus
**~350 ms (~0.94×)** for a lone request. A `170 ms` cushion has no margin
against that; `550 ms` does, and it is latency-free because the pre-buffer
already lands on the second window.
- Measured in the log at the seam (12:56:36–38): chunks of `16384` samples
(`queue_len=16384`, `prebuffer=7497` = 170 ms) with two concurrent requests
(`text_len=119` and `19`), each followed by an underrun-pause line.
### Notes
- Both levers are pure configuration; the binary is unchanged, so `./server.sh
restart` (no rebuild) is enough to apply them.
- The `550 ms` cushion is the ~557 ms gap to the third window start, so raising
it further only adds latency (it would wait for the *next* window).
---
## 2026-10-16 — Drama-3 vocoder retuned for seam quality → stride `4` / followup `8` / overlap `16`; `TTS_PREBUFFER_MS` 450 → 170 ms
### Changed
- **The Drama-3 launcher (`/root/drama3/start-drama3.sh`, pm2 `start-drama3`) now
carries more decoder context per window.** The previous config (stride `4`,
flat ramp `"[4,4,4]"`, overlap `8`, crossfade `256`) was replaced by:
```
--vocoder.factory.stream_stride 4
--vocoder.factory.stream_followup_stride_ramp "[8,8,8]"
--vocoder.factory.stream_followup_stride 8
--vocoder.factory.stream_overlap_tokens 16
--vocoder.factory.stream_crossfade_samples 512
--tts_engine.factory.enable_async_decode true
```
- **`TTS_PREBUFFER_MS` lowered `450` → `170`** (`.env`, `.env.example`,
[`src/config.rs`](src/config.rs)). With stride `4` the *first* vocoder window is
4 codec frames = **0.186 s**, so `170 ms` starts playback on that first window
and **TTFA collapses to the server's own TTFB (~0.38 s)**.
- **README:** the *"Starting the Drama 3 backend (SGLang-Omni)"* subsection and
the *"Mixer & jitter buffering"* pre-buffer note were rewritten around the new
stride/overlap values and the ~0.186 s first / ~0.372 s steady window cadence.
### Why
- The previous latency-first config (stride `4`, overlap `8`) produced **audible
micro-gaps and a slight detune / falsetto on long replies**. Root cause: with
only `8` tokens of decoder warm-up context *per window*, and one TTS request
per reply chunk (PCM concatenated without crossfade at the request seams), the
streaming decoder's state was too shallow and its timbre/pitch wobbled at each
window and request boundary.
- **Seam context restored without breaking real-time:** raising the overlap to
`16` at stride `4` pushes decode to 5× real time, which the box **cannot**
sustain (ratio rose to 1.18–1.98×). Raising *strides and overlap together* —
stride `4` (first window) / followup `8` (steady) with overlap `16` — keeps the
decode load at a sustainable **`(16+8)/8 = 3×`** (~0.95× ratio) while giving
each steady window twice the context, and `crossfade 512` smooths the seams
further.
- A/B measurements on the live endpoint (`measure_tts.py`, warm server, voice
cloning, single request vs sentence-chunked like the engine) on a long
multi-sentence reply: **the amplitude jump at the seam boundaries dropped from
the 99.5+ percentile of the signal to 96–98**, i.e. the seams no longer stand
out. TTFB stays **~0.38 s** (the first window is unchanged) and the real-time
ratio stays **~0.94–0.98×**.
### Notes
- The first window is unchanged (4 frames), so the residual latency is still
dominated by the server's **fixed ~0.38 s TTFB**; the steady window grew from
0.186 s to **0.372 s** (8 frames) and arrives ~350 ms apart, so the pre-buffer
of `170 ms` still starts playback on the very first window. Further latency
gains would need a faster first token (model / preprocessing), not a smaller
stride.
- `measure_tts.py` was added to the repo: it streams `/v1/audio/speech` and prints
the per-window arrival schedule, TTFB and real-time ratio (plus a seam-jump
summary), so future A/B runs are one command.
---
## 2026-10-16 — Mivocero: karaoke word highlighting on the assistant reply
### Changed