Files

T

2026-10-17 22:01:31 +01:00

106 KiB

Raw Blame History

Audio Agent Engine — Updates

This file documents all notable changes to the audio-agent-engine project. Entries are written in English and ordered newest-first.

2026-10-17 — Feat: custom voice accepts an uploaded audio file + typed transcript

Added

Upload source on the /customvoice page. A Source toggle now offers two ways to build the reference: Record now (the existing microphone flow, read a predefined phrase for ≥ 10 s) or Upload a file — pick any audio file ffmpeg can read (accept="audio/*": wav, mp3, m4a, ogg, flac…) and type (or paste) its transcript. In upload mode the transcript is required and gates the Use it button, since the typed text is what drives the clone. Both modes share the same Use it / Reset actions and preview player.
text query parameter on POST /api/customvoice. The endpoint is now POST /api/customvoice?lang=<code>[&text=<transcript>]. A non-blank text overrides the language's predefined reading phrase for the transcript written to the sibling .txt and passed to set_reference; when absent or blank the predefined phrase is used as before (so a plain recording is unchanged). Backend: SetParams gained text: Option<String> and set_custom_voice resolves the transcript from it (src/customvoice.rs).

Notes

No format constraints are imposed on the upload: the same ffmpeg probe used for recordings handles whatever container is sent. The 32 MB body cap already covers uncompressed WAVs.
README updated (feature bullet, usage comment, Custom voice builder flow and API/route tables, text parameter).

2026-10-17 — Fix: short utterances ("uno", "due") dropped by the STT gate

Fixed

The probability average now counts voiced windows only. The gate's avg_prob was the mean of the raw VAD probability over every window from speech start to segment close — including the VAD_MIN_SILENCE_MS (700 ms) hangover that ends the utterance. For a long phrase that silence is a small fraction, but for a short word it is most of the segment: "uno" (~350 ms of voice + ~700 ms of silence) averaged 0.065–0.199 and was discarded as too_soft, even though the user was plainly speaking. In the same session the long phrases passed at 0.32–0.60. The average is now taken only over windows whose probability is ≥ VAD_NEG_THRESHOLD (src/vad/silero.rs), so the voiced portion scores its true value and short real words clear the gate while noise still does not.

Notes

VAD_MIN_AVG_PROB stays at 0.25; its meaning is now stricter (it no longer benefits from the silence dilution), but genuine short words score well above it. README / .env / .env.example / src/vad/mod.rs docs aligned.
Tune with VAD_MIN_AVG_PROB: raise it if noisy segments become hallucinated turns, lower it if genuinely soft speech is still dropped.

2026-10-17 — Fix: too many utterances dropped by the STT gate

Fixed

VAD_MIN_AVG_PROB lowered 0.45 → 0.25. Log analysis showed 62 drops, all too_soft (none for duration), still occurring after a restart. Accepted utterances measured avg_prob 0.47–0.76 while dropped ones measured 0.08–0.45 — including plainly real speech cut by a hair (2136 ms @ 0.428, 3576 ms @ 0.436). The gate sat almost on top of the per-frame speech threshold (VAD_THRESHOLD=0.5): a real utterance hovers around the threshold on its onset/release, so its average lands just under 0.45 and the segment is discarded. The 0.25 default leaves a proper margin — genuine soft speech (0.3–0.45) now passes, while clearly noisy segments (below 0.25) are still filtered. Update VAD_MIN_AVG_PROB in .env (default, .env.example, src/vad/mod.rs and README aligned).

Notes

Tune with VAD_MIN_AVG_PROB: lower it further if genuine speech is still dropped, raise it if noisy segments are turning into hallucinated turns. 0.0 disables the probability gate.

2026-10-17 — Fix: custom voice still louder than the default voice

Fixed

Custom voice level pushed below the reference. Matching the shipped reference at the integrated level was not enough: a measured take lands at ≈-23.25 LUFS against solis.wav's ≈-23.44 LUFS, i.e. the same loudness, yet it still sounded louder. The difference is dynamic range and peaks — the user take measures ≈6.2 LRA / -2.2 dBTP versus 1.6 LRA / -7.8 dBTP for the mastered reference, so its peaks sit ~5.5 dB hotter and read as "louder". The normalization target is now -26 LUFS with a -5 dBTP ceiling, which drops a take ~3 dB and pulls its peaks down to ≈-5 dBTP so a custom voice sits just under the default instead of on top of it.
Defaults changed: CUSTOMVOICE_TARGET_LUFS -23 → -26, CUSTOMVOICE_TRUE_PEAK -1.5 → -5 (code, .env, .env.example, README).

Notes

Lower CUSTOMVOICE_TARGET_LUFS further for a quieter custom voice; the value is applied at record time, so a take must be re-recorded (or the endpoint re-posted) for a change to take effect.
The custom voice lives only in memory (TtsEngine::set_reference): a restart reverts the reference to the configured TTS_REFERENCE_AUDIO voice.

2026-10-17 — Fix: custom voice louder than the default voice

Fixed

Loudness target aligned with the shipped reference. The custom-voice chain normalized to -18 LUFS while voice_profiles/solis.wav measures ≈-23.4 LUFS, so a custom voice came out ~5.4 dB louder than the voice it replaced. The default is now -23 (CUSTOMVOICE_TARGET_LUFS), matching the reference — swapping voices no longer changes how loud the agent sounds.
Mono downmix moved inside the filter chain. -ac 1 used to be applied at the output stage, i.e. after loudnorm: a stereo upload was measured as stereo and then folded to mono, landing the written file ~3 dB below the target. aformat=channel_layouts=mono now runs first, so loudnorm measures the signal that is actually written.

Notes

Verified: solis.wav (stereo, -23.4 LUFS) and a quiet mono take (-48.6 LUFS) both render to exactly -23.00 LUFS, mono 44.1 kHz. cargo test → 25 passed.

2026-10-17 — Post-processing for the custom voice (level match + cleanup)

Added

Post-processing chain for a recorded custom voice (src/customvoice.rs), applied before the reference is installed: highpass=f=80 (rumble/DC), optional afftdn denoise + deesser, leading/trailing silence trim (silenceremove + areverse) and a two-pass EBU R128 loudnorm with linear=true.
New env knobs: CUSTOMVOICE_POSTPROCESS (default true), CUSTOMVOICE_DENOISE (default true), CUSTOMVOICE_TARGET_LUFS (default -18), CUSTOMVOICE_TRUE_PEAK (default -1.5).

Notes

The loudness stage measures the cleaned signal first, then applies the result as a single constant gain — a per-frame (single-pass) norm is what makes takes jump in volume, which is the problem this fixes. Verified: a take at -48.6 LUFS and one at -17.5 LUFS both render to ≈ -18 LUFS with TP at -1.5 dBTP.
Any stage failing (unreadable take, all-silence clip, filter quirk) degrades to the plain transcode with a warning — a take is never lost.
Error handling verified: garbage input → 502 with a clean message, unknown language → 400. cargo test → 25 passed.

2026-10-17 — Custom voice builder (`/customvoice`) + Mivocero side menu

Added

Custom-voice builder page at GET /customvoice (src/web/customvoice.html). It asks the user for a language, shows a predefined phrase to read (long enough for cloning, ~10 s), records it with MediaRecorder and installs the take as the active voice. The Stop button is disabled until 10 s have elapsed (timer + progress bar), the take auto-stops at 20 s, and it can be played back and discarded before use.
Backend endpoints (src/customvoice.rs): GET /api/customvoice/phrases (reading phrases per language + the configured default), POST /api/customvoice?lang=<code> (raw recording body), POST /api/customvoice/clear. The body cap of the upload route is raised to 32 MB.
Side menu on /mivocero (☰ drawer) linking to the chat and to the custom-voice page; the same drawer is on /customvoice.

Notes

Phrases live server-side, so the reference transcript always matches the text the user read. POST /api/customvoice transcodes the upload to mono 16-bit PCM WAV at OUTPUT_SAMPLE_RATE (44.1 kHz) with ffmpeg, writes it next to TTS_REFERENCE_AUDIO (the directory the Fish Audio backend can read) with a sibling <name>.txt, and installs it via TtsEngine::set_reference — live for new replies, no restart.
POST /api/customvoice/clear restores the reference configured at startup (TTS_REFERENCE_AUDIO) rather than dropping it entirely.
Verified: uploading an audio file returns {"ok":true,…} and produces a 44.1 kHz/mono pcm_s16le WAV with its .txt; clear logs the restore of solis.wav. Tests: cargo test → 25 passed.

2026-10-17 — Tool calling survives a mis-matched server-side tool parser

Fixed

Tool calls dropped on the streaming path are recovered. The production vLLM server runs --tool-call-parser hermes, which expects the <tool_call>{"name":…,"arguments":{…}}</tool_call> JSON shape; the Qwen3 model emits the XML shape instead, so the parser throws and the call is discarded. In streaming the turn then ended with an empty reply (finish_reason: "stop") and no tool_call ever fired — the assistant either stayed silent or narrated a call it never made.
Non-streaming fallback in the turn loop (src/orchestrator/turn_manager.rs): a round that comes back empty while tools are configured is re-asked once without streaming via the new LlmClient::chat_reply (src/llm.rs), which returns both the visible text and the parsed tool calls.
Structured calls are parsed too — tools::parse_openai_tool_calls (src/tools.rs) reads the standard message.tool_calls[] array, so the engine keeps working when the server parser does match (and returns finish_reason: "tool_calls"). The native <tool_call> text path is unchanged.

Notes

End-to-end verified: "Che tempo fa a New York adesso?" now emits tool_call {name: get_weather, arguments: {city: "New York"}} and answers with live wttr.in data. A tool-free turn is unchanged (no extra request).
The clean fix remains server-side — --tool-call-parser qwen3_coder matches the Qwen3 XML format. The fallback makes the engine correct regardless, at the cost of one extra request only on an otherwise-empty round.
Tests: cargo test → 25 passed (2 new in tools::tests).

2026-10-17 — Longer end-of-utterance window (`VAD_MIN_SILENCE_MS` 300 → 700 ms)

Changed

VAD_MIN_SILENCE_MS raised from 300 to 700 ms in .env / .env.example, and the fallback default in src/vad/mod.rs aligned to 700. This is the silence window the VAD waits for before closing an utterance (see SileroVad::step, silence_run_ms >= min_silence_ms).
README — updated the config table and the SpeechEnd trigger row.

Notes

With 300 ms a brief thinking pause (~0,3 s) closed the turn and sent a partial sentence to the STT/LLM. 700 ms tolerates a natural mid-sentence pause before the agent answers. The trade-off is a slightly later reply (the agent now waits ~0,7 s of silence instead of ~0,3 s), which the filler / start-filler cues already mask.
The backchannel micro-pause window is unaffected: its pause_max_ms (260) is still below the new cap (min_silence_ms - 32 = 668).

2026-10-17 — Mivocero: barge-in indicator + karaoke on the user bubbles

Added

Server {"type":"barge_in","source":"vad","prob":…} event in src/ws/session.rs — emitted when the server-side VAD onset interrupts the agent. The onset is detected in the session task, so the browser had no other way to know a barge-in had happened.
Barge-in indicator in src/web/mivocero.html — a red Barge-in pill in the header that flashes and bumps a session counter. Driven by the barge_in event (VAD onset) and by the local Stop button; the counter resets on each connect.

Changed

Karaoke on the user bubbles in src/web/mivocero.html — the user's own messages (STT transcript and typed text) are now rendered word by word like the assistant's, with colours tuned for the blue bubble. Because the STT returns the whole transcript only once the utterance has ended, the words are lit by a short left-to-right sweep (USER_REVEAL_MS_PER_WORD, ~90 ms/word) instead of following the audio playhead.
README — documented the barge_in event in both the server→client table and the audit out row, and updated the Mivocero section (karaoke, barge-in indicator).

2026-10-17 — VAD barge-in gated on audible output (no more silent cancelled replies)

Added

AudioMixer::is_audible() in src/audio/mixer.rs — reports whether the agent has audible output: the reply is playing out (tts_primed) or a latency-mask cue (filler / start filler) is queued. Unlike is_active() it is false while the reply is still being synthesised or sits in the TTS pre-buffer — nothing has reached the user's ears yet.
TurnManager::is_audible() in src/orchestrator/turn_manager.rs — forwards the mixer method for the barge-in gate.

Changed

VAD-driven barge-in gate in src/ws/session.rs — a SpeechOnset now interrupts only when TurnManager::is_audible() is true, instead of is_speaking() (is_active()). is_speaking() is retained for the backchannel gate, where cueing over any in-flight TTS would still be wrong.

Notes

Root cause: is_speaking() is already true as soon as the TTS request is in flight (tts_stream_active), so a VAD onset landing during the synthesis / pre-buffer window (~1.2–1.6 s) cancelled the reply before a single sample was played. On the analysed session (s-1792251858371-0001, ~210 s, 21 turns) three replies streamed their text to the client but produced zero playback, and three more were truncated after 0.1–0.7 s; several of the onsets were false triggers (the utterance was later dropped by the gate with avg_prob 0.28 / 0.21).
A reply cancelled this way is not orphaned: the next turn's on_eou already cancels the previous in-flight synthesis and restarts the stream, so the audio follows the newest turn instead of being lost.

2026-10-17 — Filler fade-out on reply start (no more overlap with the answer)

Changed

Mixer filler fade-out in src/audio/mixer.rs — when the reply's playback starts (TTS pre-buffer reached) while an end-of-turn filler or a start filler is still queued, the cue is now faded out over FILLER_FADEOUT_FRAMES (4 frames = 80 ms) and then cut, instead of being ducked to filler_gain_duck and left playing underneath the whole answer. The filler is only a latency mask, so it now hands off cleanly to the reply.
- New AudioMixer::filler_fade_frames counter, armed on the priming transition and reset by mark_stream_start / flush.
START_FILLER_GO_DELAY raised from 1200 to 1800 ms in .env / .env.example — the cue now only fires when the first audio chunk is genuinely late, instead of on the boundary where it used to overlap the beginning of the reply.

Notes

Root cause: with START_FILLER_GO_DELAY=1200 the first TTS audio arrived 0.75–1.4 s after the cue, while the sf03 clip is ~2.83 s long — so the cue played for its first second at full gain and its tail then overlapped the start of the answer. On the analysed session this happened on 5 of 6 start fillers (all sf03).

2026-10-17 — Utterance gate: filter spurious VAD turns before STT

Added

Utterance gate in src/ws/session.rs — a VAD segment is now dropped before transcription when it is too short or too "soft", so the STT can no longer turn a VAD false-trigger (breath, click, acoustic echo, noise) into a hallucinated text turn that would each otherwise cost a full LLM turn. Two independent thresholds, both env-driven:
- VAD_MIN_UTTERANCE_MS (VadSettings::min_utterance_ms, default 300, 0 disables) — drop segments shorter than this.
- VAD_MIN_AVG_PROB (VadSettings::min_avg_prob, default 0.45, 0.0 disables) — drop segments whose average speech probability is below this. A dropped segment is logged and recorded in the audit as utterance_dropped (duration_ms, avg_prob, too_short, too_soft).
Empty-transcript guard in src/orchestrator/turn_manager.rs::on_eou — a transcript that decodes to an empty string (or has no tokens at all) no longer opens an LLM turn.

Configuration

New .env keys next to the other VAD_* options:

VAD_MIN_UTTERANCE_MS=300
VAD_MIN_AVG_PROB=0.45

Notes

The gate runs only when VAD_SEGMENT_STT=true. Raise the probability to be stricter; lower the duration to keep very short backchannels ("sì", "ok").
Documentation updated: README (VAD configuration table, VAD wiring section, audit-event table) and project memory.

2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)

Added

cues WebSocket event — sent once at session start by src/ws/session.rs, it reports the effective policy of the three cues for the active language: the filler (delay_ms, gain_idle, gain_duck, clip count, bank dir), the start filler (go_delay_ms, clip count, bank dir) and the backchannel (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories, clip count). The backchannel pause_max_ms is capped to the VAD EOU window exactly like BackchannelScheduler::pause_max; the bank paths are the language subdirectories (assets/fillers/<lang>). Purely informational.
Cue methods panel in src/web/mivocero.html — a collapsible <details> section under the cue tiles with one card per cue: the method used (when it fires, how a clip is chosen) and the live parameters from the cues event, so the panel always matches the running configuration instead of hard-coded values.

Notes

The panel is display-only; it changes no behaviour. Values reflect the process configuration at session start (env-derived), so a .env change is visible after the next session.

2026-10-17 — Per-session audit log (structured JSONL)

Added

src/audit.rs — a best-effort per-session audit sink. Each WebSocket session is recorded to its own JSONL file (SESSION_AUDIT_DIR/<id>.jsonl, id s-<epoch_ms>-<seq>), one record per line: {"ts":…,"elapsed_ms":…,"event":…,"data":…}. Records are flushed as they are written, and any failure (directory/file creation, write) only disables the sink for that session — the session itself is never affected.
Event coverage (complete):
- session_start — full non-secret configuration snapshot plus the active runtime capabilities (STT / VAD / tokenizer / LLM + model);
- control — inbound user_text turns;
- out — every event sent to the client, tapped at the single sender point in src/ws/session.rs so nothing is missed (user, assistant_delta, assistant_end, tool_call, cue, error, audio_source state, …);
- interrupt — client barge-in (JSON or legacy text form);
- barge_in — VAD-driven barge-in (with the onset probability);
- vad_speech_start / vad_speech_end — utterance boundaries;
- media / reference / speak — external-audio control, voice-reference changes and direct TTS requests;
- language_switch — detected language change (from, to, generated) — src/orchestrator/turn_manager.rs;
- error — transport/receiver errors;
- session_end — reason, duration, session id, file and per-event counts.
Config — SESSION_AUDIT_ENABLED (default true) and SESSION_AUDIT_DIR (default logs/sessions); documented in .env, .env.example and the README. logs/ is git-ignored.

Changed

TurnManager::new takes an Arc<SessionAudit> (used for the language-switch record); every other event reaches the log through the sender tap.

Notes

The audit path is off the audio hot path (the sink only touches the outbound text events, not the 20 ms frames) and is disabled cleanly when the LLM/VAD/… are off. Intended for post-mortem analysis and for tuning the latency knobs.

2026-10-17 — Cue emission indicators in the Mivocero demo

Added

{"type":"cue"} WebSocket event — the server now tells the client each time a latency-mask / backchannel cue is actually emitted into the mixer:
- filler (end-of-turn) — src/orchestrator/turn_manager.rs (schedule_filler),
- start_filler (turn start) — src/orchestrator/turn_manager.rs (schedule_start_filler),
- backchannel (during user speech) — src/ws/session.rs (mic task). The payload is {"type":"cue","kind":…,"name":…} and, for backchannel, also "category":…. A cue that is skipped (audio already arrived / playing, or a barge-in) sends nothing.
BackchannelScheduler::maybe_emit now returns (name, category) instead of just the name, so the backchannel cue carries its category.
Mivocero cue indicators — src/web/mivocero.html gained a row of three tiles (Filler, Start filler, Backchannel) that flash (colour-coded dot + border glow) and bump a per-session counter on every cue event, with a detail line showing the backchannel category (or the clip name). Counters reset on each connect.

Notes

Only real emissions are reported, so the panel is a faithful view of what the user actually hears — useful for tuning FILLER_DELAY_MS / START_FILLER_GO_DELAY and the backchannel cadence.

2026-10-17 — Per-language subdirectory for every clip bank (base included)

Changed

src/main.rs — the startup bootstrap now generates and loads the three base banks (filler, start-filler, backchannel) from <dir>/<LANG_DEFAULT>/ via assets::lang_dir instead of the top-level directory. Every language — including the default one — now lives in its own subdirectory, so the filler, start-filler and backchannel share the exact same layout for every language and no language is special-cased to the top level.
src/assets.rs — lang_dir doc updated accordingly.

Notes

On the first start after this change the base clips are re-synthesized into assets/<bank>/<lang>/. To avoid the re-synthesis, move the existing top-level *.raw files (and texts.txt) into the matching language subdirectory by hand — this was done for the es deployment.
No other language's clips are touched: the per-language subdirectory remains what isolates each language's refresh (remove_stale_clips).

2026-10-17 — Multilingual clip banks (backchannel, filler & start filler)

Added

src/lang.rs — new multilingual module. It holds built-in phrase tables for backchannel (category + phrase), filler (interjections) and start filler (turn-start acknowledgments, with Drama-3 style tags) in Spanish, English, Italian, French, German and Portuguese, exposes accessors for them, and implements the auxiliary LLM calls: detect_language (returns only the ISO 639-1 code, or und) and generate_backchannel_texts / generate_filler_texts / generate_start_filler_texts (LLM fallback for languages absent from the tables). It also defines LangBanks (a language's loaded clip banks, including its start-filler bank) and LanguageState (the session's active language plus a generation counter, kept in a parking-lot RwLock).
On-demand per-language generation — when the user's language differs from the active one, that language's backchannel/filler/start-filler clips are synthesized into a per-language subdirectory (assets/backchannel/<lang>/, assets/fillers/<lang>/, assets/start_filler/<lang>/) and the session switches to them. Because the refresh pass only prunes clips inside the directory it is given, no other language's clips are ever removed; the base language keeps living in the top-level directories, so no migration is needed.
New env vars — LANG_DEFAULT (default es), LANG_DETECT_ENABLED (default true), LANG_MIN_CHARS (default 12), LANG_REDETECT_EVERY (default 5) and LANG_LLM_FALLBACK_ENABLED (default true).

Changed

src/llm.rs — new LlmClient::simple_chat for short, non-streaming auxiliary requests (explicit system + user message, no tools), used by the language detection and phrase-generation calls.
src/config.rs — five new LANG_* fields, parsed in from_env.
src/assets.rs — lang_dir helper plus ensure_lang_backchannel_clips / ensure_lang_filler_clips / ensure_lang_start_filler_clips, which scope the text-bank refresh to <base>/<lang>/ so each language is refreshed independently.
src/orchestrator/backchannel.rs — the clip/category mapping is extracted into map_clips, and BackchannelScheduler::retarget re-points a live scheduler to a new bank when the language changes. DEFAULT_CATEGORY is now pub(crate) so lang.rs can reuse it.
src/orchestrator/turn_manager.rs — TurnManager now holds an Arc<LanguageState> instead of a single filler bank; next_filler and the start filler read from the active language (schedule_start_filler picks from langs.current().start_filler). A new maybe_detect_language runs on each user turn (first turn + every LANG_REDETECT_EVERY) and spawns switch_language_if_needed, which detects, generates/loads the target language's banks (start filler optional: a missing table just leaves that cue silent) and swaps the active state — in the background, so the reply is never blocked.
src/ws/session.rs — builds the shared LanguageState; the mic task polls the language generation counter each frame and calls retarget on the backchannel scheduler only when it changes.
.env / .env.example — new "Multilingual clip banks" section; README.md — new "Multilingual clip banks" section.

Why

The banks were pre-synthesized in a single language, so a user speaking any other language heard Spanish cues — including the start filler. Generating the matching language on demand keeps the acknowledgment natural in a multilingual deployment, while the per-language directory layout guarantees that adding a language never destroys the clips already generated for the others.

Notes

Detection needs the LLM (LANG_DETECT_ENABLED is a no-op without it); the LLM phrase fallback is used only for languages outside the built-in tables.
Generating a new language's bank calls the TTS backend once per phrase, which can take a few seconds on the first switch; it is idempotent (an already-generated language is only reloaded).

2026-10-16 — Mivocero: auto-disconnect an idle session after 2 minutes

Added

Idle guard in the Mivocero client (src/web/mivocero.html) — the session closes itself after IDLE_TIMEOUT_MS (2 minutes) without any activity. touchActivity() re-arms the timer and is called on every server event, on every received audio frame and on every user action sent (sendJson), so an ongoing reply or a playing external source keeps the session alive. On expiry it logs the reason, appends an explanatory message to the chat and calls disconnect(), which closes the WebSocket and releases the microphone.

Changed

Reconnect after close — ws.onclose now clears the ws reference. Without it the Connect button stayed bound to the stale socket, so a session closed by the idle guard (or by any disconnect) could not be reopened without reloading the page.

Why

A microphone-backed session left open keeps the device busy and the server streaming silent 20 ms frames forever. An inactivity timeout releases both without user action, while still allowing a long reply or a background radio to run uninterrupted.

Notes

The timer counts any traffic in either direction: TTS audio is sent as binary frames, so speaking does not trip the timeout.
The timeout is a client-side constant (IDLE_TIMEOUT_MS in the HTML); change it there and rebuild (the page is embedded with include_str!).

2026-10-16 — Tool calling: configurable HTTP tools for the LLM

Added

src/tools.rs — new tool-calling module. It loads tool definitions from a JSON file and provides: a ToolRegistry (builds the OpenAI tools array, executes a call over HTTP), a client-side parser for the model's native <tool_call> format, and a ToolCallStripper that removes tool-call blocks from the streamed reply — holding back a block that is still incomplete, so it is never spoken by the TTS nor shown in the chat box even when split across deltas (same hold-back pattern as the chain-of-thought stripper).
tools.example.json — ready-to-use definitions: get_weather (GET with a {city} URL placeholder, wttr.in) and send_webhook (POST with a custom header), covering both request shapes.
New env vars — LLM_TOOLS_ENABLED (auto: on when LLM_TOOLS_FILE is set), LLM_TOOLS_FILE, LLM_TOOL_MAX_ITERATIONS (default 4), LLM_TOOL_TIMEOUT_SECS (default 15) and LLM_TOOL_RESULT_MAX_CHARS (default 2000).
ChatMessage::tool — role: "tool" message used to hand a tool result back to the model.
WebSocket protocol — informational {"type":"tool_call","name":"…","arguments":{…}} event emitted when the model calls a tool.

Changed

LlmClient carries an optional Arc<ToolRegistry> (src/llm.rs); when present, every request advertises tools + tool_choice:"auto", so the model knows which functions it may call. The registry is built by load_tools(config); a missing or invalid file logs a warning and leaves tool calling disabled without breaking the pipeline.
TurnManager::run_llm_turn now runs a bounded tool loop: each round streams a reply through the ToolCallStripper; if a call was emitted, the assistant message (with its native call) plus one tool message per result are appended and the model is re-prompted, up to LLM_TOOL_MAX_ITERATIONS. The TTS pipeline is created once and every round's visible text flows into it, so the assistant can say something short before calling. Only the spoken text is written to the rolling history.

Why

The deployment's vLLM server runs without a tool parser (--enable-auto-tool-choice --tool-call-parser …), so it never returns structured tool_calls / finish_reason:"tool_calls". Tools are still advertised in the request and the Qwen model emits the call as text in its native format; parsing it client-side keeps the feature working with any server and requires no server-side reconfiguration.
Tool definitions live in a JSON file rather than .env so the verbose JSON schemas stay readable and editable.

Notes

Tool errors (unknown tool, transport failure, non-2xx status) are returned to the model as text so it can recover; a tool never fails the turn.
GET sends arguments not consumed by a {arg} placeholder as query parameters; POST sends them as a JSON body. headers are sent verbatim.

2026-10-16 — External audio channel: inject a URL / radio / YouTube source

Added

src/media.rs — new MediaSource module implementing an external audio source for a WebSocket session. A source is decoded by an external process and its raw PCM is pushed into the mixer's new external ring buffer. ffmpeg decodes direct URLs / radio / HLS (-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1, with -reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5); YouTube links are first resolved to a direct URL with yt-dlp (-f bestaudio/best -g --no-playlist). Child processes are killed on stop / replacement / disconnection (kill_on_drop(true) + CancellationToken), and a monotonic generation counter prevents a stale task from clearing a newer source.
Mixer external channel — AudioMixer gains external_queue and a runtime gain (src/audio/mixer.rs). The mixing formula becomes … + (external × external_gain); set_external_gain updates the gain (an atomic, so it takes effect on the next frame without rebuilding the mixer). The external queue is not cleared by flush() and does not count towards is_active(): a barge-in stops the agent, not the background audio.
New env vars — EXTERNAL_AUDIO_GAIN (default 0.6), MEDIA_FFMPEG_BIN (default ffmpeg), MEDIA_YTDLP_BIN (default yt-dlp) and MEDIA_YTDLP_ARGS (extra yt-dlp args, e.g. --cookies …).
WebSocket protocol — new audio_source message: {"type":"audio_source","action":"play","url":"…"}, {"type":"audio_source","action":"stop"} and {"type":"audio_source","action":"volume","gain":0.6}. The server reports state back with {"type":"audio_source","state":"loading|playing|stopped|error","url":"…"} (src/ws/session.rs).
Mivocero client — footer bar with a URL input, Play / Stop buttons, a volume slider (0–150 %) and a live state label, driving the external channel (src/web/mivocero.html).

Changed

TurnManager::set_external_gain forwards the client volume to the mixer.
New module declared in src/main.rs (mod media;).

Why

Lets the agent play a music/radio/YouTube bed under the voice — useful for demos (radio, background music) — while keeping the channel fully independent of the speech pipeline so it survives barge-in.

Notes

Only needed when the external channel is used: ffmpeg for URL/radio/HLS and yt-dlp for YouTube. YouTube may require cookies in some environments; pass them via MEDIA_YTDLP_ARGS=--cookies /path/cookies.txt.
A bounded source (VOD) is throttled to the mixer's drain rate via backpressure (remaining_capacity() + 20 ms sleeps), so a fast decoder cannot overflow the ring buffer.

2026-10-16 — Drama 3 de-emphasis: sampling knobs + neutral style tag

Added

TTS_TEMPERATURE (default 0.45), TTS_TOP_P (default 0.75) and TTS_TOP_K (unset) — optional sampling overrides forwarded in the /v1/audio/speech payload by the fish engine. An empty value omits the field so the server default applies. New env_opt_f32 / env_opt_u32 helpers in src/config.rs implement the "unset → fallback, empty → None" semantics.
TTS_STYLE_TAG (default [conversational, casual]) — a neutral style tag prepended to every synthesis request whose text does not already start with a [...] tag (FishAudioTts::styled). Empty disables the prefix.

Changed

SpeechRequest gains optional temperature, top_p and top_k fields (serialized only when set); the request body now uses the styled text as input. Startup logs TTS de-emphasis settings …, and the per-request debug log shows the sampling values and whether the text was prefixed (styled=).

Why

Drama 3 over-articulates: LLM replies could come back with a theatrical, radio-drama cadence. Exaggerated intonations live in the tails of the acoustic token distribution, so trimming the sampling tails (temperature / top_p) flattens the delivery, and a neutral style tag steers untagged (LLM) prose into a conversational register. Clip banks carry their own tags and are untouched.

Notes

repetition_penalty is deliberately not exposed: this server rejects any value other than 1.0 (Drama 3 request: repetition_penalty must be 1.0). min_p, seed and a nested sampling_params are rejected too.
The voice-cloning reference (solis.wav) is left unchanged for now — its "locutor profesional" prosody is a further, separate lever.
.env gotcha surfaced during this change. dotenvy fails on unquoted values containing spaces, and a single bad line makes it discard the whole file: the process then silently starts with built-in defaults (no LLM, plain HTTP) while server.sh still reports the scheme from its own .env grep. The style tag is therefore quoted (TTS_STYLE_TAG="[conversational, casual]").

2026-10-16 — Backchannel: random reaction depth inside the micro-pause

Changed

BackchannelScheduler::maybe_emit now draws the depth at which it reacts per pause: the first time a micro-pause is seen, a target is picked uniformly in [BACKCHANNEL_PAUSE_MIN_MS, BACKCHANNEL_PAUSE_MAX_MS] and the cue fires only once silence_ms() reaches it (new pause_target_ms field, cleared when the pause ends or the turn resets). Previously the cue fired at the first analysis window inside the window, so with a 32 ms VAD step every cue landed at exactly BACKCHANNEL_PAUSE_MIN_MS.
A pause that grows past the upper bound (on its way to end-of-utterance) is abandoned without firing.

Why

Verification on a live session showed every cue at pause_ms=96, i.e. all cues reacting at exactly the minimum — a residual regularity in an otherwise naturalized feature. Randomizing the depth removes it: cues now land anywhere in the window (observed pause_ms of 96, 128, 192, …).

Notes

No new environment variables: the behaviour is driven by the existing BACKCHANNEL_PAUSE_MIN_MS / BACKCHANNEL_PAUSE_MAX_MS window.
Because the VAD advances in 32 ms windows, the emitted pause_ms is a multiple of 32 ms; the randomly drawn target is what varies.

2026-10-16 — Backchannel sounds more natural (micro-pause cues, jittered timing, categories)

Added

BACKCHANNEL_JITTER_MS (default 1200), BACKCHANNEL_PAUSE_MIN_MS (default 96), BACKCHANNEL_PAUSE_MAX_MS (default 260), BACKCHANNEL_PROB (default 0.65) and BACKCHANNEL_VOLUME_JITTER (default 0.2) — new .env knobs (.env, .env.example, BackchannelSettings) that shape the cue timing, spacing, hit rate and loudness drift.
Category prefix in BACKCHANNEL_TEXTS: a line may be written category::phrase (e.g. ack::[calm] Mm-hmm..., agree::[agreeing] Claro...); parse_texts stores the category alongside the phrase and defaults to ack when the prefix is absent.
VadDetector::silence_ms — exposes the running silence inside the current utterance (multiples of the 32 ms analysis window), i.e. the length of the micro-pause the speaker is currently in.
FillerBank::names — read accessor for the clip names, used to map the bank order onto the phrase/category list.

Changed

BackchannelScheduler::maybe_emit now takes the VAD micro-pause length and fires a cue only inside a PAUSE_MIN_MS..PAUSE_MAX_MS window (the upper bound is capped just below VAD_MIN_SILENCE_MS so it never collides with end-of-utterance), at most once per pause (latched with pause_settled).
Cue timing is randomized: the first cue of a turn waits MIN_SPEECH_MS + rand(0..JITTER_MS), and the gap to the next cue is INTERVAL_MS ± JITTER_MS/2 — instead of a fixed interval.
Each qualifying pause passes a probabilistic gate (PROB); a miss does not retry on the same pause.
The clip is chosen with an anti-repetition rule (skips the last two clips) and its samples are scaled by ±VOLUME_JITTER before being pushed to backchannel_queue.
Selection follows a category progression through the turn — it prefers a category other than the last one used and favours categories not yet used this turn (e.g. ack then agree), falling back to a random clip.
BACKCHANNEL_MAX_PER_TURN default lowered 8 → 3, so the agent emits a couple of cues per turn rather than chattering.

Why

A fixed interval with a fixed clip made the backchannel feel mechanical and detached from the speaker. Real listeners react to pauses, not to a metronome: cueing inside the speaker's own micro-pauses, with jittered spacing, an occasional deliberate skip, mild volume drift and a little progression (acknowledge → agree) reads as attentive listening instead of a loop.

Notes

The naturalness knobs are all optional; omitting them keeps the previous behaviour except for the MAX_PER_TURN default (now 3).
Nested clamps in from_env: interval_ms >= 500, pause_max >= pause_min, and gain/prob/volume_jitter are clamped to sane ranges.
A debug line is logged only when a cue is actually emitted: backchannel emitted during user speech … pause_ms=… (enable audio_agent_engine::orchestrator=debug in RUST_LOG).

2026-10-16 — Backchannel becomes a random phrase bank driven by `BACKCHANNEL_TEXTS`

Added

BACKCHANNEL_TEXTS (.env, .env.example, BackchannelSettings): the backchannel bank is now a list of phrases, one per line (a | separator is also accepted), with Drama-3 style tags passed straight through to the TTS. .env ships 8 Spanish phrases ([calm] Mm-hmm..., [attentive] Ajá..., …); the Rust default is DEFAULT_BACKCHANNEL_TEXTS.
assets::ensure_backchannel_clips, implemented on top of the new generic assets::ensure_text_bank_clips: synthesizes one clip per phrase at startup (bc01, bc02, …, TEXT_BANK_CONCURRENCY = 4 at a time) and removes clips left over from a previous list. The list the current clips were made from is stored in BACKCHANNEL_DIR/texts.txt; when it is unchanged and every clip is present the whole synthesis is skipped, so a restart costs ~2 s instead of ~1 s per phrase. The index is only written once every clip succeeded, so a partial failure is retried at the next startup.

Changed

BackchannelScheduler::maybe_emit picks the clip at random (FillerBank::random, xorshift64 seeded from the clock) instead of round-robin, so the acknowledgment varies from turn to turn — the same behaviour the start filler already had. The next_index cursor was dropped.
assets::ensure_start_filler_clips is now a thin wrapper over ensure_text_bank_clips, so the start filler and the backchannel share one implementation. Its logs gained a label field (refreshing backchannel clips at startup …).
The backchannel no longer uses the fixed assets::BACKCHANNEL_CLIPS list (removed); its defaults moved to DEFAULT_BACKCHANNEL_TEXTS.

Why

A fixed 4-clip bank repeated the same "mm-hmm / ajá" over and over; a curated list with tags keeps the agent sounding natural and is editable straight from .env, exactly like the start filler.
The old bootstrap (ensure_clips) only generated missing clips, so changing the phrases required deleting files by hand; the phrase list is now the source of truth.

Notes

Clips are named by position, so switching from the old names (mmhmm, aja, claro, yaveo) to bc01…removes the four legacy files on the first restart after this change (logged as removed stale backchannel clip).
BACKCHANNEL_GAIN, BACKCHANNEL_MIN_SPEECH_MS, BACKCHANNEL_INTERVAL_MS and BACKCHANNEL_MAX_PER_TURN are unchanged.

2026-10-16 — Fix: start filler fired on every turn (gate on chunk arrival, not playback)

Fixed

TurnManager::schedule_start_filler gated the cue on AudioMixer::tts_playback_started, which only flips once the TTS pre-buffer (TTS_PREBUFFER_MS) is full — i.e. when playback starts, ~1.9 s after the dispatch in the reference deployment. The first response chunk actually arrives at ~0.7–1.0 s, so a START_FILLER_GO_DELAY of 1200 ms was almost always shorter than the gate's own signal and the acknowledgment was queued on essentially every turn.
The gate now uses the new AudioMixer::tts_audio_arrived, latched as soon as the first TTS samples are queued for the turn (checked on the 20 ms ticker), before the pre-buffer is reached. Both tts_audio_arrived and tts_playback_started are reset by mark_stream_start / flush. A fast reply that produces audio within START_FILLER_GO_DELAY therefore suppresses the cue, as intended.

Changed

The skip log line is now first response audio already arrived — start filler skipped (was response audio already playing — start filler skipped).
The end-of-turn filler is unchanged: it still keys off tts_playback_started (its 2500 ms delay is long enough that playback start is the right signal).

Notes

No configuration change; START_FILLER_GO_DELAY=1200 now means what the .env comment always said ("first audio chunk of the reply"). Verified in the log: a text turn in which the first chunk reached the mixer ~0.4 s after the TTS request logged first response audio already arrived — start filler skipped with no queued start filler for the turn.

2026-10-16 — Start filler becomes conditional: `START_FILLER_GO_DELAY` (1200 ms)

Changed

The start filler is no longer played on every turn: it is now a latency mask, scheduled exactly like the end-of-turn filler but on its own channel and with its own, shorter delay. TurnManager::schedule_start_filler (replacing play_start_filler) sleeps START_FILLER_GO_DELAY after the turn is dispatched and queues a random phrase only if the first response audio has not started playing (AudioMixer::tts_playback_started still false); otherwise the cue is dropped. (The gate was later changed to tts_audio_arrived — see the entry above, which fixes this cue firing on every turn.) A pending cue is cancelled on barge-in / new turn through the turn's CancellationToken, and mixer.flush clears it once queued.
The scheduling now happens in run_llm_turn / dispatch_tts next to schedule_filler (where the turn's cancellation token and mark_stream_start live) instead of at the top of handle_user_turn, because the delay must be measured from the dispatch of the reply and be cancellable with it.

Added

START_FILLER_GO_DELAY (.env, .env.example, default 1200 ms): how long to wait for the first response audio before falling back to the start filler.

Why

The cue is only useful when the reply is late; a fast reply (LLM + TTS first chunk under 1200 ms) is now answered directly, without an unnecessary "Un momento…" in front of it.

Notes

The 22-clip bank, the random pick and the startup refresh are unchanged.

2026-10-16 — Start filler: random phrase bank driven by `START_FILLER_TEXTS`

Added

START_FILLER_TEXTS (.env, .env.example, src/config.rs) replaces the single START_FILLER_TEXT: a list of phrases, one per line (a | separator is also accepted), e.g. the Spanish set "[calm] Un momento...", "[thoughtful] A ver, veamos...", … with Drama-3 style tags passed straight through to the TTS. .env ships 22 phrases.
assets::ensure_start_filler_clips: synthesizes one clip per phrase at startup (sf01, sf02, …, generated START_FILLER_CONCURRENCY = 4 at a time) and removes clips left over from a previous list. The list the current clips were made from is stored in START_FILLER_DIR/texts.txt; when it is unchanged and every clip is present the whole synthesis is skipped, so a restart costs ~2 s instead of ~1 s per phrase. The index is only written once every clip succeeded, so a partial failure is retried at the next startup.
FillerBank::random: uniform pick over the bank using a per-bank xorshift64 state seeded from the clock; TurnManager::play_start_filler now uses it, so the acknowledgment differs from turn to turn instead of always repeating the same sentence. (Caller renamed to schedule_start_filler by the entry above, which also made the cue conditional.)

Why

A single fixed acknowledgment ("Un momento…") becomes annoying across a long conversation; a random phrase from a curated list keeps the agent sounding natural, and the phrase list is editable straight from .env.
Re-synthesizing 22 clips on every startup added ~21 s to the boot time (measured), so the bootstrap now tracks the phrase list and only re-synthesizes when it changed (or a clip is missing).

Notes

force was dropped from assets::ensure_clips: the filler / backchannel banks keep the "generate only if missing" behaviour, and the start filler has its own list-driven bootstrap.
server.sh startup wait raised from ~20 s to ~60 s: the first startup after a list change synthesizes the whole bank before the socket is bound.
A failed synthesis keeps the previous clip of that position; a barge-in still clears the start-filler channel (AudioMixer::flush), and START_FILLER_ENABLED=false disables it entirely.

2026-10-16 — Start filler: immediate turn-start acknowledgment on its own mixer channel

Added

START_FILLER_ENABLED / START_FILLER_TEXT / START_FILLER_DIR (.env, .env.example, src/config.rs). The start filler is a pre-generated clip queued the moment a user turn begins — no delay — on a dedicated mixer channel, so it acknowledges the turn while the LLM/TTS are still working, independently of the delayed latency filler. (Superseded by the entry above: START_FILLER_TEXT became the START_FILLER_TEXTS list and the clip is now picked at random.)
AudioMixer: new start_filler_queue channel, mixed as mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN), so the start filler shares the filler gain (idle 1.0, ducked 0.15 under TTS). Included in is_active() and cleared by flush().
TurnManager::play_start_filler, called from handle_user_turn right after the in-flight reply is cancelled.
Startup bootstrap synthesizes START_FILLER_DIR/start_filler.raw with the configured voice if it is missing (src/assets.rs); the clip is preloaded into its own FillerBank, kept out of the round-robin FILLERS_DIR bank.

Why

The delayed latency filler only fires after FILLER_DELAY_MS (and only if the reply is still silent), so with the current fast TTFB (~0.73 s) the agent can stay silent for a moment after the user stops speaking. The start filler provides an immediate acknowledgment instead, without touching the existing delayed filler.

Notes

(Superseded — see the entry above.) The clip was re-synthesized on every startup (ensure_clips(..., force = true)), so editing START_FILLER_TEXT took effect at the next restart with no manual deletion. A new clip was written only on success, so a failed synthesis left the previous clip in place; the filler / backchannel banks kept the "generate only if missing" behaviour.
A barge-in clears the start filler along with every other channel (AudioMixer::flush); START_FILLER_ENABLED=false disables it entirely.

2026-10-16 — Fixed micro-gaps at the start of each sentence → `TTS_PREBUFFER_MS` 170 → 550 ms, `LLM_STREAM_CHUNK_CHARS` 120 → 240

Changed

TTS_PREBUFFER_MS raised 170 → 550 (.env, .env.example). With the current vocoder windows (first 185.8 ms, steady 371.6 ms) any pre-buffer in (186, 557] ms primes playback on the same second window, so 550 ms takes the entire ~~557 ms cushion at no extra TTFA (~~0.73 s).
LLM_STREAM_CHUNK_CHARS raised 120 → 240 (.env, .env.example). The first-chunk threshold stays at 20 (early flush preserved), but every following chunk is now twice as long → fewer TTS requests, hence fewer chunk seams.
README (Mixer & jitter buffering, the TTS_PREBUFFER_MS / LLM_STREAM_CHUNK_CHARS tables and the TTFA note) updated to the new defaults and rationale. No Rust code changed — both values are read from .env at runtime.

Why

After the seam-quality retune left the audible micro-interruptions at the start of sentences. Diagnosis from run/server.log: the defect is a TTS queue underrun at the chunk seams, not a timbre artifact. Each sentence / fragment is a separate HTTP request whose PCM is concatenated with no crossfade; the relay advances chunk-by-chunk, so the first dip in production empties the queue and the mixer logs TTS queue underrun — pausing playback until audio resumes (94 occurrences in the sampled log; 0 running low warnings), then restarts with need = 1 → an audible micro-gap at the start of the next sentence.
Why the queue empties: with TTS_PIPELINE_DEPTH = 3 on a single GPU the measured per-window cadence under concurrency is 552–613 ms (~1.5×) versus ~350 ms (~0.94×) for a lone request. A 170 ms cushion has no margin against that; 550 ms does, and it is latency-free because the pre-buffer already lands on the second window.
Measured in the log at the seam (12:56:36–38): chunks of 16384 samples (queue_len=16384, prebuffer=7497 = 170 ms) with two concurrent requests (text_len=119 and 19), each followed by an underrun-pause line.

Notes

Both levers are pure configuration; the binary is unchanged, so ./server.sh restart (no rebuild) is enough to apply them.
The 550 ms cushion is the ~557 ms gap to the third window start, so raising it further only adds latency (it would wait for the next window).

2026-10-16 — Drama-3 vocoder retuned for seam quality → stride `4` / followup `8` / overlap `16`; `TTS_PREBUFFER_MS` 450 → 170 ms

Changed

The Drama-3 launcher (/root/drama3/start-drama3.sh, pm2 start-drama3) now carries more decoder context per window. The previous config (stride 4, flat ramp "[4,4,4]", overlap 8, crossfade 256) was replaced by:

--vocoder.factory.stream_stride 4
--vocoder.factory.stream_followup_stride_ramp "[8,8,8]"
--vocoder.factory.stream_followup_stride 8
--vocoder.factory.stream_overlap_tokens 16
--vocoder.factory.stream_crossfade_samples 512
--tts_engine.factory.enable_async_decode true

TTS_PREBUFFER_MS lowered 450 → 170 (.env, .env.example, src/config.rs). With stride 4 the first vocoder window is 4 codec frames = 0.186 s, so 170 ms starts playback on that first window and TTFA collapses to the server's own TTFB (~0.38 s).
README: the "Starting the Drama 3 backend (SGLang-Omni)" subsection and the "Mixer & jitter buffering" pre-buffer note were rewritten around the new stride/overlap values and the ~0.186 s first / ~0.372 s steady window cadence.

Why

The previous latency-first config (stride 4, overlap 8) produced audible micro-gaps and a slight detune / falsetto on long replies. Root cause: with only 8 tokens of decoder warm-up context per window, and one TTS request per reply chunk (PCM concatenated without crossfade at the request seams), the streaming decoder's state was too shallow and its timbre/pitch wobbled at each window and request boundary.
Seam context restored without breaking real-time: raising the overlap to 16 at stride 4 pushes decode to 5× real time, which the box cannot sustain (ratio rose to 1.18–1.98×). Raising strides and overlap together — stride 4 (first window) / followup 8 (steady) with overlap 16 — keeps the decode load at a sustainable (16+8)/8 = 3× (~0.95× ratio) while giving each steady window twice the context, and crossfade 512 smooths the seams further.
A/B measurements on the live endpoint (measure_tts.py, warm server, voice cloning, single request vs sentence-chunked like the engine) on a long multi-sentence reply: the amplitude jump at the seam boundaries dropped from the 99.5+ percentile of the signal to 96–98, i.e. the seams no longer stand out. TTFB stays ~0.38 s (the first window is unchanged) and the real-time ratio stays ~0.94–0.98×.

Notes

The first window is unchanged (4 frames), so the residual latency is still dominated by the server's fixed ~0.38 s TTFB; the steady window grew from 0.186 s to 0.372 s (8 frames) and arrives ~350 ms apart, so the pre-buffer of 170 ms still starts playback on the very first window. Further latency gains would need a faster first token (model / preprocessing), not a smaller stride.
measure_tts.py was added to the repo: it streams /v1/audio/speech and prints the per-window arrival schedule, TTFB and real-time ratio (plus a seam-jump summary), so future A/B runs are one command.

2026-10-16 — Mivocero: karaoke word highlighting on the assistant reply

Changed

The /mivocero client now reveals the assistant reply in karaoke style (src/web/mivocero.html) instead of appending plain text. Each word is a <span>; words light up as they are spoken. The highlight is driven by the audio playhead: the playback AudioWorklet now counts the samples it has actually emitted and reports them (played) on its existing per-block message. The client maps that count onto a character offset with an estimated speaking rate (KARAOKE_CPS, ~15 chars/s) and marks each word done / now / pending.
The old blinking-caret "pending" style was removed; the whole reply is shown at once (dim) and progressively illuminated.
A safety net finishes the highlight: once assistant_end has been received and playback has been idle for 600 ms after it had actually started, any word still pending is marked done, so a reply always ends fully lit. New turns and interrupts cancel the loop.

Notes

The mapping from audio to text is inherently approximate (the server streams text and PCM on the same socket without per-word timing), so KARAOKE_CPS is a tunable estimate for the visual effect only — it never touches the audio.
played counts only real samples, so the playback jitter buffer's silence and any underrun pauses do not advance the highlight.

2026-10-16 — End-to-end validation → `TTS_PREBUFFER_MS` 700 → 450 ms; Drama-3 startup documented

Changed

TTS_PREBUFFER_MS lowered 700 → 450 (.env, .env.example, src/config.rs). With the vocoder ramp "[10,12,16]" the first window is 10 codec frames = 0.464 s, so 450 ms sits just below it and playback now starts on that first window instead of waiting for the second one that 700 ms required.
README: new "Starting the Drama 3 backend (SGLang-Omni)" subsection under the fish plugin, with the exact sgl-omni serve command, the pm2 workflow, the meaning of the two --vocoder.factory.* flags and the two CLI gotchas (no stages. prefix; the ramp tuple passed as a JSON string).

Why

An end-to-end browser test (RUST_LOG=…,audio_agent_engine::tts=debug,…) on the live engine confirmed the narrower windows: bursts of 10/12/16 frames (0.464 / 0.557 / 0.743 s) instead of a single 2.97 s block, prebuffer=30870 samples (= 700 ms) and time-to-first-audio ≈ 1.1 s from dispatch (~0.64 s of which is the server's TTFB). Barge-in fired correctly on three interrupts.
Three tail underruns were still observed at the end of short replies: e.g. 2.46 s of audio produced in ~2.6 s (≈1.07× real time including the fixed ~0.64 s TTFB), so the queue drained just before the last words. The mixer recovered via resilient resume, but the gap is audible. 450 ms targets the first 0.464 s window and trades a little more tail risk for a markedly earlier start.

Notes

The residual deficit is dominated by the server's fixed ~0.64 s time-to-first-byte, which no pre-buffer value can remove. Going lower would need a faster first token (model / preprocessing), not a smaller stride.

2026-10-16 — Finer TTS delivery granularity (Drama-3 vocoder window) → `TTS_PREBUFFER_MS` 700 ms

Changed

The SGLang Drama-3 server now emits audio in ~0.74 s windows instead of ~2.97 s bursts. Two flags were added to the server launcher (/root/drama3/start-drama3.sh, managed by pm2 as start-drama3): --vocoder.factory.stream_followup_stride 16 and --vocoder.factory.stream_followup_stride_ramp "[10,12,16]" — upstream defaults were 64 and (10,12,32).
TTS_PREBUFFER_MS default lowered 900 → 700 (.env, .env.example, src/config.rs). 700 ms is deliberately just under one 0.743 s window, so playback starts on the first burst rather than waiting for a second one.

Why

Drama-3's streaming granularity is a vocoder setting, not an engine one: Drama3VocoderFactoryArgs buffers stream_followup_stride codec frames before decoding and emitting, and frame_length = hop_length * 4 = 2048 samples (46.4 ms at 44.1 kHz). The upstream 64 therefore means a 131 072-sample burst every ~2.7 s.
Measured against the live endpoint (single, uncontended request, PCM stream):

burst period ratio first audio

before (stride 64) 131 072 samples / 2.97 s 2.70–2.78 s 1.08× 0.96 s

after (stride 16) 32 768 samples / 0.743 s 0.69 s (σ ≈ 0.01 s) 1.04× 1.34 s
The finer window is what makes the small pre-buffer viable: the producer still runs slightly faster than real time (1.04×), but it now tops the queue up every 0.69 s instead of every ~2.7 s, so arrival jitter is bounded by one window and no longer forces a multi-second buffer.
Prefix caching (RadixAttention) was evaluated and rejected: the client already sends a constant prefix (model / voice / references), yet an A/B on the endpoint showed no TTFB difference (ref 0.62–0.64 s, no-ref 0.52 s) — that start-up is decode/vocoder, not prompt prefill.

	burst	period	ratio	first audio
before (`stride` 64)	131 072 samples / 2.97 s	2.70–2.78 s	1.08×	0.96 s
after (`stride` 16)	32 768 samples / 0.743 s	0.69 s (σ ≈ 0.01 s)	1.04×	1.34 s

Notes

Each window re-decodes stream_overlap_tokens (32) frames of context, so a smaller stride costs more vocoder compute: 96 frames decoded per 64 emitted becomes 48 per 16. The measured ratio moved 1.08× → 1.04×, still above real time; stride 8 (0.371 s windows, 40 frames decoded per 8 emitted) is the next step down and should only be tried with the ratio re-measured.
The window lengths feed the vocoder's CUDA-graph capture set, so changing them requires a server restart.
stream_stride (first window) and stream_followup_stride_ramp (the windows after it) were left near their defaults — only the steady-state value drives the granularity of a long reply.
(Superseded: the TTS_PREBUFFER_MS default is now 450, targeting the ramp's 0.464 s first window — see the entry above.)

2026-10-16 — Resilient mixer resume + `TTS_PREBUFFER_MS` default 900 ms

Changed

The mixer no longer re-fills the whole jitter buffer after a mid-reply underrun. Previously, when the TTS queue emptied for more than the 200 ms drain grace, pull_frame cleared tts_primed and tts_stream_end, then required a full tts_prebuffer_samples refill before playback resumed — adding a second, self-inflicted pause on top of the real underrun. It also left the mixer "primed" between turns because mark_stream_start never re-armed tts_primed. Now:
- the full pre-buffer is awaited only at the first playback of a turn; once playback has started, an underrun pauses and resumes as soon as any audio is queued again (need = 1);
- tts_stream_end is no longer cleared on drain, so a genuine end-of-stream is still honoured;
- mark_stream_start re-arms tts_primed, tts_playback_started and tts_drain_frames at the start of every turn.
TTS_PREBUFFER_MS default raised 400 → 900 (.env, .env.example, src/config.rs). With 400 the reply was still prone to drop-outs on long answers.

Why

Direct measurement of the SGLang Drama-3 endpoint (a single, uncontended request) showed it delivers ~1.08× real time, in bursts every ~2.7 s (≈131 072 samples / 2.97 s per burst). Because the producer barely outruns the consumer and delivers in bursts, gapless playback requires a jitter buffer of roughly the burst period (~3 s); smaller buffers show a short real drop-out (pure silence) on long replies.
The mixer fix removes the avoidable part of the pause (the full refill after an underrun) and recovers at the first byte, which is what makes the remaining gap equal to the backend stall rather than stall + refill.
Default 900 is a deliberate latency-first choice: it keeps time-to-first-audio low (~1.8 s) and accepts a rare short drop-out on very long replies. Raising TTS_PREBUFFER_MS towards ~3000 trades that latency back for gaplessness. (Superseded: the default is now 700, once the server's vocoder window was narrowed to ~0.74 s — see the entry above.)

Notes

The residual drop-out is a property of the TTS backend (~1× real time), not of the mixer: no buffering smaller than the backend's burst period can bridge a delivery stall.
Relay diagnostics at debug level in run_llm_turn log per-chunk relay timing (TTS pipeline ...) to help localise seams.

2026-10-16 — Pipelined TTS: chunks are synthesized ahead of playback

Changed

The TTS sink no longer waits for one chunk's stream to finish before requesting the next. Previously run_llm_turn looped while let Some(chunk) = rx.recv() { tts.synthesize(chunk).await }, so a chunk's synthesis only started after the previous chunk had finished streaming end-to-end. Because the TTS engine needs ~0.6 s to emit its first audio byte, every chunk boundary left a gap roughly that long, which the old 1.5 s pre-buffer used to hide.
The sink is now split into two pieces:
- a collector that spawns one synthesis task per chunk as soon as the LLM produces it (bounded by a semaphore), each streaming into its own staging buffer;
- a relay that forwards those staging buffers into the mixer queue strictly in submission order, so playback stays in order and gapless. A chunk's synthesis therefore overlaps the previous chunk's playback, hiding the per-request start-up latency.

Added

New TTS_PIPELINE_DEPTH configuration (default 3) in src/config.rs: how many TTS requests may be in flight at once. 1 restores the previous strictly serial behaviour.
SampleRingBuffer::remaining_capacity() (src/audio/ring_buffer.rs) so the relay never pushes past the mixer queue's free space — queued (i.e. currently playing) audio is never dropped to make room.

Why

Direct measurement against the SGLang Drama-3 endpoint showed a fixed ~0.62 s start-up latency per request (request → first audio byte), with audio then streamed at roughly real time. Two concurrent requests are served in parallel (start-up rises only to ~0.84–1.0 s), so overlapping requests genuinely removes the inter-chunk gaps instead of just moving them.
The gaps were independent of chunk size (120 → 60 chars left the same ~0.8 s hole), which confirmed the cause was serialization, not chunk length.

Notes

Staging buffers are sized at 30 s of audio per in-flight request, which is comfortably more than any single chunk.
With the pipeline in place the pre-buffer only has to cover arrival jitter rather than the TTS engine's per-request start-up, so it can be kept small. (Superseded: the default was later raised to 900 — see the entry above.)

2026-10-16 — Mixer TTS pre-buffer is now configurable (default 400 ms)

Changed

The TTS jitter buffer moved from a hardcoded 1.5 s constant to the TTS_PREBUFFER_MS setting (default 400). AudioMixer (in src/audio/mixer.rs) now takes tts_prebuffer_samples as a constructor argument instead of reading the removed TTS_PREBUFFER_SAMPLES constant, and run_session derives it from the config.

Why

Log analysis of a voice turn ("Conta da dieci a uno.") showed the first TTS request left ~1.2 s after the EOU and the first audio bytes arrived ~1.9 s, but playback only started at +4.2 s: the mixer had 1.486 s (65 536 samples) queued at +2.8 s, 14 ms short of the 1.5 s threshold, so it waited for the next Fish chunk, which landed 1.39 s later. The pre-buffer, not the text chunker, dominated the perceived latency.
Fish Audio now delivers near real time, so 1.5 s was far more buffer than needed. 400 ms removes ~1.1 s of time-to-first-audio while still absorbing normal inter-chunk jitter (the 200 ms drain grace is unchanged).

Notes

Tune from .env: raise TTS_PREBUFFER_MS if you hear drop-outs, lower it for an even snappier start.
The low-buffer warning threshold now scales with the configured value (prebuffer / 4).

2026-10-16 — TTS starts on the first words, not on the first sentence

Changed

The streaming reply is now chunked on a word boundary instead of waiting for sentence punctuation. SentenceSplitter (in src/orchestrator/turn_manager.rs) keeps the sentence-terminator rule (., !, ?, ;, newline) but adds an early flush: the first chunk is cut as soon as LLM_STREAM_FIRST_CHUNK_CHARS characters are buffered, and every following chunk at LLM_STREAM_CHUNK_CHARS. The cut lands on the first whitespace at or past the threshold, so a chunk always holds at least that many characters and never splits a word. Previously the first chunk only left at a ./!/?, so a long opening sentence delayed the first spoken word.

Added

New LLM_STREAM_FIRST_CHUNK_CHARS (default 20) and LLM_STREAM_CHUNK_CHARS (default 120) configuration in src/config.rs. A small first threshold makes the agent start speaking on the first few words; the larger follow-up threshold keeps prosody good once the audio is flowing. Setting either to 0 disables that flush (chunks then end on sentence punctuation only).

Notes

Verified on the running server: for a chat-box turn the first TTS request is dispatched ~0.5 s after the user text, synthesizing "Non ho occhi fisici né" (23 chars), well before the rest of the sentence ("accesso a telecamere o immagini, quindi non posso vedere nulla …") was generated. Before the change the whole opening sentence had to complete first.
The remaining time-to-first-audio is dominated by the mixer's 1.5 s TTS pre-buffer plus the TTS engine's first-chunk latency, not by the chunker.

2026-10-15 — Filler is now a latency guard: switchable and delay-gated

Changed

The end-of-turn filler is no longer played unconditionally. It exists to mask the wait for the reply, so playing it when the answer is already speaking only gets in the way of the first words. TurnManager::schedule_filler (in src/orchestrator/turn_manager.rs) starts a FILLER_DELAY_MS timer when the turn is dispatched (both the LLM and the direct-TTS path) and queues the clip only if the response audio has not started by then. At most one filler per turn.
New FILLER_ENABLED / FILLER_DELAY_MS configuration in src/config.rs. FILLER_ENABLED=false disables the filler entirely (no clip is ever queued); FILLER_DELAY_MS (default 2500) is the latency guard, tunable from .env.
The immediate filler pushes in handle_user_turn and speak were removed in favour of the single scheduled path, so the playground's direct-TTS path follows the same rule.

Added

AudioMixer::tts_playback_started() in src/audio/mixer.rs — set when the TTS pre-buffer is reached (i.e. the response audio actually starts), reset by mark_stream_start and flush. This is what the filler decision reads.

Notes

The pending filler timer is bound to the turn's CancellationToken, so a barge-in (interrupt, VAD onset, a new turn, or disconnect) drops it.
Verified from .env on the running server: FILLER_DELAY_MS=100 → pushed filler (delayed); FILLER_DELAY_MS=8000 → response audio already playing — filler skipped; FILLER_ENABLED=false → no filler activity at all.
With the current LLM_ENABLE_THINKING=false + Fish Audio setup, a short reply starts playing within 2.5 s, so the filler correctly stays silent; raise FILLER_DELAY_MS (e.g. 1000) if you want it to fire more often.

2026-10-15 — LLM reasoning stage + `/mivocero` speech-to-speech chat client

Added

OpenAI-compatible LLM client — src/llm.rs. New LlmClient posting to {LLM_BASE_URL}/chat/completions, with an Authorization: Bearer {LLM_API_KEY} header and a bounded TCP connect timeout (no global request timeout, so SSE streams can stay open). Two entry points, selected at runtime via LLM_STREAM:
- chat_stream — SSE token streaming (stream: true), handling data: lines and data: [DONE];
- chat — a single blocking request.
Chain-of-thought stripping — reasoning models (Qwen3, …) may emit their plan in a separate reasoning_content delta (vLLM) or inline between think markers (llama.cpp and others). reasoning_content is ignored and inline think blocks are removed by an incremental ThinkStripper that is stable across delta boundaries (it withholds a tail the size of the opening marker so a marker split between two deltas is still detected). Applied to both the chat box text and the synthesized audio.
LLM stage in TurnManager — the user turn (STT transcript or chat-box text) is sent to the model and the reply is spoken by the TTS. A rolling history window (LLM_HISTORY_MESSAGES) is kept per connection and prefixed with LLM_SYSTEM_PROMPT. A SentenceSplitter chunks the streaming reply on ., !, ?, ; and newlines so the TTS starts on the first sentence (SENTENCE_SOFT_MAX = 220 chars forces a split on whitespace for punctuation-free runs).
Barge-in covers the LLM request — an interrupt (client JSON, VAD onset or disconnect) cancels the in-flight HTTP request through a CancellationToken in the reqwest tokio::select!, cancels the TTS tasks and flushes the mixer.
Bidirectional text protocol on /ws — client → server {"type":"user_text","text":…} and {"type":"interrupt"}; server → client {"type":"user","text":…}, {"type":"assistant_delta","text":…}, {"type":"assistant_end"} and {"type":"error","text":…}. Legacy text commands (INTERRUPT, SET_REFERENCE:, CLEAR_REFERENCE, raw text → direct TTS) are unchanged, so the playground keeps working.
/mivocero client — src/web/mivocero.html, embedded via src/web/mod.rs and served by src/main.rs. A speech-to-speech example agent with a microphone (same AudioWorklet capture/playback path as the playground), a chat box that renders the user transcript and the streaming assistant reply, a text composer and an interrupt button.
LLM_* configuration — Config fields and read_llm_env() in src/config.rs: LLM_ENABLED, LLM_BASE_URL, LLM_API_KEY, LLM_MODEL, LLM_MAX_TOKENS, LLM_TEMPERATURE, LLM_STREAM, LLM_HISTORY_MESSAGES, LLM_ENABLE_THINKING, LLM_STRIP_REASONING, LLM_DIRECT_ANSWERS, LLM_SYSTEM_PROMPT. LLM_ENABLED defaults to true when both LLM_BASE_URL and LLM_MODEL are set.
Three independent chain-of-thought controls (each switchable from .env) — a single mechanism does not cover every deployment:
1. LLM_ENABLE_THINKING — LlmClient sends chat_template_kwargs: {"enable_thinking": <bool>} when the variable is set, turning Qwen3-style thinking off/on server-side (vLLM/SGLang). Unset omits the vendor-specific field so strict OpenAI-compatible servers keep accepting the body.
2. LLM_STRIP_REASONING (default true) — reasoning_content deltas are dropped and inline think blocks removed; false passes the raw output (both fields) through.
3. LLM_DIRECT_ANSWERS (default true) — appends DIRECT_ANSWER_DIRECTIVE to the system prompt. Needed when the server runs no reasoning parser: the chain-of-thought is then written straight into content as ordinary prose and is neither reasoning_content nor tagged, so filtering cannot catch it.

Fixed

Chat-box turns blocked the receive loop — handle_control_message in src/ws/session.rs awaited TurnManager::on_user_text inline, so while a reply was being generated the session could not read any further client message: an interrupt sent during generation was only honoured once the turn had finished. Text turns now run in their own task, and the interrupt cancels the reply immediately (verified: 0 deltas emitted after the interrupt, session still usable for the next turn).
SSE cancellation latency — LlmClient::chat_stream now races the next stream chunk against the CancellationToken, so a barge-in drops the HTTP response instead of waiting for the model's next token.

Notes

Without an LLM the pipeline degrades to a plain STT → TTS echo (LLM_ENABLED=false).
src/orchestrator/turn_manager.rs gained unit tests for sentence splitting (terminators, split across deltas, long punctuation-free runs) and src/llm.rs for think-block stripping and for the opt-in thinking field — cargo test passes (13 passed, 1 ignored).
Verified against Qwen/Qwen3.6-35B-A3B on a local vLLM (http://10.8.0.4:8000/v1): without LLM_ENABLE_THINKING=false the model narrates "Here's a thinking process: 1. Analyze User Input …" directly in content, which the TTS reads aloud; with it, the reply is the answer only.

2026-10-15 — Startup clip bootstrap (auto-generate filler / backchannel)

Added

Startup asset bootstrap — src/assets.rs. On boot the server checks every expected filler / backchannel clip and, if missing, synthesizes it with the configured TTS engine and writes it as *.raw (16-bit LE mono PCM at 44.1 kHz) into the target directory (created if needed). Existing clips are never overwritten, so a fresh checkout works with no manual generation step and a customized/partial bank is preserved.
Default clip definitions: assets::FILLER_CLIPS (uhm, eh, mmm, aver, bueno, pues, vaya) and assets::BACKCHANNEL_CLIPS (mmhmm, aja, claro, yaveo). Backchannel clips are generated only when BACKCHANNEL_ENABLED=true.
Wired in src/main.rs before the banks are loaded; the TTS engine is now created earlier so it can be reused for generation.

Notes

Generation is best-effort and never blocks startup: a synthesis failure logs a warning per clip and the server still starts (that bank stays silent).
The scratch capture buffer during generation is a 30 s SampleRingBuffer drained after synthesis.

2026-10-15 — Filler bank (round-robin end-of-utterance fillers)

Added

Filler bank — assets/fillers/*.raw. The end-of-utterance filler is no longer a single hard-coded clip; TurnManager now selects clips round-robin from the bank via FillerBank::nth (a per-turn cursor, TurnManager::next_filler). With a single clip the behaviour is unchanged.
Clips synthesized with the cloned reference voice (Spanish, 44.1 kHz, 16-bit LE mono PCM): uhm, eh, mmm, aver, bueno, pues, vaya (~0.1–0.9 s).

Notes

FillerBank::get is kept for reference/debugging but is no longer on the hot path (#[allow(dead_code)]).

2026-10-15 — During-utterance backchannel

Added

Backchannel subsystem — src/orchestrator/backchannel.rs. Emits short acknowledgments ("mm-hmm", "ajá", "claro") while the user is speaking, at a low fixed gain, to signal that the agent is listening without taking the turn. Distinct from the end-of-utterance filler injected by the TurnManager.
BackchannelScheduler — per-session decision logic fed by the VAD: on_speech_start on VadEvent::SpeechStart, on_speech_end on SpeechEnd, and maybe_emit() on every frame while speaking. A cue is emitted only when the user has spoken for at least BACKCHANNEL_MIN_SPEECH_MS, at least BACKCHANNEL_INTERVAL_MS elapsed since the last cue, fewer than BACKCHANNEL_MAX_PER_TURN cues were emitted this turn, and the agent is not speaking (!TurnManager::is_speaking()).
Preloaded clip bank — assets/backchannel/*.raw (16-bit LE mono PCM at 44.1 kHz), synthesized with the cloned reference voice, loaded into a shared FillerBank and played round-robin.
Third mixer channel — AudioMixer gained a dedicated backchannel_queue and a fixed backchannel_gain. The mix is now tts + (filler × filler_gain) + (backchannel × backchannel_gain); the queue is cleared by flush() (barge-in).
FillerBank generalization — one type now serves both the filler and the backchannel banks (load(dir, label), round-robin nth(index), is_empty()); the default filler lookup ("uhm") is unchanged.
VadDetector::in_speech() — exposes whether the detector is inside a confirmed utterance, used to gate backchannel emission.
New .env knobs (documented in .env.example): BACKCHANNEL_ENABLED, BACKCHANNEL_DIR, BACKCHANNEL_GAIN, BACKCHANNEL_MIN_SPEECH_MS, BACKCHANNEL_INTERVAL_MS, BACKCHANNEL_MAX_PER_TURN.

Notes

BACKCHANNEL_GAIN is clamped to 0..1 and BACKCHANNEL_INTERVAL_MS is clamped to >= BACKCHANNEL_MIN_SPEECH_MS at load time.
Wiring lives in src/ws/session.rs (mic/VAD task); the bank is created once in src/main.rs and shared via AppState.

2026-10-15 — Server-side VAD (Silero V5, ONNX)

Added

VAD subsystem — src/vad/mod.rs + src/vad/silero.rs. Runs Silero VAD v5 via ONNX Runtime on the same 16 kHz mono stream that feeds the STT engine. The model consumes 32 ms windows (512 samples) with a 64-sample context and a recurrent state [2,1,128]; the graph shapes and the sr input rank are introspected at load time.
Onset/offset state machine with hysteresis (VAD_THRESHOLD / VAD_NEG_THRESHOLD) emitting three events: SpeechOnset (fast, for barge-in), SpeechStart, and SpeechEnd (carrying the captured PCM plus a pre-roll pad so the first phoneme is not clipped).
Fast server-side barge-in — a SpeechOnset interrupts the agent when it is currently speaking (TurnManager::is_speaking), cancelling the in-flight TTS and flushing the mixer.
VAD-driven STT segmentation — SttWorker::push_utterance receives whole utterances instead of raw frames, so the STT no longer needs its own RMS segmentation (VAD_SEGMENT_STT=true, default).
Shared ONNX environment — src/onnx.rs. ONNX Runtime allows a single environment per process, so the STT and VAD pipelines now share a lazily created Environment (cheap Arc clones) instead of each building one.
New .env knobs (all documented in .env.example): VAD_ENABLED, VAD_MODEL_PATH, VAD_THRESHOLD, VAD_NEG_THRESHOLD, VAD_MIN_SPEECH_MS, VAD_MIN_SILENCE_MS, VAD_SPEECH_PAD_MS, VAD_BARGE_IN_MIN_SPEECH_MS, VAD_SEGMENT_STT, VAD_BARGE_IN, VAD_DEBUG.
AudioMixer::is_active / tts_stream_active flag tracking whether a TTS stream is in flight, independent of the queue length, so barge-in can gate on "the agent is speaking".

Fixed

Duplicate analysis window appended to the utterance at speech start (the pre-roll ring already contained it) — the utterance is now seeded from the pre-roll ring only.
server.sh now also appends audio_agent_engine::vad=debug to RUST_LOG (mirroring the earlier TTS fix), so VAD debug logs are never filtered out.

Verified

models/silero_vad.onnx is loaded at startup (state_shape=[2,1,128], sr_rank=0). A live WSS session streaming 44.1 kHz speech produced, in run/server.log:

INFO  ws::session: VAD active for this session threshold=0.5 min_speech_ms=250 min_silence_ms=300 …
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
DEBUG vad::silero: VAD speech start speech_ms=256 prob="1.000" preroll_samples=4992
DEBUG vad::silero: VAD speech end duration_ms=1144 samples=18304 avg_prob="0.609"
INFO  ws::session: VAD utterance → STT duration_ms=1144 samples=18304
INFO  orchestrator::turn_manager: EOU detected tokens=5 text=Me escucha.

Talking over the agent (after a text TTS prompt) produced:

DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
INFO  ws::session: VAD barge-in — interrupting agent prob="0.999"
INFO  orchestrator::turn_manager: interrupt received, flushing audio

2026-10-15 — TTS DEBUG logs actually reach `run/server.log`

Problem

The debug! diagnostics added to the TTS subsystem never appeared in the log, even though the code and the DEFAULT_LOG_FILTER fallback were in place. Root cause: the parent shell (IDE) already exports RUST_LOG=info, so EnvFilter::try_from_default_env() succeeded and the info,audio_agent_engine::tts=debug fallback was never used. In server.sh the same happened — ${RUST_LOG:-info,audio_agent_engine::tts=debug} never expanded the default because RUST_LOG was already set.

Fixed

server.sh now appends the audio_agent_engine::tts=debug directive to whatever RUST_LOG it inherits (unless that target is already mentioned), so the TTS debug logs cannot be silently filtered out. start logs the effective filter.

Verified

A live WSS session (Hola, …) that is kept open for the whole synthesis now emits, in run/server.log:

DEBUG audio_agent_engine::tts::fish_audio: TTS request dispatched endpoint=… has_reference=true text=…
DEBUG audio_agent_engine::tts::fish_audio: TTS time-to-first-byte ttfb_ms=0 bytes=8186
DEBUG audio_agent_engine::tts::fish_audio: TTS chunk received chunk=1 bytes=8186 samples=4093 queue_len=4093
DEBUG audio_agent_engine::tts::fish_audio: TTS stream summary chunk_count=9 total_samples=180224 audio_s="4.09" elapsed_ms=3347

Note: closing the WebSocket mid-stream triggers turn_manager.interrupt() (barge-in), which cancels the TTS stream before any chunk arrives — so only TTS request dispatched is logged in that case.

2026-10-15 — Optional HTTPS/WSS + clearer microphone diagnostics

Problem

Clicking "Connect & Start Mic" in the playground failed with a generic mic error, with no WebSocket connection ever reaching the server. Browsers only expose navigator.mediaDevices.getUserMedia in a secure context (HTTPS or localhost); over plain HTTP on any other origin the API is simply absent, so startAudio() threw before the socket was created.

Added

Optional TLS termination — src/tls.rs. Set both TLS_CERT and TLS_KEY (PEM) in .env and the server serves the same Axum router over HTTPS/WSS via tokio-rustls + hyper (HTTP/1 upgrades enabled, so WebSocket handshakes work). Without them it stays on plain HTTP. The startup log now says listening (HTTPS) / listening (HTTP).
./server.sh cert [--force] — generates a self-signed certificate under certs/ whose SAN covers localhost, 127.0.0.1, ::1 and the host's IPs (EXTRA_SAN to append more). start/restart auto-generate it if TLS is enabled but the files are missing; status shows the scheme and cert state.

Changed

Playground diagnostics — src/web/playground.html now checks for a secure context before requesting the mic and reports the precise cause (insecure origin / NotAllowedError / NotFoundError) with a remediation hint, instead of a bare mic error.
New config fields Config::tls_cert / Config::tls_key (+ tls_enabled()), read from TLS_CERT / TLS_KEY.

Notes

No new crates are downloaded: hyper, hyper-util, tokio-rustls, rustls and rustls-pki-types were already present as transitive deps of reqwest (ring provider), so they are declared with default-features = false.

2026-10-15 — `server.sh` control script (start / stop / restart)

Added server.sh, a small bash helper so the server does not have to be managed by hand.

Usage

./server.sh start [--build]     # start (auto-builds if binary missing)
./server.sh stop                # graceful stop (SIGKILL after 10 s)
./server.sh restart [--build]   # stop + start
./server.sh status              # process / port / HTTP status
./server.sh logs [N]            # tail run/server.log (default 50, follow)
./server.sh build               # build the release binary only

PID is tracked in run/server.pid; logs are appended to run/server.log.
start waits until the port is accepting connections (the model load takes ~1.5 s) before reporting success.
Overridable env vars: PROFILE (release|debug), RUST_LOG, PORT.
The script cds to its own directory first, so .env and the relative models/parakeet path resolve correctly regardless of the caller's cwd.

Documented in the README (Quick Start).

2026-10-15 — Parakeet TDT STT reimplemented in pure Rust (ONNX)

The previous STT was a placeholder that ran argmax over a single .onnx file — it could not actually transcribe. It has been replaced with a complete, dependency-free (no Python/NeMo) implementation of the NVIDIA Parakeet TDT 0.6B v3 transducer pipeline in src/stt/parakeet.rs, mirroring the reference in istupakov/onnx-asr.

Change

3-stage ONNX pipeline, all sessions kept alive by the ort Environment:
1. nemo128.onnx — log-mel features waveforms/waveforms_lens → features/features_lens.
2. encoder-model.int8.onnx (preferred, auto-detected) or encoder-model.onnx — audio_signal/length → outputs[1,1024,T']/encoded_lengths.
3. decoder_joint-model.onnx — greedy TDT decoding.
Runtime graph introspection: d_model, num_durations, state_layers and state_hidden are read from the ONNX I/O shapes instead of being hardcoded.
Greedy TDT decode loop (token head + duration head; blank <blk> = 8192; max_tokens_per_step from config.json, default 10). Encoder output is re-packed from [1, d_model, T'] to contiguous [T', d_model] frames.
Internal text decoding: vocab.txt (<piece> <id>, SentencePiece ▁ → space) plus the reference spacing normalizer (\A\s|\s\B|(\s)\b). The Transcript struct now carries a text: String.
Config: STT_MODEL_PATH now points to a directory (models/parakeet by default, was models/parakeet-tdt.onnx). STT_TOKENIZER_PATH defaults to empty. Updated .env, .env.example and the README.
TurnManager: on_eou now uses transcript.text directly, falling back to the external tokenizer only when no text was produced.

Notes / fixes found during integration

The decoder_joint graph expects targets and target_length as int32 (not int64); encoder_outputs and both state tensors are float.
The int8 encoder (~650 MB, self-contained) is preferred over the fp32 encoder (42 MB + a 2.4 GB .onnx.data sidecar) for speed and simpler deployment.

Verification

Opt-in end-to-end test decodes a real 16 kHz mono WAV through the model:

$ PARAKEET_TEST_WAV=/tmp/stt_test/solis16k.wav \
    cargo test --release -- --ignored test_infer_real_audio --nocapture
tokens: 67 | text: "Bienvenidos a esta fascinante exploración del universo.
  Como locutor profesional, les explicaré hoy cómo las ondas electromagnéticas
  viajan por el espacio, transformando nuestra comprensión del cosmos."
test stt::parakeet::tests::test_infer_real_audio ... ok

The transcription matches the reference transcript verbatim (the source clip was trimmed to 12 s). cargo test (unit tests) also passes.

2026-10-15 — Fix: short utterances swallowed by the TTS pre-buffer

Short phrases produced no audio at all. The mixer only started playing TTS once the queue held TTS_PREBUFFER_SAMPLES (1.5 s). If an utterance's total length was below that threshold (e.g. "Cristóbal Colón" ≈ 1.11 s / 49152 samples), the pre-buffer was never reached, so tts_primed stayed false and the queued samples were never drained.

Change

src/audio/mixer.rs: added an tts_stream_end flag with mark_stream_start() / mark_stream_end(). When the stream has ended, the mixer primes as soon as any sample is queued (stream_ended && tts_len > 0), bypassing the pre-buffer for short utterances. The flag is reset by flush(), mark_stream_start() and when the queue drains.
src/orchestrator/turn_manager.rs: the spawned TTS tasks now call mixer.mark_stream_start() before synthesis and mixer.mark_stream_end() after it returns — only when the request was not cancelled, so barge-in does not falsely signal completion.

Verification

TTS stream complete chunk_count=6 total_samples=49152 duration_s="0.64"
TTS pre-buffer reached — starting TTS playback tts_queue_len=49152 prebuffer=66150 stream_ended=true

A raw WebSocket client sending "Cristóbal Colón" now receives ~1.12 s of non-silent audio (previously the same request yielded only silence frames).

2026-10-15 — Dual-stack listener (fix `localhost` / `::1` connection refused)

The HTTP server bound only to 0.0.0.0 (IPv4). On hosts where localhost resolves to ::1 first (confirmed via getent hosts localhost → ::1 localhost), clients and preview/port-forwarding proxies that connect to localhost via IPv6 received connection refused, surfacing as "service is unavailable".

Change

src/main.rs: the listener now binds to the IPv6 wildcard SocketAddr::from((Ipv6Addr::UNSPECIFIED, port)) instead of 0.0.0.0. With net.ipv6.bindv6only = 0 (Linux default, verified on this host) this is dual-stack: it accepts both IPv6 (::1 / localhost) and IPv4 clients.

Verification

ss -ltnp | grep 8787  → LISTEN *:8787
curl http://[::1]:8787/     → 200
curl http://localhost:8787/ → 200
curl http://127.0.0.1:8787/ → 200

2026-10-14 — Modular TTS plugin system (engine-agnostic via `.env`)

The TTS layer was hard-coded to Fish Audio / SGLang Drama 3. To support switching engines (e.g. Kokoro, or any future backend) without touching orchestration code, the TTS subsystem is now a pluggable trait selected at runtime via the TTS_ENGINE environment variable.

New `TtsEngine` trait (`src/tts/mod.rs`)

#[async_trait]
pub trait TtsEngine: Send + Sync {
    fn name(&self) -> &str;
    fn sample_rate(&self) -> u32;
    async fn synthesize(&self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken) -> Result<()>;
    fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}  // default no-op
}

Every plugin streams 16-bit mono PCM at OUTPUT_SAMPLE_RATE (44.1 kHz) into the shared mixer queue. If an engine's native rate differs, it resamples internally (the mixer runs at a fixed rate).
synthesize() must honor the CancellationToken for barge-in.
set_reference() defaults to a no-op; only engines that support voice cloning override it.

Factory (`src/tts/mod.rs::create_tts_engine`)

TTS_ENGINE=fish      → FishAudioTts  (Fish Audio / SGLang Drama 3)
TTS_ENGINE=kokoro    → KokoroTts     (stub, implement in src/tts/kokoro.rs)

Unknown values panic at startup with a clear message.

Bundled plugins

Plugin	File	Status
`fish` (Fish Audio Drama 3)	`src/tts/fish_audio.rs`	Production-ready (renamed from `sglang_client.rs`)
`kokoro`	`src/tts/kokoro.rs`	Stub — HTTP streaming call left as `TODO` with a commented example; includes a 24 kHz → 44.1 kHz linear resampler helper

`.env` loading

Added dotenvy to auto-load a .env file at startup (existing env vars take precedence).
Added .env.example documenting all env vars including TTS_ENGINE, KOKORO_URL, KOKORO_VOICE.

Files changed

src/tts/mod.rs — new TtsEngine trait + create_tts_engine factory.
src/tts/fish_audio.rs — renamed from sglang_client.rs; FishAudioTts now implements TtsEngine.
src/tts/kokoro.rs — new Kokoro plugin stub (example for adding engines).
src/tts/sglang_client.rs — deleted (moved to fish_audio.rs).
src/config.rs — new tts_engine field, TTS_ENGINE env var (default fish).
src/orchestrator/turn_manager.rs — tts: SglangTtsClient → tts: Arc<dyn TtsEngine>.
src/ws/handler.rs — AppState.tts → Arc<dyn TtsEngine>.
src/main.rs — uses create_tts_engine(&config); loads .env via dotenvy.
Cargo.toml — added async-trait, dotenvy.
.env.example — new file documenting all configuration env vars.

How to add a new engine

Create src/tts/myengine.rs and implement TtsEngine.
Add pub mod myengine; to src/tts/mod.rs.
Register it in the create_tts_engine match.
Set TTS_ENGINE=myengine in .env.

Build / verification

cargo build — 0 errors, 2 dead-code warnings (Kokoro stub fields).
cargo test — 4/4 passed.
Server starts with TTS_ENGINE=fish (default): log shows selected TTS engine engine="fish" and voice-cloning reference loaded.

2026-10-14 — Cancel TTS on WebSocket disconnect

When the client disconnected, the in-flight TTS synthesis task continued streaming from Fish Audio even though the WebSocket session was already closed, wasting bandwidth and server resources.

Fix (`src/ws/session.rs`)

Added turn_manager.interrupt() in the session cleanup path (after the receiver ends, before aborting the sender/mic tasks).
This cancels the TTS CancellationToken, flushes the mixer queues, and stops the streaming HTTP request immediately.

2026-10-14 — Fix WebSocket frame timing (Burst ticker)

The 20 ms audio sender ticker used MissedTickBehavior::Skip, which dropped tick beats whenever the Tokio event loop was busy (e.g. processing a large Fish Audio TTS chunk). Each skipped tick meant a 20 ms gap in frame delivery to the browser, producing audible micro-interruptions even though the server's TTS queue was well-buffered.

Fix (`src/ws/session.rs`)

Changed MissedTickBehavior::Skip → MissedTickBehavior::Burst.
Missed ticks now trigger immediate back-to-back sends so the browser receives all due frames. The browser's jitter buffer (500 ms) absorbs the burst without underrunning.

2026-10-14 — Micro-interruption fix (grace period on TTS drain)

Even with the 1.5 s server pre-buffer and 500 ms browser jitter buffer, brief audio gaps could still occur when the TTS queue momentarily hit zero between Fish Audio chunks. The mixer immediately reset its tts_primed state, forcing a full 1.5 s re-prime on the next chunk. The browser playback processor did the same, re-entering buffering mode on any underrun.

Server fix (`src/audio/mixer.rs`)

Added tts_drain_frames counter (Arc<AtomicUsize>) and TTS_DRAIN_GRACE_FRAMES = 10 (200 ms).
When the TTS queue is empty while primed, the mixer now outputs silence for up to 10 consecutive frames before resetting tts_primed. This absorbs brief inter-chunk gaps without re-triggering the pre-buffer.
flush() also resets tts_drain_frames.

Browser fix (`src/web/playground.html`)

PlaybackProcessor now tracks underrunSamples and only re-enters buffering mode after 100 ms (sampleRate * 0.10) of continuous underrun. Brief underruns produce silent output but keep playback running, so audio resumes immediately when the next WebSocket frame arrives.

2026-10-14 — Prevent overlapping TTS streams (double-Speak fix)

When the user clicked Speak twice in quick succession, two TTS synthesis tasks ran concurrently and both pushed PCM into the same mixer queue. This produced overlapping/garbled audio and roughly doubled the total duration (one observed run produced 83 s of audio for a ~42 s text).

Server fix (`src/orchestrator/turn_manager.rs`)

speak() now calls self.mixer.flush() immediately after cancel_current_tts(). This clears any remaining PCM from the previous utterance so the new one starts from a clean queue.

Client fix (`src/web/playground.html`)

The Speak button is now disabled (shows "Speaking...") as soon as a request is sent.
A 1.5 s idle timer re-enables it once the server stops sending audio frames, preventing accidental double submissions.

2026-10-14 — Jitter buffer fixes (no more audio drop-outs)

Fish Audio Drama 3 streams PCM in unevenly-spaced chunks, which caused brief audible drop-outs (silence gaps) during playback. Fixed with a double jitter buffer.

Server-side TTS pre-buffer (`src/audio/mixer.rs`)

Added TTS_PREBUFFER_SAMPLES ≈ 200 ms of audio (44.1 kHz × 0.20).
The mixer now tracks a tts_primed flag (Arc<AtomicBool>). TTS samples are not consumed until the TTS queue holds at least the pre-buffer amount.
During the priming window only the filler ("uhm") backchannel is emitted, so the user hears an immediate response while TTS audio builds up.
When the TTS queue empties (stream exhausted), tts_primed resets to false for the next request. flush() also resets it (barge-in).
AudioMixer::pull_frame now uses tts_queue.len() instead of is_empty() for priming decisions.

Browser-side playback jitter buffer (`src/web/playground.html`)

The PlaybackProcessor AudioWorklet now starts in a buffering state and only begins playing once the ring buffer holds ≥ 200 ms of samples.
On underrun (buffer drained mid-playback) it returns to buffering to avoid stuttering, then resumes once enough audio has re-accumulated.
Added available() helper to compute the fill level of the ring buffer.

Build / verification

cargo build — 0 warnings.
cargo test — 4/4 passed.

2026-10-14 — Voice cloning + 44.1 kHz native output

Sample rate aligned to Fish Audio native output

Changed OUTPUT_SAMPLE_RATE from 24 000 Hz to 44 100 Hz to match Fish Audio Drama 3's native PCM output.
Removed the TTS anti-aliasing resampler from sglang_client.rs — Fish Audio PCM samples are pushed directly into the mixer queue. This eliminates any resampling artifacts ("metallic" sound) and pitch/speed mismatches.
The browser AudioContext now runs at 44 100 Hz, so playback is sample-accurate.
OUTPUT_FRAME_SAMPLES is now 882 (44.1 kHz × 20 ms), OUTPUT_FRAME_BYTES = 1764.
Mic → STT path: browser capture resamples to 44.1 kHz, server resamples 44.1 kHz → 16 kHz for Parakeet (linear interpolation, STT-only path so aliasing is not a concern).

Voice cloning (reference audio)

Added support for Fish Audio's voice-cloning references field.

Config (env vars, applied at startup):

Env var	Description
`TTS_REFERENCE_AUDIO`	Path to a WAV file on the Fish Audio server filesystem.
`TTS_REFERENCE_TEXT`	Reference transcript. If it points to an existing `.txt` file, its contents are loaded; otherwise the raw string is used. If unset, the server looks for a `<audio-stem>.txt` sibling of the WAV.

Default: solis.wav + auto-loaded solis.txt from /root/pipecat-voice/voice_profiles/.

Runtime override (playground → server WebSocket text messages):

SET_REFERENCE:<path>|<text> — set or change the reference. <text> may be empty to re-use the sibling .txt.
CLEAR_REFERENCE — disable voice cloning (use the model's default voice).

Files changed:

src/config.rs — new tts_reference_audio / tts_reference_text fields, Config::tts_reference() resolver with sibling-.txt fallback, new env-var parsing.
src/tts/sglang_client.rs — Reference struct, references field in SpeechRequest (omitted when None), runtime set_reference() backed by a parking_lot::RwLock<Option<Reference>>.
src/orchestrator/turn_manager.rs — forwards set_reference() to the TTS client.
src/ws/session.rs — handles SET_REFERENCE: / CLEAR_REFERENCE control messages.
src/web/playground.html — new "Voice Cloning" card with WAV path + transcript inputs and Set/Clear buttons; reference status indicator.

Build / verification

cargo build — 0 warnings.
cargo test — 4/4 passed (tokenizer).
Direct Fish Audio references call confirmed HTTP 200 + x-sample-rate: 44100.
End-to-end WebSocket test: SET_REFERENCE followed by a text-to-speak request streams 44.1 kHz PCM back to the client.

106 KiB Raw Blame History Unescape Escape

Audio Agent Engine — Updates

2026-10-17 — Feat: custom voice accepts an uploaded audio file + typed transcript

Added

Notes

2026-10-17 — Fix: short utterances ("uno", "due") dropped by the STT gate

Fixed

Notes

2026-10-17 — Fix: too many utterances dropped by the STT gate

Fixed

Notes

2026-10-17 — Fix: custom voice still louder than the default voice

Fixed

Notes

2026-10-17 — Fix: custom voice louder than the default voice

Fixed

Notes

2026-10-17 — Post-processing for the custom voice (level match + cleanup)

Added

Notes

2026-10-17 — Custom voice builder (/customvoice) + Mivocero side menu

Added

Notes

2026-10-17 — Tool calling survives a mis-matched server-side tool parser

Fixed

Notes

2026-10-17 — Longer end-of-utterance window (VAD_MIN_SILENCE_MS 300 → 700 ms)

Changed

Notes

2026-10-17 — Mivocero: barge-in indicator + karaoke on the user bubbles

Added

Changed

2026-10-17 — VAD barge-in gated on audible output (no more silent cancelled replies)

Added

Changed

Notes

2026-10-17 — Filler fade-out on reply start (no more overlap with the answer)

Changed

Notes

2026-10-17 — Utterance gate: filter spurious VAD turns before STT

Added

Configuration

Notes

2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)

Added

Notes

2026-10-17 — Per-session audit log (structured JSONL)

Added

Changed

Notes

2026-10-17 — Cue emission indicators in the Mivocero demo

Added

Notes

2026-10-17 — Per-language subdirectory for every clip bank (base included)

Changed

Notes

2026-10-17 — Multilingual clip banks (backchannel, filler & start filler)

Added

Changed

Why

Notes

2026-10-16 — Mivocero: auto-disconnect an idle session after 2 minutes

Added

Changed

Why

Notes

2026-10-16 — Tool calling: configurable HTTP tools for the LLM

Added

Changed

Why

Notes

2026-10-16 — External audio channel: inject a URL / radio / YouTube source

Added

Changed

Why

Notes

2026-10-16 — Drama 3 de-emphasis: sampling knobs + neutral style tag

Added

Changed

Why

106 KiB

Raw Blame History

2026-10-17 — Custom voice builder (`/customvoice`) + Mivocero side menu

2026-10-17 — Longer end-of-utterance window (`VAD_MIN_SILENCE_MS` 300 → 700 ms)

2026-10-16 — Backchannel becomes a random phrase bank driven by `BACKCHANNEL_TEXTS`

2026-10-16 — Start filler becomes conditional: `START_FILLER_GO_DELAY` (1200 ms)

2026-10-16 — Start filler: random phrase bank driven by `START_FILLER_TEXTS`

2026-10-16 — Fixed micro-gaps at the start of each sentence → `TTS_PREBUFFER_MS` 170 → 550 ms, `LLM_STREAM_CHUNK_CHARS` 120 → 240

2026-10-16 — Drama-3 vocoder retuned for seam quality → stride `4` / followup `8` / overlap `16`; `TTS_PREBUFFER_MS` 450 → 170 ms

2026-10-16 — End-to-end validation → `TTS_PREBUFFER_MS` 700 → 450 ms; Drama-3 startup documented

2026-10-16 — Finer TTS delivery granularity (Drama-3 vocoder window) → `TTS_PREBUFFER_MS` 700 ms

2026-10-16 — Resilient mixer resume + `TTS_PREBUFFER_MS` default 900 ms