# Audio Agent Engine — Updates This file documents all notable changes to the audio-agent-engine project. Entries are written in English and ordered newest-first. --- ## 2026-10-16 — Mivocero: karaoke word highlighting on the assistant reply ### Changed - **The `/mivocero` client now reveals the assistant reply in karaoke style** ([`src/web/mivocero.html`](src/web/mivocero.html)) instead of appending plain text. Each word is a ``; words light up as they are *spoken*. The highlight is driven by the **audio playhead**: the playback AudioWorklet now counts the samples it has actually emitted and reports them (`played`) on its existing per-block message. The client maps that count onto a character offset with an estimated speaking rate (`KARAOKE_CPS`, ~15 chars/s) and marks each word `done` / `now` / pending. - The old blinking-caret "pending" style was removed; the whole reply is shown at once (dim) and progressively illuminated. - A safety net finishes the highlight: once `assistant_end` has been received and playback has been idle for 600 ms *after it had actually started*, any word still pending is marked done, so a reply always ends fully lit. New turns and interrupts cancel the loop. ### Notes - The mapping from audio to text is inherently approximate (the server streams text and PCM on the same socket without per-word timing), so `KARAOKE_CPS` is a tunable estimate for the visual effect only — it never touches the audio. - `played` counts only real samples, so the playback jitter buffer's silence and any underrun pauses do not advance the highlight. --- ## 2026-10-16 — End-to-end validation → `TTS_PREBUFFER_MS` 700 → 450 ms; Drama-3 startup documented ### Changed - **`TTS_PREBUFFER_MS` lowered `700` → `450`** (`.env`, `.env.example`, [`src/config.rs`](src/config.rs)). With the vocoder ramp `"[10,12,16]"` the *first* window is 10 codec frames = 0.464 s, so `450 ms` sits just below it and playback now starts on that **first** window instead of waiting for the second one that `700 ms` required. - **README:** new *"Starting the Drama 3 backend (SGLang-Omni)"* subsection under the `fish` plugin, with the exact `sgl-omni serve` command, the **pm2** workflow, the meaning of the two `--vocoder.factory.*` flags and the two CLI gotchas (no `stages.` prefix; the ramp tuple passed as a JSON string). ### Why - An end-to-end browser test (`RUST_LOG=…,audio_agent_engine::tts=debug,…`) on the live engine confirmed the narrower windows: bursts of 10/12/16 frames (0.464 / 0.557 / 0.743 s) instead of a single 2.97 s block, `prebuffer=30870` samples (= 700 ms) and time-to-first-audio ≈ 1.1 s from dispatch (~0.64 s of which is the server's TTFB). Barge-in fired correctly on three interrupts. - Three **tail underruns** were still observed at the end of short replies: e.g. 2.46 s of audio produced in ~2.6 s (≈1.07× real time including the fixed ~0.64 s TTFB), so the queue drained just before the last words. The mixer recovered via resilient resume, but the gap is audible. `450 ms` targets the first 0.464 s window and trades a little more tail risk for a markedly earlier start. ### Notes - The residual deficit is dominated by the server's **fixed ~0.64 s time-to-first-byte**, which no pre-buffer value can remove. Going lower would need a faster first token (model / preprocessing), not a smaller stride. --- ## 2026-10-16 — Finer TTS delivery granularity (Drama-3 vocoder window) → `TTS_PREBUFFER_MS` 700 ms ### Changed - **The SGLang Drama-3 server now emits audio in ~0.74 s windows instead of ~2.97 s bursts.** Two flags were added to the server launcher (`/root/drama3/start-drama3.sh`, managed by `pm2` as `start-drama3`): `--vocoder.factory.stream_followup_stride 16` and `--vocoder.factory.stream_followup_stride_ramp "[10,12,16]"` — upstream defaults were `64` and `(10,12,32)`. - **`TTS_PREBUFFER_MS` default lowered `900` → `700`** (`.env`, `.env.example`, [`src/config.rs`](src/config.rs)). 700 ms is deliberately just under one 0.743 s window, so playback starts on the *first* burst rather than waiting for a second one. ### Why - Drama-3's streaming granularity is a **vocoder** setting, not an engine one: [`Drama3VocoderFactoryArgs`](../drama3/sglang-omni/sglang_omni/models/fishaudio_drama3/config.py) buffers `stream_followup_stride` codec frames before decoding and emitting, and `frame_length = hop_length * 4 = 2048` samples (46.4 ms at 44.1 kHz). The upstream `64` therefore means a 131 072-sample burst every ~2.7 s. - Measured against the live endpoint (single, uncontended request, PCM stream): | | burst | period | ratio | first audio | |---|---|---|---|---| | before (`stride` 64) | 131 072 samples / 2.97 s | 2.70–2.78 s | 1.08× | 0.96 s | | after (`stride` 16) | 32 768 samples / 0.743 s | 0.69 s (σ ≈ 0.01 s) | 1.04× | 1.34 s | - The finer window is what makes the small pre-buffer viable: the producer still runs slightly faster than real time (1.04×), but it now tops the queue up every 0.69 s instead of every ~2.7 s, so arrival jitter is bounded by one window and no longer forces a multi-second buffer. - Prefix caching (RadixAttention) was evaluated and **rejected**: the client already sends a constant prefix (`model` / `voice` / `references`), yet an A/B on the endpoint showed no TTFB difference (ref 0.62–0.64 s, no-ref 0.52 s) — that start-up is decode/vocoder, not prompt prefill. ### Notes - Each window re-decodes `stream_overlap_tokens` (32) frames of context, so a smaller stride costs more vocoder compute: 96 frames decoded per 64 emitted becomes 48 per 16. The measured ratio moved 1.08× → 1.04×, still above real time; `stride` 8 (0.371 s windows, 40 frames decoded per 8 emitted) is the next step down and should only be tried with the ratio re-measured. - The window lengths feed the vocoder's CUDA-graph capture set, so changing them requires a server restart. - `stream_stride` (first window) and `stream_followup_stride_ramp` (the windows after it) were left near their defaults — only the steady-state value drives the granularity of a long reply. - (Superseded: the `TTS_PREBUFFER_MS` default is now `450`, targeting the ramp's 0.464 s *first* window — see the entry above.) --- ## 2026-10-16 — Resilient mixer resume + `TTS_PREBUFFER_MS` default 900 ms ### Changed - **The mixer no longer re-fills the whole jitter buffer after a mid-reply underrun.** Previously, when the TTS queue emptied for more than the 200 ms drain grace, [`pull_frame`](src/audio/mixer.rs) cleared `tts_primed` *and* `tts_stream_end`, then required a full `tts_prebuffer_samples` refill before playback resumed — adding a second, self-inflicted pause on top of the real underrun. It also left the mixer "primed" between turns because `mark_stream_start` never re-armed `tts_primed`. Now: - the full pre-buffer is awaited **only at the first playback of a turn**; once playback has started, an underrun pauses and resumes as soon as *any* audio is queued again (`need = 1`); - `tts_stream_end` is no longer cleared on drain, so a genuine end-of-stream is still honoured; - `mark_stream_start` re-arms `tts_primed`, `tts_playback_started` and `tts_drain_frames` at the start of every turn. - **`TTS_PREBUFFER_MS` default raised `400` → `900`** (`.env`, `.env.example`, [`src/config.rs`](src/config.rs)). With `400` the reply was still prone to drop-outs on long answers. ### Why - Direct measurement of the SGLang Drama-3 endpoint (a single, uncontended request) showed it delivers **~1.08× real time, in bursts every ~2.7 s** (≈131 072 samples / 2.97 s per burst). Because the producer barely outruns the consumer and delivers in bursts, gapless playback requires a jitter buffer of roughly the burst period (~3 s); smaller buffers show a short real drop-out (pure silence) on long replies. - The mixer fix removes the *avoidable* part of the pause (the full refill after an underrun) and recovers at the first byte, which is what makes the remaining gap equal to the backend stall rather than stall + refill. - Default `900` is a deliberate latency-first choice: it keeps time-to-first-audio low (~1.8 s) and accepts a rare short drop-out on very long replies. Raising `TTS_PREBUFFER_MS` towards ~3000 trades that latency back for gaplessness. (Superseded: the default is now `700`, once the server's vocoder window was narrowed to ~0.74 s — see the entry above.) ### Notes - The residual drop-out is a property of the TTS backend (~1× real time), not of the mixer: no buffering smaller than the backend's burst period can bridge a delivery stall. - Relay diagnostics at `debug` level in [`run_llm_turn`](src/orchestrator/turn_manager.rs) log per-chunk relay timing (`TTS pipeline ...`) to help localise seams. --- ## 2026-10-16 — Pipelined TTS: chunks are synthesized ahead of playback ### Changed - **The TTS sink no longer waits for one chunk's stream to finish before requesting the next.** Previously [`run_llm_turn`](src/orchestrator/turn_manager.rs) looped `while let Some(chunk) = rx.recv() { tts.synthesize(chunk).await }`, so a chunk's synthesis only started after the previous chunk had finished streaming end-to-end. Because the TTS engine needs ~0.6 s to emit its first audio byte, every chunk boundary left a gap roughly that long, which the old 1.5 s pre-buffer used to hide. - The sink is now split into two pieces: - a **collector** that spawns one synthesis task per chunk as soon as the LLM produces it (bounded by a semaphore), each streaming into its own staging buffer; - a **relay** that forwards those staging buffers into the mixer queue **strictly in submission order**, so playback stays in order and gapless. A chunk's synthesis therefore overlaps the previous chunk's playback, hiding the per-request start-up latency. ### Added - **New `TTS_PIPELINE_DEPTH` configuration (default `3`)** in [`src/config.rs`](src/config.rs): how many TTS requests may be in flight at once. `1` restores the previous strictly serial behaviour. - `SampleRingBuffer::remaining_capacity()` ([`src/audio/ring_buffer.rs`](src/audio/ring_buffer.rs)) so the relay never pushes past the mixer queue's free space — queued (i.e. currently playing) audio is never dropped to make room. ### Why - Direct measurement against the SGLang Drama-3 endpoint showed a **fixed ~0.62 s start-up latency** per request (request → first audio byte), with audio then streamed at roughly real time. Two concurrent requests are **served in parallel** (start-up rises only to ~0.84–1.0 s), so overlapping requests genuinely removes the inter-chunk gaps instead of just moving them. - The gaps were independent of chunk size (120 → 60 chars left the same ~0.8 s hole), which confirmed the cause was serialization, not chunk length. ### Notes - Staging buffers are sized at 30 s of audio per in-flight request, which is comfortably more than any single chunk. - With the pipeline in place the pre-buffer only has to cover arrival jitter rather than the TTS engine's per-request start-up, so it can be kept small. (Superseded: the default was later raised to `900` — see the entry above.) --- ## 2026-10-16 — Mixer TTS pre-buffer is now configurable (default 400 ms) ### Changed - **The TTS jitter buffer moved from a hardcoded 1.5 s constant to the `TTS_PREBUFFER_MS` setting (default `400`).** `AudioMixer` (in [`src/audio/mixer.rs`](src/audio/mixer.rs)) now takes `tts_prebuffer_samples` as a constructor argument instead of reading the removed `TTS_PREBUFFER_SAMPLES` constant, and [`run_session`](src/ws/session.rs) derives it from the config. ### Why - Log analysis of a voice turn (`"Conta da dieci a uno."`) showed the first TTS request left ~1.2 s after the EOU and the first audio bytes arrived ~1.9 s, but playback only started at **+4.2 s**: the mixer had 1.486 s (65 536 samples) queued at +2.8 s, **14 ms short of the 1.5 s threshold**, so it waited for the next Fish chunk, which landed 1.39 s later. The pre-buffer, not the text chunker, dominated the perceived latency. - Fish Audio now delivers near real time, so 1.5 s was far more buffer than needed. 400 ms removes ~1.1 s of time-to-first-audio while still absorbing normal inter-chunk jitter (the 200 ms drain grace is unchanged). ### Notes - Tune from `.env`: raise `TTS_PREBUFFER_MS` if you hear drop-outs, lower it for an even snappier start. - The low-buffer warning threshold now scales with the configured value (`prebuffer / 4`). --- ## 2026-10-16 — TTS starts on the first words, not on the first sentence ### Changed - **The streaming reply is now chunked on a word boundary instead of waiting for sentence punctuation.** `SentenceSplitter` (in [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)) keeps the sentence-terminator rule (`.`, `!`, `?`, `;`, newline) but adds an early flush: the **first** chunk is cut as soon as `LLM_STREAM_FIRST_CHUNK_CHARS` characters are buffered, and every following chunk at `LLM_STREAM_CHUNK_CHARS`. The cut lands on the first whitespace at or past the threshold, so a chunk always holds at least that many characters and never splits a word. Previously the first chunk only left at a `.`/`!`/`?`, so a long opening sentence delayed the first spoken word. ### Added - **New `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`) and `LLM_STREAM_CHUNK_CHARS` (default `120`) configuration** in [`src/config.rs`](src/config.rs). A small first threshold makes the agent start speaking on the first few words; the larger follow-up threshold keeps prosody good once the audio is flowing. Setting either to `0` disables that flush (chunks then end on sentence punctuation only). ### Notes - Verified on the running server: for a chat-box turn the first TTS request is dispatched ~0.5 s after the user text, synthesizing `"Non ho occhi fisici né"` (23 chars), well before the rest of the sentence (`"accesso a telecamere o immagini, quindi non posso vedere nulla …"`) was generated. Before the change the whole opening sentence had to complete first. - The remaining time-to-first-audio is dominated by the mixer's 1.5 s TTS pre-buffer plus the TTS engine's first-chunk latency, not by the chunker. --- ## 2026-10-15 — Filler is now a latency guard: switchable and delay-gated ### Changed - **The end-of-turn filler is no longer played unconditionally.** It exists to mask the wait for the reply, so playing it when the answer is already speaking only gets in the way of the first words. `TurnManager::schedule_filler` (in [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)) starts a `FILLER_DELAY_MS` timer when the turn is dispatched (both the LLM and the direct-TTS path) and queues the clip **only if the response audio has not started by then**. At most one filler per turn. - **New `FILLER_ENABLED` / `FILLER_DELAY_MS` configuration** in [`src/config.rs`](src/config.rs). `FILLER_ENABLED=false` disables the filler entirely (no clip is ever queued); `FILLER_DELAY_MS` (default `2500`) is the latency guard, tunable from `.env`. - The immediate filler pushes in `handle_user_turn` and `speak` were removed in favour of the single scheduled path, so the playground's direct-TTS path follows the same rule. ### Added - **`AudioMixer::tts_playback_started()`** in [`src/audio/mixer.rs`](src/audio/mixer.rs) — set when the TTS pre-buffer is reached (i.e. the response audio actually starts), reset by `mark_stream_start` and `flush`. This is what the filler decision reads. ### Notes - The pending filler timer is bound to the turn's `CancellationToken`, so a barge-in (`interrupt`, VAD onset, a new turn, or disconnect) drops it. - Verified from `.env` on the running server: `FILLER_DELAY_MS=100` → `pushed filler (delayed)`; `FILLER_DELAY_MS=8000` → `response audio already playing — filler skipped`; `FILLER_ENABLED=false` → no filler activity at all. - With the current `LLM_ENABLE_THINKING=false` + Fish Audio setup, a short reply starts playing within 2.5 s, so the filler correctly stays silent; raise `FILLER_DELAY_MS` (e.g. 1000) if you want it to fire more often. --- ## 2026-10-15 — LLM reasoning stage + `/mivocero` speech-to-speech chat client ### Added - **OpenAI-compatible LLM client** — [`src/llm.rs`](src/llm.rs). New `LlmClient` posting to `{LLM_BASE_URL}/chat/completions`, with an `Authorization: Bearer {LLM_API_KEY}` header and a bounded TCP connect timeout (no global request timeout, so SSE streams can stay open). Two entry points, selected at runtime via `LLM_STREAM`: - `chat_stream` — SSE token streaming (`stream: true`), handling `data:` lines and `data: [DONE]`; - `chat` — a single blocking request. - **Chain-of-thought stripping** — reasoning models (Qwen3, …) may emit their plan in a separate `reasoning_content` delta (vLLM) or inline between think markers (llama.cpp and others). `reasoning_content` is ignored and inline think blocks are removed by an incremental `ThinkStripper` that is stable across delta boundaries (it withholds a tail the size of the opening marker so a marker split between two deltas is still detected). Applied to both the chat box text and the synthesized audio. - **LLM stage in [`TurnManager`](src/orchestrator/turn_manager.rs)** — the user turn (STT transcript or chat-box text) is sent to the model and the reply is spoken by the TTS. A rolling history window (`LLM_HISTORY_MESSAGES`) is kept per connection and prefixed with `LLM_SYSTEM_PROMPT`. A `SentenceSplitter` chunks the streaming reply on `.`, `!`, `?`, `;` and newlines so the TTS starts on the first sentence (`SENTENCE_SOFT_MAX` = 220 chars forces a split on whitespace for punctuation-free runs). - **Barge-in covers the LLM request** — an interrupt (client JSON, VAD onset or disconnect) cancels the in-flight HTTP request through a `CancellationToken` in the `reqwest` `tokio::select!`, cancels the TTS tasks and flushes the mixer. - **Bidirectional text protocol on `/ws`** — client → server `{"type":"user_text","text":…}` and `{"type":"interrupt"}`; server → client `{"type":"user","text":…}`, `{"type":"assistant_delta","text":…}`, `{"type":"assistant_end"}` and `{"type":"error","text":…}`. Legacy text commands (`INTERRUPT`, `SET_REFERENCE:`, `CLEAR_REFERENCE`, raw text → direct TTS) are unchanged, so the playground keeps working. - **`/mivocero` client** — [`src/web/mivocero.html`](src/web/mivocero.html), embedded via [`src/web/mod.rs`](src/web/mod.rs) and served by [`src/main.rs`](src/main.rs). A speech-to-speech example agent with a microphone (same AudioWorklet capture/playback path as the playground), a chat box that renders the user transcript and the **streaming** assistant reply, a text composer and an interrupt button. - **`LLM_*` configuration** — `Config` fields and `read_llm_env()` in [`src/config.rs`](src/config.rs): `LLM_ENABLED`, `LLM_BASE_URL`, `LLM_API_KEY`, `LLM_MODEL`, `LLM_MAX_TOKENS`, `LLM_TEMPERATURE`, `LLM_STREAM`, `LLM_HISTORY_MESSAGES`, `LLM_ENABLE_THINKING`, `LLM_STRIP_REASONING`, `LLM_DIRECT_ANSWERS`, `LLM_SYSTEM_PROMPT`. `LLM_ENABLED` defaults to `true` when both `LLM_BASE_URL` and `LLM_MODEL` are set. - **Three independent chain-of-thought controls** (each switchable from `.env`) — a single mechanism does not cover every deployment: 1. `LLM_ENABLE_THINKING` — `LlmClient` sends `chat_template_kwargs: {"enable_thinking": }` when the variable is set, turning Qwen3-style thinking off/on server-side (vLLM/SGLang). Unset omits the vendor-specific field so strict OpenAI-compatible servers keep accepting the body. 2. `LLM_STRIP_REASONING` (default `true`) — `reasoning_content` deltas are dropped and inline think blocks removed; `false` passes the raw output (both fields) through. 3. `LLM_DIRECT_ANSWERS` (default `true`) — appends [`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt. Needed when the server runs **no** reasoning parser: the chain-of-thought is then written straight into `content` as ordinary prose and is neither `reasoning_content` nor tagged, so filtering cannot catch it. ### Fixed - **Chat-box turns blocked the receive loop** — `handle_control_message` in [`src/ws/session.rs`](src/ws/session.rs) awaited `TurnManager::on_user_text` inline, so while a reply was being generated the session could not read any further client message: an `interrupt` sent during generation was only honoured once the turn had finished. Text turns now run in their own task, and the interrupt cancels the reply immediately (verified: 0 deltas emitted after the interrupt, session still usable for the next turn). - **SSE cancellation latency** — `LlmClient::chat_stream` now races the next stream chunk against the `CancellationToken`, so a barge-in drops the HTTP response instead of waiting for the model's next token. ### Notes - Without an LLM the pipeline degrades to a plain STT → TTS echo (`LLM_ENABLED=false`). - `src/orchestrator/turn_manager.rs` gained unit tests for sentence splitting (terminators, split across deltas, long punctuation-free runs) and `src/llm.rs` for think-block stripping and for the opt-in thinking field — `cargo test` passes (13 passed, 1 ignored). - Verified against `Qwen/Qwen3.6-35B-A3B` on a local vLLM (`http://10.8.0.4:8000/v1`): without `LLM_ENABLE_THINKING=false` the model narrates "Here's a thinking process: 1. **Analyze User Input** …" directly in `content`, which the TTS reads aloud; with it, the reply is the answer only. --- ## 2026-10-15 — Startup clip bootstrap (auto-generate filler / backchannel) ### Added - **Startup asset bootstrap** — [`src/assets.rs`](src/assets.rs). On boot the server checks every expected filler / backchannel clip and, if missing, **synthesizes it with the configured TTS engine** and writes it as `*.raw` (16-bit LE mono PCM at 44.1 kHz) into the target directory (created if needed). Existing clips are never overwritten, so a fresh checkout works with no manual generation step and a customized/partial bank is preserved. - Default clip definitions: [`assets::FILLER_CLIPS`](src/assets.rs) (`uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya`) and [`assets::BACKCHANNEL_CLIPS`](src/assets.rs) (`mmhmm`, `aja`, `claro`, `yaveo`). Backchannel clips are generated only when `BACKCHANNEL_ENABLED=true`. - Wired in [`src/main.rs`](src/main.rs) **before** the banks are loaded; the TTS engine is now created earlier so it can be reused for generation. ### Notes - Generation is best-effort and never blocks startup: a synthesis failure logs a warning per clip and the server still starts (that bank stays silent). - The scratch capture buffer during generation is a 30 s [`SampleRingBuffer`](src/audio/ring_buffer.rs) drained after synthesis. --- ## 2026-10-15 — Filler bank (round-robin end-of-utterance fillers) ### Added - **Filler bank** — `assets/fillers/*.raw`. The end-of-utterance filler is no longer a single hard-coded clip; [`TurnManager`](src/orchestrator/turn_manager.rs) now selects clips **round-robin** from the bank via [`FillerBank::nth`](src/orchestrator/turn_manager.rs) (a per-turn cursor, `TurnManager::next_filler`). With a single clip the behaviour is unchanged. - Clips synthesized with the cloned reference voice (Spanish, 44.1 kHz, 16-bit LE mono PCM): `uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya` (~0.1–0.9 s). ### Notes - `FillerBank::get` is kept for reference/debugging but is no longer on the hot path (`#[allow(dead_code)]`). --- ## 2026-10-15 — During-utterance backchannel ### Added - **Backchannel subsystem** — [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs). Emits short acknowledgments ("mm-hmm", "ajá", "claro") *while the user is speaking*, at a low fixed gain, to signal that the agent is listening without taking the turn. Distinct from the end-of-utterance **filler** injected by the [`TurnManager`](src/orchestrator/turn_manager.rs). - **`BackchannelScheduler`** — per-session decision logic fed by the VAD: `on_speech_start` on `VadEvent::SpeechStart`, `on_speech_end` on `SpeechEnd`, and `maybe_emit()` on every frame while speaking. A cue is emitted only when the user has spoken for at least `BACKCHANNEL_MIN_SPEECH_MS`, at least `BACKCHANNEL_INTERVAL_MS` elapsed since the last cue, fewer than `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn, and the agent is **not** speaking (`!TurnManager::is_speaking()`). - **Preloaded clip bank** — `assets/backchannel/*.raw` (16-bit LE mono PCM at 44.1 kHz), synthesized with the cloned reference voice, loaded into a shared [`FillerBank`](src/orchestrator/turn_manager.rs) and played round-robin. - **Third mixer channel** — [`AudioMixer`](src/audio/mixer.rs) gained a dedicated `backchannel_queue` and a fixed `backchannel_gain`. The mix is now `tts + (filler × filler_gain) + (backchannel × backchannel_gain)`; the queue is cleared by `flush()` (barge-in). - **`FillerBank` generalization** — one type now serves both the filler and the backchannel banks (`load(dir, label)`, round-robin `nth(index)`, `is_empty()`); the default filler lookup ("uhm") is unchanged. - **`VadDetector::in_speech()`** — exposes whether the detector is inside a confirmed utterance, used to gate backchannel emission. - **New `.env` knobs** (documented in [`.env.example`](.env.example)): `BACKCHANNEL_ENABLED`, `BACKCHANNEL_DIR`, `BACKCHANNEL_GAIN`, `BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS`, `BACKCHANNEL_MAX_PER_TURN`. ### Notes - `BACKCHANNEL_GAIN` is clamped to `0..1` and `BACKCHANNEL_INTERVAL_MS` is clamped to `>= BACKCHANNEL_MIN_SPEECH_MS` at load time. - Wiring lives in [`src/ws/session.rs`](src/ws/session.rs) (mic/VAD task); the bank is created once in [`src/main.rs`](src/main.rs) and shared via `AppState`. --- ## 2026-10-15 — Server-side VAD (Silero V5, ONNX) ### Added - **VAD subsystem** — [`src/vad/mod.rs`](src/vad/mod.rs) + [`src/vad/silero.rs`](src/vad/silero.rs). Runs **Silero VAD v5** via ONNX Runtime on the same 16 kHz mono stream that feeds the STT engine. The model consumes 32 ms windows (512 samples) with a 64-sample context and a recurrent `state [2,1,128]`; the graph shapes and the `sr` input rank are introspected at load time. - **Onset/offset state machine** with hysteresis (`VAD_THRESHOLD` / `VAD_NEG_THRESHOLD`) emitting three events: `SpeechOnset` (fast, for barge-in), `SpeechStart`, and `SpeechEnd` (carrying the captured PCM plus a pre-roll pad so the first phoneme is not clipped). - **Fast server-side barge-in** — a `SpeechOnset` interrupts the agent when it is currently speaking (`TurnManager::is_speaking`), cancelling the in-flight TTS and flushing the mixer. - **VAD-driven STT segmentation** — `SttWorker::push_utterance` receives whole utterances instead of raw frames, so the STT no longer needs its own RMS segmentation (`VAD_SEGMENT_STT=true`, default). - **Shared ONNX environment** — [`src/onnx.rs`](src/onnx.rs). ONNX Runtime allows a single environment per process, so the STT and VAD pipelines now share a lazily created `Environment` (cheap `Arc` clones) instead of each building one. - **New `.env` knobs** (all documented in [`.env.example`](.env.example)): `VAD_ENABLED`, `VAD_MODEL_PATH`, `VAD_THRESHOLD`, `VAD_NEG_THRESHOLD`, `VAD_MIN_SPEECH_MS`, `VAD_MIN_SILENCE_MS`, `VAD_SPEECH_PAD_MS`, `VAD_BARGE_IN_MIN_SPEECH_MS`, `VAD_SEGMENT_STT`, `VAD_BARGE_IN`, `VAD_DEBUG`. - **`AudioMixer::is_active`** / `tts_stream_active` flag tracking whether a TTS stream is in flight, independent of the queue length, so barge-in can gate on "the agent is speaking". ### Fixed - Duplicate analysis window appended to the utterance at speech start (the pre-roll ring already contained it) — the utterance is now seeded from the pre-roll ring only. - `server.sh` now also appends `audio_agent_engine::vad=debug` to `RUST_LOG` (mirroring the earlier TTS fix), so VAD debug logs are never filtered out. ### Verified `models/silero_vad.onnx` is loaded at startup (`state_shape=[2,1,128]`, `sr_rank=0`). A live WSS session streaming 44.1 kHz speech produced, in `run/server.log`: ```text INFO ws::session: VAD active for this session threshold=0.5 min_speech_ms=250 min_silence_ms=300 … DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999" DEBUG vad::silero: VAD speech start speech_ms=256 prob="1.000" preroll_samples=4992 DEBUG vad::silero: VAD speech end duration_ms=1144 samples=18304 avg_prob="0.609" INFO ws::session: VAD utterance → STT duration_ms=1144 samples=18304 INFO orchestrator::turn_manager: EOU detected tokens=5 text=Me escucha. ``` Talking over the agent (after a text TTS prompt) produced: ```text DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999" INFO ws::session: VAD barge-in — interrupting agent prob="0.999" INFO orchestrator::turn_manager: interrupt received, flushing audio ``` --- ## 2026-10-15 — TTS DEBUG logs actually reach `run/server.log` ### Problem The `debug!` diagnostics added to the TTS subsystem never appeared in the log, even though the code and the `DEFAULT_LOG_FILTER` fallback were in place. Root cause: the parent shell (IDE) already exports `RUST_LOG=info`, so `EnvFilter::try_from_default_env()` succeeded and the `info,audio_agent_engine::tts=debug` fallback was never used. In `server.sh` the same happened — `${RUST_LOG:-info,audio_agent_engine::tts=debug}` never expanded the default because `RUST_LOG` was already set. ### Fixed - [`server.sh`](server.sh) now **appends** the `audio_agent_engine::tts=debug` directive to whatever `RUST_LOG` it inherits (unless that target is already mentioned), so the TTS debug logs cannot be silently filtered out. `start` logs the effective filter. ### Verified A live WSS session (`Hola, …`) that is kept open for the whole synthesis now emits, in `run/server.log`: ```text DEBUG audio_agent_engine::tts::fish_audio: TTS request dispatched endpoint=… has_reference=true text=… DEBUG audio_agent_engine::tts::fish_audio: TTS time-to-first-byte ttfb_ms=0 bytes=8186 DEBUG audio_agent_engine::tts::fish_audio: TTS chunk received chunk=1 bytes=8186 samples=4093 queue_len=4093 DEBUG audio_agent_engine::tts::fish_audio: TTS stream summary chunk_count=9 total_samples=180224 audio_s="4.09" elapsed_ms=3347 ``` > Note: closing the WebSocket mid-stream triggers `turn_manager.interrupt()` > (barge-in), which cancels the TTS stream before any chunk arrives — so only > `TTS request dispatched` is logged in that case. --- ## 2026-10-15 — Optional HTTPS/WSS + clearer microphone diagnostics ### Problem Clicking "Connect & Start Mic" in the playground failed with a generic `mic error`, with no WebSocket connection ever reaching the server. Browsers only expose `navigator.mediaDevices.getUserMedia` in a **secure context** (HTTPS or `localhost`); over plain HTTP on any other origin the API is simply absent, so `startAudio()` threw before the socket was created. ### Added - **Optional TLS termination** — [`src/tls.rs`](src/tls.rs). Set both `TLS_CERT` and `TLS_KEY` (PEM) in `.env` and the server serves the same Axum router over HTTPS/WSS via `tokio-rustls` + `hyper` (HTTP/1 upgrades enabled, so WebSocket handshakes work). Without them it stays on plain HTTP. The startup log now says `listening (HTTPS)` / `listening (HTTP)`. - **`./server.sh cert [--force]`** — generates a self-signed certificate under `certs/` whose SAN covers `localhost`, `127.0.0.1`, `::1` and the host's IPs (`EXTRA_SAN` to append more). `start`/`restart` auto-generate it if TLS is enabled but the files are missing; `status` shows the scheme and cert state. ### Changed - **Playground diagnostics** — [`src/web/playground.html`](src/web/playground.html) now checks for a secure context *before* requesting the mic and reports the precise cause (`insecure origin` / `NotAllowedError` / `NotFoundError`) with a remediation hint, instead of a bare `mic error`. - New config fields `Config::tls_cert` / `Config::tls_key` (+ `tls_enabled()`), read from `TLS_CERT` / `TLS_KEY`. ### Notes - No new crates are downloaded: `hyper`, `hyper-util`, `tokio-rustls`, `rustls` and `rustls-pki-types` were already present as transitive deps of `reqwest` (`ring` provider), so they are declared with `default-features = false`. --- ## 2026-10-15 — `server.sh` control script (start / stop / restart) Added [`server.sh`](server.sh), a small bash helper so the server does not have to be managed by hand. ### Usage ```bash ./server.sh start [--build] # start (auto-builds if binary missing) ./server.sh stop # graceful stop (SIGKILL after 10 s) ./server.sh restart [--build] # stop + start ./server.sh status # process / port / HTTP status ./server.sh logs [N] # tail run/server.log (default 50, follow) ./server.sh build # build the release binary only ``` - PID is tracked in `run/server.pid`; logs are appended to `run/server.log`. - `start` waits until the port is accepting connections (the model load takes ~1.5 s) before reporting success. - Overridable env vars: `PROFILE` (`release`|`debug`), `RUST_LOG`, `PORT`. - The script `cd`s to its own directory first, so `.env` and the relative `models/parakeet` path resolve correctly regardless of the caller's cwd. Documented in the README (Quick Start). --- ## 2026-10-15 — Parakeet TDT STT reimplemented in pure Rust (ONNX) The previous STT was a placeholder that ran `argmax` over a single `.onnx` file — it could not actually transcribe. It has been replaced with a complete, dependency-free (no Python/NeMo) implementation of the NVIDIA **Parakeet TDT 0.6B v3** transducer pipeline in [`src/stt/parakeet.rs`](src/stt/parakeet.rs), mirroring the reference in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr). ### Change - **3-stage ONNX pipeline**, all sessions kept alive by the `ort` `Environment`: 1. `nemo128.onnx` — log-mel features `waveforms`/`waveforms_lens` → `features`/`features_lens`. 2. `encoder-model.int8.onnx` (preferred, auto-detected) or `encoder-model.onnx` — `audio_signal`/`length` → `outputs`[1,1024,T']/`encoded_lengths`. 3. `decoder_joint-model.onnx` — greedy **TDT** decoding. - **Runtime graph introspection**: `d_model`, `num_durations`, `state_layers` and `state_hidden` are read from the ONNX I/O shapes instead of being hardcoded. - **Greedy TDT decode loop** (token head + duration head; blank `` = 8192; `max_tokens_per_step` from `config.json`, default 10). Encoder output is re-packed from `[1, d_model, T']` to contiguous `[T', d_model]` frames. - **Internal text decoding**: `vocab.txt` (` `, SentencePiece `▁` → space) plus the reference spacing normalizer (`\A\s|\s\B|(\s)\b`). The `Transcript` struct now carries a `text: String`. - **Config**: `STT_MODEL_PATH` now points to a **directory** (`models/parakeet` by default, was `models/parakeet-tdt.onnx`). `STT_TOKENIZER_PATH` defaults to empty. Updated `.env`, `.env.example` and the README. - **TurnManager**: [`on_eou`](src/orchestrator/turn_manager.rs) now uses `transcript.text` directly, falling back to the external tokenizer only when no text was produced. ### Notes / fixes found during integration - The `decoder_joint` graph expects `targets` and `target_length` as **int32** (not int64); `encoder_outputs` and both state tensors are float. - The int8 encoder (~650 MB, self-contained) is preferred over the fp32 encoder (42 MB + a 2.4 GB `.onnx.data` sidecar) for speed and simpler deployment. ### Verification Opt-in end-to-end test decodes a real 16 kHz mono WAV through the model: ``` $ PARAKEET_TEST_WAV=/tmp/stt_test/solis16k.wav \ cargo test --release -- --ignored test_infer_real_audio --nocapture tokens: 67 | text: "Bienvenidos a esta fascinante exploración del universo. Como locutor profesional, les explicaré hoy cómo las ondas electromagnéticas viajan por el espacio, transformando nuestra comprensión del cosmos." test stt::parakeet::tests::test_infer_real_audio ... ok ``` The transcription matches the reference transcript verbatim (the source clip was trimmed to 12 s). `cargo test` (unit tests) also passes. --- ## 2026-10-15 — Fix: short utterances swallowed by the TTS pre-buffer Short phrases produced **no audio at all**. The mixer only started playing TTS once the queue held `TTS_PREBUFFER_SAMPLES` (1.5 s). If an utterance's total length was below that threshold (e.g. `"Cristóbal Colón"` ≈ 1.11 s / 49152 samples), the pre-buffer was never reached, so `tts_primed` stayed `false` and the queued samples were never drained. ### Change - `src/audio/mixer.rs`: added an `tts_stream_end` flag with `mark_stream_start()` / `mark_stream_end()`. When the stream has ended, the mixer primes as soon as *any* sample is queued (`stream_ended && tts_len > 0`), bypassing the pre-buffer for short utterances. The flag is reset by `flush()`, `mark_stream_start()` and when the queue drains. - `src/orchestrator/turn_manager.rs`: the spawned TTS tasks now call `mixer.mark_stream_start()` before synthesis and `mixer.mark_stream_end()` after it returns — only when the request was **not** cancelled, so barge-in does not falsely signal completion. ### Verification ``` TTS stream complete chunk_count=6 total_samples=49152 duration_s="0.64" TTS pre-buffer reached — starting TTS playback tts_queue_len=49152 prebuffer=66150 stream_ended=true ``` A raw WebSocket client sending `"Cristóbal Colón"` now receives ~1.12 s of non-silent audio (previously the same request yielded only silence frames). --- ## 2026-10-15 — Dual-stack listener (fix `localhost` / `::1` connection refused) The HTTP server bound only to `0.0.0.0` (IPv4). On hosts where `localhost` resolves to `::1` first (confirmed via `getent hosts localhost` → `::1 localhost`), clients and preview/port-forwarding proxies that connect to `localhost` via IPv6 received `connection refused`, surfacing as *"service is unavailable"*. ### Change - `src/main.rs`: the listener now binds to the IPv6 wildcard `SocketAddr::from((Ipv6Addr::UNSPECIFIED, port))` instead of `0.0.0.0`. With `net.ipv6.bindv6only = 0` (Linux default, verified on this host) this is dual-stack: it accepts both IPv6 (`::1` / `localhost`) and IPv4 clients. ### Verification ``` ss -ltnp | grep 8787 → LISTEN *:8787 curl http://[::1]:8787/ → 200 curl http://localhost:8787/ → 200 curl http://127.0.0.1:8787/ → 200 ``` --- ## 2026-10-14 — Modular TTS plugin system (engine-agnostic via `.env`) The TTS layer was hard-coded to Fish Audio / SGLang Drama 3. To support switching engines (e.g. Kokoro, or any future backend) without touching orchestration code, the TTS subsystem is now a pluggable trait selected at runtime via the `TTS_ENGINE` environment variable. ### New `TtsEngine` trait (`src/tts/mod.rs`) ```rust #[async_trait] pub trait TtsEngine: Send + Sync { fn name(&self) -> &str; fn sample_rate(&self) -> u32; async fn synthesize(&self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken) -> Result<()>; fn set_reference(&self, _audio_path: Option, _text: Option) {} // default no-op } ``` - Every plugin streams **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44.1 kHz)** into the shared mixer queue. If an engine's native rate differs, it resamples internally (the mixer runs at a fixed rate). - `synthesize()` must honor the `CancellationToken` for barge-in. - `set_reference()` defaults to a no-op; only engines that support voice cloning override it. ### Factory (`src/tts/mod.rs::create_tts_engine`) ``` TTS_ENGINE=fish → FishAudioTts (Fish Audio / SGLang Drama 3) TTS_ENGINE=kokoro → KokoroTts (stub, implement in src/tts/kokoro.rs) ``` Unknown values panic at startup with a clear message. ### Bundled plugins | Plugin | File | Status | |---|---|---| | `fish` (Fish Audio Drama 3) | `src/tts/fish_audio.rs` | Production-ready (renamed from `sglang_client.rs`) | | `kokoro` | `src/tts/kokoro.rs` | Stub — HTTP streaming call left as `TODO` with a commented example; includes a 24 kHz → 44.1 kHz linear resampler helper | ### `.env` loading - Added `dotenvy` to auto-load a `.env` file at startup (existing env vars take precedence). - Added `.env.example` documenting all env vars including `TTS_ENGINE`, `KOKORO_URL`, `KOKORO_VOICE`. ### Files changed - `src/tts/mod.rs` — new `TtsEngine` trait + `create_tts_engine` factory. - `src/tts/fish_audio.rs` — renamed from `sglang_client.rs`; `FishAudioTts` now implements `TtsEngine`. - `src/tts/kokoro.rs` — new Kokoro plugin stub (example for adding engines). - `src/tts/sglang_client.rs` — deleted (moved to `fish_audio.rs`). - `src/config.rs` — new `tts_engine` field, `TTS_ENGINE` env var (default `fish`). - `src/orchestrator/turn_manager.rs` — `tts: SglangTtsClient` → `tts: Arc`. - `src/ws/handler.rs` — `AppState.tts` → `Arc`. - `src/main.rs` — uses `create_tts_engine(&config)`; loads `.env` via `dotenvy`. - `Cargo.toml` — added `async-trait`, `dotenvy`. - `.env.example` — new file documenting all configuration env vars. ### How to add a new engine 1. Create `src/tts/myengine.rs` and implement `TtsEngine`. 2. Add `pub mod myengine;` to `src/tts/mod.rs`. 3. Register it in the `create_tts_engine` match. 4. Set `TTS_ENGINE=myengine` in `.env`. ### Build / verification - `cargo build` — 0 errors, 2 dead-code warnings (Kokoro stub fields). - `cargo test` — 4/4 passed. - Server starts with `TTS_ENGINE=fish` (default): log shows `selected TTS engine engine="fish"` and voice-cloning reference loaded. --- ## 2026-10-14 — Cancel TTS on WebSocket disconnect When the client disconnected, the in-flight TTS synthesis task continued streaming from Fish Audio even though the WebSocket session was already closed, wasting bandwidth and server resources. ### Fix (`src/ws/session.rs`) - Added `turn_manager.interrupt()` in the session cleanup path (after the receiver ends, before aborting the sender/mic tasks). - This cancels the TTS `CancellationToken`, flushes the mixer queues, and stops the streaming HTTP request immediately. --- ## 2026-10-14 — Fix WebSocket frame timing (Burst ticker) The 20 ms audio sender ticker used `MissedTickBehavior::Skip`, which dropped tick beats whenever the Tokio event loop was busy (e.g. processing a large Fish Audio TTS chunk). Each skipped tick meant a 20 ms gap in frame delivery to the browser, producing audible micro-interruptions even though the server's TTS queue was well-buffered. ### Fix (`src/ws/session.rs`) - Changed `MissedTickBehavior::Skip` → `MissedTickBehavior::Burst`. - Missed ticks now trigger immediate back-to-back sends so the browser receives all due frames. The browser's jitter buffer (500 ms) absorbs the burst without underrunning. --- ## 2026-10-14 — Micro-interruption fix (grace period on TTS drain) Even with the 1.5 s server pre-buffer and 500 ms browser jitter buffer, brief audio gaps could still occur when the TTS queue momentarily hit zero between Fish Audio chunks. The mixer immediately reset its `tts_primed` state, forcing a full 1.5 s re-prime on the next chunk. The browser playback processor did the same, re-entering buffering mode on any underrun. ### Server fix (`src/audio/mixer.rs`) - Added `tts_drain_frames` counter (`Arc`) and `TTS_DRAIN_GRACE_FRAMES = 10` (200 ms). - When the TTS queue is empty while primed, the mixer now outputs silence for up to 10 consecutive frames before resetting `tts_primed`. This absorbs brief inter-chunk gaps without re-triggering the pre-buffer. - `flush()` also resets `tts_drain_frames`. ### Browser fix (`src/web/playground.html`) - `PlaybackProcessor` now tracks `underrunSamples` and only re-enters buffering mode after 100 ms (`sampleRate * 0.10`) of continuous underrun. Brief underruns produce silent output but keep playback running, so audio resumes immediately when the next WebSocket frame arrives. --- ## 2026-10-14 — Prevent overlapping TTS streams (double-Speak fix) When the user clicked **Speak** twice in quick succession, two TTS synthesis tasks ran concurrently and both pushed PCM into the same mixer queue. This produced overlapping/garbled audio and roughly doubled the total duration (one observed run produced 83 s of audio for a ~42 s text). ### Server fix (`src/orchestrator/turn_manager.rs`) - `speak()` now calls `self.mixer.flush()` immediately after `cancel_current_tts()`. This clears any remaining PCM from the previous utterance so the new one starts from a clean queue. ### Client fix (`src/web/playground.html`) - The **Speak** button is now disabled (shows "Speaking...") as soon as a request is sent. - A 1.5 s idle timer re-enables it once the server stops sending audio frames, preventing accidental double submissions. --- ## 2026-10-14 — Jitter buffer fixes (no more audio drop-outs) Fish Audio Drama 3 streams PCM in unevenly-spaced chunks, which caused brief audible drop-outs (silence gaps) during playback. Fixed with a double jitter buffer. ### Server-side TTS pre-buffer (`src/audio/mixer.rs`) - Added `TTS_PREBUFFER_SAMPLES` ≈ 200 ms of audio (44.1 kHz × 0.20). - The mixer now tracks a `tts_primed` flag (`Arc`). TTS samples are **not** consumed until the TTS queue holds at least the pre-buffer amount. - During the priming window only the filler ("uhm") backchannel is emitted, so the user hears an immediate response while TTS audio builds up. - When the TTS queue empties (stream exhausted), `tts_primed` resets to `false` for the next request. `flush()` also resets it (barge-in). - `AudioMixer::pull_frame` now uses `tts_queue.len()` instead of `is_empty()` for priming decisions. ### Browser-side playback jitter buffer (`src/web/playground.html`) - The `PlaybackProcessor` AudioWorklet now starts in a `buffering` state and only begins playing once the ring buffer holds ≥ 200 ms of samples. - On underrun (buffer drained mid-playback) it returns to `buffering` to avoid stuttering, then resumes once enough audio has re-accumulated. - Added `available()` helper to compute the fill level of the ring buffer. ### Build / verification - `cargo build` — 0 warnings. - `cargo test` — 4/4 passed. --- ## 2026-10-14 — Voice cloning + 44.1 kHz native output ### Sample rate aligned to Fish Audio native output - Changed `OUTPUT_SAMPLE_RATE` from **24 000 Hz** to **44 100 Hz** to match Fish Audio Drama 3's native PCM output. - Removed the TTS anti-aliasing resampler from `sglang_client.rs` — Fish Audio PCM samples are pushed directly into the mixer queue. This eliminates any resampling artifacts ("metallic" sound) and pitch/speed mismatches. - The browser `AudioContext` now runs at 44 100 Hz, so playback is sample-accurate. - `OUTPUT_FRAME_SAMPLES` is now 882 (44.1 kHz × 20 ms), `OUTPUT_FRAME_BYTES` = 1764. - Mic → STT path: browser capture resamples to 44.1 kHz, server resamples 44.1 kHz → 16 kHz for Parakeet (linear interpolation, STT-only path so aliasing is not a concern). ### Voice cloning (reference audio) Added support for Fish Audio's voice-cloning `references` field. **Config (env vars, applied at startup):** | Env var | Description | |---|---| | `TTS_REFERENCE_AUDIO` | Path to a WAV file on the Fish Audio server filesystem. | | `TTS_REFERENCE_TEXT` | Reference transcript. If it points to an existing `.txt` file, its contents are loaded; otherwise the raw string is used. If unset, the server looks for a `.txt` sibling of the WAV. | **Default:** `solis.wav` + auto-loaded `solis.txt` from `/root/pipecat-voice/voice_profiles/`. **Runtime override (playground → server WebSocket text messages):** - `SET_REFERENCE:|` — set or change the reference. `` may be empty to re-use the sibling `.txt`. - `CLEAR_REFERENCE` — disable voice cloning (use the model's default voice). **Files changed:** - `src/config.rs` — new `tts_reference_audio` / `tts_reference_text` fields, `Config::tts_reference()` resolver with sibling-`.txt` fallback, new env-var parsing. - `src/tts/sglang_client.rs` — `Reference` struct, `references` field in `SpeechRequest` (omitted when `None`), runtime `set_reference()` backed by a `parking_lot::RwLock>`. - `src/orchestrator/turn_manager.rs` — forwards `set_reference()` to the TTS client. - `src/ws/session.rs` — handles `SET_REFERENCE:` / `CLEAR_REFERENCE` control messages. - `src/web/playground.html` — new "Voice Cloning" card with WAV path + transcript inputs and Set/Clear buttons; reference status indicator. ### Build / verification - `cargo build` — 0 warnings. - `cargo test` — 4/4 passed (tokenizer). - Direct Fish Audio `references` call confirmed HTTP 200 + `x-sample-rate: 44100`. - End-to-end WebSocket test: `SET_REFERENCE` followed by a text-to-speak request streams 44.1 kHz PCM back to the client.