piperust/UPDATES.md

# Audio Agent Engine — Updates

This file documents all notable changes to the audio-agent-engine project.
Entries are written in English and ordered newest-first.

---

## 2026-10-16 — Resilient mixer resume + `TTS_PREBUFFER_MS` default 900 ms

### Changed

- **The mixer no longer re-fills the whole jitter buffer after a mid-reply
  underrun.** Previously, when the TTS queue emptied for more than the 200 ms
  drain grace, [`pull_frame`](src/audio/mixer.rs) cleared `tts_primed` *and*
  `tts_stream_end`, then required a full `tts_prebuffer_samples` refill before
  playback resumed — adding a second, self-inflicted pause on top of the real
  underrun. It also left the mixer "primed" between turns because
  `mark_stream_start` never re-armed `tts_primed`.
  Now:
  - the full pre-buffer is awaited **only at the first playback of a turn**; once
    playback has started, an underrun pauses and resumes as soon as *any* audio
    is queued again (`need = 1`);
  - `tts_stream_end` is no longer cleared on drain, so a genuine end-of-stream is
    still honoured;
  - `mark_stream_start` re-arms `tts_primed`, `tts_playback_started` and
    `tts_drain_frames` at the start of every turn.
- **`TTS_PREBUFFER_MS` default raised `400` → `900`** (`.env`, `.env.example`,
  [`src/config.rs`](src/config.rs)). With `400` the reply was still prone to
  drop-outs on long answers.

### Why

- Direct measurement of the SGLang Drama-3 endpoint (a single, uncontended
  request) showed it delivers **~1.08× real time, in bursts every ~2.7 s**
  (≈131 072 samples / 2.97 s per burst). Because the producer barely outruns the
  consumer and delivers in bursts, gapless playback requires a jitter buffer of
  roughly the burst period (~3 s); smaller buffers show a short real drop-out
  (pure silence) on long replies.
- The mixer fix removes the *avoidable* part of the pause (the full refill after
  an underrun) and recovers at the first byte, which is what makes the remaining
  gap equal to the backend stall rather than stall + refill.
- Default `900` is a deliberate latency-first choice: it keeps time-to-first-audio
  low (~1.8 s) and accepts a rare short drop-out on very long replies. Raising
  `TTS_PREBUFFER_MS` towards ~3000 trades that latency back for gaplessness.

### Notes

- The residual drop-out is a property of the TTS backend (~1× real time), not of
  the mixer: no buffering smaller than the backend's burst period can bridge a
  delivery stall.
- Relay diagnostics at `debug` level in [`run_llm_turn`](src/orchestrator/turn_manager.rs)
  log per-chunk relay timing (`TTS pipeline ...`) to help localise seams.

---

## 2026-10-16 — Pipelined TTS: chunks are synthesized ahead of playback

### Changed

- **The TTS sink no longer waits for one chunk's stream to finish before
  requesting the next.** Previously [`run_llm_turn`](src/orchestrator/turn_manager.rs)
  looped `while let Some(chunk) = rx.recv() { tts.synthesize(chunk).await }`, so
  a chunk's synthesis only started after the previous chunk had finished
  streaming end-to-end. Because the TTS engine needs ~0.6 s to emit its first
  audio byte, every chunk boundary left a gap roughly that long, which the old
  1.5 s pre-buffer used to hide.
- The sink is now split into two pieces:
  - a **collector** that spawns one synthesis task per chunk as soon as the LLM
    produces it (bounded by a semaphore), each streaming into its own staging
    buffer;
  - a **relay** that forwards those staging buffers into the mixer queue
    **strictly in submission order**, so playback stays in order and gapless.
  A chunk's synthesis therefore overlaps the previous chunk's playback, hiding
  the per-request start-up latency.

### Added

- **New `TTS_PIPELINE_DEPTH` configuration (default `3`)** in
  [`src/config.rs`](src/config.rs): how many TTS requests may be in flight at
  once. `1` restores the previous strictly serial behaviour.
- `SampleRingBuffer::remaining_capacity()` ([`src/audio/ring_buffer.rs`](src/audio/ring_buffer.rs))
  so the relay never pushes past the mixer queue's free space — queued (i.e.
  currently playing) audio is never dropped to make room.

### Why

- Direct measurement against the SGLang Drama-3 endpoint showed a **fixed
  ~0.62 s start-up latency** per request (request → first audio byte), with
  audio then streamed at roughly real time. Two concurrent requests are
  **served in parallel** (start-up rises only to ~0.84–1.0 s), so overlapping
  requests genuinely removes the inter-chunk gaps instead of just moving them.
- The gaps were independent of chunk size (120 → 60 chars left the same ~0.8 s
  hole), which confirmed the cause was serialization, not chunk length.

### Notes

- Staging buffers are sized at 30 s of audio per in-flight request, which is
  comfortably more than any single chunk.
- With the pipeline in place the pre-buffer only has to cover arrival jitter
  rather than the TTS engine's per-request start-up, so it can be kept small.
  (Superseded: the default was later raised to `900` — see the entry above.)

---

## 2026-10-16 — Mixer TTS pre-buffer is now configurable (default 400 ms)

### Changed

- **The TTS jitter buffer moved from a hardcoded 1.5 s constant to the
  `TTS_PREBUFFER_MS` setting (default `400`).** `AudioMixer` (in
  [`src/audio/mixer.rs`](src/audio/mixer.rs)) now takes `tts_prebuffer_samples`
  as a constructor argument instead of reading the removed
  `TTS_PREBUFFER_SAMPLES` constant, and [`run_session`](src/ws/session.rs)
  derives it from the config.

### Why

- Log analysis of a voice turn (`"Conta da dieci a uno."`) showed the first TTS
  request left ~1.2 s after the EOU and the first audio bytes arrived ~1.9 s,
  but playback only started at **+4.2 s**: the mixer had 1.486 s (65 536
  samples) queued at +2.8 s, **14 ms short of the 1.5 s threshold**, so it
  waited for the next Fish chunk, which landed 1.39 s later. The pre-buffer,
  not the text chunker, dominated the perceived latency.
- Fish Audio now delivers near real time, so 1.5 s was far more buffer than
  needed. 400 ms removes ~1.1 s of time-to-first-audio while still absorbing
  normal inter-chunk jitter (the 200 ms drain grace is unchanged).

### Notes

- Tune from `.env`: raise `TTS_PREBUFFER_MS` if you hear drop-outs, lower it for
  an even snappier start.
- The low-buffer warning threshold now scales with the configured value
  (`prebuffer / 4`).

---

## 2026-10-16 — TTS starts on the first words, not on the first sentence

### Changed

- **The streaming reply is now chunked on a word boundary instead of waiting
  for sentence punctuation.** `SentenceSplitter` (in
  [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs)) keeps
  the sentence-terminator rule (`.`, `!`, `?`, `;`, newline) but adds an early
  flush: the **first** chunk is cut as soon as
  `LLM_STREAM_FIRST_CHUNK_CHARS` characters are buffered, and every following
  chunk at `LLM_STREAM_CHUNK_CHARS`. The cut lands on the first whitespace at or
  past the threshold, so a chunk always holds at least that many characters and
  never splits a word. Previously the first chunk only left at a `.`/`!`/`?`,
  so a long opening sentence delayed the first spoken word.

### Added

- **New `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`) and
  `LLM_STREAM_CHUNK_CHARS` (default `120`) configuration** in
  [`src/config.rs`](src/config.rs). A small first threshold makes the agent
  start speaking on the first few words; the larger follow-up threshold keeps
  prosody good once the audio is flowing. Setting either to `0` disables that
  flush (chunks then end on sentence punctuation only).

### Notes

- Verified on the running server: for a chat-box turn the first TTS request is
  dispatched ~0.5 s after the user text, synthesizing `"Non ho occhi fisici né"`
  (23 chars), well before the rest of the sentence
  (`"accesso a telecamere o immagini, quindi non posso vedere nulla …"`) was
  generated. Before the change the whole opening sentence had to complete first.
- The remaining time-to-first-audio is dominated by the mixer's 1.5 s TTS
  pre-buffer plus the TTS engine's first-chunk latency, not by the chunker.

---

## 2026-10-15 — Filler is now a latency guard: switchable and delay-gated

### Changed

- **The end-of-turn filler is no longer played unconditionally.** It exists to
  mask the wait for the reply, so playing it when the answer is already speaking
  only gets in the way of the first words. `TurnManager::schedule_filler`
  (in [`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs))
  starts a `FILLER_DELAY_MS` timer when the turn is dispatched (both the LLM and
  the direct-TTS path) and queues the clip **only if the response audio has not
  started by then**. At most one filler per turn.
- **New `FILLER_ENABLED` / `FILLER_DELAY_MS` configuration** in
  [`src/config.rs`](src/config.rs). `FILLER_ENABLED=false` disables the filler
  entirely (no clip is ever queued); `FILLER_DELAY_MS` (default `2500`) is the
  latency guard, tunable from `.env`.
- The immediate filler pushes in `handle_user_turn` and `speak` were removed in
  favour of the single scheduled path, so the playground's direct-TTS path
  follows the same rule.

### Added

- **`AudioMixer::tts_playback_started()`** in
  [`src/audio/mixer.rs`](src/audio/mixer.rs) — set when the TTS pre-buffer is
  reached (i.e. the response audio actually starts), reset by
  `mark_stream_start` and `flush`. This is what the filler decision reads.

### Notes

- The pending filler timer is bound to the turn's `CancellationToken`, so a
  barge-in (`interrupt`, VAD onset, a new turn, or disconnect) drops it.
- Verified from `.env` on the running server: `FILLER_DELAY_MS=100` →
  `pushed filler (delayed)`; `FILLER_DELAY_MS=8000` → `response audio already
  playing — filler skipped`; `FILLER_ENABLED=false` → no filler activity at all.
- With the current `LLM_ENABLE_THINKING=false` + Fish Audio setup, a short reply
  starts playing within 2.5 s, so the filler correctly stays silent; raise
  `FILLER_DELAY_MS` (e.g. 1000) if you want it to fire more often.

---

## 2026-10-15 — LLM reasoning stage + `/mivocero` speech-to-speech chat client

### Added

- **OpenAI-compatible LLM client** — [`src/llm.rs`](src/llm.rs). New
  `LlmClient` posting to `{LLM_BASE_URL}/chat/completions`, with an
  `Authorization: Bearer {LLM_API_KEY}` header and a bounded TCP connect
  timeout (no global request timeout, so SSE streams can stay open). Two
  entry points, selected at runtime via `LLM_STREAM`:
  - `chat_stream` — SSE token streaming (`stream: true`), handling `data:`
    lines and `data: [DONE]`;
  - `chat` — a single blocking request.
- **Chain-of-thought stripping** — reasoning models (Qwen3, …) may emit their
  plan in a separate `reasoning_content` delta (vLLM) or inline between
  think markers (llama.cpp and others). `reasoning_content` is ignored and
  inline think blocks are removed by an incremental `ThinkStripper` that is
  stable across delta boundaries (it withholds a tail the size of the opening
  marker so a marker split between two deltas is still detected). Applied to
  both the chat box text and the synthesized audio.
- **LLM stage in [`TurnManager`](src/orchestrator/turn_manager.rs)** — the user
  turn (STT transcript or chat-box text) is sent to the model and the reply is
  spoken by the TTS. A rolling history window (`LLM_HISTORY_MESSAGES`) is kept
  per connection and prefixed with `LLM_SYSTEM_PROMPT`. A `SentenceSplitter`
  chunks the streaming reply on `.`, `!`, `?`, `;` and newlines so the TTS
  starts on the first sentence (`SENTENCE_SOFT_MAX` = 220 chars forces a split
  on whitespace for punctuation-free runs).
- **Barge-in covers the LLM request** — an interrupt (client JSON, VAD onset or
  disconnect) cancels the in-flight HTTP request through a
  `CancellationToken` in the `reqwest` `tokio::select!`, cancels the TTS tasks
  and flushes the mixer.
- **Bidirectional text protocol on `/ws`** — client → server
  `{"type":"user_text","text":…}` and `{"type":"interrupt"}`; server → client
  `{"type":"user","text":…}`, `{"type":"assistant_delta","text":…}`,
  `{"type":"assistant_end"}` and `{"type":"error","text":…}`. Legacy text
  commands (`INTERRUPT`, `SET_REFERENCE:`, `CLEAR_REFERENCE`, raw text → direct
  TTS) are unchanged, so the playground keeps working.
- **`/mivocero` client** — [`src/web/mivocero.html`](src/web/mivocero.html),
  embedded via [`src/web/mod.rs`](src/web/mod.rs) and served by
  [`src/main.rs`](src/main.rs). A speech-to-speech example agent with a
  microphone (same AudioWorklet capture/playback path as the playground), a
  chat box that renders the user transcript and the **streaming** assistant
  reply, a text composer and an interrupt button.
- **`LLM_*` configuration** — `Config` fields and `read_llm_env()` in
  [`src/config.rs`](src/config.rs): `LLM_ENABLED`, `LLM_BASE_URL`,
  `LLM_API_KEY`, `LLM_MODEL`, `LLM_MAX_TOKENS`, `LLM_TEMPERATURE`,
  `LLM_STREAM`, `LLM_HISTORY_MESSAGES`, `LLM_ENABLE_THINKING`,
  `LLM_STRIP_REASONING`, `LLM_DIRECT_ANSWERS`, `LLM_SYSTEM_PROMPT`.
  `LLM_ENABLED` defaults to `true` when both `LLM_BASE_URL` and `LLM_MODEL`
  are set.
- **Three independent chain-of-thought controls** (each switchable from `.env`)
  — a single mechanism does not cover every deployment:
  1. `LLM_ENABLE_THINKING` — `LlmClient` sends
     `chat_template_kwargs: {"enable_thinking": <bool>}` when the variable is
     set, turning Qwen3-style thinking off/on server-side (vLLM/SGLang). Unset
     omits the vendor-specific field so strict OpenAI-compatible servers keep
     accepting the body.
  2. `LLM_STRIP_REASONING` (default `true`) — `reasoning_content` deltas are
     dropped and inline think blocks removed; `false` passes the raw output
     (both fields) through.
  3. `LLM_DIRECT_ANSWERS` (default `true`) — appends
     [`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt. Needed
     when the server runs **no** reasoning parser: the chain-of-thought is then
     written straight into `content` as ordinary prose and is neither
     `reasoning_content` nor tagged, so filtering cannot catch it.

### Fixed

- **Chat-box turns blocked the receive loop** — `handle_control_message` in
  [`src/ws/session.rs`](src/ws/session.rs) awaited `TurnManager::on_user_text`
  inline, so while a reply was being generated the session could not read any
  further client message: an `interrupt` sent during generation was only
  honoured once the turn had finished. Text turns now run in their own task,
  and the interrupt cancels the reply immediately (verified: 0 deltas emitted
  after the interrupt, session still usable for the next turn).
- **SSE cancellation latency** — `LlmClient::chat_stream` now races the next
  stream chunk against the `CancellationToken`, so a barge-in drops the HTTP
  response instead of waiting for the model's next token.

### Notes

- Without an LLM the pipeline degrades to a plain STT → TTS echo
  (`LLM_ENABLED=false`).
- `src/orchestrator/turn_manager.rs` gained unit tests for sentence splitting
  (terminators, split across deltas, long punctuation-free runs) and
  `src/llm.rs` for think-block stripping and for the opt-in thinking field —
  `cargo test` passes (13 passed, 1 ignored).
- Verified against `Qwen/Qwen3.6-35B-A3B` on a local vLLM
  (`http://10.8.0.4:8000/v1`): without `LLM_ENABLE_THINKING=false` the model
  narrates "Here's a thinking process: 1. **Analyze User Input** …" directly in
  `content`, which the TTS reads aloud; with it, the reply is the answer only.

---

## 2026-10-15 — Startup clip bootstrap (auto-generate filler / backchannel)

### Added

- **Startup asset bootstrap** — [`src/assets.rs`](src/assets.rs). On boot the
  server checks every expected filler / backchannel clip and, if missing,
  **synthesizes it with the configured TTS engine** and writes it as
  `*.raw` (16-bit LE mono PCM at 44.1 kHz) into the target directory (created
  if needed). Existing clips are never overwritten, so a fresh checkout works
  with no manual generation step and a customized/partial bank is preserved.
- Default clip definitions:
  [`assets::FILLER_CLIPS`](src/assets.rs) (`uhm`, `eh`, `mmm`, `aver`, `bueno`,
  `pues`, `vaya`) and [`assets::BACKCHANNEL_CLIPS`](src/assets.rs) (`mmhmm`,
  `aja`, `claro`, `yaveo`). Backchannel clips are generated only when
  `BACKCHANNEL_ENABLED=true`.
- Wired in [`src/main.rs`](src/main.rs) **before** the banks are loaded; the TTS
  engine is now created earlier so it can be reused for generation.

### Notes

- Generation is best-effort and never blocks startup: a synthesis failure logs a
  warning per clip and the server still starts (that bank stays silent).
- The scratch capture buffer during generation is a 30 s
  [`SampleRingBuffer`](src/audio/ring_buffer.rs) drained after synthesis.

---

## 2026-10-15 — Filler bank (round-robin end-of-utterance fillers)

### Added

- **Filler bank** — `assets/fillers/*.raw`. The end-of-utterance filler is no
  longer a single hard-coded clip; [`TurnManager`](src/orchestrator/turn_manager.rs)
  now selects clips **round-robin** from the bank via
  [`FillerBank::nth`](src/orchestrator/turn_manager.rs) (a per-turn cursor,
  `TurnManager::next_filler`). With a single clip the behaviour is unchanged.
- Clips synthesized with the cloned reference voice (Spanish, 44.1 kHz, 16-bit
  LE mono PCM): `uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya` (~0.1–0.9 s).

### Notes

- `FillerBank::get` is kept for reference/debugging but is no longer on the
  hot path (`#[allow(dead_code)]`).

---

## 2026-10-15 — During-utterance backchannel

### Added

- **Backchannel subsystem** — [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs).
  Emits short acknowledgments ("mm-hmm", "ajá", "claro") *while the user is
  speaking*, at a low fixed gain, to signal that the agent is listening without
  taking the turn. Distinct from the end-of-utterance **filler** injected by the
  [`TurnManager`](src/orchestrator/turn_manager.rs).
- **`BackchannelScheduler`** — per-session decision logic fed by the VAD:
  `on_speech_start` on `VadEvent::SpeechStart`, `on_speech_end` on `SpeechEnd`,
  and `maybe_emit()` on every frame while speaking. A cue is emitted only when
  the user has spoken for at least `BACKCHANNEL_MIN_SPEECH_MS`, at least
  `BACKCHANNEL_INTERVAL_MS` elapsed since the last cue, fewer than
  `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn, and the agent is
  **not** speaking (`!TurnManager::is_speaking()`).
- **Preloaded clip bank** — `assets/backchannel/*.raw` (16-bit LE mono PCM at
  44.1 kHz), synthesized with the cloned reference voice, loaded into a shared
  [`FillerBank`](src/orchestrator/turn_manager.rs) and played round-robin.
- **Third mixer channel** — [`AudioMixer`](src/audio/mixer.rs) gained a dedicated
  `backchannel_queue` and a fixed `backchannel_gain`. The mix is now
  `tts + (filler × filler_gain) + (backchannel × backchannel_gain)`; the queue is
  cleared by `flush()` (barge-in).
- **`FillerBank` generalization** — one type now serves both the filler and the
  backchannel banks (`load(dir, label)`, round-robin `nth(index)`, `is_empty()`);
  the default filler lookup ("uhm") is unchanged.
- **`VadDetector::in_speech()`** — exposes whether the detector is inside a
  confirmed utterance, used to gate backchannel emission.
- **New `.env` knobs** (documented in [`.env.example`](.env.example)):
  `BACKCHANNEL_ENABLED`, `BACKCHANNEL_DIR`, `BACKCHANNEL_GAIN`,
  `BACKCHANNEL_MIN_SPEECH_MS`, `BACKCHANNEL_INTERVAL_MS`,
  `BACKCHANNEL_MAX_PER_TURN`.

### Notes

- `BACKCHANNEL_GAIN` is clamped to `0..1` and `BACKCHANNEL_INTERVAL_MS` is
  clamped to `>= BACKCHANNEL_MIN_SPEECH_MS` at load time.
- Wiring lives in [`src/ws/session.rs`](src/ws/session.rs) (mic/VAD task); the
  bank is created once in [`src/main.rs`](src/main.rs) and shared via `AppState`.

---

## 2026-10-15 — Server-side VAD (Silero V5, ONNX)

### Added

- **VAD subsystem** — [`src/vad/mod.rs`](src/vad/mod.rs) +
  [`src/vad/silero.rs`](src/vad/silero.rs). Runs **Silero VAD v5** via ONNX
  Runtime on the same 16 kHz mono stream that feeds the STT engine. The model
  consumes 32 ms windows (512 samples) with a 64-sample context and a recurrent
  `state [2,1,128]`; the graph shapes and the `sr` input rank are introspected at
  load time.
- **Onset/offset state machine** with hysteresis (`VAD_THRESHOLD` /
  `VAD_NEG_THRESHOLD`) emitting three events: `SpeechOnset` (fast, for
  barge-in), `SpeechStart`, and `SpeechEnd` (carrying the captured PCM plus a
  pre-roll pad so the first phoneme is not clipped).
- **Fast server-side barge-in** — a `SpeechOnset` interrupts the agent when it is
  currently speaking (`TurnManager::is_speaking`), cancelling the in-flight TTS
  and flushing the mixer.
- **VAD-driven STT segmentation** — `SttWorker::push_utterance` receives whole
  utterances instead of raw frames, so the STT no longer needs its own RMS
  segmentation (`VAD_SEGMENT_STT=true`, default).
- **Shared ONNX environment** — [`src/onnx.rs`](src/onnx.rs). ONNX Runtime allows
  a single environment per process, so the STT and VAD pipelines now share a
  lazily created `Environment` (cheap `Arc` clones) instead of each building one.
- **New `.env` knobs** (all documented in [`.env.example`](.env.example)):
  `VAD_ENABLED`, `VAD_MODEL_PATH`, `VAD_THRESHOLD`, `VAD_NEG_THRESHOLD`,
  `VAD_MIN_SPEECH_MS`, `VAD_MIN_SILENCE_MS`, `VAD_SPEECH_PAD_MS`,
  `VAD_BARGE_IN_MIN_SPEECH_MS`, `VAD_SEGMENT_STT`, `VAD_BARGE_IN`, `VAD_DEBUG`.
- **`AudioMixer::is_active`** / `tts_stream_active` flag tracking whether a TTS
  stream is in flight, independent of the queue length, so barge-in can gate on
  "the agent is speaking".

### Fixed

- Duplicate analysis window appended to the utterance at speech start (the
  pre-roll ring already contained it) — the utterance is now seeded from the
  pre-roll ring only.
- `server.sh` now also appends `audio_agent_engine::vad=debug` to `RUST_LOG`
  (mirroring the earlier TTS fix), so VAD debug logs are never filtered out.

### Verified

`models/silero_vad.onnx` is loaded at startup (`state_shape=[2,1,128]`,
`sr_rank=0`). A live WSS session streaming 44.1 kHz speech produced, in
`run/server.log`:

```text
INFO  ws::session: VAD active for this session threshold=0.5 min_speech_ms=250 min_silence_ms=300 …
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
DEBUG vad::silero: VAD speech start speech_ms=256 prob="1.000" preroll_samples=4992
DEBUG vad::silero: VAD speech end duration_ms=1144 samples=18304 avg_prob="0.609"
INFO  ws::session: VAD utterance → STT duration_ms=1144 samples=18304
INFO  orchestrator::turn_manager: EOU detected tokens=5 text=Me escucha.
```

Talking over the agent (after a text TTS prompt) produced:

```text
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
INFO  ws::session: VAD barge-in — interrupting agent prob="0.999"
INFO  orchestrator::turn_manager: interrupt received, flushing audio
```

---

## 2026-10-15 — TTS DEBUG logs actually reach `run/server.log`

### Problem

The `debug!` diagnostics added to the TTS subsystem never appeared in the log,
even though the code and the `DEFAULT_LOG_FILTER` fallback were in place. Root
cause: the parent shell (IDE) already exports `RUST_LOG=info`, so
`EnvFilter::try_from_default_env()` succeeded and the
`info,audio_agent_engine::tts=debug` fallback was never used. In `server.sh`
the same happened — `${RUST_LOG:-info,audio_agent_engine::tts=debug}` never
expanded the default because `RUST_LOG` was already set.

### Fixed

- [`server.sh`](server.sh) now **appends** the
  `audio_agent_engine::tts=debug` directive to whatever `RUST_LOG` it
  inherits (unless that target is already mentioned), so the TTS debug logs
  cannot be silently filtered out. `start` logs the effective filter.

### Verified

A live WSS session (`Hola, …`) that is kept open for the whole synthesis now
emits, in `run/server.log`:

```text
DEBUG audio_agent_engine::tts::fish_audio: TTS request dispatched endpoint=… has_reference=true text=…
DEBUG audio_agent_engine::tts::fish_audio: TTS time-to-first-byte ttfb_ms=0 bytes=8186
DEBUG audio_agent_engine::tts::fish_audio: TTS chunk received chunk=1 bytes=8186 samples=4093 queue_len=4093
DEBUG audio_agent_engine::tts::fish_audio: TTS stream summary chunk_count=9 total_samples=180224 audio_s="4.09" elapsed_ms=3347
```

> Note: closing the WebSocket mid-stream triggers `turn_manager.interrupt()`
> (barge-in), which cancels the TTS stream before any chunk arrives — so only
> `TTS request dispatched` is logged in that case.

---

## 2026-10-15 — Optional HTTPS/WSS + clearer microphone diagnostics

### Problem

Clicking "Connect & Start Mic" in the playground failed with a generic
`mic error`, with no WebSocket connection ever reaching the server. Browsers
only expose `navigator.mediaDevices.getUserMedia` in a **secure context**
(HTTPS or `localhost`); over plain HTTP on any other origin the API is simply
absent, so `startAudio()` threw before the socket was created.

### Added

- **Optional TLS termination** — [`src/tls.rs`](src/tls.rs). Set both `TLS_CERT`
  and `TLS_KEY` (PEM) in `.env` and the server serves the same Axum router over
  HTTPS/WSS via `tokio-rustls` + `hyper` (HTTP/1 upgrades enabled, so WebSocket
  handshakes work). Without them it stays on plain HTTP. The startup log now
  says `listening (HTTPS)` / `listening (HTTP)`.
- **`./server.sh cert [--force]`** — generates a self-signed certificate under
  `certs/` whose SAN covers `localhost`, `127.0.0.1`, `::1` and the host's IPs
  (`EXTRA_SAN` to append more). `start`/`restart` auto-generate it if TLS is
  enabled but the files are missing; `status` shows the scheme and cert state.

### Changed

- **Playground diagnostics** — [`src/web/playground.html`](src/web/playground.html)
  now checks for a secure context *before* requesting the mic and reports the
  precise cause (`insecure origin` / `NotAllowedError` / `NotFoundError`) with a
  remediation hint, instead of a bare `mic error`.
- New config fields `Config::tls_cert` / `Config::tls_key` (+ `tls_enabled()`),
  read from `TLS_CERT` / `TLS_KEY`.

### Notes

- No new crates are downloaded: `hyper`, `hyper-util`, `tokio-rustls`, `rustls`
  and `rustls-pki-types` were already present as transitive deps of `reqwest`
  (`ring` provider), so they are declared with `default-features = false`.

---

## 2026-10-15 — `server.sh` control script (start / stop / restart)

Added [`server.sh`](server.sh), a small bash helper so the server does not have
to be managed by hand.

### Usage

```bash
./server.sh start [--build]     # start (auto-builds if binary missing)
./server.sh stop                # graceful stop (SIGKILL after 10 s)
./server.sh restart [--build]   # stop + start
./server.sh status              # process / port / HTTP status
./server.sh logs [N]            # tail run/server.log (default 50, follow)
./server.sh build               # build the release binary only
```

- PID is tracked in `run/server.pid`; logs are appended to `run/server.log`.
- `start` waits until the port is accepting connections (the model load takes
  ~1.5 s) before reporting success.
- Overridable env vars: `PROFILE` (`release`|`debug`), `RUST_LOG`, `PORT`.
- The script `cd`s to its own directory first, so `.env` and the relative
  `models/parakeet` path resolve correctly regardless of the caller's cwd.

Documented in the README (Quick Start).

---

## 2026-10-15 — Parakeet TDT STT reimplemented in pure Rust (ONNX)

The previous STT was a placeholder that ran `argmax` over a single `.onnx`
file — it could not actually transcribe. It has been replaced with a complete,
dependency-free (no Python/NeMo) implementation of the NVIDIA **Parakeet TDT
0.6B v3** transducer pipeline in [`src/stt/parakeet.rs`](src/stt/parakeet.rs),
mirroring the reference in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr).

### Change

- **3-stage ONNX pipeline**, all sessions kept alive by the `ort` `Environment`:
  1. `nemo128.onnx` — log-mel features `waveforms`/`waveforms_lens` → `features`/`features_lens`.
  2. `encoder-model.int8.onnx` (preferred, auto-detected) or `encoder-model.onnx` —
     `audio_signal`/`length` → `outputs`[1,1024,T']/`encoded_lengths`.
  3. `decoder_joint-model.onnx` — greedy **TDT** decoding.
- **Runtime graph introspection**: `d_model`, `num_durations`, `state_layers` and
  `state_hidden` are read from the ONNX I/O shapes instead of being hardcoded.
- **Greedy TDT decode loop** (token head + duration head; blank `<blk>` = 8192;
  `max_tokens_per_step` from `config.json`, default 10). Encoder output is
  re-packed from `[1, d_model, T']` to contiguous `[T', d_model]` frames.
- **Internal text decoding**: `vocab.txt` (`<piece> <id>`, SentencePiece `▁` →
  space) plus the reference spacing normalizer (`\A\s|\s\B|(\s)\b`). The
  `Transcript` struct now carries a `text: String`.
- **Config**: `STT_MODEL_PATH` now points to a **directory** (`models/parakeet`
  by default, was `models/parakeet-tdt.onnx`). `STT_TOKENIZER_PATH` defaults to
  empty. Updated `.env`, `.env.example` and the README.
- **TurnManager**: [`on_eou`](src/orchestrator/turn_manager.rs) now uses
  `transcript.text` directly, falling back to the external tokenizer only when
  no text was produced.

### Notes / fixes found during integration

- The `decoder_joint` graph expects `targets` and `target_length` as **int32**
  (not int64); `encoder_outputs` and both state tensors are float.
- The int8 encoder (~650 MB, self-contained) is preferred over the fp32 encoder
  (42 MB + a 2.4 GB `.onnx.data` sidecar) for speed and simpler deployment.

### Verification

Opt-in end-to-end test decodes a real 16 kHz mono WAV through the model:

```
$ PARAKEET_TEST_WAV=/tmp/stt_test/solis16k.wav \
    cargo test --release -- --ignored test_infer_real_audio --nocapture
tokens: 67 | text: "Bienvenidos a esta fascinante exploración del universo.
  Como locutor profesional, les explicaré hoy cómo las ondas electromagnéticas
  viajan por el espacio, transformando nuestra comprensión del cosmos."
test stt::parakeet::tests::test_infer_real_audio ... ok
```

The transcription matches the reference transcript verbatim (the source clip was
trimmed to 12 s). `cargo test` (unit tests) also passes.

---

## 2026-10-15 — Fix: short utterances swallowed by the TTS pre-buffer

Short phrases produced **no audio at all**. The mixer only started playing TTS
once the queue held `TTS_PREBUFFER_SAMPLES` (1.5 s). If an utterance's total
length was below that threshold (e.g. `"Cristóbal Colón"` ≈ 1.11 s / 49152
samples), the pre-buffer was never reached, so `tts_primed` stayed `false` and
the queued samples were never drained.

### Change

- `src/audio/mixer.rs`: added an `tts_stream_end` flag with
  `mark_stream_start()` / `mark_stream_end()`. When the stream has ended, the
  mixer primes as soon as *any* sample is queued (`stream_ended && tts_len > 0`),
  bypassing the pre-buffer for short utterances. The flag is reset by
  `flush()`, `mark_stream_start()` and when the queue drains.
- `src/orchestrator/turn_manager.rs`: the spawned TTS tasks now call
  `mixer.mark_stream_start()` before synthesis and `mixer.mark_stream_end()`
  after it returns — only when the request was **not** cancelled, so barge-in
  does not falsely signal completion.

### Verification

```
TTS stream complete chunk_count=6 total_samples=49152 duration_s="0.64"
TTS pre-buffer reached — starting TTS playback tts_queue_len=49152 prebuffer=66150 stream_ended=true
```

A raw WebSocket client sending `"Cristóbal Colón"` now receives ~1.12 s of
non-silent audio (previously the same request yielded only silence frames).

---

## 2026-10-15 — Dual-stack listener (fix `localhost` / `::1` connection refused)

The HTTP server bound only to `0.0.0.0` (IPv4). On hosts where `localhost`
resolves to `::1` first (confirmed via `getent hosts localhost` → `::1 localhost`),
clients and preview/port-forwarding proxies that connect to `localhost` via IPv6
received `connection refused`, surfacing as *"service is unavailable"*.

### Change

- `src/main.rs`: the listener now binds to the IPv6 wildcard
  `SocketAddr::from((Ipv6Addr::UNSPECIFIED, port))` instead of `0.0.0.0`.
  With `net.ipv6.bindv6only = 0` (Linux default, verified on this host) this is
  dual-stack: it accepts both IPv6 (`::1` / `localhost`) and IPv4 clients.

### Verification

```
ss -ltnp | grep 8787  → LISTEN *:8787
curl http://[::1]:8787/     → 200
curl http://localhost:8787/ → 200
curl http://127.0.0.1:8787/ → 200
```

---

## 2026-10-14 — Modular TTS plugin system (engine-agnostic via `.env`)

The TTS layer was hard-coded to Fish Audio / SGLang Drama 3. To support
switching engines (e.g. Kokoro, or any future backend) without touching
orchestration code, the TTS subsystem is now a pluggable trait selected at
runtime via the `TTS_ENGINE` environment variable.

### New `TtsEngine` trait (`src/tts/mod.rs`)

```rust
#[async_trait]
pub trait TtsEngine: Send + Sync {
    fn name(&self) -> &str;
    fn sample_rate(&self) -> u32;
    async fn synthesize(&self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken) -> Result<()>;
    fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}  // default no-op
}
```

- Every plugin streams **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44.1 kHz)**
  into the shared mixer queue. If an engine's native rate differs, it resamples
  internally (the mixer runs at a fixed rate).
- `synthesize()` must honor the `CancellationToken` for barge-in.
- `set_reference()` defaults to a no-op; only engines that support voice
  cloning override it.

### Factory (`src/tts/mod.rs::create_tts_engine`)

```
TTS_ENGINE=fish      → FishAudioTts  (Fish Audio / SGLang Drama 3)
TTS_ENGINE=kokoro    → KokoroTts     (stub, implement in src/tts/kokoro.rs)
```

Unknown values panic at startup with a clear message.

### Bundled plugins

| Plugin | File | Status |
|---|---|---|
| `fish` (Fish Audio Drama 3) | `src/tts/fish_audio.rs` | Production-ready (renamed from `sglang_client.rs`) |
| `kokoro` | `src/tts/kokoro.rs` | Stub — HTTP streaming call left as `TODO` with a commented example; includes a 24 kHz → 44.1 kHz linear resampler helper |

### `.env` loading

- Added `dotenvy` to auto-load a `.env` file at startup (existing env vars take
  precedence).
- Added `.env.example` documenting all env vars including `TTS_ENGINE`,
  `KOKORO_URL`, `KOKORO_VOICE`.

### Files changed

- `src/tts/mod.rs` — new `TtsEngine` trait + `create_tts_engine` factory.
- `src/tts/fish_audio.rs` — renamed from `sglang_client.rs`; `FishAudioTts` now implements `TtsEngine`.
- `src/tts/kokoro.rs` — new Kokoro plugin stub (example for adding engines).
- `src/tts/sglang_client.rs` — deleted (moved to `fish_audio.rs`).
- `src/config.rs` — new `tts_engine` field, `TTS_ENGINE` env var (default `fish`).
- `src/orchestrator/turn_manager.rs` — `tts: SglangTtsClient` → `tts: Arc<dyn TtsEngine>`.
- `src/ws/handler.rs` — `AppState.tts` → `Arc<dyn TtsEngine>`.
- `src/main.rs` — uses `create_tts_engine(&config)`; loads `.env` via `dotenvy`.
- `Cargo.toml` — added `async-trait`, `dotenvy`.
- `.env.example` — new file documenting all configuration env vars.

### How to add a new engine

1. Create `src/tts/myengine.rs` and implement `TtsEngine`.
2. Add `pub mod myengine;` to `src/tts/mod.rs`.
3. Register it in the `create_tts_engine` match.
4. Set `TTS_ENGINE=myengine` in `.env`.

### Build / verification

- `cargo build` — 0 errors, 2 dead-code warnings (Kokoro stub fields).
- `cargo test` — 4/4 passed.
- Server starts with `TTS_ENGINE=fish` (default): log shows `selected TTS engine engine="fish"` and voice-cloning reference loaded.

---

## 2026-10-14 — Cancel TTS on WebSocket disconnect

When the client disconnected, the in-flight TTS synthesis task continued
streaming from Fish Audio even though the WebSocket session was already closed,
wasting bandwidth and server resources.

### Fix (`src/ws/session.rs`)

- Added `turn_manager.interrupt()` in the session cleanup path (after the
  receiver ends, before aborting the sender/mic tasks).
- This cancels the TTS `CancellationToken`, flushes the mixer queues, and stops the
  streaming HTTP request immediately.

---

## 2026-10-14 — Fix WebSocket frame timing (Burst ticker)

The 20 ms audio sender ticker used `MissedTickBehavior::Skip`, which dropped
tick beats whenever the Tokio event loop was busy (e.g. processing a large
Fish Audio TTS chunk). Each skipped tick meant a 20 ms gap in frame delivery
to the browser, producing audible micro-interruptions even though the server's
TTS queue was well-buffered.

### Fix (`src/ws/session.rs`)

- Changed `MissedTickBehavior::Skip` → `MissedTickBehavior::Burst`.
- Missed ticks now trigger immediate back-to-back sends so the browser
  receives all due frames. The browser's jitter buffer (500 ms) absorbs the
  burst without underrunning.

---

## 2026-10-14 — Micro-interruption fix (grace period on TTS drain)

Even with the 1.5 s server pre-buffer and 500 ms browser jitter buffer,
brief audio gaps could still occur when the TTS queue momentarily hit zero
between Fish Audio chunks. The mixer immediately reset its `tts_primed`
state, forcing a full 1.5 s re-prime on the next chunk. The browser
playback processor did the same, re-entering buffering mode on any
underrun.

### Server fix (`src/audio/mixer.rs`)

- Added `tts_drain_frames` counter (`Arc<AtomicUsize>`) and
  `TTS_DRAIN_GRACE_FRAMES = 10` (200 ms).
- When the TTS queue is empty while primed, the mixer now outputs silence
  for up to 10 consecutive frames before resetting `tts_primed`. This
  absorbs brief inter-chunk gaps without re-triggering the pre-buffer.
- `flush()` also resets `tts_drain_frames`.

### Browser fix (`src/web/playground.html`)

- `PlaybackProcessor` now tracks `underrunSamples` and only re-enters
  buffering mode after 100 ms (`sampleRate * 0.10`) of continuous
  underrun. Brief underruns produce silent output but keep playback
  running, so audio resumes immediately when the next WebSocket frame
  arrives.

---

## 2026-10-14 — Prevent overlapping TTS streams (double-Speak fix)

When the user clicked **Speak** twice in quick succession, two TTS synthesis
tasks ran concurrently and both pushed PCM into the same mixer queue. This
produced overlapping/garbled audio and roughly doubled the total duration
(one observed run produced 83 s of audio for a ~42 s text).

### Server fix (`src/orchestrator/turn_manager.rs`)

- `speak()` now calls `self.mixer.flush()` immediately after
  `cancel_current_tts()`. This clears any remaining PCM from the previous
  utterance so the new one starts from a clean queue.

### Client fix (`src/web/playground.html`)

- The **Speak** button is now disabled (shows "Speaking...") as soon as a
  request is sent.
- A 1.5 s idle timer re-enables it once the server stops sending audio
  frames, preventing accidental double submissions.

---

## 2026-10-14 — Jitter buffer fixes (no more audio drop-outs)

Fish Audio Drama 3 streams PCM in unevenly-spaced chunks, which caused brief
audible drop-outs (silence gaps) during playback. Fixed with a double jitter
buffer.

### Server-side TTS pre-buffer (`src/audio/mixer.rs`)

- Added `TTS_PREBUFFER_SAMPLES` ≈ 200 ms of audio (44.1 kHz × 0.20).
- The mixer now tracks a `tts_primed` flag (`Arc<AtomicBool>`). TTS samples are
  **not** consumed until the TTS queue holds at least the pre-buffer amount.
- During the priming window only the filler ("uhm") backchannel is emitted, so
  the user hears an immediate response while TTS audio builds up.
- When the TTS queue empties (stream exhausted), `tts_primed` resets to `false`
  for the next request. `flush()` also resets it (barge-in).
- `AudioMixer::pull_frame` now uses `tts_queue.len()` instead of `is_empty()`
  for priming decisions.

### Browser-side playback jitter buffer (`src/web/playground.html`)

- The `PlaybackProcessor` AudioWorklet now starts in a `buffering` state and
  only begins playing once the ring buffer holds ≥ 200 ms of samples.
- On underrun (buffer drained mid-playback) it returns to `buffering` to avoid
  stuttering, then resumes once enough audio has re-accumulated.
- Added `available()` helper to compute the fill level of the ring buffer.

### Build / verification

- `cargo build` — 0 warnings.
- `cargo test` — 4/4 passed.

---

## 2026-10-14 — Voice cloning + 44.1 kHz native output

### Sample rate aligned to Fish Audio native output

- Changed `OUTPUT_SAMPLE_RATE` from **24 000 Hz** to **44 100 Hz** to match Fish Audio Drama 3's native PCM output.
- Removed the TTS anti-aliasing resampler from `sglang_client.rs` — Fish Audio PCM samples are pushed directly into the mixer queue. This eliminates any resampling artifacts ("metallic" sound) and pitch/speed mismatches.
- The browser `AudioContext` now runs at 44 100 Hz, so playback is sample-accurate.
- `OUTPUT_FRAME_SAMPLES` is now 882 (44.1 kHz × 20 ms), `OUTPUT_FRAME_BYTES` = 1764.
- Mic → STT path: browser capture resamples to 44.1 kHz, server resamples 44.1 kHz → 16 kHz for Parakeet (linear interpolation, STT-only path so aliasing is not a concern).

### Voice cloning (reference audio)

Added support for Fish Audio's voice-cloning `references` field.

**Config (env vars, applied at startup):**

| Env var | Description |
|---|---|
| `TTS_REFERENCE_AUDIO` | Path to a WAV file on the Fish Audio server filesystem. |
| `TTS_REFERENCE_TEXT` | Reference transcript. If it points to an existing `.txt` file, its contents are loaded; otherwise the raw string is used. If unset, the server looks for a `<audio-stem>.txt` sibling of the WAV. |

**Default:** `solis.wav` + auto-loaded `solis.txt` from `/root/pipecat-voice/voice_profiles/`.

**Runtime override (playground → server WebSocket text messages):**

- `SET_REFERENCE:<path>|<text>` — set or change the reference. `<text>` may be empty to re-use the sibling `.txt`.
- `CLEAR_REFERENCE` — disable voice cloning (use the model's default voice).

**Files changed:**

- `src/config.rs` — new `tts_reference_audio` / `tts_reference_text` fields, `Config::tts_reference()` resolver with sibling-`.txt` fallback, new env-var parsing.
- `src/tts/sglang_client.rs` — `Reference` struct, `references` field in `SpeechRequest` (omitted when `None`), runtime `set_reference()` backed by a `parking_lot::RwLock<Option<Reference>>`.
- `src/orchestrator/turn_manager.rs` — forwards `set_reference()` to the TTS client.
- `src/ws/session.rs` — handles `SET_REFERENCE:` / `CLEAR_REFERENCE` control messages.
- `src/web/playground.html` — new "Voice Cloning" card with WAV path + transcript inputs and Set/Clear buttons; reference status indicator.

### Build / verification

- `cargo build` — 0 warnings.
- `cargo test` — 4/4 passed (tokenizer).
- Direct Fish Audio `references` call confirmed HTTP 200 + `x-sample-rate: 44100`.
- End-to-end WebSocket test: `SET_REFERENCE` followed by a text-to-speak request streams 44.1 kHz PCM back to the client.