piperust/UPDATES.md

Audio Agent Engine — Updates

This file documents all notable changes to the audio-agent-engine project. Entries are written in English and ordered newest-first.

---

2026-10-16 — Resilient mixer resume + TTS_PREBUFFER_MS default 900 ms

Changed

• The mixer no longer re-fills the whole jitter buffer after a mid-reply underrun. Previously, when the TTS queue emptied for more than the 200 ms drain grace, pull_frame cleared tts_primed and tts_stream_end, then required a full tts_prebuffer_samples refill before playback resumed — adding a second, self-inflicted pause on top of the real underrun. It also left the mixer "primed" between turns because mark_stream_start never re-armed tts_primed. Now:
  • the full pre-buffer is awaited only at the first playback of a turn; once playback has started, an underrun pauses and resumes as soon as any audio is queued again (need = 1);
  • tts_stream_end is no longer cleared on drain, so a genuine end-of-stream is still honoured;
  • mark_stream_start re-arms tts_primed, tts_playback_started and tts_drain_frames at the start of every turn.
• TTS_PREBUFFER_MS default raised 400 → 900 (.env, .env.example, src/config.rs). With 400 the reply was still prone to drop-outs on long answers.

Why

• Direct measurement of the SGLang Drama-3 endpoint (a single, uncontended request) showed it delivers ~1.08× real time, in bursts every ~2.7 s (≈131 072 samples / 2.97 s per burst). Because the producer barely outruns the consumer and delivers in bursts, gapless playback requires a jitter buffer of roughly the burst period (~3 s); smaller buffers show a short real drop-out (pure silence) on long replies.
• The mixer fix removes the avoidable part of the pause (the full refill after an underrun) and recovers at the first byte, which is what makes the remaining gap equal to the backend stall rather than stall + refill.
• Default 900 is a deliberate latency-first choice: it keeps time-to-first-audio low (~1.8 s) and accepts a rare short drop-out on very long replies. Raising TTS_PREBUFFER_MS towards ~3000 trades that latency back for gaplessness.

Notes

• The residual drop-out is a property of the TTS backend (~1× real time), not of the mixer: no buffering smaller than the backend's burst period can bridge a delivery stall.
• Relay diagnostics at debug level in run_llm_turn log per-chunk relay timing (TTS pipeline ...) to help localise seams.

---

2026-10-16 — Pipelined TTS: chunks are synthesized ahead of playback

Changed

• The TTS sink no longer waits for one chunk's stream to finish before requesting the next. Previously run_llm_turn looped while let Some(chunk) = rx.recv() { tts.synthesize(chunk).await }, so a chunk's synthesis only started after the previous chunk had finished streaming end-to-end. Because the TTS engine needs ~0.6 s to emit its first audio byte, every chunk boundary left a gap roughly that long, which the old 1.5 s pre-buffer used to hide.
• The sink is now split into two pieces:
  • a collector that spawns one synthesis task per chunk as soon as the LLM produces it (bounded by a semaphore), each streaming into its own staging buffer;
  • a relay that forwards those staging buffers into the mixer queue strictly in submission order, so playback stays in order and gapless. A chunk's synthesis therefore overlaps the previous chunk's playback, hiding the per-request start-up latency.

Added

• New TTS_PIPELINE_DEPTH configuration (default 3) in src/config.rs: how many TTS requests may be in flight at once. 1 restores the previous strictly serial behaviour.
• SampleRingBuffer::remaining_capacity() (src/audio/ring_buffer.rs) so the relay never pushes past the mixer queue's free space — queued (i.e. currently playing) audio is never dropped to make room.

Why

• Direct measurement against the SGLang Drama-3 endpoint showed a fixed ~0.62 s start-up latency per request (request → first audio byte), with audio then streamed at roughly real time. Two concurrent requests are served in parallel (start-up rises only to ~0.84–1.0 s), so overlapping requests genuinely removes the inter-chunk gaps instead of just moving them.
• The gaps were independent of chunk size (120 → 60 chars left the same ~0.8 s hole), which confirmed the cause was serialization, not chunk length.

Notes

• Staging buffers are sized at 30 s of audio per in-flight request, which is comfortably more than any single chunk.
• With the pipeline in place the pre-buffer only has to cover arrival jitter rather than the TTS engine's per-request start-up, so it can be kept small. (Superseded: the default was later raised to 900 — see the entry above.)

---

2026-10-16 — Mixer TTS pre-buffer is now configurable (default 400 ms)

Changed

• The TTS jitter buffer moved from a hardcoded 1.5 s constant to the TTS_PREBUFFER_MS setting (default 400). AudioMixer (in src/audio/mixer.rs) now takes tts_prebuffer_samples as a constructor argument instead of reading the removed TTS_PREBUFFER_SAMPLES constant, and run_session derives it from the config.

Why

• Log analysis of a voice turn ("Conta da dieci a uno.") showed the first TTS request left ~1.2 s after the EOU and the first audio bytes arrived ~1.9 s, but playback only started at +4.2 s: the mixer had 1.486 s (65 536 samples) queued at +2.8 s, 14 ms short of the 1.5 s threshold, so it waited for the next Fish chunk, which landed 1.39 s later. The pre-buffer, not the text chunker, dominated the perceived latency.
• Fish Audio now delivers near real time, so 1.5 s was far more buffer than needed. 400 ms removes ~1.1 s of time-to-first-audio while still absorbing normal inter-chunk jitter (the 200 ms drain grace is unchanged).

Notes

• Tune from .env: raise TTS_PREBUFFER_MS if you hear drop-outs, lower it for an even snappier start.
• The low-buffer warning threshold now scales with the configured value (prebuffer / 4).

---

2026-10-16 — TTS starts on the first words, not on the first sentence

Changed

• The streaming reply is now chunked on a word boundary instead of waiting for sentence punctuation. SentenceSplitter (in src/orchestrator/turn_manager.rs) keeps the sentence-terminator rule (., !, ?, ;, newline) but adds an early flush: the first chunk is cut as soon as LLM_STREAM_FIRST_CHUNK_CHARS characters are buffered, and every following chunk at LLM_STREAM_CHUNK_CHARS. The cut lands on the first whitespace at or past the threshold, so a chunk always holds at least that many characters and never splits a word. Previously the first chunk only left at a ./!/?, so a long opening sentence delayed the first spoken word.

Added

• New LLM_STREAM_FIRST_CHUNK_CHARS (default 20) and LLM_STREAM_CHUNK_CHARS (default 120) configuration in src/config.rs. A small first threshold makes the agent start speaking on the first few words; the larger follow-up threshold keeps prosody good once the audio is flowing. Setting either to 0 disables that flush (chunks then end on sentence punctuation only).

Notes

• Verified on the running server: for a chat-box turn the first TTS request is dispatched ~0.5 s after the user text, synthesizing "Non ho occhi fisici né" (23 chars), well before the rest of the sentence ("accesso a telecamere o immagini, quindi non posso vedere nulla …") was generated. Before the change the whole opening sentence had to complete first.
• The remaining time-to-first-audio is dominated by the mixer's 1.5 s TTS pre-buffer plus the TTS engine's first-chunk latency, not by the chunker.

---

2026-10-15 — Filler is now a latency guard: switchable and delay-gated

Changed

• The end-of-turn filler is no longer played unconditionally. It exists to mask the wait for the reply, so playing it when the answer is already speaking only gets in the way of the first words. TurnManager::schedule_filler (in src/orchestrator/turn_manager.rs) starts a FILLER_DELAY_MS timer when the turn is dispatched (both the LLM and the direct-TTS path) and queues the clip only if the response audio has not started by then. At most one filler per turn.
• New FILLER_ENABLED / FILLER_DELAY_MS configuration in src/config.rs. FILLER_ENABLED=false disables the filler entirely (no clip is ever queued); FILLER_DELAY_MS (default 2500) is the latency guard, tunable from .env.
• The immediate filler pushes in handle_user_turn and speak were removed in favour of the single scheduled path, so the playground's direct-TTS path follows the same rule.

Added

• AudioMixer::tts_playback_started() in src/audio/mixer.rs — set when the TTS pre-buffer is reached (i.e. the response audio actually starts), reset by mark_stream_start and flush. This is what the filler decision reads.

Notes

• The pending filler timer is bound to the turn's CancellationToken, so a barge-in (interrupt, VAD onset, a new turn, or disconnect) drops it.
• Verified from .env on the running server: FILLER_DELAY_MS=100 → pushed filler (delayed); FILLER_DELAY_MS=8000 → response audio already playing — filler skipped; FILLER_ENABLED=false → no filler activity at all.
• With the current LLM_ENABLE_THINKING=false + Fish Audio setup, a short reply starts playing within 2.5 s, so the filler correctly stays silent; raise FILLER_DELAY_MS (e.g. 1000) if you want it to fire more often.

---

2026-10-15 — LLM reasoning stage + /mivocero speech-to-speech chat client

Added

• OpenAI-compatible LLM client — src/llm.rs. New LlmClient posting to {LLM_BASE_URL}/chat/completions, with an Authorization: Bearer {LLM_API_KEY} header and a bounded TCP connect timeout (no global request timeout, so SSE streams can stay open). Two entry points, selected at runtime via LLM_STREAM:
  • chat_stream — SSE token streaming (stream: true), handling data: lines and data: [DONE];
  • chat — a single blocking request.
• Chain-of-thought stripping — reasoning models (Qwen3, …) may emit their plan in a separate reasoning_content delta (vLLM) or inline between think markers (llama.cpp and others). reasoning_content is ignored and inline think blocks are removed by an incremental ThinkStripper that is stable across delta boundaries (it withholds a tail the size of the opening marker so a marker split between two deltas is still detected). Applied to both the chat box text and the synthesized audio.
• LLM stage in TurnManager — the user turn (STT transcript or chat-box text) is sent to the model and the reply is spoken by the TTS. A rolling history window (LLM_HISTORY_MESSAGES) is kept per connection and prefixed with LLM_SYSTEM_PROMPT. A SentenceSplitter chunks the streaming reply on ., !, ?, ; and newlines so the TTS starts on the first sentence (SENTENCE_SOFT_MAX = 220 chars forces a split on whitespace for punctuation-free runs).
• Barge-in covers the LLM request — an interrupt (client JSON, VAD onset or disconnect) cancels the in-flight HTTP request through a CancellationToken in the reqwest tokio::select!, cancels the TTS tasks and flushes the mixer.
• Bidirectional text protocol on /ws — client → server {"type":"user_text","text":…} and {"type":"interrupt"}; server → client {"type":"user","text":…}, {"type":"assistant_delta","text":…}, {"type":"assistant_end"} and {"type":"error","text":…}. Legacy text commands (INTERRUPT, SET_REFERENCE:, CLEAR_REFERENCE, raw text → direct TTS) are unchanged, so the playground keeps working.
• /mivocero client — src/web/mivocero.html, embedded via src/web/mod.rs and served by src/main.rs. A speech-to-speech example agent with a microphone (same AudioWorklet capture/playback path as the playground), a chat box that renders the user transcript and the streaming assistant reply, a text composer and an interrupt button.
• LLM_* configuration — Config fields and read_llm_env() in src/config.rs: LLM_ENABLED, LLM_BASE_URL, LLM_API_KEY, LLM_MODEL, LLM_MAX_TOKENS, LLM_TEMPERATURE, LLM_STREAM, LLM_HISTORY_MESSAGES, LLM_ENABLE_THINKING, LLM_STRIP_REASONING, LLM_DIRECT_ANSWERS, LLM_SYSTEM_PROMPT. LLM_ENABLED defaults to true when both LLM_BASE_URL and LLM_MODEL are set.
• Three independent chain-of-thought controls (each switchable from .env) — a single mechanism does not cover every deployment:
  1. LLM_ENABLE_THINKING — LlmClient sends chat_template_kwargs: {"enable_thinking": <bool>} when the variable is set, turning Qwen3-style thinking off/on server-side (vLLM/SGLang). Unset omits the vendor-specific field so strict OpenAI-compatible servers keep accepting the body.
  2. LLM_STRIP_REASONING (default true) — reasoning_content deltas are dropped and inline think blocks removed; false passes the raw output (both fields) through.
  3. LLM_DIRECT_ANSWERS (default true) — appends DIRECT_ANSWER_DIRECTIVE to the system prompt. Needed when the server runs no reasoning parser: the chain-of-thought is then written straight into content as ordinary prose and is neither reasoning_content nor tagged, so filtering cannot catch it.

Fixed

• Chat-box turns blocked the receive loop — handle_control_message in src/ws/session.rs awaited TurnManager::on_user_text inline, so while a reply was being generated the session could not read any further client message: an interrupt sent during generation was only honoured once the turn had finished. Text turns now run in their own task, and the interrupt cancels the reply immediately (verified: 0 deltas emitted after the interrupt, session still usable for the next turn).
• SSE cancellation latency — LlmClient::chat_stream now races the next stream chunk against the CancellationToken, so a barge-in drops the HTTP response instead of waiting for the model's next token.

Notes

• Without an LLM the pipeline degrades to a plain STT → TTS echo (LLM_ENABLED=false).
• src/orchestrator/turn_manager.rs gained unit tests for sentence splitting (terminators, split across deltas, long punctuation-free runs) and src/llm.rs for think-block stripping and for the opt-in thinking field — cargo test passes (13 passed, 1 ignored).
• Verified against Qwen/Qwen3.6-35B-A3B on a local vLLM (http://10.8.0.4:8000/v1): without LLM_ENABLE_THINKING=false the model narrates "Here's a thinking process: 1. Analyze User Input …" directly in content, which the TTS reads aloud; with it, the reply is the answer only.

---

2026-10-15 — Startup clip bootstrap (auto-generate filler / backchannel)

Added

• Startup asset bootstrap — src/assets.rs. On boot the server checks every expected filler / backchannel clip and, if missing, synthesizes it with the configured TTS engine and writes it as *.raw (16-bit LE mono PCM at 44.1 kHz) into the target directory (created if needed). Existing clips are never overwritten, so a fresh checkout works with no manual generation step and a customized/partial bank is preserved.
• Default clip definitions: assets::FILLER_CLIPS (uhm, eh, mmm, aver, bueno, pues, vaya) and assets::BACKCHANNEL_CLIPS (mmhmm, aja, claro, yaveo). Backchannel clips are generated only when BACKCHANNEL_ENABLED=true.
• Wired in src/main.rs before the banks are loaded; the TTS engine is now created earlier so it can be reused for generation.

Notes

• Generation is best-effort and never blocks startup: a synthesis failure logs a warning per clip and the server still starts (that bank stays silent).
• The scratch capture buffer during generation is a 30 s SampleRingBuffer drained after synthesis.

---

2026-10-15 — Filler bank (round-robin end-of-utterance fillers)

Added

• Filler bank — assets/fillers/*.raw. The end-of-utterance filler is no longer a single hard-coded clip; TurnManager now selects clips round-robin from the bank via FillerBank::nth (a per-turn cursor, TurnManager::next_filler). With a single clip the behaviour is unchanged.
• Clips synthesized with the cloned reference voice (Spanish, 44.1 kHz, 16-bit LE mono PCM): uhm, eh, mmm, aver, bueno, pues, vaya (~0.1–0.9 s).

Notes

• FillerBank::get is kept for reference/debugging but is no longer on the hot path (#[allow(dead_code)]).

---

2026-10-15 — During-utterance backchannel

Added

• Backchannel subsystem — src/orchestrator/backchannel.rs. Emits short acknowledgments ("mm-hmm", "ajá", "claro") while the user is speaking, at a low fixed gain, to signal that the agent is listening without taking the turn. Distinct from the end-of-utterance filler injected by the TurnManager.
• BackchannelScheduler — per-session decision logic fed by the VAD: on_speech_start on VadEvent::SpeechStart, on_speech_end on SpeechEnd, and maybe_emit() on every frame while speaking. A cue is emitted only when the user has spoken for at least BACKCHANNEL_MIN_SPEECH_MS, at least BACKCHANNEL_INTERVAL_MS elapsed since the last cue, fewer than BACKCHANNEL_MAX_PER_TURN cues were emitted this turn, and the agent is not speaking (!TurnManager::is_speaking()).
• Preloaded clip bank — assets/backchannel/*.raw (16-bit LE mono PCM at 44.1 kHz), synthesized with the cloned reference voice, loaded into a shared FillerBank and played round-robin.
• Third mixer channel — AudioMixer gained a dedicated backchannel_queue and a fixed backchannel_gain. The mix is now tts + (filler × filler_gain) + (backchannel × backchannel_gain); the queue is cleared by flush() (barge-in).
• FillerBank generalization — one type now serves both the filler and the backchannel banks (load(dir, label), round-robin nth(index), is_empty()); the default filler lookup ("uhm") is unchanged.
• VadDetector::in_speech() — exposes whether the detector is inside a confirmed utterance, used to gate backchannel emission.
• New .env knobs (documented in .env.example): BACKCHANNEL_ENABLED, BACKCHANNEL_DIR, BACKCHANNEL_GAIN, BACKCHANNEL_MIN_SPEECH_MS, BACKCHANNEL_INTERVAL_MS, BACKCHANNEL_MAX_PER_TURN.

Notes

• BACKCHANNEL_GAIN is clamped to 0..1 and BACKCHANNEL_INTERVAL_MS is clamped to >= BACKCHANNEL_MIN_SPEECH_MS at load time.
• Wiring lives in src/ws/session.rs (mic/VAD task); the bank is created once in src/main.rs and shared via AppState.

---

2026-10-15 — Server-side VAD (Silero V5, ONNX)

Added

• VAD subsystem — src/vad/mod.rs + src/vad/silero.rs. Runs Silero VAD v5 via ONNX Runtime on the same 16 kHz mono stream that feeds the STT engine. The model consumes 32 ms windows (512 samples) with a 64-sample context and a recurrent state [2,1,128]; the graph shapes and the sr input rank are introspected at load time.
• Onset/offset state machine with hysteresis (VAD_THRESHOLD / VAD_NEG_THRESHOLD) emitting three events: SpeechOnset (fast, for barge-in), SpeechStart, and SpeechEnd (carrying the captured PCM plus a pre-roll pad so the first phoneme is not clipped).
• Fast server-side barge-in — a SpeechOnset interrupts the agent when it is currently speaking (TurnManager::is_speaking), cancelling the in-flight TTS and flushing the mixer.
• VAD-driven STT segmentation — SttWorker::push_utterance receives whole utterances instead of raw frames, so the STT no longer needs its own RMS segmentation (VAD_SEGMENT_STT=true, default).
• Shared ONNX environment — src/onnx.rs. ONNX Runtime allows a single environment per process, so the STT and VAD pipelines now share a lazily created Environment (cheap Arc clones) instead of each building one.
• New .env knobs (all documented in .env.example): VAD_ENABLED, VAD_MODEL_PATH, VAD_THRESHOLD, VAD_NEG_THRESHOLD, VAD_MIN_SPEECH_MS, VAD_MIN_SILENCE_MS, VAD_SPEECH_PAD_MS, VAD_BARGE_IN_MIN_SPEECH_MS, VAD_SEGMENT_STT, VAD_BARGE_IN, VAD_DEBUG.
• AudioMixer::is_active / tts_stream_active flag tracking whether a TTS stream is in flight, independent of the queue length, so barge-in can gate on "the agent is speaking".

Fixed

• Duplicate analysis window appended to the utterance at speech start (the pre-roll ring already contained it) — the utterance is now seeded from the pre-roll ring only.
• server.sh now also appends audio_agent_engine::vad=debug to RUST_LOG (mirroring the earlier TTS fix), so VAD debug logs are never filtered out.

Verified

models/silero_vad.onnx is loaded at startup (state_shape=[2,1,128], sr_rank=0). A live WSS session streaming 44.1 kHz speech produced, in run/server.log:

INFO  ws::session: VAD active for this session threshold=0.5 min_speech_ms=250 min_silence_ms=300 …
DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
DEBUG vad::silero: VAD speech start speech_ms=256 prob="1.000" preroll_samples=4992
DEBUG vad::silero: VAD speech end duration_ms=1144 samples=18304 avg_prob="0.609"
INFO  ws::session: VAD utterance → STT duration_ms=1144 samples=18304
INFO  orchestrator::turn_manager: EOU detected tokens=5 text=Me escucha.

Talking over the agent (after a text TTS prompt) produced:

DEBUG vad::silero: VAD speech onset speech_ms=160 prob="0.999"
INFO  ws::session: VAD barge-in — interrupting agent prob="0.999"
INFO  orchestrator::turn_manager: interrupt received, flushing audio

---

2026-10-15 — TTS DEBUG logs actually reach run/server.log

Problem

The debug! diagnostics added to the TTS subsystem never appeared in the log, even though the code and the DEFAULT_LOG_FILTER fallback were in place. Root cause: the parent shell (IDE) already exports RUST_LOG=info, so EnvFilter::try_from_default_env() succeeded and the info,audio_agent_engine::tts=debug fallback was never used. In server.sh the same happened — ${RUST_LOG:-info,audio_agent_engine::tts=debug} never expanded the default because RUST_LOG was already set.

Fixed

• server.sh now appends the audio_agent_engine::tts=debug directive to whatever RUST_LOG it inherits (unless that target is already mentioned), so the TTS debug logs cannot be silently filtered out. start logs the effective filter.

Verified

A live WSS session (Hola, …) that is kept open for the whole synthesis now emits, in run/server.log:

DEBUG audio_agent_engine::tts::fish_audio: TTS request dispatched endpoint=… has_reference=true text=…
DEBUG audio_agent_engine::tts::fish_audio: TTS time-to-first-byte ttfb_ms=0 bytes=8186
DEBUG audio_agent_engine::tts::fish_audio: TTS chunk received chunk=1 bytes=8186 samples=4093 queue_len=4093
DEBUG audio_agent_engine::tts::fish_audio: TTS stream summary chunk_count=9 total_samples=180224 audio_s="4.09" elapsed_ms=3347

Note: closing the WebSocket mid-stream triggers turn_manager.interrupt() (barge-in), which cancels the TTS stream before any chunk arrives — so only TTS request dispatched is logged in that case.

---

2026-10-15 — Optional HTTPS/WSS + clearer microphone diagnostics

Problem

Clicking "Connect & Start Mic" in the playground failed with a generic mic error, with no WebSocket connection ever reaching the server. Browsers only expose navigator.mediaDevices.getUserMedia in a secure context (HTTPS or localhost); over plain HTTP on any other origin the API is simply absent, so startAudio() threw before the socket was created.

Added

• Optional TLS termination — src/tls.rs. Set both TLS_CERT and TLS_KEY (PEM) in .env and the server serves the same Axum router over HTTPS/WSS via tokio-rustls + hyper (HTTP/1 upgrades enabled, so WebSocket handshakes work). Without them it stays on plain HTTP. The startup log now says listening (HTTPS) / listening (HTTP).
• ./server.sh cert [--force] — generates a self-signed certificate under certs/ whose SAN covers localhost, 127.0.0.1, ::1 and the host's IPs (EXTRA_SAN to append more). start/restart auto-generate it if TLS is enabled but the files are missing; status shows the scheme and cert state.

Changed

• Playground diagnostics — src/web/playground.html now checks for a secure context before requesting the mic and reports the precise cause (insecure origin / NotAllowedError / NotFoundError) with a remediation hint, instead of a bare mic error.
• New config fields Config::tls_cert / Config::tls_key (+ tls_enabled()), read from TLS_CERT / TLS_KEY.

Notes

• No new crates are downloaded: hyper, hyper-util, tokio-rustls, rustls and rustls-pki-types were already present as transitive deps of reqwest (ring provider), so they are declared with default-features = false.

---

2026-10-15 — server.sh control script (start / stop / restart)

Added server.sh, a small bash helper so the server does not have to be managed by hand.

Usage

./server.sh start [--build]     # start (auto-builds if binary missing)
./server.sh stop                # graceful stop (SIGKILL after 10 s)
./server.sh restart [--build]   # stop + start
./server.sh status              # process / port / HTTP status
./server.sh logs [N]            # tail run/server.log (default 50, follow)
./server.sh build               # build the release binary only

• PID is tracked in run/server.pid; logs are appended to run/server.log.
• start waits until the port is accepting connections (the model load takes ~1.5 s) before reporting success.
• Overridable env vars: PROFILE (release|debug), RUST_LOG, PORT.
• The script cds to its own directory first, so .env and the relative models/parakeet path resolve correctly regardless of the caller's cwd.

Documented in the README (Quick Start).

---

2026-10-15 — Parakeet TDT STT reimplemented in pure Rust (ONNX)

The previous STT was a placeholder that ran argmax over a single .onnx file — it could not actually transcribe. It has been replaced with a complete, dependency-free (no Python/NeMo) implementation of the NVIDIA Parakeet TDT 0.6B v3 transducer pipeline in src/stt/parakeet.rs, mirroring the reference in istupakov/onnx-asr.

Change

• 3-stage ONNX pipeline, all sessions kept alive by the ort Environment:
  1. nemo128.onnx — log-mel features waveforms/waveforms_lens → features/features_lens.
  2. encoder-model.int8.onnx (preferred, auto-detected) or encoder-model.onnx — audio_signal/length → outputs[1,1024,T']/encoded_lengths.
  3. decoder_joint-model.onnx — greedy TDT decoding.
• Runtime graph introspection: d_model, num_durations, state_layers and state_hidden are read from the ONNX I/O shapes instead of being hardcoded.
• Greedy TDT decode loop (token head + duration head; blank <blk> = 8192; max_tokens_per_step from config.json, default 10). Encoder output is re-packed from [1, d_model, T'] to contiguous [T', d_model] frames.
• Internal text decoding: vocab.txt (<piece> <id>, SentencePiece ▁ → space) plus the reference spacing normalizer (\A\s|\s\B|(\s)\b). The Transcript struct now carries a text: String.
• Config: STT_MODEL_PATH now points to a directory (models/parakeet by default, was models/parakeet-tdt.onnx). STT_TOKENIZER_PATH defaults to empty. Updated .env, .env.example and the README.
• TurnManager: on_eou now uses transcript.text directly, falling back to the external tokenizer only when no text was produced.

Notes / fixes found during integration

• The decoder_joint graph expects targets and target_length as int32 (not int64); encoder_outputs and both state tensors are float.
• The int8 encoder (~650 MB, self-contained) is preferred over the fp32 encoder (42 MB + a 2.4 GB .onnx.data sidecar) for speed and simpler deployment.

Verification

Opt-in end-to-end test decodes a real 16 kHz mono WAV through the model:

$ PARAKEET_TEST_WAV=/tmp/stt_test/solis16k.wav \
    cargo test --release -- --ignored test_infer_real_audio --nocapture
tokens: 67 | text: "Bienvenidos a esta fascinante exploración del universo.
  Como locutor profesional, les explicaré hoy cómo las ondas electromagnéticas
  viajan por el espacio, transformando nuestra comprensión del cosmos."
test stt::parakeet::tests::test_infer_real_audio ... ok

The transcription matches the reference transcript verbatim (the source clip was trimmed to 12 s). cargo test (unit tests) also passes.

---

2026-10-15 — Fix: short utterances swallowed by the TTS pre-buffer

Short phrases produced no audio at all. The mixer only started playing TTS once the queue held TTS_PREBUFFER_SAMPLES (1.5 s). If an utterance's total length was below that threshold (e.g. "Cristóbal Colón" ≈ 1.11 s / 49152 samples), the pre-buffer was never reached, so tts_primed stayed false and the queued samples were never drained.

Change

• src/audio/mixer.rs: added an tts_stream_end flag with mark_stream_start() / mark_stream_end(). When the stream has ended, the mixer primes as soon as any sample is queued (stream_ended && tts_len > 0), bypassing the pre-buffer for short utterances. The flag is reset by flush(), mark_stream_start() and when the queue drains.
• src/orchestrator/turn_manager.rs: the spawned TTS tasks now call mixer.mark_stream_start() before synthesis and mixer.mark_stream_end() after it returns — only when the request was not cancelled, so barge-in does not falsely signal completion.

Verification

TTS stream complete chunk_count=6 total_samples=49152 duration_s="0.64"
TTS pre-buffer reached — starting TTS playback tts_queue_len=49152 prebuffer=66150 stream_ended=true

A raw WebSocket client sending "Cristóbal Colón" now receives ~1.12 s of non-silent audio (previously the same request yielded only silence frames).

---

2026-10-15 — Dual-stack listener (fix localhost / ::1 connection refused)

The HTTP server bound only to 0.0.0.0 (IPv4). On hosts where localhost resolves to ::1 first (confirmed via getent hosts localhost → ::1 localhost), clients and preview/port-forwarding proxies that connect to localhost via IPv6 received connection refused, surfacing as "service is unavailable".

Change

• src/main.rs: the listener now binds to the IPv6 wildcard SocketAddr::from((Ipv6Addr::UNSPECIFIED, port)) instead of 0.0.0.0. With net.ipv6.bindv6only = 0 (Linux default, verified on this host) this is dual-stack: it accepts both IPv6 (::1 / localhost) and IPv4 clients.

Verification

ss -ltnp | grep 8787  → LISTEN *:8787
curl http://[::1]:8787/     → 200
curl http://localhost:8787/ → 200
curl http://127.0.0.1:8787/ → 200

---

2026-10-14 — Modular TTS plugin system (engine-agnostic via .env)

The TTS layer was hard-coded to Fish Audio / SGLang Drama 3. To support switching engines (e.g. Kokoro, or any future backend) without touching orchestration code, the TTS subsystem is now a pluggable trait selected at runtime via the TTS_ENGINE environment variable.

New TtsEngine trait (src/tts/mod.rs)

#[async_trait]
pub trait TtsEngine: Send + Sync {
    fn name(&self) -> &str;
    fn sample_rate(&self) -> u32;
    async fn synthesize(&self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken) -> Result<()>;
    fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}  // default no-op
}

• Every plugin streams 16-bit mono PCM at OUTPUT_SAMPLE_RATE (44.1 kHz) into the shared mixer queue. If an engine's native rate differs, it resamples internally (the mixer runs at a fixed rate).
• synthesize() must honor the CancellationToken for barge-in.
• set_reference() defaults to a no-op; only engines that support voice cloning override it.

Factory (src/tts/mod.rs::create_tts_engine)

TTS_ENGINE=fish      → FishAudioTts  (Fish Audio / SGLang Drama 3)
TTS_ENGINE=kokoro    → KokoroTts     (stub, implement in src/tts/kokoro.rs)

Unknown values panic at startup with a clear message.

Bundled plugins

Plugin	File	Status
fish (Fish Audio Drama 3)	src/tts/fish_audio.rs	Production-ready (renamed from sglang_client.rs)
kokoro	src/tts/kokoro.rs	Stub — HTTP streaming call left as TODO with a commented example; includes a 24 kHz → 44.1 kHz linear resampler helper

.env loading

• Added dotenvy to auto-load a .env file at startup (existing env vars take precedence).
• Added .env.example documenting all env vars including TTS_ENGINE, KOKORO_URL, KOKORO_VOICE.

Files changed

• src/tts/mod.rs — new TtsEngine trait + create_tts_engine factory.
• src/tts/fish_audio.rs — renamed from sglang_client.rs; FishAudioTts now implements TtsEngine.
• src/tts/kokoro.rs — new Kokoro plugin stub (example for adding engines).
• src/tts/sglang_client.rs — deleted (moved to fish_audio.rs).
• src/config.rs — new tts_engine field, TTS_ENGINE env var (default fish).
• src/orchestrator/turn_manager.rs — tts: SglangTtsClient → tts: Arc<dyn TtsEngine>.
• src/ws/handler.rs — AppState.tts → Arc<dyn TtsEngine>.
• src/main.rs — uses create_tts_engine(&config); loads .env via dotenvy.
• Cargo.toml — added async-trait, dotenvy.
• .env.example — new file documenting all configuration env vars.

How to add a new engine

1. Create src/tts/myengine.rs and implement TtsEngine.
2. Add pub mod myengine; to src/tts/mod.rs.
3. Register it in the create_tts_engine match.
4. Set TTS_ENGINE=myengine in .env.

Build / verification

• cargo build — 0 errors, 2 dead-code warnings (Kokoro stub fields).
• cargo test — 4/4 passed.
• Server starts with TTS_ENGINE=fish (default): log shows selected TTS engine engine="fish" and voice-cloning reference loaded.

---

2026-10-14 — Cancel TTS on WebSocket disconnect

When the client disconnected, the in-flight TTS synthesis task continued streaming from Fish Audio even though the WebSocket session was already closed, wasting bandwidth and server resources.

Fix (src/ws/session.rs)

• Added turn_manager.interrupt() in the session cleanup path (after the receiver ends, before aborting the sender/mic tasks).
• This cancels the TTS CancellationToken, flushes the mixer queues, and stops the streaming HTTP request immediately.

---

2026-10-14 — Fix WebSocket frame timing (Burst ticker)

The 20 ms audio sender ticker used MissedTickBehavior::Skip, which dropped tick beats whenever the Tokio event loop was busy (e.g. processing a large Fish Audio TTS chunk). Each skipped tick meant a 20 ms gap in frame delivery to the browser, producing audible micro-interruptions even though the server's TTS queue was well-buffered.

Fix (src/ws/session.rs)

• Changed MissedTickBehavior::Skip → MissedTickBehavior::Burst.
• Missed ticks now trigger immediate back-to-back sends so the browser receives all due frames. The browser's jitter buffer (500 ms) absorbs the burst without underrunning.

---

2026-10-14 — Micro-interruption fix (grace period on TTS drain)

Even with the 1.5 s server pre-buffer and 500 ms browser jitter buffer, brief audio gaps could still occur when the TTS queue momentarily hit zero between Fish Audio chunks. The mixer immediately reset its tts_primed state, forcing a full 1.5 s re-prime on the next chunk. The browser playback processor did the same, re-entering buffering mode on any underrun.

Server fix (src/audio/mixer.rs)

• Added tts_drain_frames counter (Arc<AtomicUsize>) and TTS_DRAIN_GRACE_FRAMES = 10 (200 ms).
• When the TTS queue is empty while primed, the mixer now outputs silence for up to 10 consecutive frames before resetting tts_primed. This absorbs brief inter-chunk gaps without re-triggering the pre-buffer.
• flush() also resets tts_drain_frames.

Browser fix (src/web/playground.html)

• PlaybackProcessor now tracks underrunSamples and only re-enters buffering mode after 100 ms (sampleRate * 0.10) of continuous underrun. Brief underruns produce silent output but keep playback running, so audio resumes immediately when the next WebSocket frame arrives.

---

2026-10-14 — Prevent overlapping TTS streams (double-Speak fix)

When the user clicked Speak twice in quick succession, two TTS synthesis tasks ran concurrently and both pushed PCM into the same mixer queue. This produced overlapping/garbled audio and roughly doubled the total duration (one observed run produced 83 s of audio for a ~42 s text).

Server fix (src/orchestrator/turn_manager.rs)

• speak() now calls self.mixer.flush() immediately after cancel_current_tts(). This clears any remaining PCM from the previous utterance so the new one starts from a clean queue.

Client fix (src/web/playground.html)

• The Speak button is now disabled (shows "Speaking...") as soon as a request is sent.
• A 1.5 s idle timer re-enables it once the server stops sending audio frames, preventing accidental double submissions.

---

2026-10-14 — Jitter buffer fixes (no more audio drop-outs)

Fish Audio Drama 3 streams PCM in unevenly-spaced chunks, which caused brief audible drop-outs (silence gaps) during playback. Fixed with a double jitter buffer.

Server-side TTS pre-buffer (src/audio/mixer.rs)

• Added TTS_PREBUFFER_SAMPLES ≈ 200 ms of audio (44.1 kHz × 0.20).
• The mixer now tracks a tts_primed flag (Arc<AtomicBool>). TTS samples are not consumed until the TTS queue holds at least the pre-buffer amount.
• During the priming window only the filler ("uhm") backchannel is emitted, so the user hears an immediate response while TTS audio builds up.
• When the TTS queue empties (stream exhausted), tts_primed resets to false for the next request. flush() also resets it (barge-in).
• AudioMixer::pull_frame now uses tts_queue.len() instead of is_empty() for priming decisions.

Browser-side playback jitter buffer (src/web/playground.html)

• The PlaybackProcessor AudioWorklet now starts in a buffering state and only begins playing once the ring buffer holds ≥ 200 ms of samples.
• On underrun (buffer drained mid-playback) it returns to buffering to avoid stuttering, then resumes once enough audio has re-accumulated.
• Added available() helper to compute the fill level of the ring buffer.

Build / verification

• cargo build — 0 warnings.
• cargo test — 4/4 passed.

---

2026-10-14 — Voice cloning + 44.1 kHz native output

Sample rate aligned to Fish Audio native output

• Changed OUTPUT_SAMPLE_RATE from 24 000 Hz to 44 100 Hz to match Fish Audio Drama 3's native PCM output.
• Removed the TTS anti-aliasing resampler from sglang_client.rs — Fish Audio PCM samples are pushed directly into the mixer queue. This eliminates any resampling artifacts ("metallic" sound) and pitch/speed mismatches.
• The browser AudioContext now runs at 44 100 Hz, so playback is sample-accurate.
• OUTPUT_FRAME_SAMPLES is now 882 (44.1 kHz × 20 ms), OUTPUT_FRAME_BYTES = 1764.
• Mic → STT path: browser capture resamples to 44.1 kHz, server resamples 44.1 kHz → 16 kHz for Parakeet (linear interpolation, STT-only path so aliasing is not a concern).

Voice cloning (reference audio)

Added support for Fish Audio's voice-cloning references field.

Config (env vars, applied at startup):

Env var	Description
TTS_REFERENCE_AUDIO	Path to a WAV file on the Fish Audio server filesystem.
TTS_REFERENCE_TEXT	Reference transcript. If it points to an existing .txt file, its contents are loaded; otherwise the raw string is used. If unset, the server looks for a <audio-stem>.txt sibling of the WAV.

Default: solis.wav + auto-loaded solis.txt from /root/pipecat-voice/voice_profiles/.

Runtime override (playground → server WebSocket text messages):

• SET_REFERENCE:<path>|<text> — set or change the reference. <text> may be empty to re-use the sibling .txt.
• CLEAR_REFERENCE — disable voice cloning (use the model's default voice).

Files changed:

• src/config.rs — new tts_reference_audio / tts_reference_text fields, Config::tts_reference() resolver with sibling-.txt fallback, new env-var parsing.
• src/tts/sglang_client.rs — Reference struct, references field in SpeechRequest (omitted when None), runtime set_reference() backed by a parking_lot::RwLock<Option<Reference>>.
• src/orchestrator/turn_manager.rs — forwards set_reference() to the TTS client.
• src/ws/session.rs — handles SET_REFERENCE: / CLEAR_REFERENCE control messages.
• src/web/playground.html — new "Voice Cloning" card with WAV path + transcript inputs and Set/Clear buttons; reference status indicator.

Build / verification

• cargo build — 0 warnings.
• cargo test — 4/4 passed (tokenizer).
• Direct Fish Audio references call confirmed HTTP 200 + x-sample-rate: 44100.
• End-to-end WebSocket test: SET_REFERENCE followed by a text-to-speak request streams 44.1 kHz PCM back to the client.