# Audio Agent Engine A production-ready **real-time, full-duplex voice agent engine** written in Rust (Tokio + Axum + ONNX Runtime). It streams microphone audio from the browser, runs speech recognition with end-of-utterance detection, injects a low-latency backchannel filler, and streams synthesized speech back to the browser — all mixed on a strict 20 ms ticker. ``` Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU │ ▼ TurnManager (orchestrator) │ ▼ LLM reasoning (OpenAI-compatible) │ ┌──────────────────────┼──────────────────────┐ ▼ ▼ ▼ Filler / backchannel TTS engine plugin Mixer (20 ms ticker) (instant "uhm" + cues) (Fish Audio / Kokoro) │ └──────────────────────┴──────────────────────┘ │ ▼ Mixed 44.1 kHz / 16-bit mono PCM ──► Browser ``` --- ## Features - **Full-duplex WebSocket audio** — 44.1 kHz / 16-bit mono PCM, 20 ms frames. - **Pluggable TTS engines** — switch engines (Fish Audio, Kokoro, …) via `.env` without changing orchestration code. See [TTS Plugin System](#tts-plugin-system). - **Latency filler** — a preloaded filler is played on end-of-utterance *only if the reply is late*: it is scheduled `FILLER_DELAY_MS` (default 2500 ms) after the turn starts and dropped if the response audio has already begun. Clips are played round-robin from `FILLERS_DIR`; `FILLER_ENABLED=false` disables it. - **Start filler** — a second latency mask on its own mixer channel: if the first audio chunk of the reply has not reached the mixer within `START_FILLER_GO_DELAY` (default 1200 ms) the agent plays a short acknowledgment ("Un momento...") while the LLM/TTS are still working, and drops it as soon as audio arrives in time. One phrase is picked **at random** from `START_FILLER_TEXTS`, so the cue varies turn to turn. Independent of the delayed latency filler above; `START_FILLER_ENABLED=false` disables it. See [Fillers](#fillers--end-of-utterance-acknowledgments). - **During-utterance backchannel** — short acknowledgments ("mm-hmm", "ajá") are mixed in at a reduced gain *while the user is speaking*, so the agent signals it is listening without taking the turn. Like the start filler the bank is a phrase list (`BACKCHANNEL_TEXTS`) with Drama-3 tags, one clip per phrase synthesized at startup. To sound natural the scheduler cues only on a VAD **micro-pause**, randomizes the timing, gates each pause probabilistically, avoids repeating the recent clips, drifts the volume slightly and follows a **category** progression through the turn (`category::phrase` lines). - **External audio injection** — a dedicated mixer channel can play a background source (direct audio URL, Icecast/Shoutcast radio, HLS, or a YouTube link) alongside the agent's voice, with an independent runtime volume and its own decoder process (`ffmpeg` for URLs/radio, `yt-dlp` for YouTube). See [External audio injection](#external-audio-injection). - **Barge-in / interrupt** — cancels in-flight synthesis and flushes audio, triggered client-side or server-side by the VAD. - **Voice cloning** — reference audio + transcript (Fish Audio `references`). - **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to `TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's start-up latency behind the previous chunk's audio. - **Double jitter buffer** — server pre-buffer + browser playback buffer absorb uneven TTS chunk arrival, eliminating drop-outs. - **Silero VAD (ONNX)** — server-side speech segmentation and fast barge-in. - **Parakeet STT (ONNX)** with tokenizer decoding and EOU detection. - **LLM reasoning** — the user transcript is sent to any OpenAI-compatible chat-completions server (vLLM, SGLang, llama.cpp, Ollama, OpenAI) and the reply is spoken by the TTS. Streaming (SSE) chunks the reply as it arrives (first chunk after a few words, tunable) for the lowest time-to-first-audio; chain-of-thought (`reasoning_content` / ` thinking`) is stripped from both the audio and the chat box. - **Tool calling** — the model can call external HTTP tools described in a JSON file (`LLM_TOOLS_FILE`) before answering. Calls are parsed client-side from the model's native `` format, executed over HTTP and fed back to the model; the tool-call block is never spoken. See [Tool calling](#tool-calling). - **Embedded playground** — a single-page HTML client served at `/`. - **Mivocero speech-to-speech client** — an example agent client at `/mivocero` with a microphone, a live chat box (user transcript + streaming assistant reply) and a text composer, useful to drive a full STT → LLM → TTS turn. - **Per-session audit log** — each WebSocket session can be recorded to a structured JSONL file: the configuration snapshot, every inbound control message and outbound event, the cue emissions, the language switches, the barge-ins and an end-of-session summary. Best-effort and off the audio hot path. See [Session audit log](#session-audit-log). --- ## Requirements - Rust **1.75+** (async traits are provided via the `async-trait` crate). - A TTS backend reachable over HTTP (e.g. Fish Audio / SGLang Drama 3). - *(Optional)* A Parakeet TDT ONNX model directory (`models/parakeet/`) for speech recognition. The server runs without STT (TTS still works via the playground text box) if the model is absent. - *(Optional)* An OpenAI-compatible chat-completions endpoint (e.g. a local vLLM / SGLang server) for the LLM reasoning stage. Without it the pipeline is a plain STT → TTS echo. - *(Optional)* `ffmpeg` on `PATH` (or `MEDIA_FFMPEG_BIN`) to inject external audio from a URL / radio stream, and `yt-dlp` (or `MEDIA_YTDLP_BIN`) to inject a YouTube link. Both are only needed for the external audio channel. --- ## Quick Start ```bash cd audio-agent-engine # 1. Configure (optional — defaults target Fish Audio Drama 3) cp .env.example .env $EDITOR .env # 2. Build and run cargo run --release # 3. Open the clients # http://localhost:8787/ playground (TTS / audio test) # http://localhost:8787/mivocero speech-to-speech agent with chat box ``` Environment variables can also be `export`ed directly; a `.env` file is loaded automatically at startup (existing env vars take precedence). ### Server control script [`server.sh`](server.sh) wraps start/stop/restart so you don't have to manage the process by hand: ```bash ./server.sh start # start (builds automatically if the binary is missing) ./server.sh start --build # rebuild, then start ./server.sh stop # graceful stop (SIGKILL after 10 s) ./server.sh restart # stop + start ./server.sh status # process / port / scheme / HTTP status ./server.sh logs 100 # tail the last 100 lines (follows) ./server.sh cert # generate a self-signed HTTPS certificate ``` It writes the PID to `run/server.pid` and appends stdout/stderr to `run/server.log`. Overridable env vars: `PROFILE` (`release`|`debug`, default `release`), `RUST_LOG` (default `info`; `audio_agent_engine::tts=debug` is always appended so the TTS streaming/latency diagnostics show up), `PORT` (default `8787`). ### HTTPS (microphone access) Browsers only expose the microphone (`getUserMedia`) in a **secure context** — HTTPS or `localhost`. If you open the playground over plain HTTP on a non-localhost origin the "Connect & Start Mic" button fails with `mic error`. Set both variables to serve HTTPS/WSS instead of HTTP: ```dotenv TLS_CERT=certs/cert.pem TLS_KEY=certs/key.pem ``` `./server.sh cert` generates a self-signed pair (SAN covers `localhost`, `127.0.0.1`, `::1` and this host's IPs; use `EXTRA_SAN=IP:1.2.3.4` to add more). Then open `https://localhost:8787/` and accept the certificate warning — the page becomes a secure context and the mic works. Comment both variables out to fall back to plain HTTP. The server logs `listening (HTTPS)` or `listening (HTTP)` at startup accordingly. --- ## Configuration All configuration is via environment variables (or a `.env` file). See [`.env.example`](.env.example) for a copy-paste template. | Variable | Default | Description | |---|---|---| | `SERVER_PORT` | `8787` | TCP port the Axum server listens on. | | `TTS_ENGINE` | `fish` | **TTS engine plugin** — `fish` or `kokoro`. See below. | | `SGLANG_URL` | `http://10.8.0.3:8000` | Base URL of the Fish Audio / SGLang server (`fish` engine). | | `TTS_MODEL` | `fishaudio/drama-3` | Model identifier sent to `/v1/audio/speech`. | | `TTS_VOICE` | `default` | Voice identifier. | | `TTS_REFERENCE_AUDIO` | `…/voice_profiles/solis.wav` | Voice-cloning reference WAV (path on the TTS server). | | `TTS_REFERENCE_TEXT` | *(empty)* | Reference transcript. If it names an existing `.txt`, its contents are loaded; otherwise the raw string is used. Empty → auto-load `.txt`. | | `TTS_TEMPERATURE` | `0.45` | Sampling temperature sent to the SGLang speech endpoint (`fish` engine). Lower values flatten the delivery — the recitative "radio drama" register lives in the distribution tails. Empty omits the field (server default). | | `TTS_TOP_P` | `0.75` | Nucleus sampling `top_p` sent to SGLang. Trims the tail of the acoustic token distribution, where exaggerated intonations live. Empty omits the field. | | `TTS_TOP_K` | *(unset)* | Optional `top_k` sampling. Unset/empty omits the field (server default). | | `TTS_STYLE_TAG` | `[conversational, casual]` | Neutral style tag prepended to every synthesis request that does not already start with a `[...]` tag. LLM replies carry no tag (the system prompt forbids stage directions), so they get this register; clip banks with their own tag are left untouched. Empty disables the prefix. **Quote the value in `.env`** (it contains a space). | | `KOKORO_URL` | `http://localhost:8880` | Kokoro HTTP endpoint (`kokoro` engine). | | `KOKORO_VOICE` | `af_heart` | Kokoro voice name. | | `STT_MODEL_PATH` | `models/parakeet` | Directory of the Parakeet TDT ONNX pipeline (see below). | | `STT_TOKENIZER_PATH` | *(empty)* | Optional external tokenizer. Parakeet decodes text internally; leave empty. | | `VAD_ENABLED` | `true` | Master switch for the Silero VAD. When `false`, the STT worker falls back to its internal RMS segmentation. | | `VAD_MODEL_PATH` | `models/silero_vad.onnx` | Silero VAD v5 ONNX model (16 kHz, 512-sample windows). | | `VAD_THRESHOLD` | `0.5` | Speech probability above which a window counts as voice. | | `VAD_NEG_THRESHOLD` | `0.35` | Probability below which a window counts as silence (hysteresis; clamped to `VAD_THRESHOLD`). | | `VAD_MIN_SPEECH_MS` | `250` | Speech required to confirm an utterance start. | | `VAD_MIN_SILENCE_MS` | `300` | Silence required to close an utterance (end-of-utterance window). | | `VAD_SPEECH_PAD_MS` | `30` | Audio kept before the detected onset so the first phoneme is not clipped. | | `VAD_BARGE_IN_MIN_SPEECH_MS` | `150` | Speech required to fire the fast barge-in onset (clamped to `VAD_MIN_SPEECH_MS`). | | `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. | | `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. | | `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. | | `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. Clips live in a per-language subdirectory `/` (e.g. `assets/fillers/es/`). | | `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. | | `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. | | `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. | | `FILLER_GAIN_DUCK` | `0.15` | Filler gain while TTS is active (ducking). Also applies to the start-filler channel. | | `START_FILLER_ENABLED` | `true` | Master switch for the **start filler** — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. `false` => no cue is ever queued. | | `START_FILLER_GO_DELAY` | `1200` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. | | `START_FILLER_TEXTS` | `[calm] Un momento...` | Phrases spoken by the start filler, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and a **random** one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, `[sigh]`, …) are passed through to the TTS. | | `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `/` (e.g. `assets/start_filler/es/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. | | `TTS_PREBUFFER_MS` | `550` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first `185.8` ms, then `371.6` ms), any value in `(186, 557]` ms primes on the *same* second window, so `550` takes the whole cushion at no extra TTFA (~`0.73 s`). Below the first window (~`186` ms) playback starts on that window (TTFA ≈ server TTFB, ~`0.38 s`) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. | | `TTS_PIPELINE_DEPTH` | `3` | How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. `1` restores strictly serial synthesis. | | `BACKCHANNEL_ENABLED` | `true` | Master switch for during-utterance acknowledgments. | | `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of the generated `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz; `bc01.raw`, `bc02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `/` (e.g. `assets/backchannel/es/`). | | `BACKCHANNEL_TEXTS` | `ack::[calm] Mm-hmm...` … | Phrases spoken by the backchannel, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and one is played per cue. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, …) are passed through to the TTS. A line may carry an optional **`category::`** prefix (e.g. `agree::[agreeing] Claro...`) used to progress through the turn; a line without it belongs to the default `ack` category. | | `BACKCHANNEL_GAIN` | `0.35` | Mix gain applied to the backchannel channel (0..1). | | `BACKCHANNEL_MIN_SPEECH_MS` | `1500` | Confirmed speech required before the first cue may be considered. | | `BACKCHANNEL_INTERVAL_MS` | `3000` | Base spacing between two cues of the same turn; randomized by ±`BACKCHANNEL_JITTER_MS`/2. | | `BACKCHANNEL_JITTER_MS` | `1200` | Timing jitter: delays the first cue by `0..JITTER` after `BACKCHANNEL_MIN_SPEECH_MS` and spreads the spacing. | | `BACKCHANNEL_PAUSE_MIN_MS` | `96` | Lower bound of the VAD micro-pause window (ms). The reaction **depth** inside a pause is drawn at random in `[MIN, MAX]`, so cues don't all land at exactly `MIN`. Below this the user is likely still mid-word. | | `BACKCHANNEL_PAUSE_MAX_MS` | `260` | Upper bound of the micro-pause window, capped below `VAD_MIN_SILENCE_MS` so a cue never lands on the pause that closes the turn. | | `BACKCHANNEL_PROB` | `0.65` | Probability (0..1) of cueing on a qualifying micro-pause — keeps the agent from acknowledging every pause. | | `BACKCHANNEL_VOLUME_JITTER` | `0.2` | Random per-cue volume scaling (± fraction), so cues don't sit at a fixed level. | | `BACKCHANNEL_MAX_PER_TURN` | `3` | Maximum backchannels emitted per user turn. | | `LANG_DEFAULT` | `es` | Base language of the **filler, start-filler and backchannel** banks — its clips live in the `/` subdirectory (e.g. `assets/backchannel/es/`). Other languages are generated on demand into their own per-language subdirectories. See [Multilingual clip banks](#multilingual-clip-banks). | | `LANG_DETECT_ENABLED` | `true` | Master switch for LLM-based language detection. Requires the LLM (`LLM_ENABLED=true`). | | `LANG_MIN_CHARS` | `12` | Minimum transcript length (chars) before a sample is sent to the detector. | | `LANG_REDETECT_EVERY` | `5` | The first user turn is always checked; afterwards the language is re-checked once every N turns. | | `LANG_LLM_FALLBACK_ENABLED` | `true` | Phrase tables cover `es/en/it/fr/de/pt`; for any other language the phrase list is generated with the LLM. `false` keeps only the tables. | | `EXTERNAL_AUDIO_GAIN` | `0.6` | Mix gain applied to the external audio channel (0..1) at session start. Overridable at runtime from the Mivocero volume slider. | | `MEDIA_FFMPEG_BIN` | `ffmpeg` | `ffmpeg` binary used to decode a URL / radio / HLS stream into PCM. | | `MEDIA_YTDLP_BIN` | `yt-dlp` | `yt-dlp` binary used to resolve a YouTube link to a direct stream URL. | | `MEDIA_YTDLP_ARGS` | *(empty)* | Extra whitespace-separated args appended to the `yt-dlp` invocation (e.g. `--cookies /path/cookies.txt` when YouTube asks for bot verification, or a custom `--extractor-args`). | | `LLM_ENABLED` | *(auto)* | Master switch for the LLM reasoning stage. Default: enabled when `LLM_BASE_URL` and `LLM_MODEL` are both set. When `false`, the pipeline is a plain STT → TTS echo. | | `LLM_BASE_URL` | *(empty)* | Base URL of the OpenAI-compatible endpoint; the client posts to `{LLM_BASE_URL}/chat/completions`. | | `LLM_API_KEY` | `EMPTY` | Bearer token sent in the `Authorization` header (local servers usually ignore it). | | `LLM_MODEL` | *(empty)* | Model identifier sent in the request body. | | `LLM_MAX_TOKENS` | `1024` | `max_tokens` for the completion. | | `LLM_TEMPERATURE` | `0.7` | Sampling temperature. | | `LLM_STREAM` | `true` | `true` → SSE token streaming, TTS starts on the first words; `false` → one blocking request, the whole reply is synthesized at once. | | `LLM_STREAM_FIRST_CHUNK_CHARS` | `20` | Characters buffered before the **first** chunk is sent to the TTS, cut on a word boundary. Small = the agent starts speaking on the first few words. `0` disables the early flush. | | `LLM_STREAM_CHUNK_CHARS` | `240` | Same threshold for every following chunk (larger chunks give better prosody and fewer TTS seams). `0` disables. | | `LLM_HISTORY_MESSAGES` | `16` | Rolling conversation window (user+assistant messages) kept across turns. | | `LLM_ENABLE_THINKING` | *(unset)* | `false` sends `chat_template_kwargs: {"enable_thinking": false}` — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers). | | `LLM_STRIP_REASONING` | `true` | Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores `reasoning_content` deltas and removes inline think blocks. `false` passes the raw output through. | | `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. | | `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). | | `LLM_TOOLS_ENABLED` | *(auto)* | Master switch for tool calling. Default: enabled when `LLM_TOOLS_FILE` is a non-empty path. | | `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method`, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). | | `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops. | | `LLM_TOOL_TIMEOUT_SECS` | `15` | Timeout of a single tool HTTP request. | | `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | Tool responses are truncated to this many characters before being handed back to the model. | | `TLS_CERT` | *(empty)* | PEM certificate chain. Set together with `TLS_KEY` to serve HTTPS/WSS. | | `TLS_KEY` | *(empty)* | PEM private key matching `TLS_CERT`. | | `SESSION_AUDIT_ENABLED` | `true` | Write a structured JSONL audit file per WebSocket session (config snapshot, inbound control messages, every outbound event, cues, language switches, barge-ins, end-of-session summary). Best-effort — a write failure never affects the session. See [Session audit log](#session-audit-log). | | `SESSION_AUDIT_DIR` | `logs/sessions` | Directory holding the per-session audit files (`.jsonl`). Relative to the process working directory; git-ignored. | --- ## TTS Plugin System The TTS layer is **engine-agnostic**. Every engine is a module under [`src/tts/`](src/tts) that implements the [`TtsEngine`](src/tts/mod.rs) trait. The active engine is selected at runtime by the `TTS_ENGINE` environment variable — the orchestration layer (`TurnManager`, `AppState`) only ever sees an `Arc`. ### The `TtsEngine` trait ```rust #[async_trait] pub trait TtsEngine: Send + Sync { /// Human-readable engine name (used for logging). fn name(&self) -> &str; /// Rate (Hz) at which this engine pushes samples into the mixer queue. /// Must equal `config::OUTPUT_SAMPLE_RATE` (44_100). fn sample_rate(&self) -> u32; /// Synthesize `text` and stream PCM into `tts_queue`. /// Must honor `cancel` for barge-in and return promptly. async fn synthesize( &self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken, ) -> Result<()>; /// Runtime voice-cloning reference update (default: no-op). fn set_reference(&self, _audio_path: Option, _text: Option) {} } ``` **Contract for implementors:** 1. Push **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44 100 Hz)** into `tts_queue`. If the engine's native rate differs, **resample internally** — the mixer runs at a fixed rate and does not resample. 2. Honor the `CancellationToken`: on cancellation, stop pushing and return `Ok(())` promptly (barge-in / interruption). 3. Override `set_reference()` only if the engine supports voice cloning. ### Engine selection (factory) [`create_tts_engine()`](src/tts/mod.rs) maps `TTS_ENGINE` to a concrete plugin. Values are case-insensitive: | `TTS_ENGINE` | Plugin type | Notes | |---|---|---| | `fish` · `fishaudio` · `fish_audio` · `sglang` | [`FishAudioTts`](src/tts/fish_audio.rs) | Fish Audio / SGLang Drama 3, OpenAI-compatible streaming. Production-ready. | | `kokoro` | [`KokoroTts`](src/tts/kokoro.rs) | Kokoro TTS. **Stub** — complete the HTTP call (see below). | An unknown value panics at startup with a clear message. ### Bundled plugins #### `fish` — Fish Audio / SGLang Drama 3 [`src/tts/fish_audio.rs`](src/tts/fish_audio.rs) - `POST {SGLANG_URL}/v1/audio/speech` with `response_format: "pcm"`, `stream: true`. - Reads the actual output rate from the response header `x-sample-rate` (Drama 3 emits **44.1 kHz**, which matches the mixer — no resampling needed). - Supports voice cloning via the `references` field. - Honors barge-in cancellation. ##### Starting the Drama 3 backend (SGLang-Omni) The `fish` engine talks to an **SGLang-Omni** server running Drama 3. The vocoder emits audio in **windows** of codec frames (2048 samples = 46.4 ms each); the window sizes are the single biggest lever on latency, so the server must be started with them tuned: ```bash sgl-omni serve \ --model-path fishaudio/drama-3 \ --host 0.0.0.0 --port 8000 \ --tts_engine.gpu_memory_fraction 0.45 \ --tts_engine.factory.max_running_requests 4 \ --tts_engine.factory.enable_async_decode true \ --vocoder.factory.stream_stride 4 \ --vocoder.factory.stream_followup_stride_ramp "[8,8,8]" \ --vocoder.factory.stream_followup_stride 8 \ --vocoder.factory.stream_overlap_tokens 16 \ --vocoder.factory.stream_crossfade_samples 512 ``` In this deployment the launcher is `/root/drama3/start-drama3.sh` and the server is kept alive by **pm2** as the process `start-drama3`: ```bash pm2 restart start-drama3 --update-env # after editing the launcher pm2 logs start-drama3 --lines 100 # CUDA-graph rebuild takes ~2 min at start ``` | Flag | Value | Meaning | |---|---|---| | `stream_stride` | `4` | **First** window, in codec frames (decoded with *no* overlap context). Sets the server TTFB floor: at `4` it is ~`0.38 s`, at `16` it is ~`1.2 s`, because the TTS engine must emit that many codec tokens before the vocoder can decode anything. | | `stream_followup_stride` | `8` | Steady-state window: `8` frames = `0.372 s`, arriving ~`0.35 s` apart (the server runs at ~`0.95×` real time), so the client queue grows and never starves. Larger windows mean **fewer seams** per second, which is what removes the audible artifacts. Upstream default `64` ⇒ `2.97 s` bursts. | | `stream_followup_stride_ramp` | `"[8,8,8]"` | Windows 2..4, as a JSON tuple. Keep it flat: a *rising* ramp (e.g. `[4,6,10]`, or the upstream `(10,12,32)`) makes later windows arrive after the small ones have drained, which gaps playback at the start of a reply. | | `stream_overlap_tokens` | `16` | Decoder warm-up context. It is decoded and **discarded** on output, so it only costs compute. This is the seam-quality lever: with too little context the streaming decoder wobbles in timbre/pitch at each seam (an audible "detune"/falsetto on long replies). `16` with stride `8` is a `(16+8)/8 = 3×` real-time decode load and stays sustainable (~`0.95×` ratio); the same `16` with stride `4` would be 5× and is **not** sustainable. Upstream default `32`. | | `stream_crossfade_samples` | `512` | ~`11.6 ms` blend at each window seam, smoothing the hand-off between windows. | Measured on this deployment (with the voice-cloning reference), server-side: | Config | TTFB | Real-time ratio | |---|---|---| | `stream_stride 4`, followup `8`, overlap `16`, crossfade `512` *(this deployment)* | **~0.38 s** | ~`0.94–0.98×` | | `stream_stride 4`, followup `4`, overlap `8`, crossfade `256` | ~`0.38 s` | ~`0.95×` (but audible detune / micro-gaps on long replies) | | `stream_stride 16`, overlap `16` | ~`1.2 s` | ~`1.0×` | The engine's Time-to-First-Audio is essentially `TTFB + (TTS_PREBUFFER_MS − first window)`, quantized to the window the pre-buffer lands on. With this config the first window is `185.8` ms and the second `371.6` ms, so any `TTS_PREBUFFER_MS` in `(186, 557]` ms primes on the **same** second window → TTFA ≈ `0.73 s`. The deployment ships `550 ms`: the whole cushion, no extra latency. Dropping to a value below the first window (~`186` ms) starts playback on the first window (TTFA ≈ `0.38 s`) but leaves little margin, and under concurrent TTS requests the per-request throughput dips below real time (measured ~`1.5×` vs ~`0.94×` alone), so each chunk seam starves the queue and pauses playback — an audible micro-gap at the start of each sentence. Two gotchas when writing the command: - The flags are **dotted keys into the pydantic config**, so they go *without* a `stages.` prefix — `--vocoder.factory.stream_followup_stride`, not `--stages.vocoder.factory.stream_followup_stride` (the latter is rejected). - The ramp tuple must be passed as a **JSON string** — `"[8,8,8]"` — because the config manager parses non-scalar paths with JSON. ##### De-emphasis — reducing Drama 3's recitative register Drama 3 tends to over-articulate: an LLM reply can come back with a theatrical, radio-drama cadence rather than a conversational tone. The engine applies two configurable mitigations, both in [`src/tts/fish_audio.rs`](src/tts/fish_audio.rs): 1. **Sampling knobs** — `temperature` / `top_p` / `top_k` are sent in the `/v1/audio/speech` payload (`TTS_TEMPERATURE` default `0.45`, `TTS_TOP_P` default `0.75`). Exaggerated intonations live in the tails of the acoustic token distribution, so trimming them flattens the delivery. Lowering `temperature` toward `0.3` flattens it further; `0.8+` makes it unstable and theatrical again. An empty value omits the field (server default). Note that this server **rejects `repetition_penalty`** with anything other than `1.0`, so that knob cannot be used. 2. **Neutral style tag** — `TTS_STYLE_TAG` (default `[conversational, casual]`) is prepended to every request whose text does not already start with a `[...]` tag. LLM replies carry no tag (the system prompt forbids stage directions), so they get this register; the filler/backchannel banks carry their own tags and are left untouched. Empty disables the prefix. Quote the value in `.env`, as it contains a space. > **`.env` parsing note.** The file is read with `dotenvy`, which **fails on > unquoted values containing spaces** — and a single bad line makes it discard the > *entire* file (the process then silently falls back to built-in defaults: no > LLM, plain HTTP). Always quote values that contain spaces, e.g. > `TTS_STYLE_TAG="[conversational, casual]"`. The remaining lever — replacing the voice-cloning reference with a more conversational one — is not applied here: the default `solis.wav` is a "locutor profesional" reading, whose prosody is part of the recitative tone. #### `kokoro` — Kokoro TTS *(stub)* [`src/tts/kokoro.rs`](src/tts/kokoro.rs) - Config via `KOKORO_URL` and `KOKORO_VOICE`. - `synthesize()` currently returns an error — uncomment and adapt the provided example to call your Kokoro HTTP server. - Kokoro typically outputs **24 kHz**; the stub includes a `resample_24k_to_output()` helper (linear interpolation) that converts to the mixer rate before pushing. Replace it with a windowed-sinc/polyphase resampler for higher fidelity if needed. ### Adding a new engine 1. Create `src/tts/my_engine.rs` and implement [`TtsEngine`](src/tts/mod.rs): ```rust use crate::audio::SampleRingBuffer; use crate::config::{Config, OUTPUT_SAMPLE_RATE}; use crate::tts::TtsEngine; use anyhow::Result; use async_trait::async_trait; use tokio_util::sync::CancellationToken; pub struct MyEngine { /* http client, url, voice, … */ } impl MyEngine { pub fn new(config: &Config) -> Self { /* read env / config */ } } #[async_trait] impl TtsEngine for MyEngine { fn name(&self) -> &str { "my_engine" } fn sample_rate(&self) -> u32 { OUTPUT_SAMPLE_RATE } async fn synthesize( &self, text: &str, tts_queue: &SampleRingBuffer, cancel: &CancellationToken, ) -> Result<()> { // stream PCM, resample to OUTPUT_SAMPLE_RATE, push into tts_queue, // and stop when `cancel.is_cancelled()`. Ok(()) } } ``` 2. Register the module and the factory arm in [`src/tts/mod.rs`](src/tts/mod.rs): ```rust pub mod my_engine; // … "my_engine" => Arc::new(MyEngine::new(config)), ``` 3. Select it at runtime: ```bash TTS_ENGINE=my_engine cargo run --release ``` --- ## STT — Parakeet TDT (pure Rust) Speech recognition is implemented entirely in Rust via ONNX Runtime — no Python or NeMo dependency. [`src/stt/parakeet.rs`](src/stt/parakeet.rs) runs the full NVIDIA **Parakeet TDT 0.6B v3** pipeline: ```text 16 kHz mono f32 └─ nemo128.onnx → mel features [1, 128, T] └─ encoder-model*.onnx → encoder out [1, 1024, T'] └─ decoder_joint-model.onnx (greedy TDT) → token IDs └─ vocab.txt → text ``` The decoder is a **transducer (TDT)**: at each encoder frame it predicts a token *and* a duration step. A positive step advances several frames at once; the blank token (``, id 8192) means "nothing emitted here". This matches the reference implementation in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr). ### Model directory `STT_MODEL_PATH` must point to a directory containing: | File | Purpose | |---|---| | `nemo128.onnx` | Log-mel feature extractor. | | `encoder-model.int8.onnx` *(preferred)* or `encoder-model.onnx` | Conformer encoder. The int8 encoder (~650 MB, self-contained) is used automatically when present; otherwise the fp32 encoder (~42 MB + a `.onnx.data` sidecar) is loaded. | | `decoder_joint-model.onnx` | TDT prediction network + joint. | | `vocab.txt` | ` ` token table (8193 entries). | | `config.json` | Optional; supplies `max_tokens_per_step` (default 10). | Model files can be fetched from [`istupakov/parakeet-tdt-0.6b-v3-onnx`](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx) into `models/parakeet/`. ### Runtime behaviour - The worker ([`SttWorker`](src/stt/parakeet.rs)) consumes 16 kHz frames. By default the session's **Silero VAD** segments the audio and hands whole utterances to `push_utterance`; without VAD it falls back to RMS-based segmentation and triggers inference (`spawn_blocking`) after a `EOU_SILENCE_MS` (300 ms) silence window. - Text is decoded **inside** the engine (SentencePiece `▁` → space, spacing normalized), so no external tokenizer is required — `transcript.text` is used directly by the [`TurnManager`](src/orchestrator/turn_manager.rs). ### Testing An opt-in end-to-end test decodes a 16 kHz mono WAV through the real model: ```bash PARAKEET_TEST_WAV=/path/to/16k_mono.wav \ cargo test --release -- --ignored test_infer_real_audio --nocapture ``` --- ## VAD — Silero V5 (ONNX) Voice Activity Detection runs entirely on the server, in Rust, on the same 16 kHz mono stream that feeds the STT engine. It is implemented in [`src/vad/`](src/vad) and detects when the user starts and stops speaking. ### Model `VAD_MODEL_PATH` points to `silero_vad.onnx` (Silero VAD **v5**). The model consumes fixed **32 ms** windows (512 samples @ 16 kHz) preceded by a 64-sample context carried over from the previous window, plus a recurrent state tensor `[2, 1, 128]` and a scalar sample-rate input. Each window yields a speech probability. The graph's tensor shapes and the sample-rate input rank are introspected at load time, so the wrapper adapts to the exported model rather than hard-coding shapes. ### Events and state machine [`VadDetector`](src/vad/silero.rs) turns per-window probabilities into three events: | Event | Trigger | Used for | |---|---|---| | `SpeechOnset` | `VAD_BARGE_IN_MIN_SPEECH_MS` (150 ms) of speech | Fast **barge-in**: interrupt the agent as soon as the user starts talking. | | `SpeechStart` | `VAD_MIN_SPEECH_MS` (250 ms) of speech | Confirms the utterance start. | | `SpeechEnd` | `VAD_MIN_SILENCE_MS` (300 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. | Onset/offset use hysteresis: a window counts as voice above `VAD_THRESHOLD` and as silence below `VAD_NEG_THRESHOLD`, which prevents flapping around a single threshold. A **pre-roll ring** (`VAD_SPEECH_PAD_MS + VAD_MIN_SPEECH_MS + one window`) is kept ahead of the detected onset so the first phonemes are never clipped. ### Barge-in When `VAD_BARGE_IN=true` and the agent is currently speaking ([`TurnManager::is_speaking`](src/orchestrator/turn_manager.rs), backed by [`AudioMixer::is_active`](src/audio/mixer.rs)), a `SpeechOnset` calls `TurnManager::interrupt()` — cancelling the in-flight TTS via its `CancellationToken` and flushing the mixer queues. ### Wiring The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one `VadDetector` per WebSocket session (the model itself is shared) and: - with `VAD_SEGMENT_STT=true` (default): forwards `SpeechEnd` utterances to `SttWorker::push_utterance`, and does **not** forward raw frames (avoids double segmentation); - with `VAD_SEGMENT_STT=false`: falls back to the STT worker's RMS segmentation, using the VAD only for barge-in; - if `VAD_ENABLED=false` or the model fails to load: legacy behaviour (RMS segmentation only). All VAD logs live under the `audio_agent_engine::vad` target; `./server.sh` appends `audio_agent_engine::vad=debug` to `RUST_LOG`, and `VAD_DEBUG=true` additionally logs every window's probability. --- ## Startup clip bootstrap The filler, start-filler and backchannel banks are plain `*.raw` PCM files (16-bit LE mono PCM at 44.1 kHz). At startup the clip banks are bootstrapped with the configured TTS engine and the files are written to disk (the directory is created if needed), so a fresh checkout works without a manual generation step. Every bank lives in a per-language subdirectory `/` (see [Multilingual clip banks](#multilingual-clip-banks)); the startup bootstrap targets the `LANG_DEFAULT` subdirectory (e.g. `assets/fillers/es/`). - Filler uses [`assets::ensure_clips`](src/assets.rs): each expected clip is checked and **synthesized only if missing**, so an existing bank is left untouched. Defaults live in [`assets::FILLER_CLIPS`](src/assets.rs) (7 clips: `uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya`). The generated voice matches the configured TTS voice-cloning reference. - The **start filler** and the **backchannel** use the generic [`assets::ensure_text_bank_clips`](src/assets.rs) (through [`ensure_start_filler_clips`](src/assets.rs) / [`ensure_backchannel_clips`](src/assets.rs)): both are driven by their phrase list (`START_FILLER_TEXTS` / `BACKCHANNEL_TEXTS`) instead of a fixed clip set, so editing the list from `.env` is enough to change what is spoken. The list the current clips were made from is recorded in `/texts.txt`; when it is unchanged and every clip is still on disk the synthesis is **skipped** (startup stays at ~2 s). Editing, adding or dropping a phrase — or deleting a clip — triggers a **full refresh** at the next restart, and clips left over from a previous list are removed. Clips are named by position (`sf01`, `sf02`, … for the start filler; `bc01`, `bc02`, … for the backchannel) and generated a few at a time (`TEXT_BANK_CONCURRENCY`); a failed synthesis keeps the previous clip of that position, and the index is only written once every clip succeeded, so a partial failure is retried next time. - Backchannel clips are only generated when `BACKCHANNEL_ENABLED=true`; its default phrase list lives in [`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs). - Generation is best-effort: if the TTS backend is unreachable each clip logs a warning and the server still starts (that bank simply stays silent). ```text INFO audio_agent_engine::assets: generating clips at startup label="filler" dir=assets/fillers/es clips=7 engine="fish-audio" INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/es/uhm.raw ... INFO audio_agent_engine::assets: refreshing backchannel clips at startup label="backchannel" dir=assets/backchannel/es clips=8 engine="fish-audio" INFO audio_agent_engine::assets: generated backchannel clip label="backchannel" name="bc01" samples=28672 path=assets/backchannel/es/bc01.raw ... INFO audio_agent_engine::assets: removed stale backchannel clip label="backchannel" path=assets/backchannel/es/mmhmm.raw INFO audio_agent_engine::assets: refreshing start-filler clips at startup label="start-filler" dir=assets/start_filler/es clips=22 engine="fish-audio" INFO audio_agent_engine::assets: generated start-filler clip label="start-filler" name="sf01" samples=36864 path=assets/start_filler/es/sf01.raw ... INFO audio_agent_engine::assets: removed stale start-filler clip label="start-filler" path=assets/start_filler/es/start_filler.raw INFO audio_agent_engine: clip banks bootstrapped generated=30 ``` On a later startup with unchanged lists: ```text INFO audio_agent_engine::assets: backchannel clips up to date — skipping synthesis label="backchannel" dir=assets/backchannel/es clips=8 INFO audio_agent_engine::assets: start-filler clips up to date — skipping synthesis label="start-filler" dir=assets/start_filler/es clips=22 ``` --- ## Fillers — end-of-utterance acknowledgments A **filler** is a short acknowledgment ("uhm", "eh") used as a **latency mask** while the reply is being generated. Because it exists to cover a *gap*, it is not played unconditionally: when the turn is dispatched ([`TurnManager::schedule_filler`](src/orchestrator/turn_manager.rs)) a timer of `FILLER_DELAY_MS` (default `2500`) is started, and the clip is queued **only if the response audio has not started playing by then** — talking over the first words of the answer is worse than a moment of silence. The check uses the mixer's `tts_playback_started` flag, which is set when the TTS pre-buffer is reached and reset at every `mark_stream_start` / `flush`. - **At most one filler per turn**, and a barge-in (`interrupt`, VAD onset, new turn, disconnect) cancels the pending timer. - `FILLER_ENABLED=false` disables the filler entirely (no clip is ever queued). - Raise `FILLER_DELAY_MS` for slower LLM/TTS setups, lower it if the filler rarely fires. Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`) and ducks it to `FILLER_GAIN_DUCK` (`0.15`) if TTS starts mid-clip. `FILLERS_DIR` (default `assets/fillers`) holds `*.raw` clips — **16-bit LE mono PCM at 44.1 kHz** — selected **round-robin** (`TurnManager::next_filler` → [`FillerBank::nth`](src/orchestrator/turn_manager.rs)), so consecutive turns use different fillers. With a single clip the behaviour is unchanged. Missing clips are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)). ### Start filler (turn-start acknowledgment when the reply is late) A second latency mask, on its own mixer channel. It works exactly like the filler above, but with its own — shorter — delay ([`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)): `START_FILLER_GO_DELAY` (default **1200 ms**) after the turn is dispatched, if the first audio chunk of the reply has **not** reached the mixer yet (`AudioMixer::tts_audio_arrived` is still `false`) the cue is queued; if audio arrived within that window, the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (at the moment anything is pushed on the TTS queue, checked on the 20 ms ticker), not on playback start: unlike `tts_playback_started`, which only flips once the `TTS_PREBUFFER_MS` jitter buffer is full, it does not wait out the pre-buffer. With the reference deployment the first chunk lands ~`0.7`–`1.0 s` after the dispatch while playback starts ~`1.9 s` later, so gating on playback made the cue fire on every turn; gating on arrival lets a reply that produces audio within `START_FILLER_GO_DELAY` suppress it. A pending cue is also dropped on barge-in / new turn via the turn's `CancellationToken`. This means a fast reply is never preceded by an acknowledgment — only a late one is masked. It lives on its own mixer channel (`start_filler_queue`), so it plays independently of the round-robin latency filler above, but it shares the filler gain, so it ducks to `FILLER_GAIN_DUCK` once the reply starts. Disabled by `START_FILLER_ENABLED=false`. Its clips live in `START_FILLER_DIR` (default `assets/start_filler`) and are **not** part of the round-robin `FILLERS_DIR` bank. One clip is synthesized per phrase of `START_FILLER_TEXTS` (default `"[calm] Un momento..."`) with the configured voice, and [`FillerBank::random`](src/orchestrator/turn_manager.rs) picks one **uniformly at random**, so the acknowledgment varies instead of repeating the same sentence. Editing the phrase list re-synthesizes the bank at the next restart (see [Startup clip bootstrap](#startup-clip-bootstrap)); a failed synthesis leaves the previous clip in place. ```text START_FILLER_TEXTS="[calm] Un momento... [thoughtful] A ver, veamos... [sigh] Vale, ya voy..." ``` --- ## Backchannel — during-utterance acknowledgments A **backchannel** is a short acknowledgment ("mm-hmm", "ajá", "claro") played *while the user is still speaking*, at a low gain, to signal that the agent is listening — without taking the turn. It is distinct from the **filler** and the **start filler** injected around end-of-utterance by the [`TurnManager`](src/orchestrator/turn_manager.rs) (delayed mask from `FILLERS_DIR` played round-robin, and a shorter-delay cue from `START_FILLER_DIR`; see [Fillers](#fillers--end-of-utterance-acknowledgments)). It is implemented in [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs). ### Clips `BACKCHANNEL_DIR` (default `assets/backchannel`) holds the generated `*.raw` clips — **16-bit LE mono PCM at 44.1 kHz** — one per phrase of `BACKCHANNEL_TEXTS`, named by position (`bc01`, `bc02`, …). Each phrase may carry an optional `category::` prefix; clips keep the list order, so the i-th clip inherits the i-th phrase's category. The default list ([`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs), Spanish with Drama-3 tags) is synthesized with the same cloned voice as the TTS. The clips are loaded once at startup into a [`FillerBank`](src/orchestrator/turn_manager.rs) and shared across sessions. Missing clips are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)). ### Trigger policy [`BackchannelScheduler`](src/orchestrator/backchannel.rs) is fed by the VAD: | Signal | Action | |---|---| | `VadEvent::SpeechStart` | begin a turn (`on_speech_start`, randomizes the first-cue instant) | | `VadEvent::SpeechEnd` | close the turn, reset counters (`on_speech_end`) | | every frame while speaking | `maybe_emit(pause_ms)` attempts a cue | `maybe_emit()` emits a clip **only when all** of the following hold: - `BACKCHANNEL_ENABLED=true` and the clip bank is non-empty; - the agent is **not** speaking (`!TurnManager::is_speaking()`) — cuing over the agent's own TTS would be an interruption, not a backchannel (checked by the caller); - the user is in a **micro-pause**: `VadDetector::silence_ms()` reached the per-pause reaction depth, a random value drawn in `[BACKCHANNEL_PAUSE_MIN_MS, BACKCHANNEL_PAUSE_MAX_MS]`, so the cue never lands mid-word *and* doesn't always react at the same depth. A zero-length pause (voice ongoing) re-arms the gate; - the first cue is due (a per-turn randomized instant, `BACKCHANNEL_MIN_SPEECH_MS + 0..BACKCHANNEL_JITTER_MS` after the speech start); - a randomized `BACKCHANNEL_INTERVAL_MS ± JITTER/2` gap elapsed since the previous cue; - fewer than `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn; - the **probabilistic gate** (`BACKCHANNEL_PROB`) does not skip this pause — so the agent acknowledges some pauses but not all. ### Naturalness Six mechanisms keep the cues from sounding like a metronome: | Mechanism | How | |---|---| | Cue on micro-pauses | gated on the VAD `silence_ms()` (a `PAUSE_MIN..PAUSE_MAX` window, 32 ms steps, capped below `VAD_MIN_SILENCE_MS`), not on a timer | | Randomized reaction depth | per pause, the depth to react at is drawn at random in `[PAUSE_MIN, PAUSE_MAX]`, so the cue doesn't always land at exactly `PAUSE_MIN` | | Randomized timing | the first cue and every following gap are jittered by `BACKCHANNEL_JITTER_MS` | | Probabilistic gate | `BACKCHANNEL_PROB` skips a share of the qualifying pauses | | Anti-repetition | the last two clip names are excluded from the pick, across turns | | Volume drift | each cue is scaled by a random ±`BACKCHANNEL_VOLUME_JITTER` factor | The phrase list is further grouped in **categories**: the scheduler prefers a category other than the previous cue's, then a category not yet used this turn, so a turn tends to start with an acknowledgment (`ack`) and drift toward agreement (`agree`) instead of repeating the same kind of nod. ### Mixing The cue is pushed into the mixer's dedicated `backchannel_queue` and summed at a fixed low gain (`BACKCHANNEL_GAIN`, default `0.35`): ```text mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) ``` The queue is included in `AudioMixer::flush()`, so a barge-in clears any pending cue along with the TTS and filler audio. At session start the effective policy is logged under `audio_agent_engine::orchestrator::backchannel` (`backchannel scheduler ready …`, with the parsed categories), and every emitted cue is logged under `audio_agent_engine::ws::session` (`backchannel emitted during user speech`, with the clip name and the pause length). --- ## Multilingual clip banks The filler, start-filler and backchannel banks are pre-synthesized in a single language — the base language `LANG_DEFAULT` (default `es`). When the user is detected speaking a different language, the engine generates that language's banks on demand and switches the session to them, **without touching** the clips of any other language. ### Layout Every language — **including the base `LANG_DEFAULT`** — lives in its own subdirectory named after its ISO 639-1 code under `FILLERS_DIR` / `START_FILLER_DIR` / `BACKCHANNEL_DIR`: ```text assets/backchannel/es/ <- base language (LANG_DEFAULT), pre-synthesized assets/fillers/es/ assets/start_filler/es/ assets/backchannel/en/ <- English, generated on demand assets/backchannel/it/ <- Italian, generated on demand assets/fillers/en/ assets/start_filler/en/ ``` Because the refresh pass (`remove_stale_clips`) only prunes `*.raw` files inside the directory it is given, scoping each language — base included — to its own subdirectory is what guarantees a regeneration never deletes another language's clips. No language is special-cased to the top level. ### Phrase source Phrases come from built-in tables (Spanish, English, Italian, French, German and Portuguese) in [`src/lang.rs`](src/lang.rs) — one table for the backchannel, one for the end-of-turn filler and one for the start filler; for any other language they are generated with the LLM (`LANG_LLM_FALLBACK_ENABLED=true`). Each language's list is then synthesized through the same [`ensure_text_bank_clips`](src/assets.rs) pipeline used at startup, so an already-generated language is only reloaded (its `texts.txt` index matches) and never re-synthesized. ### Detection & switching - A dedicated, short LLM call returns only the ISO 639-1 code ([`lang::detect_language`](src/lang.rs)); it runs on the first user turn and then once every `LANG_REDETECT_EVERY` turns (default `5`), so a mid-conversation switch is picked up without paying for a detection call on every turn. - Detection only runs when `LANG_DETECT_ENABLED=true`, the LLM is present, at least one language-aware cue (backchannel / filler / start filler) is enabled and the transcript is at least `LANG_MIN_CHARS` long (shorter samples are too noisy to classify). - If the detected code equals the base language the session falls back to the startup banks; otherwise the language's clips are ensured/loaded into a new bank and the session switches to it. - The active language is shared between the consumers through an `Arc` ([`src/lang.rs`](src/lang.rs)): the end-of-turn and start fillers ([`TurnManager`](src/orchestrator/turn_manager.rs)) read the active bank per turn, while the backchannel scheduler is re-pointed via [`BackchannelScheduler::retarget`](src/orchestrator/backchannel.rs) whenever the language generation counter changes. Detection runs in the background, so it never blocks the reply — it takes effect from the following turn. ### Configuration | Variable | Default | Purpose | |---|---|---| | `LANG_DEFAULT` | `es` | Language of the startup filler / start-filler / backchannel banks (each in its own `/` subdirectory) | | `LANG_DETECT_ENABLED` | `true` | Master switch for LLM language detection | | `LANG_MIN_CHARS` | `12` | Minimum transcript length before detecting | | `LANG_REDETECT_EVERY` | `5` | Re-check the language every N user turns | | `LANG_LLM_FALLBACK_ENABLED` | `true` | Generate phrases via LLM for languages absent from the tables | --- ## LLM reasoning When `LLM_ENABLED=true` (the default once `LLM_BASE_URL` + `LLM_MODEL` are set), [`TurnManager`](src/orchestrator/turn_manager.rs) inserts a reasoning stage between STT and TTS: ``` microphone → VAD → STT ─┐ ├─→ LLM (chat completions) → chunker → TTS → mixer → browser chat box ──────────────┘ ``` - **Client** — [`src/llm.rs`](src/llm.rs). A minimal OpenAI-compatible client (`POST {LLM_BASE_URL}/chat/completions`) with two modes, selectable at runtime via `LLM_STREAM`: - `chat_stream` — SSE token streaming (`stream: true`); deltas are handled as they arrive. - `chat` — a single blocking request (`stream: false`). - **Chain-of-thought handling** — reasoning models (Qwen3, …) may leak their plan in three different ways. Each is handled by an independent, `.env`- switchable mechanism, because a single one does not cover every deployment: 1. `LLM_ENABLE_THINKING=false` — server-side switch: the request carries `chat_template_kwargs: {"enable_thinking": false}` (vLLM/SGLang), so the model answers directly and generates no reasoning at all. Left unset, the field is not sent, keeping the body valid for strict OpenAI-compatible servers. 2. `LLM_STRIP_REASONING=true` (default) — client-side filter: separate `reasoning_content` deltas (vLLM with a reasoning parser) are dropped and inline think-tag blocks are removed by an incremental stripper that is stable across delta boundaries. 3. `LLM_DIRECT_ANSWERS=true` (default) — prompt-level guard: appends [`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt, telling the model not to narrate its reasoning. This also covers servers with no reasoning parser at all, where the chain-of-thought is written straight into `content` as ordinary prose. - **Streaming chunking** — the assistant text is split on `.`, `!`, `?`, `;` and newlines as it streams, and each chunk is dispatched to the TTS as soon as it is ready. A word-boundary flush bounds the latency for punctuation-free runs: the **first** chunk is cut once `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`) characters are buffered — so the agent starts speaking on the first few words instead of waiting for a whole sentence — and every following chunk at `LLM_STREAM_CHUNK_CHARS` (default `240`). Set either to `0` to disable that flush (chunks then end on sentence punctuation only). Larger follow-up chunks mean fewer TTS requests, hence fewer chunk seams to starve the queue. - **Pipelined synthesis** — chunks are *not* synthesized one after another. Each chunk is requested as soon as the LLM produces it (up to `TTS_PIPELINE_DEPTH`, default `3`, in flight at once) and streams into its own staging buffer; a relay forwards those buffers to the mixer **in submission order**. This overlaps each request's start-up latency with the previous chunk's playback, so raising the pre-buffer is no longer needed to avoid inter-chunk gaps. - **History** — a rolling window of the last `LLM_HISTORY_MESSAGES` user/assistant messages is kept per connection and prepended with `LLM_SYSTEM_PROMPT`. - **Barge-in** — an interrupt (client JSON, VAD onset, or disconnect) cancels both the in-flight HTTP request ([`CancellationToken`] in the `reqwest` `tokio::select!`, including while waiting for the next SSE chunk) and the TTS tasks, then flushes the mixer. Chat-box turns run in their own task so the session's receive loop stays responsive and keeps accepting `interrupt` messages while a reply is being generated. - **Failure handling** — request errors are reported to the client as `{"type":"error","text":…}` and the turn ends cleanly. --- ## Tool calling With `LLM_TOOLS_FILE` pointing at a JSON file, the model can call external HTTP services before answering. Tools are plain descriptions of an HTTP request, so any service (weather API, webhook, home automation, internal REST endpoint) can be exposed without touching the code: ```json [ { "name": "get_weather", "description": "Current weather and temperature for a city.", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] }, "url": "https://wttr.in/{city}?format=3", "method": "GET", "headers": { "X-Api-Key": "…" } } ] ``` - `url` may contain `{arg}` placeholders substituted from the call arguments. - `GET` sends the arguments that were **not** consumed by a placeholder as query parameters; `POST` (the default) sends them as a JSON body. - `headers` are optional and sent verbatim (API keys, etc.). See [`tools.example.json`](tools.example.json) for a ready-to-use file. ### Why a client-side parser The deployment's OpenAI-compatible server (vLLM) runs **without** a tool parser (`--enable-auto-tool-choice --tool-call-parser …`), so it never returns structured `tool_calls` or `finish_reason: "tool_calls"`. The tools are still advertised in every request (`tools` + `tool_choice: "auto"`), so the Qwen model knows about them and emits the call **as text** in its native format: ```text Paris ``` [`src/tools.rs`](src/tools.rs) parses that format client-side and works with any server, no server-side reconfiguration needed. ### Turn loop Each turn runs a bounded loop (`LLM_TOOL_MAX_ITERATIONS`, default `4`): 1. Stream one reply. A `ToolCallStripper` removes `` blocks from the streamed text — a block that is still incomplete is **held back**, so it is never spoken by the TTS nor shown in the chat box even when split across deltas. The visible text is spoken as usual, so the assistant can say something short ("Let me check…") before calling. 2. If the round produced no tool call, the turn ends. 3. Otherwise the assistant message (with its native call) plus one `tool` role message per result are appended, and the loop re-prompts the model, which can then answer using the results. Tool errors (unknown tool, transport failure, non-2xx status) are returned to the model as text, so it can recover and still produce an answer; the turn never fails because of a tool. Only the *spoken* text is written to the rolling history — the tool exchange stays local to the turn. The `tool_call` event is also forwarded to the client for observability: ```json { "type": "tool_call", "name": "get_weather", "arguments": { "city": "Paris" } } ``` --- ## External audio injection The mixer has a dedicated **external channel** (`external_queue`) that plays an audio source injected from the client — a direct audio URL, an Icecast/Shoutcast radio stream, an HLS playlist, or a YouTube link — mixed together with the agent's voice. It is implemented by [`src/media.rs`](src/media.rs) (`MediaSource`) and driven over the WebSocket by the `audio_source` message (see [WebSocket Protocol](#websocket-protocol)). The Mivocero client exposes it in its footer bar. ### Why `ffmpeg` / `yt-dlp` The channel does not decode audio itself: it spawns an external decoder process whose raw PCM output feeds the ring buffer. - **`ffmpeg`** (URL / radio / HLS) — reads the source and writes `-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1`, i.e. exactly the mixer's format (44.1 kHz, 16-bit LE, mono). URL inputs use `-reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5` so a dropped radio connection is retried instead of ending the stream. - **`yt-dlp`** (YouTube) — used only to *resolve* a YouTube page to a direct media URL (`-f bestaudio/best -g --no-playlist`); that URL is then handed to `ffmpeg`. YouTube-specific cookies/extractor tweaks go in `MEDIA_YTDLP_ARGS` (e.g. `--cookies /path/cookies.txt` if the media server asks to "confirm you're not a bot"). Both binaries are configurable (`MEDIA_FFMPEG_BIN`, `MEDIA_YTDLP_BIN`) and are only required when the external channel is actually used. ### Backpressure A live radio always produces audio at real time, but a bounded VOD source (a YouTube video, a file) decodes far faster than real time. The decode task respects the channel's `remaining_capacity()` and sleeps 20 ms whenever the queue is full, so a fast source is throttled to the mixer's drain rate instead of overflowing the ring buffer (which would drop the oldest samples). ### Independence from the agent The external channel is deliberately **not** cleared by `AudioMixer::flush()`: a barge-in stops the *agent* (TTS + fillers), not the background music. It also does not count towards `AudioMixer::is_active()` — the agent is not considered to be "speaking" because the music is playing. Its volume is a runtime atomic (`AudioMixer::set_external_gain`) so the client slider takes effect on the next mixed frame. ### Lifecycle | Step | What happens | |---|---| | `play` | Any current source is stopped; state `loading` is emitted. For a YouTube URL `yt-dlp` resolves the direct URL first. | | ready | `ffmpeg` is spawned and reading begins; state `playing` is emitted. | | streaming | Decoded PCM is pushed into `external_queue` under backpressure. | | `stop` / new `play` / disconnect | The `CancellationToken` is cancelled and the child processes are killed (`kill_on_drop(true)`); `stopped` (or `error`) is emitted. | A monotonic generation counter guards against a stale task clearing the state of a newer source. --- ## Mivocero chat client A ready-to-use speech-to-speech example client is served at **`/mivocero`** (embedded from [`src/web/mivocero.html`](src/web/mivocero.html)): - **Microphone** — the same AudioWorklet capture path as the playground (44.1 kHz, 16-bit LE mono, 20 ms frames). - **Chat box** — shows the user transcript (from STT) and the assistant reply **as it streams**, plus messages typed in the composer. - **Karaoke highlighting** — the assistant bubble is rendered word by word: each word lights up as it is *spoken*, driven by the audio playhead (the playback worklet reports the samples it has actually emitted). Words already spoken are bright, the word currently sounding is accented, and the rest stay dim. The rate is an estimate (`KARAOKE_CPS`, ~15 chars/s) — it only affects the visual highlight, never the audio. - **Text composer** — typing and sending a message runs a full LLM → TTS turn without using the microphone (`{"type":"user_text","text":…}`). - **Interrupt button** — flushes audio and cancels the in-flight reply (`{"type":"interrupt"}`). - **External audio bar** — footer controls to inject a background source by URL (direct audio / radio / YouTube), with **Play** / **Stop** buttons, a volume slider (0–150 %, sent as `EXTERNAL_AUDIO_GAIN` × slider) and a live state label driven by the server's `audio_source` events. Demonstrates the [external audio channel](#external-audio-injection). - **Cue indicators** — a row of three tiles (**Filler**, **Start filler**, **Backchannel**) that flash and bump a session counter each time the server emits the matching cue, driven by the `{"type":"cue"}` events. The detail line shows the backchannel category (or the clip name), so the emission cadence of the latency masks is visible live. Counters reset on each connect. - **Cue methods panel** — a collapsible **Cue methods** section below the tiles that documents the method used by each cue (when it fires, how a clip is chosen) and shows the **live parameters** reported by the server in the `{"type":"cues"}` event sent at session start: filler delay / gains / clip count, start-filler go-delay, and the backchannel gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap and categories — all read from the effective runtime configuration (no hard-coded values). - **Idle auto-disconnect** — the session closes itself after **2 minutes** (`IDLE_TIMEOUT_MS`) without any traffic in either direction. The timer is reset by every server event *and* audio frame received, and by every user action sent (typed message, Stop, media controls) — so an ongoing reply or a playing external source counts as activity. On expiry the WebSocket is closed, the microphone is released, an explanatory message is appended to the chat and the **Connect** button becomes available again. Open `http://localhost:8787/mivocero` (or `https://…` for remote microphone access — see [HTTPS](#https-microphone-access)). --- ## Session audit log Every WebSocket session can be recorded to its own structured **JSONL** file — one JSON object per line — under `SESSION_AUDIT_DIR` (default `logs/sessions`), named `s--.jsonl`. It is meant for post-mortem analysis and for tuning the latency knobs (why a filler fired, when a cue landed, how long the first audio took). Each record carries a wall-clock timestamp, the elapsed time since the session started, an event discriminator and a payload: ```json {"ts":1729170000000,"elapsed_ms":12,"event":"session_start","data":{"config":{…},"runtime":{…}}} {"ts":1729170000123,"elapsed_ms":135,"event":"control","data":{"type":"user_text","text":"…"}} {"ts":1729170000456,"elapsed_ms":468,"event":"out","data":{"type":"assistant_delta","text":"Hola,"}} {"ts":1729170000500,"elapsed_ms":512,"event":"out","data":{"type":"cue","kind":"start_filler","name":"sf02"}} {"ts":1729170009000,"elapsed_ms":9012,"event":"session_end","data":{"reason":"client_disconnected","counts":{…}}} ``` | `event` | When | `data` | |---|---|---| | `session_start` | Session opens | `config` — the full non-secret configuration snapshot; `runtime` — which of STT / VAD / tokenizer / LLM are actually active (plus the LLM model). | | `control` | `user_text` control message received | The text turn. | | `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. | | `interrupt` | Client barge-in (`interrupt` JSON or legacy `INTERRUPT`) | `source` (`client` / `client_legacy`). | | `barge_in` | VAD-driven barge-in | `prob` (onset speech probability). | | `vad_speech_start` / `vad_speech_end` | Utterance boundaries | `prob`; `duration_ms`, `avg_prob`. | | `media` | External audio control | `action` (`play`/`stop`/`volume`) with `url` / `gain`. | | `reference` | Voice-cloning reference change | `action` (`set`/`clear`) and the audio path. | | `speak` | Direct TTS request (playground / fallback text) | `text`. | | `language_switch` | Detected language changed | `from`, `to`, `generated` (whether clip banks were synthesized). | | `error` | Transport / receiver error | `where`, `text`. | | `session_end` | Session closes | `reason`, `duration_ms`, `session_id`, `file`, `counts` (per-event totals). | The sink is **best-effort**: if the file cannot be opened, or a write fails, the session runs unchanged (the failure is logged once in the tracing output and the sink then goes quiet). Files are appended and flushed after every record, so the log survives a crash or a `pm2 restart`. The directory is git-ignored. --- ## WebSocket Protocol Connect to `ws://:8787/ws`. ### Client → Server | Message | Description | |---|---| | **Binary** | Raw 16-bit LE mono PCM microphone frames (44.1 kHz). Resampled to 16 kHz server-side for STT. | | **Text — `{"type":"user_text","text":"…"}`** | Text turn from the chat box: runs the same LLM → TTS path as a spoken utterance (no STT). | | **Text — `{"type":"interrupt"}`** | Barge-in: cancel the in-flight LLM reply / TTS and flush all queued audio. | | **Text — `{"type":"audio_source","action":"play","url":"…"}`** | Inject an external audio source (direct URL / radio / HLS / YouTube) into the mixer's external channel, replacing any current one. | | **Text — `{"type":"audio_source","action":"stop"}`** | Stop the external source and clear the channel. | | **Text — `{"type":"audio_source","action":"volume","gain":0.6}`** | Set the external channel gain (0..1) at runtime. | | **Text — `INTERRUPT`** *(also `interrupt`, `BAARGEIN`, `bargein`)* | Legacy barge-in form (kept for the playground). | | **Text — `SET_REFERENCE:\|`** | Set/change the voice-cloning reference. `` may be omitted to re-use the sibling `.txt`. | | **Text — `CLEAR_REFERENCE`** | Disable voice cloning. | | **Text — anything else** | Synthesize the text directly (useful for testing TTS without STT). | ### Server → Client | Message | Description | |---|---| | **Binary** | One 20 ms mixed frame of 16-bit LE mono PCM @ 44.1 kHz (`1764` bytes). Silence is sent when no audio is queued, keeping the stream alive. | | **Text — `{"type":"user","text":"…"}`** | The user's turn (STT transcript or chat-box message) — rendered in the chat box. | | **Text — `{"type":"assistant_delta","text":"…"}`** | Streaming chunk of the assistant reply (chain-of-thought already removed). | | **Text — `{"type":"assistant_end"}`** | The assistant turn is complete. | | **Text — `{"type":"tool_call","name":"…","arguments":{…}}`** | The model called a tool; the call has been executed (result is not echoed in the chat box, only in the model's next round). Purely informational. | | **Text — `{"type":"error","text":"…"}`** | LLM request failed; the turn was aborted. | | **Text — `{"type":"audio_source","state":"…","url":"…"}`** | External audio channel state change: `loading` (decoding is being set up / YouTube URL resolving), `playing` (PCM is streaming), `stopped`, or `error` (with a `text` message). | | **Text — `{"type":"cue","kind":"…","name":"…","category":"…"}`** | A latency-mask / backchannel cue was actually emitted into the mixer: `kind` is `filler` (end-of-turn), `start_filler` (turn start) or `backchannel` (during user speech); `name` is the clip name and `category` (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators. | | **Text — `{"type":"cues","lang":"…","config":{…}}`** | Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (`delay_ms`, gains, clip count), the start-filler `go_delay_ms` and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's **Cue methods** panel so it shows the live values instead of hard-coded ones. | --- ## Audio Pipeline Constants are defined in [`src/config.rs`](src/config.rs): | Constant | Value | Meaning | |---|---|---| | `OUTPUT_SAMPLE_RATE` | `44_100` Hz | Server → browser rate (matches Fish Audio Drama 3 native output). | | `STT_SAMPLE_RATE` | `16_000` Hz | Rate expected by Parakeet. | | `FRAME_DURATION_MS` | `20` ms | Mixer ticker period. | | `OUTPUT_FRAME_SAMPLES` | `882` | Samples per frame (44.1 kHz × 20 ms). | | `OUTPUT_FRAME_BYTES` | `1764` | Bytes per frame (16-bit mono). | | `EOU_SILENCE_MS` | `300` ms | Silence window that marks end-of-utterance. | | `VAD_RMS_THRESHOLD` | `0.01` | RMS above which audio counts as voice. | ### Mixer & jitter buffering ([`src/audio/mixer.rs`](src/audio/mixer.rs)) - Mixing formula: `mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) + (external × external_gain)`. The filler gain is `1.0` when TTS is idle, else `filler_gain_duck` (default `0.15`); the backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`). The start-filler channel is summed with the filler, so it shares the same gain. The external channel gain starts at `EXTERNAL_AUDIO_GAIN` (default `0.6`) and is adjustable at runtime (`AudioMixer::set_external_gain`). Output is clamped to `i16`. - **Pre-buffer:** TTS playback does not start until the queue holds ≥ `TTS_PREBUFFER_MS` (default **550 ms**) of synthesized audio. It absorbs uneven TTS chunk arrival; lower it to start speaking sooner, raise it for fewer drop-outs. Note: Drama-3's *granularity* is set by the server's vocoder window (`--vocoder.factory.stream_stride` / `stream_followup_stride`, in codec frames of 2048 samples = 46.4 ms). With the deployment used here (stride `4`, followup `8`, overlap `16`) the first window is 4 frames = 0.186 s and the steady windows are 8 frames = 0.372 s, delivered ~0.35 s apart (the server runs at ~0.95× real time). Because `TTFA ≈ TTFB + (prebuffer − window)`, any pre-buffer in `(186, 557]` ms primes on the **same** second window → TTFA ≈ `0.73 s`; the default `550 ms` takes that whole cushion at no extra latency. A pre-buffer below ~`186 ms` starts playback on the first window (TTFA ≈ `0.38 s`) but leaves almost no margin: under concurrent TTS requests one GPU serves several requests and the per-request throughput dips below real time (~`1.5×` vs ~`0.94×` alone), so every chunk seam starves the queue and pauses playback → an audible micro-gap at the start of each sentence. With the upstream default (`64`, a 2.97 s burst every ~2.7 s) gapless playback would instead require a ~3 s pre-buffer; a large constant stride (`16`) both raises the TTFB (~1.2 s) and bursts, and a growing ramp gaps the tail while the small early windows drain. - **Resilient resume:** the full pre-buffer is only required at the *first* playback of a reply. If the queue empties mid-reply (after the 200 ms drain grace) playback pauses, but resumes as soon as *any* audio is queued again — it does not wait to refill the whole jitter buffer. `mark_stream_start` re-arms the pre-buffer at the beginning of every turn. - **Drain grace:** brief inter-chunk empties (< 200 ms) emit silence without pausing playback, avoiding audible drop-outs on ordinary arrival jitter. - **Ticker:** `MissedTickBehavior::Burst` — missed ticks are caught up in a burst rather than dropped, so no 20 ms frames are lost. --- ## Project Layout ``` audio-agent-engine/ ├── Cargo.toml ├── .env.example # documented configuration template ├── server.sh # start / stop / restart / status / logs / cert ├── certs/ # self-signed TLS cert+key (generated) ├── UPDATES.md # changelog (English, newest-first) ├── tools.example.json # tool definitions for LLM tool calling (copy/edit) ├── assets/fillers// # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin), one dir per language ├── assets/start_filler// # sf01.raw … turn-start start-filler clips + texts.txt index ├── assets/backchannel// # bc01.raw … backchannel clips + texts.txt index ├── models/ # Parakeet ONNX + Silero VAD + tokenizer └── src/ ├── main.rs # Axum server, routes, graceful shutdown ├── config.rs # env-var config + audio constants ├── assets.rs # startup clip bootstrap (auto-generate fillers/backchannel/start filler) ├── tls.rs # optional HTTPS/WSS termination (rustls) ├── onnx.rs # shared process-wide ONNX Runtime environment ├── audio/ # ring_buffer, mixer ├── tts/ # ← plugin system │ ├── mod.rs # TtsEngine trait + create_tts_engine factory │ ├── fish_audio.rs # Fish Audio / SGLang Drama 3 plugin │ └── kokoro.rs # Kokoro plugin (stub) ├── stt/ # Parakeet ONNX + tokenizer + worker ├── vad/ # Silero VAD v5 + onset/offset state machine ├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping) ├── tools.rs # configurable HTTP tool calling (native parser + executor) ├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp ├── audit.rs # per-session JSONL audit log (config snapshot, events, summary) ├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel ├── ws/ # handler, session (per-connection audio loop) └── web/ # embedded playground.html + mivocero.html ``` ### HTTP routes | Route | Description | |---|---| | `GET /` | Playground single-page client (embedded at compile time). | | `GET /mivocero` | Mivocero speech-to-speech example client with chat box (embedded at compile time). | | `GET /health` | Liveness probe (`ok`). | | `GET /ws` | WebSocket upgrade → per-session audio loop. | --- ## Development ```bash # Build cargo build # Run tests (tokenizer unit tests) cargo test # Run with debug logging RUST_LOG=debug cargo run # Release build (LTO + thin codegen) cargo run --release ``` --- ## Related Documentation - [`UPDATES.md`](UPDATES.md) — full changelog (English, newest-first). - [`src/tts/mod.rs`](src/tts/mod.rs) — trait + factory source of truth.