Audio Agent Engine

A production-ready real-time, full-duplex voice agent engine written in Rust (Tokio + Axum + ONNX Runtime). It streams microphone audio from the browser, runs speech recognition with end-of-utterance detection, injects a low-latency backchannel filler, and streams synthesized speech back to the browser — all mixed on a strict 20 ms ticker.

Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
                                            │
                                            ▼
                                   TurnManager (orchestrator)
                                            │
                                            ▼
                            LLM reasoning (OpenAI-compatible)
                                            │
                     ┌──────────────────────┼──────────────────────┐
                     ▼                      ▼                      ▼
            Filler / backchannel     TTS engine plugin       Mixer (20 ms ticker)
          (instant "uhm" + cues)   (Fish Audio / Kokoro)              │
                     └──────────────────────┴──────────────────────┘
                                            │
                                            ▼
                              Mixed 44.1 kHz / 16-bit mono PCM ──► Browser

---

Features

• Full-duplex WebSocket audio — 44.1 kHz / 16-bit mono PCM, 20 ms frames.
• JavaScript client SDK — a dependency-free, single-file ESM module (browser
  • Node) wrapping the WebSocket session and the REST surface, so the client can live on another server. See Client SDK (JavaScript).
• Pluggable TTS engines — switch engines (Fish Audio, Kokoro, …) via .env without changing orchestration code. See TTS Plugin System.
• Latency filler — a preloaded filler is played on end-of-utterance only if the reply is late: it is scheduled FILLER_DELAY_MS (default 2500 ms) after the turn starts and dropped if the response audio has already begun. Clips are played round-robin from FILLERS_DIR; FILLER_ENABLED=false disables it.
• Start filler — a second latency mask on its own mixer channel: if the first audio chunk of the reply has not reached the mixer within START_FILLER_GO_DELAY (default 1800 ms) the agent plays a short acknowledgment ("Un momento...") while the LLM/TTS are still working, and drops it as soon as audio arrives in time. One phrase is picked at random from START_FILLER_TEXTS, so the cue varies turn to turn. Independent of the delayed latency filler above; START_FILLER_ENABLED=false disables it. See Fillers.
• During-utterance backchannel — short acknowledgments ("mm-hmm", "ajá") are mixed in at a reduced gain while the user is speaking, so the agent signals it is listening without taking the turn. Like the start filler the bank is a phrase list (BACKCHANNEL_TEXTS) with Drama-3 tags, one clip per phrase synthesized at startup. To sound natural the scheduler cues only on a VAD micro-pause, randomizes the timing, gates each pause probabilistically, avoids repeating the recent clips, drifts the volume slightly and follows a category progression through the turn (category::phrase lines).
• External audio injection — a dedicated mixer channel can play a background source (direct audio URL, Icecast/Shoutcast radio, HLS, or a YouTube link) alongside the agent's voice, with an independent runtime volume and its own decoder process (ffmpeg for URLs/radio, yt-dlp for YouTube). See External audio injection.
• Barge-in / interrupt — cancels in-flight synthesis and flushes audio, triggered client-side or server-side by the VAD.
• Voice cloning — reference audio + transcript (Fish Audio references).
• Custom voice builder — a dedicated /customvoice page (linked from the Mivocero side menu) asks the user to pick a language, then either read a predefined phrase for at least 10 s or upload an audio file with its typed transcript; the take is transcoded to a WAV reference and installed as the active voice at runtime. See Custom voice builder.
• Pipelined TTS — reply chunks are synthesized ahead of playback (up to TTS_PIPELINE_DEPTH at once) and replayed in order, hiding each request's start-up latency behind the previous chunk's audio.
• OpenAPI / Swagger UI — the REST endpoints are documented by an utoipa OpenAPI document rendered as a Swagger UI at /docs (raw JSON at /api-docs/openapi.json). See API documentation.
• File transcription endpoint — POST /api/transcribe decodes an uploaded audio file with ffmpeg and returns the Parakeet transcript of the whole take (reusing the live session's STT model). See Transcription endpoint.
• Double jitter buffer — server pre-buffer + browser playback buffer absorb uneven TTS chunk arrival, eliminating drop-outs.
• Silero VAD (ONNX) — server-side speech segmentation and fast barge-in.
• Parakeet STT (ONNX) with tokenizer decoding and EOU detection.
• LLM reasoning — the user transcript is sent to any OpenAI-compatible chat-completions server (vLLM, SGLang, llama.cpp, Ollama, OpenAI) and the reply is spoken by the TTS. Streaming (SSE) chunks the reply as it arrives (first chunk after a few words, tunable) for the lowest time-to-first-audio; chain-of-thought (reasoning_content / thinking) is stripped from both the audio and the chat box.
• Tool calling — the model can call external HTTP tools described in a JSON file (LLM_TOOLS_FILE) before answering, or installed at runtime with POST /api/tools (in-memory override; without it the file definitions are used). Calls are parsed client-side from either the structured tool_calls array or the model's native <tool_call> format (a round that streams back empty is re-asked without streaming), executed over HTTP and fed back to the model; the tool-call block is never spoken. See Tool calling.
• Per-session system prompt — each WebSocket session gets a server-assigned id (delivered to the client in a {"type":"session","id":"…"} message) and the LLM system prompt can be read and overridden for that session — or process-wide — at runtime through GET/POST /api/prompt, without a restart. See Runtime system prompt.
• Embedded playground — a single-page HTML client served at /.
• Mivocero speech-to-speech client — an example agent client at /mivocero with a microphone, a live chat box (user transcript + streaming assistant reply) and a text composer, useful to drive a full STT → LLM → TTS turn.
• Per-session audit log — each WebSocket session can be recorded to a structured JSONL file: the configuration snapshot, every inbound control message and outbound event, the cue emissions, the language switches, the barge-ins and an end-of-session summary. Best-effort and off the audio hot path. See Session audit log.

---

Requirements

• Rust 1.75+ (async traits are provided via the async-trait crate).
• A TTS backend reachable over HTTP (e.g. Fish Audio / SGLang Drama 3).
• (Optional) A Parakeet TDT ONNX model directory (models/parakeet/) for speech recognition. The server runs without STT (TTS still works via the playground text box) if the model is absent.
• (Optional) An OpenAI-compatible chat-completions endpoint (e.g. a local vLLM / SGLang server) for the LLM reasoning stage. Without it the pipeline is a plain STT → TTS echo.
• (Optional) ffmpeg on PATH (or MEDIA_FFMPEG_BIN) to inject external audio from a URL / radio stream, and yt-dlp (or MEDIA_YTDLP_BIN) to inject a YouTube link. Both are only needed for the external audio channel.

---

Quick Start

cd audio-agent-engine

# 1. Configure (optional — defaults target Fish Audio Drama 3)
cp .env.example .env
$EDITOR .env

# 2. Build and run
cargo run --release

# 3. Open the clients
#    http://localhost:8787/             playground (TTS / audio test)
#    http://localhost:8787/mivocero     speech-to-speech agent with chat box
#    http://localhost:8787/customvoice  custom-voice builder (record / upload a reference)
#    http://localhost:8787/docs         Swagger UI (REST API documentation)

Environment variables can also be exported directly; a .env file is loaded automatically at startup (existing env vars take precedence).

Server control script

server.sh wraps start/stop/restart so you don't have to manage the process by hand:

./server.sh start            # start (builds automatically if the binary is missing)
./server.sh start --build    # rebuild, then start
./server.sh stop             # graceful stop (SIGKILL after 10 s)
./server.sh restart          # stop + start
./server.sh status           # process / port / scheme / HTTP status
./server.sh logs 100         # tail the last 100 lines (follows)
./server.sh cert             # generate a self-signed HTTPS certificate

It writes the PID to run/server.pid and appends stdout/stderr to run/server.log. Overridable env vars: PROFILE (release|debug, default release), RUST_LOG (default info; audio_agent_engine::tts=debug is always appended so the TTS streaming/latency diagnostics show up), PORT (default 8787).

HTTPS (microphone access)

Browsers only expose the microphone (getUserMedia) in a secure context — HTTPS or localhost. If you open the playground over plain HTTP on a non-localhost origin the "Connect & Start Mic" button fails with mic error.

Set both variables to serve HTTPS/WSS instead of HTTP:

TLS_CERT=certs/cert.pem
TLS_KEY=certs/key.pem

./server.sh cert generates a self-signed pair (SAN covers localhost, 127.0.0.1, ::1 and this host's IPs; use EXTRA_SAN=IP:1.2.3.4 to add more). Then open https://localhost:8787/ and accept the certificate warning — the page becomes a secure context and the mic works. Comment both variables out to fall back to plain HTTP. The server logs listening (HTTPS) or listening (HTTP) at startup accordingly.

---

Configuration

All configuration is via environment variables (or a .env file). See .env.example for a copy-paste template.

Variable	Default	Description
SERVER_PORT	8787	TCP port the Axum server listens on.
TTS_ENGINE	fish	TTS engine plugin — fish or kokoro. See below.
SGLANG_URL	http://10.8.0.3:8000	Base URL of the Fish Audio / SGLang server (fish engine).
TTS_MODEL	fishaudio/drama-3	Model identifier sent to /v1/audio/speech.
TTS_VOICE	default	Voice identifier.
TTS_REFERENCE_AUDIO	…/voice_profiles/solis.wav	Voice-cloning reference WAV (path on the TTS server).
TTS_REFERENCE_TEXT	(empty)	Reference transcript. If it names an existing .txt, its contents are loaded; otherwise the raw string is used. Empty → auto-load <audio-stem>.txt.
TTS_TEMPERATURE	0.45	Sampling temperature sent to the SGLang speech endpoint (fish engine). Lower values flatten the delivery — the recitative "radio drama" register lives in the distribution tails. Empty omits the field (server default).
TTS_TOP_P	0.75	Nucleus sampling top_p sent to SGLang. Trims the tail of the acoustic token distribution, where exaggerated intonations live. Empty omits the field.
TTS_TOP_K	(unset)	Optional top_k sampling. Unset/empty omits the field (server default).
TTS_STYLE_TAG	[conversational, casual]	Neutral style tag prepended to every synthesis request that does not already start with a [...] tag. LLM replies carry no tag (the system prompt forbids stage directions), so they get this register; clip banks with their own tag are left untouched. Empty disables the prefix. Quote the value in .env (it contains a space).
KOKORO_URL	http://localhost:8880	Kokoro HTTP endpoint (kokoro engine).
KOKORO_VOICE	af_heart	Kokoro voice name.
STT_MODEL_PATH	models/parakeet	Directory of the Parakeet TDT ONNX pipeline (see below).
STT_TOKENIZER_PATH	(empty)	Optional external tokenizer. Parakeet decodes text internally; leave empty.
VAD_ENABLED	true	Master switch for the Silero VAD. When false, the STT worker falls back to its internal RMS segmentation.
VAD_MODEL_PATH	models/silero_vad.onnx	Silero VAD v5 ONNX model (16 kHz, 512-sample windows).
VAD_THRESHOLD	0.5	Speech probability above which a window counts as voice.
VAD_NEG_THRESHOLD	0.35	Probability below which a window counts as silence (hysteresis; clamped to VAD_THRESHOLD).
VAD_MIN_SPEECH_MS	250	Speech required to confirm an utterance start.
VAD_MIN_SILENCE_MS	700	Silence required to close an utterance (end-of-utterance window).
VAD_SPEECH_PAD_MS	30	Audio kept before the detected onset so the first phoneme is not clipped.
VAD_BARGE_IN_MIN_SPEECH_MS	150	Speech required to fire the fast barge-in onset (clamped to VAD_MIN_SPEECH_MS).
VAD_MIN_UTTERANCE_MS	300	Utterance gate: drop a segment shorter than this before transcription (0 disables).
VAD_MIN_AVG_PROB	0.25	Utterance gate: drop a segment whose average speech probability is below this before transcription (0.0 disables). The average is taken over the voiced windows only (probability ≥ VAD_NEG_THRESHOLD), so the trailing hangover silence does not dilute short words like "uno" / "due". Kept well below VAD_THRESHOLD.
VAD_SEGMENT_STT	true	Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation.
VAD_BARGE_IN	true	Interrupt the agent when the user starts speaking over it.
VAD_DEBUG	false	Log the speech probability of every analysis window.
FILLERS_DIR	assets/fillers	Directory of *.raw filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. Clips live in a per-language, per-voice-profile subdirectory <dir>/<lang>/<voice_profile> (e.g. assets/fillers/es/maricel/).
FILLER_ENABLED	true	Master switch for the end-of-turn filler. false => no filler is ever queued.
FILLER_DELAY_MS	2500	How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer.
FILLER_GAIN_IDLE	1.0	Filler gain when TTS is idle.
FILLER_GAIN_DUCK	0.15	Fallback filler gain while TTS is already active. Normally a still-queued filler is faded out (80 ms) and cut when the reply's playback starts, so this only applies to a cue queued after playback has begun. Also applies to the start-filler channel.
START_FILLER_ENABLED	true	Master switch for the start filler — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. false => no cue is ever queued.
START_FILLER_GO_DELAY	1800	How long (ms) the agent waits for the first audio chunk of the reply to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk arrival (AudioMixer::tts_audio_arrived), not on playback start, so it does not wait for the TTS_PREBUFFER_MS jitter buffer. Same idea as FILLER_DELAY_MS, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. If the cue does fire and the reply then starts, it is faded out (80 ms) and cut so it does not overlap the answer.
START_FILLER_TEXTS	[calm] Un momento...	Phrases spoken by the start filler, one per line (a | separator also works). One clip is synthesized per phrase and a random one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags ([calm], [sigh], …) are passed through to the TTS.
START_FILLER_DIR	assets/start_filler	Directory holding the start-filler clips (sf01.raw, sf02.raw, …, plus a texts.txt list index). Clips live in a per-language, per-voice-profile subdirectory <dir>/<lang>/<voice_profile> (e.g. assets/start_filler/es/maricel/). Kept out of FILLERS_DIR so the clips are not part of the round-robin filler bank.
ASSET_VOICE_PROFILE	(auto)	Optional explicit cue-asset profile name. When unset, the engine derives it from the active voice setup (prefer TTS_VOICE when it is not default, otherwise fall back to the reference-audio file stem, then to default).
TTS_PREBUFFER_MS	550	Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first 185.8 ms, then 371.6 ms), any value in (186, 557] ms primes on the same second window, so 550 takes the whole cushion at no extra TTFA (0.73 s). Below the first window (186 ms) playback starts on that window (TTFA ≈ server TTFB, ~0.38 s) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again.
TTS_PIPELINE_DEPTH	3	How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. 1 restores strictly serial synthesis.
BACKCHANNEL_ENABLED	true	Master switch for during-utterance acknowledgments.
BACKCHANNEL_DIR	assets/backchannel	Directory of the generated *.raw backchannel PCM clips (16-bit LE mono, 44.1 kHz; bc01.raw, bc02.raw, …, plus a texts.txt list index). Clips live in a per-language, per-voice-profile subdirectory <dir>/<lang>/<voice_profile> (e.g. assets/backchannel/es/maricel/).
BACKCHANNEL_TEXTS	ack::[calm] Mm-hmm... …	Phrases spoken by the backchannel, one per line (a | separator also works). One clip is synthesized per phrase and one is played per cue. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags ([calm], …) are passed through to the TTS. A line may carry an optional category:: prefix (e.g. agree::[agreeing] Claro...) used to progress through the turn; a line without it belongs to the default ack category.
BACKCHANNEL_GAIN	0.35	Mix gain applied to the backchannel channel (0..1).
BACKCHANNEL_MIN_SPEECH_MS	1500	Confirmed speech required before the first cue may be considered.
BACKCHANNEL_INTERVAL_MS	3000	Base spacing between two cues of the same turn; randomized by ±BACKCHANNEL_JITTER_MS/2.
BACKCHANNEL_JITTER_MS	1200	Timing jitter: delays the first cue by 0..JITTER after BACKCHANNEL_MIN_SPEECH_MS and spreads the spacing.
BACKCHANNEL_PAUSE_MIN_MS	96	Lower bound of the VAD micro-pause window (ms). The reaction depth inside a pause is drawn at random in [MIN, MAX], so cues don't all land at exactly MIN. Below this the user is likely still mid-word.
BACKCHANNEL_PAUSE_MAX_MS	260	Upper bound of the micro-pause window, capped below VAD_MIN_SILENCE_MS so a cue never lands on the pause that closes the turn.
BACKCHANNEL_PROB	0.65	Probability (0..1) of cueing on a qualifying micro-pause — keeps the agent from acknowledging every pause.
BACKCHANNEL_VOLUME_JITTER	0.2	Random per-cue volume scaling (± fraction), so cues don't sit at a fixed level.
BACKCHANNEL_MAX_PER_TURN	3	Maximum backchannels emitted per user turn.
LANG_DEFAULT	es	Base language of the filler, start-filler and backchannel banks — its clips live in the <dir>/<lang>/<voice_profile> subdirectory (e.g. assets/backchannel/es/maricel/). Other languages are generated on demand into their own language/voice-profile subdirectories. See Multilingual clip banks.
LANG_DETECT_ENABLED	true	Master switch for LLM-based language detection. Requires the LLM (LLM_ENABLED=true).
LANG_MIN_CHARS	12	Minimum transcript length (chars) before a sample is sent to the detector.
LANG_REDETECT_EVERY	5	The first user turn is always checked; afterwards the language is re-checked once every N turns.
LANG_LLM_FALLBACK_ENABLED	true	Phrase tables cover es/en/it/fr/de/pt; for any other language the phrase list is generated with the LLM. false keeps only the tables.
EXTERNAL_AUDIO_GAIN	0.6	Mix gain applied to the external audio channel (0..1) at session start. Overridable at runtime from the Mivocero volume slider.
MEDIA_FFMPEG_BIN	ffmpeg	ffmpeg binary used to decode a URL / radio / HLS stream into PCM.
MEDIA_YTDLP_BIN	yt-dlp	yt-dlp binary used to resolve a YouTube link to a direct stream URL.
MEDIA_YTDLP_ARGS	(empty)	Extra whitespace-separated args appended to the yt-dlp invocation (e.g. --cookies /path/cookies.txt when YouTube asks for bot verification, or a custom --extractor-args).
LLM_ENABLED	(auto)	Master switch for the LLM reasoning stage. Default: enabled when LLM_BASE_URL and LLM_MODEL are both set. When false, the pipeline is a plain STT → TTS echo.
LLM_BASE_URL	(empty)	Base URL of the OpenAI-compatible endpoint; the client posts to {LLM_BASE_URL}/chat/completions.
LLM_API_KEY	EMPTY	Bearer token sent in the Authorization header (local servers usually ignore it).
LLM_MODEL	(empty)	Model identifier sent in the request body.
LLM_MAX_TOKENS	1024	max_tokens for the completion.
LLM_TEMPERATURE	0.7	Sampling temperature.
LLM_STREAM	true	true → SSE token streaming, TTS starts on the first words; false → one blocking request, the whole reply is synthesized at once.
LLM_STREAM_FIRST_CHUNK_CHARS	20	Characters buffered before the first chunk is sent to the TTS, cut on a word boundary. Small = the agent starts speaking on the first few words. 0 disables the early flush.
LLM_STREAM_CHUNK_CHARS	240	Same threshold for every following chunk (larger chunks give better prosody and fewer TTS seams). 0 disables.
LLM_HISTORY_MESSAGES	16	Rolling conversation window (user+assistant messages) kept across turns.
LLM_ENABLE_THINKING	(unset)	false sends chat_template_kwargs: {"enable_thinking": false} — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers).
LLM_STRIP_REASONING	true	Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores reasoning_content deltas and removes inline think blocks. false passes the raw output through.
LLM_DIRECT_ANSWERS	true	Append an "answer directly, never narrate your reasoning" directive to the system prompt.
LLM_SYSTEM_PROMPT	(built-in)	Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). Can also be overridden at runtime per session via GET/POST /api/prompt (Runtime system prompt).
LLM_TOOLS_ENABLED	(auto)	Master switch for tool calling. Default: enabled when LLM_TOOLS_FILE is a non-empty path.
LLM_TOOLS_FILE	(empty)	JSON file with the tool definitions (name, description, parameters JSON schema, url, method — any HTTP verb, headers). See Tool calling and tools.example.json.
LLM_TOOL_MAX_ITERATIONS	4	Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops.
LLM_TOOL_TIMEOUT_SECS	15	Timeout of a single tool HTTP request.
LLM_TOOL_RESULT_MAX_CHARS	2000	Tool responses are truncated to this many characters before being handed back to the model.
TLS_CERT	(empty)	PEM certificate chain. Set together with TLS_KEY to serve HTTPS/WSS.
TLS_KEY	(empty)	PEM private key matching TLS_CERT.
CORS_ALLOW_ORIGIN	(empty)	Comma-separated browser origins allowed to call the REST surface cross-origin (single * = any). Empty adds no CORS layer (default behaviour). Needed when the client SDK is served from another origin. Does not affect /ws (a WebSocket handshake is not a CORS request).
AUTH_API_BASE	https://aquachile.azurianlab.com	Base URL of the auth API proxied by POST /api/auth/phone-login (Phone login).
AUTH_PHONE_LOGIN_PATH	/api/v1/auth/phone-login	Login path on AUTH_API_BASE.
SESSION_AUDIT_ENABLED	true	Write a structured JSONL audit file per WebSocket session (config snapshot, inbound control messages, every outbound event, cues, language switches, barge-ins, end-of-session summary). Best-effort — a write failure never affects the session. See Session audit log.
SESSION_AUDIT_DIR	logs/sessions	Directory holding the per-session audit files (<id>.jsonl). Relative to the process working directory; git-ignored.

---

TTS Plugin System

The TTS layer is engine-agnostic. Every engine is a module under src/tts/ that implements the TtsEngine trait. The active engine is selected at runtime by the TTS_ENGINE environment variable — the orchestration layer (TurnManager, AppState) only ever sees an Arc<dyn TtsEngine>.

The TtsEngine trait

#[async_trait]
pub trait TtsEngine: Send + Sync {
    /// Human-readable engine name (used for logging).
    fn name(&self) -> &str;

    /// Rate (Hz) at which this engine pushes samples into the mixer queue.
    /// Must equal `config::OUTPUT_SAMPLE_RATE` (44_100).
    fn sample_rate(&self) -> u32;

    /// Synthesize `text` and stream PCM into `tts_queue`.
    /// Must honor `cancel` for barge-in and return promptly.
    async fn synthesize(
        &self,
        text: &str,
        tts_queue: &SampleRingBuffer,
        cancel: &CancellationToken,
    ) -> Result<()>;

    /// Runtime voice-cloning reference update (default: no-op).
    fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}
}

Contract for implementors:

1. Push 16-bit mono PCM at OUTPUT_SAMPLE_RATE (44 100 Hz) into tts_queue. If the engine's native rate differs, resample internally — the mixer runs at a fixed rate and does not resample.
2. Honor the CancellationToken: on cancellation, stop pushing and return Ok(()) promptly (barge-in / interruption).
3. Override set_reference() only if the engine supports voice cloning.

Engine selection (factory)

create_tts_engine() maps TTS_ENGINE to a concrete plugin. Values are case-insensitive:

TTS_ENGINE	Plugin type	Notes
fish · fishaudio · fish_audio · sglang	FishAudioTts	Fish Audio / SGLang Drama 3, OpenAI-compatible streaming. Production-ready.
kokoro	KokoroTts	Kokoro TTS. Stub — complete the HTTP call (see below).

An unknown value panics at startup with a clear message.

Bundled plugins

fish — Fish Audio / SGLang Drama 3

src/tts/fish_audio.rs

• POST {SGLANG_URL}/v1/audio/speech with response_format: "pcm", stream: true.
• Reads the actual output rate from the response header x-sample-rate (Drama 3 emits 44.1 kHz, which matches the mixer — no resampling needed).
• Supports voice cloning via the references field.
• Honors barge-in cancellation.

Starting the Drama 3 backend (SGLang-Omni)

The fish engine talks to an SGLang-Omni server running Drama 3. The vocoder emits audio in windows of codec frames (2048 samples = 46.4 ms each); the window sizes are the single biggest lever on latency, so the server must be started with them tuned:

sgl-omni serve \
    --model-path fishaudio/drama-3 \
    --host 0.0.0.0 --port 8000 \
    --tts_engine.gpu_memory_fraction 0.45 \
    --tts_engine.factory.max_running_requests 4 \
    --tts_engine.factory.enable_async_decode true \
    --vocoder.factory.stream_stride 4 \
    --vocoder.factory.stream_followup_stride_ramp "[8,8,8]" \
    --vocoder.factory.stream_followup_stride 8 \
    --vocoder.factory.stream_overlap_tokens 16 \
    --vocoder.factory.stream_crossfade_samples 512

In this deployment the launcher is /root/drama3/start-drama3.sh and the server is kept alive by pm2 as the process start-drama3:

pm2 restart start-drama3 --update-env   # after editing the launcher
pm2 logs start-drama3 --lines 100       # CUDA-graph rebuild takes ~2 min at start

Flag	Value	Meaning
stream_stride	4	First window, in codec frames (decoded with no overlap context). Sets the server TTFB floor: at 4 it is ~0.38 s, at 16 it is ~1.2 s, because the TTS engine must emit that many codec tokens before the vocoder can decode anything.
stream_followup_stride	8	Steady-state window: 8 frames = 0.372 s, arriving ~0.35 s apart (the server runs at ~0.95× real time), so the client queue grows and never starves. Larger windows mean fewer seams per second, which is what removes the audible artifacts. Upstream default 64 ⇒ 2.97 s bursts.
stream_followup_stride_ramp	"[8,8,8]"	Windows 2..4, as a JSON tuple. Keep it flat: a rising ramp (e.g. [4,6,10], or the upstream (10,12,32)) makes later windows arrive after the small ones have drained, which gaps playback at the start of a reply.
stream_overlap_tokens	16	Decoder warm-up context. It is decoded and discarded on output, so it only costs compute. This is the seam-quality lever: with too little context the streaming decoder wobbles in timbre/pitch at each seam (an audible "detune"/falsetto on long replies). 16 with stride 8 is a (16+8)/8 = 3× real-time decode load and stays sustainable (~0.95× ratio); the same 16 with stride 4 would be 5× and is not sustainable. Upstream default 32.
stream_crossfade_samples	512	~11.6 ms blend at each window seam, smoothing the hand-off between windows.

Measured on this deployment (with the voice-cloning reference), server-side:

Config	TTFB	Real-time ratio
stream_stride 4, followup 8, overlap 16, crossfade 512 (this deployment)	~0.38 s	~0.94–0.98×
stream_stride 4, followup 4, overlap 8, crossfade 256	~0.38 s	~0.95× (but audible detune / micro-gaps on long replies)
stream_stride 16, overlap 16	~1.2 s	~1.0×

The engine's Time-to-First-Audio is essentially TTFB + (TTS_PREBUFFER_MS − first window), quantized to the window the pre-buffer lands on. With this config the first window is 185.8 ms and the second 371.6 ms, so any TTS_PREBUFFER_MS in (186, 557] ms primes on the same second window → TTFA ≈ 0.73 s. The deployment ships 550 ms: the whole cushion, no extra latency. Dropping to a value below the first window (~186 ms) starts playback on the first window (TTFA ≈ 0.38 s) but leaves little margin, and under concurrent TTS requests the per-request throughput dips below real time (measured ~1.5× vs ~0.94× alone), so each chunk seam starves the queue and pauses playback — an audible micro-gap at the start of each sentence.

Two gotchas when writing the command:

• The flags are dotted keys into the pydantic config, so they go without a stages. prefix — --vocoder.factory.stream_followup_stride, not --stages.vocoder.factory.stream_followup_stride (the latter is rejected).
• The ramp tuple must be passed as a JSON string — "[8,8,8]" — because the config manager parses non-scalar paths with JSON.

De-emphasis — reducing Drama 3's recitative register

Drama 3 tends to over-articulate: an LLM reply can come back with a theatrical, radio-drama cadence rather than a conversational tone. The engine applies two configurable mitigations, both in src/tts/fish_audio.rs:

1. Sampling knobs — temperature / top_p / top_k are sent in the /v1/audio/speech payload (TTS_TEMPERATURE default 0.45, TTS_TOP_P default 0.75). Exaggerated intonations live in the tails of the acoustic token distribution, so trimming them flattens the delivery. Lowering temperature toward 0.3 flattens it further; 0.8+ makes it unstable and theatrical again. An empty value omits the field (server default). Note that this server rejects repetition_penalty with anything other than 1.0, so that knob cannot be used.
2. Neutral style tag — TTS_STYLE_TAG (default [conversational, casual]) is prepended to every request whose text does not already start with a [...] tag. LLM replies carry no tag (the system prompt forbids stage directions), so they get this register; the filler/backchannel banks carry their own tags and are left untouched. Empty disables the prefix. Quote the value in .env, as it contains a space.

.env parsing note. The file is read with dotenvy, which fails on unquoted values containing spaces — and a single bad line makes it discard the entire file (the process then silently falls back to built-in defaults: no LLM, plain HTTP). Always quote values that contain spaces, e.g. TTS_STYLE_TAG="[conversational, casual]".

The remaining lever — replacing the voice-cloning reference with a more conversational one — is not applied here: the default solis.wav is a "locutor profesional" reading, whose prosody is part of the recitative tone.

kokoro — Kokoro TTS (stub)

src/tts/kokoro.rs

• Config via KOKORO_URL and KOKORO_VOICE.
• synthesize() currently returns an error — uncomment and adapt the provided example to call your Kokoro HTTP server.
• Kokoro typically outputs 24 kHz; the stub includes a resample_24k_to_output() helper (linear interpolation) that converts to the mixer rate before pushing. Replace it with a windowed-sinc/polyphase resampler for higher fidelity if needed.

Adding a new engine

1. Create src/tts/my_engine.rs and implement TtsEngine:

   use crate::audio::SampleRingBuffer;
   use crate::config::{Config, OUTPUT_SAMPLE_RATE};
   use crate::tts::TtsEngine;
   use anyhow::Result;
   use async_trait::async_trait;
   use tokio_util::sync::CancellationToken;
   
   pub struct MyEngine { /* http client, url, voice, … */ }
   
   impl MyEngine {
       pub fn new(config: &Config) -> Self { /* read env / config */ }
   }
   
   #[async_trait]
   impl TtsEngine for MyEngine {
       fn name(&self) -> &str { "my_engine" }
       fn sample_rate(&self) -> u32 { OUTPUT_SAMPLE_RATE }
   
       async fn synthesize(
           &self,
           text: &str,
           tts_queue: &SampleRingBuffer,
           cancel: &CancellationToken,
       ) -> Result<()> {
           // stream PCM, resample to OUTPUT_SAMPLE_RATE, push into tts_queue,
           // and stop when `cancel.is_cancelled()`.
           Ok(())
       }
   }
   

2. Register the module and the factory arm in src/tts/mod.rs:

   pub mod my_engine;
   // …
   "my_engine" => Arc::new(MyEngine::new(config)),
   

3. Select it at runtime:

   TTS_ENGINE=my_engine cargo run --release
   

---

STT — Parakeet TDT (pure Rust)

Speech recognition is implemented entirely in Rust via ONNX Runtime — no Python or NeMo dependency. src/stt/parakeet.rs runs the full NVIDIA Parakeet TDT 0.6B v3 pipeline:

16 kHz mono f32
  └─ nemo128.onnx             → mel features   [1, 128, T]
       └─ encoder-model*.onnx → encoder out    [1, 1024, T']
            └─ decoder_joint-model.onnx (greedy TDT) → token IDs
                 └─ vocab.txt  → text

The decoder is a transducer (TDT): at each encoder frame it predicts a token and a duration step. A positive step advances several frames at once; the blank token (<blk>, id 8192) means "nothing emitted here". This matches the reference implementation in istupakov/onnx-asr.

Model directory

STT_MODEL_PATH must point to a directory containing:

File	Purpose
nemo128.onnx	Log-mel feature extractor.
encoder-model.int8.onnx (preferred) or encoder-model.onnx	Conformer encoder. The int8 encoder (~650 MB, self-contained) is used automatically when present; otherwise the fp32 encoder (~42 MB + a .onnx.data sidecar) is loaded.
decoder_joint-model.onnx	TDT prediction network + joint.
vocab.txt	<piece> <id> token table (8193 entries).
config.json	Optional; supplies max_tokens_per_step (default 10).

Model files can be fetched from istupakov/parakeet-tdt-0.6b-v3-onnx into models/parakeet/.

Runtime behaviour

• The worker (SttWorker) consumes 16 kHz frames. By default the session's Silero VAD segments the audio and hands whole utterances to push_utterance; without VAD it falls back to RMS-based segmentation and triggers inference (spawn_blocking) after a EOU_SILENCE_MS (300 ms) silence window.
• Text is decoded inside the engine (SentencePiece ▁ → space, spacing normalized), so no external tokenizer is required — transcript.text is used directly by the TurnManager.

Testing

An opt-in end-to-end test decodes a 16 kHz mono WAV through the real model:

PARAKEET_TEST_WAV=/path/to/16k_mono.wav \
  cargo test --release -- --ignored test_infer_real_audio --nocapture

---

VAD — Silero V5 (ONNX)

Voice Activity Detection runs entirely on the server, in Rust, on the same 16 kHz mono stream that feeds the STT engine. It is implemented in src/vad/ and detects when the user starts and stops speaking.

Model

VAD_MODEL_PATH points to silero_vad.onnx (Silero VAD v5). The model consumes fixed 32 ms windows (512 samples @ 16 kHz) preceded by a 64-sample context carried over from the previous window, plus a recurrent state tensor [2, 1, 128] and a scalar sample-rate input. Each window yields a speech probability. The graph's tensor shapes and the sample-rate input rank are introspected at load time, so the wrapper adapts to the exported model rather than hard-coding shapes.

Events and state machine

VadDetector turns per-window probabilities into three events:

Event	Trigger	Used for
SpeechOnset	VAD_BARGE_IN_MIN_SPEECH_MS (150 ms) of speech	Fast barge-in: interrupt the agent as soon as the user starts talking.
SpeechStart	VAD_MIN_SPEECH_MS (250 ms) of speech	Confirms the utterance start.
SpeechEnd	VAD_MIN_SILENCE_MS (700 ms) of silence	Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine.

Onset/offset use hysteresis: a window counts as voice above VAD_THRESHOLD and as silence below VAD_NEG_THRESHOLD, which prevents flapping around a single threshold. A pre-roll ring (VAD_SPEECH_PAD_MS + VAD_MIN_SPEECH_MS + one window) is kept ahead of the detected onset so the first phonemes are never clipped.

Barge-in

When VAD_BARGE_IN=true and the agent has audible output (TurnManager::is_audible, backed by AudioMixer::is_audible), a SpeechOnset calls TurnManager::interrupt() — cancelling the in-flight TTS via its CancellationToken and flushing the mixer queues.

The gate is on audible output, not on "speaking" in general: is_audible is false while the reply is still being synthesised or is sitting in the TTS pre-buffer, i.e. before anything has reached the user's ears. This stops a VAD transient (a breath, a click, or the tail of the user's own utterance) from cancelling a reply the user has not heard any of yet. Such a reply is not left orphaned: the next turn's on_eou already cancels the previous in-flight synthesis and restarts the stream. TurnManager::is_speaking (AudioMixer::is_active) is still used for the backchannel gate, where cueing on top of any in-flight TTS would be an interruption rather than an acknowledgment.

Wiring

The session task (src/ws/session.rs) owns one VadDetector per WebSocket session (the model itself is shared) and:

• with VAD_SEGMENT_STT=true (default): forwards SpeechEnd utterances to SttWorker::push_utterance, and does not forward raw frames (avoids double segmentation). Before forwarding, each segment passes the utterance gate (VAD_MIN_UTTERANCE_MS / VAD_MIN_AVG_PROB): segments below either threshold are dropped without transcription, which filters the VAD false-triggers (breaths, clicks, acoustic echo, noise) that the STT would otherwise turn into hallucinated text. The probability average is computed over the voiced windows only (probability ≥ VAD_NEG_THRESHOLD), so the VAD_MIN_SILENCE_MS hangover that closes the segment does not drag it down — a short word like "uno" would otherwise average far below any usable gate. A transcript that still decodes to no tokens is skipped too, so empty turns are never sent to the LLM.
• with VAD_SEGMENT_STT=false: falls back to the STT worker's RMS segmentation, using the VAD only for barge-in;
• if VAD_ENABLED=false or the model fails to load: legacy behaviour (RMS segmentation only).

All VAD logs live under the audio_agent_engine::vad target; ./server.sh appends audio_agent_engine::vad=debug to RUST_LOG, and VAD_DEBUG=true additionally logs every window's probability.

---

Startup clip bootstrap

The filler, start-filler and backchannel banks are plain *.raw PCM files (16-bit LE mono PCM at 44.1 kHz). At startup the clip banks are bootstrapped with the configured TTS engine and the files are written to disk (the directory is created if needed), so a fresh checkout works without a manual generation step. Every bank lives in a per-language subdirectory <dir>/<lang> (see Multilingual clip banks); the startup bootstrap targets the LANG_DEFAULT subdirectory (e.g. assets/fillers/es/).

• Filler uses assets::ensure_clips: each expected clip is checked and synthesized only if missing, so an existing bank is left untouched. Defaults live in assets::FILLER_CLIPS (7 clips: uhm, eh, mmm, aver, bueno, pues, vaya). The generated voice matches the configured TTS voice-cloning reference.
• The start filler and the backchannel use the generic assets::ensure_text_bank_clips (through ensure_start_filler_clips / ensure_backchannel_clips): both are driven by their phrase list (START_FILLER_TEXTS / BACKCHANNEL_TEXTS) instead of a fixed clip set, so editing the list from .env is enough to change what is spoken. The list the current clips were made from is recorded in <dir>/texts.txt; when it is unchanged and every clip is still on disk the synthesis is skipped (startup stays at ~2 s). Editing, adding or dropping a phrase — or deleting a clip — triggers a full refresh at the next restart, and clips left over from a previous list are removed. Clips are named by position (sf01, sf02, … for the start filler; bc01, bc02, … for the backchannel) and generated a few at a time (TEXT_BANK_CONCURRENCY); a failed synthesis keeps the previous clip of that position, and the index is only written once every clip succeeded, so a partial failure is retried next time.
• Backchannel clips are only generated when BACKCHANNEL_ENABLED=true; its default phrase list lives in DEFAULT_BACKCHANNEL_TEXTS.
• Generation is best-effort: if the TTS backend is unreachable each clip logs a warning and the server still starts (that bank simply stays silent).

INFO audio_agent_engine::assets: generating clips at startup label="filler" dir=assets/fillers/es clips=7 engine="fish-audio"
INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/es/uhm.raw
...
INFO audio_agent_engine::assets: refreshing backchannel clips at startup label="backchannel" dir=assets/backchannel/es clips=8 engine="fish-audio"
INFO audio_agent_engine::assets: generated backchannel clip label="backchannel" name="bc01" samples=28672 path=assets/backchannel/es/bc01.raw
...
INFO audio_agent_engine::assets: removed stale backchannel clip label="backchannel" path=assets/backchannel/es/mmhmm.raw
INFO audio_agent_engine::assets: refreshing start-filler clips at startup label="start-filler" dir=assets/start_filler/es clips=22 engine="fish-audio"
INFO audio_agent_engine::assets: generated start-filler clip label="start-filler" name="sf01" samples=36864 path=assets/start_filler/es/sf01.raw
...
INFO audio_agent_engine::assets: removed stale start-filler clip label="start-filler" path=assets/start_filler/es/start_filler.raw
INFO audio_agent_engine: clip banks bootstrapped generated=30

On a later startup with unchanged lists:

INFO audio_agent_engine::assets: backchannel clips up to date — skipping synthesis label="backchannel" dir=assets/backchannel/es clips=8
INFO audio_agent_engine::assets: start-filler clips up to date — skipping synthesis label="start-filler" dir=assets/start_filler/es clips=22

---

Fillers — end-of-utterance acknowledgments

A filler is a short acknowledgment ("uhm", "eh") used as a latency mask while the reply is being generated. Because it exists to cover a gap, it is not played unconditionally: when the turn is dispatched (TurnManager::schedule_filler) a timer of FILLER_DELAY_MS (default 2500) is started, and the clip is queued only if the response audio has not started playing by then — talking over the first words of the answer is worse than a moment of silence. The check uses the mixer's tts_playback_started flag, which is set when the TTS pre-buffer is reached and reset at every mark_stream_start / flush.

• At most one filler per turn, and a barge-in (interrupt, VAD onset, new turn, disconnect) cancels the pending timer.
• FILLER_ENABLED=false disables the filler entirely (no clip is ever queued).
• Raise FILLER_DELAY_MS for slower LLM/TTS setups, lower it if the filler rarely fires.

Once queued, the mixer plays it at FILLER_GAIN_IDLE (1.0). If TTS playback starts mid-clip, the cue is faded out over FILLER_FADEOUT_FRAMES (4 frames = 80 ms) and then cut — it is a latency mask, so it must hand off cleanly to the reply instead of playing underneath it. (As a fallback, a cue queued after playback has already started is ducked to FILLER_GAIN_DUCK.)

FILLERS_DIR (default assets/fillers) holds *.raw clips — 16-bit LE mono PCM at 44.1 kHz — selected round-robin (TurnManager::next_filler → FillerBank::nth), so consecutive turns use different fillers. With a single clip the behaviour is unchanged. Missing clips are synthesized at startup (Startup clip bootstrap).

Start filler (turn-start acknowledgment when the reply is late)

A second latency mask, on its own mixer channel. It works exactly like the filler above, but with its own — shorter — delay (TurnManager::schedule_start_filler): START_FILLER_GO_DELAY (default 1800 ms) after the turn is dispatched, if the first audio chunk of the reply has not reached the mixer yet (AudioMixer::tts_audio_arrived is still false) the cue is queued; if audio arrived within that window, the cue is dropped instead of talking over the beginning of the answer.

The gate is latched on chunk arrival (at the moment anything is pushed on the TTS queue, checked on the 20 ms ticker), not on playback start: unlike tts_playback_started, which only flips once the TTS_PREBUFFER_MS jitter buffer is full, it does not wait out the pre-buffer. With the reference deployment the first chunk lands ~0.7–1.0 s after the dispatch while playback starts ~1.9 s later, so gating on playback made the cue fire on every turn; gating on arrival lets a reply that produces audio within START_FILLER_GO_DELAY suppress it. A pending cue is also dropped on barge-in / new turn via the turn's CancellationToken. This means a fast reply is never preceded by an acknowledgment — only a late one is masked.

It lives on its own mixer channel (start_filler_queue), so it plays independently of the round-robin latency filler above, but it shares the filler gain: when the reply's playback starts, the still-queued cue is faded out (80 ms) and cut rather than left running under the answer. Disabled by START_FILLER_ENABLED=false.

Its clips live in START_FILLER_DIR (default assets/start_filler) and are not part of the round-robin FILLERS_DIR bank. One clip is synthesized per phrase of START_FILLER_TEXTS (default "[calm] Un momento...") with the configured voice, and FillerBank::random picks one uniformly at random, so the acknowledgment varies instead of repeating the same sentence. Editing the phrase list re-synthesizes the bank at the next restart (see Startup clip bootstrap); a failed synthesis leaves the previous clip in place.

START_FILLER_TEXTS="[calm] Un momento...
[thoughtful] A ver, veamos...
[sigh] Vale, ya voy..."

---

Backchannel — during-utterance acknowledgments

A backchannel is a short acknowledgment ("mm-hmm", "ajá", "claro") played while the user is still speaking, at a low gain, to signal that the agent is listening — without taking the turn. It is distinct from the filler and the start filler injected around end-of-utterance by the TurnManager (delayed mask from FILLERS_DIR played round-robin, and a shorter-delay cue from START_FILLER_DIR; see Fillers).

It is implemented in src/orchestrator/backchannel.rs.

Clips

BACKCHANNEL_DIR (default assets/backchannel) holds the generated *.raw clips — 16-bit LE mono PCM at 44.1 kHz — one per phrase of BACKCHANNEL_TEXTS, named by position (bc01, bc02, …). Each phrase may carry an optional category:: prefix; clips keep the list order, so the i-th clip inherits the i-th phrase's category. The default list (DEFAULT_BACKCHANNEL_TEXTS, Spanish with Drama-3 tags) is synthesized with the same cloned voice as the TTS. The clips are loaded once at startup into a FillerBank and shared across sessions. Missing clips are synthesized at startup (Startup clip bootstrap).

Trigger policy

BackchannelScheduler is fed by the VAD:

Signal	Action
VadEvent::SpeechStart	begin a turn (on_speech_start, randomizes the first-cue instant)
VadEvent::SpeechEnd	close the turn, reset counters (on_speech_end)
every frame while speaking	maybe_emit(pause_ms) attempts a cue

maybe_emit() emits a clip only when all of the following hold:

• BACKCHANNEL_ENABLED=true and the clip bank is non-empty;
• the agent is not speaking (!TurnManager::is_speaking()) — cuing over the agent's own TTS would be an interruption, not a backchannel (checked by the caller);
• the user is in a micro-pause: VadDetector::silence_ms() reached the per-pause reaction depth, a random value drawn in [BACKCHANNEL_PAUSE_MIN_MS, BACKCHANNEL_PAUSE_MAX_MS], so the cue never lands mid-word and doesn't always react at the same depth. A zero-length pause (voice ongoing) re-arms the gate;
• the first cue is due (a per-turn randomized instant, BACKCHANNEL_MIN_SPEECH_MS + 0..BACKCHANNEL_JITTER_MS after the speech start);
• a randomized BACKCHANNEL_INTERVAL_MS ± JITTER/2 gap elapsed since the previous cue;
• fewer than BACKCHANNEL_MAX_PER_TURN cues were emitted this turn;
• the probabilistic gate (BACKCHANNEL_PROB) does not skip this pause — so the agent acknowledges some pauses but not all.

Naturalness

Six mechanisms keep the cues from sounding like a metronome:

Mechanism	How
Cue on micro-pauses	gated on the VAD silence_ms() (a PAUSE_MIN..PAUSE_MAX window, 32 ms steps, capped below VAD_MIN_SILENCE_MS), not on a timer
Randomized reaction depth	per pause, the depth to react at is drawn at random in [PAUSE_MIN, PAUSE_MAX], so the cue doesn't always land at exactly PAUSE_MIN
Randomized timing	the first cue and every following gap are jittered by BACKCHANNEL_JITTER_MS
Probabilistic gate	BACKCHANNEL_PROB skips a share of the qualifying pauses
Anti-repetition	the last two clip names are excluded from the pick, across turns
Volume drift	each cue is scaled by a random ±BACKCHANNEL_VOLUME_JITTER factor

The phrase list is further grouped in categories: the scheduler prefers a category other than the previous cue's, then a category not yet used this turn, so a turn tends to start with an acknowledgment (ack) and drift toward agreement (agree) instead of repeating the same kind of nod.

Mixing

The cue is pushed into the mixer's dedicated backchannel_queue and summed at a fixed low gain (BACKCHANNEL_GAIN, default 0.35):

mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN)

The queue is included in AudioMixer::flush(), so a barge-in clears any pending cue along with the TTS and filler audio. At session start the effective policy is logged under audio_agent_engine::orchestrator::backchannel (backchannel scheduler ready …, with the parsed categories), and every emitted cue is logged under audio_agent_engine::ws::session (backchannel emitted during user speech, with the clip name and the pause length).

---

Multilingual clip banks

The filler, start-filler and backchannel banks are pre-synthesized in a single language — the base language LANG_DEFAULT (default es). When the user is detected speaking a different language, the engine generates that language's banks on demand and switches the session to them, without touching the clips of any other language.

Layout

Every language — including the base LANG_DEFAULT — keeps a separate bank for each cue-asset voice profile under FILLERS_DIR / START_FILLER_DIR / BACKCHANNEL_DIR:

assets/backchannel/es/maricel/     <- base language (LANG_DEFAULT), profile maricel
assets/fillers/es/maricel/
assets/start_filler/es/maricel/
assets/backchannel/es/solis/       <- same language, different voice profile
assets/backchannel/en/maricel/     <- English, generated on demand
assets/fillers/en/maricel/
assets/start_filler/en/maricel/

On startup, if a legacy single-voice bank still exists directly under <dir>/<lang>/ (for example assets/backchannel/es/bc01.raw), the engine copies those flat files into the active <dir>/<lang>/<voice_profile>/ directory when the target files are missing. The legacy files are left in place, so the migration is non-destructive.

Because the refresh pass (remove_stale_clips) only prunes *.raw files inside the directory it is given, scoping each language — base included — to its own subdirectory is what guarantees a regeneration never deletes another language's clips. No language is special-cased to the top level.

Phrase source

Phrases come from built-in tables (Spanish, English, Italian, French, German and Portuguese) in src/lang.rs — one table for the backchannel, one for the end-of-turn filler and one for the start filler; for any other language they are generated with the LLM (LANG_LLM_FALLBACK_ENABLED=true). Each language's list is then synthesized through the same ensure_text_bank_clips pipeline used at startup, so an already-generated language is only reloaded (its texts.txt index matches) and never re-synthesized.

Detection & switching

• A dedicated, short LLM call returns only the ISO 639-1 code (lang::detect_language); it runs on the first user turn and then once every LANG_REDETECT_EVERY turns (default 5), so a mid-conversation switch is picked up without paying for a detection call on every turn.
• Detection only runs when LANG_DETECT_ENABLED=true, the LLM is present, at least one language-aware cue (backchannel / filler / start filler) is enabled and the transcript is at least LANG_MIN_CHARS long (shorter samples are too noisy to classify).
• If the detected code equals the base language the session falls back to the startup banks; otherwise the language's clips are ensured/loaded into a new bank and the session switches to it.
• The active language is shared between the consumers through an Arc<LanguageState> (src/lang.rs): the end-of-turn and start fillers (TurnManager) read the active bank per turn, while the backchannel scheduler is re-pointed via BackchannelScheduler::retarget whenever the language generation counter changes. Detection runs in the background, so it never blocks the reply — it takes effect from the following turn.

Configuration

Variable	Default	Purpose
LANG_DEFAULT	es	Language of the startup filler / start-filler / backchannel banks (each in its own <dir>/<lang> subdirectory)
LANG_DETECT_ENABLED	true	Master switch for LLM language detection
LANG_MIN_CHARS	12	Minimum transcript length before detecting
LANG_REDETECT_EVERY	5	Re-check the language every N user turns
LANG_LLM_FALLBACK_ENABLED	true	Generate phrases via LLM for languages absent from the tables

---

LLM reasoning

When LLM_ENABLED=true (the default once LLM_BASE_URL + LLM_MODEL are set), TurnManager inserts a reasoning stage between STT and TTS:

microphone → VAD → STT ─┐
                        ├─→ LLM (chat completions) → chunker → TTS → mixer → browser
chat box  ──────────────┘

• Client — src/llm.rs. A minimal OpenAI-compatible client (POST {LLM_BASE_URL}/chat/completions) with two modes, selectable at runtime via LLM_STREAM:
  • chat_stream — SSE token streaming (stream: true); deltas are handled as they arrive.
  • chat — a single blocking request (stream: false).
• Chain-of-thought handling — reasoning models (Qwen3, …) may leak their plan in three different ways. Each is handled by an independent, .env- switchable mechanism, because a single one does not cover every deployment:
  1. LLM_ENABLE_THINKING=false — server-side switch: the request carries chat_template_kwargs: {"enable_thinking": false} (vLLM/SGLang), so the model answers directly and generates no reasoning at all. Left unset, the field is not sent, keeping the body valid for strict OpenAI-compatible servers.
  2. LLM_STRIP_REASONING=true (default) — client-side filter: separate reasoning_content deltas (vLLM with a reasoning parser) are dropped and inline think-tag blocks are removed by an incremental stripper that is stable across delta boundaries.
  3. LLM_DIRECT_ANSWERS=true (default) — prompt-level guard: appends DIRECT_ANSWER_DIRECTIVE to the system prompt, telling the model not to narrate its reasoning. This also covers servers with no reasoning parser at all, where the chain-of-thought is written straight into content as ordinary prose.
• Streaming chunking — the assistant text is split on ., !, ?, ; and newlines as it streams, and each chunk is dispatched to the TTS as soon as it is ready. A word-boundary flush bounds the latency for punctuation-free runs: the first chunk is cut once LLM_STREAM_FIRST_CHUNK_CHARS (default 20) characters are buffered — so the agent starts speaking on the first few words instead of waiting for a whole sentence — and every following chunk at LLM_STREAM_CHUNK_CHARS (default 240). Set either to 0 to disable that flush (chunks then end on sentence punctuation only). Larger follow-up chunks mean fewer TTS requests, hence fewer chunk seams to starve the queue.
• Pipelined synthesis — chunks are not synthesized one after another. Each chunk is requested as soon as the LLM produces it (up to TTS_PIPELINE_DEPTH, default 3, in flight at once) and streams into its own staging buffer; a relay forwards those buffers to the mixer in submission order. This overlaps each request's start-up latency with the previous chunk's playback, so raising the pre-buffer is no longer needed to avoid inter-chunk gaps.
• History — a rolling window of the last LLM_HISTORY_MESSAGES user/assistant messages is kept per connection and prepended with the effective system prompt (the session override if any, else LLM_SYSTEM_PROMPT, else the built-in default — see Runtime system prompt).
• Barge-in — an interrupt (client JSON, VAD onset, or disconnect) cancels both the in-flight HTTP request ([CancellationToken] in the reqwest tokio::select!, including while waiting for the next SSE chunk) and the TTS tasks, then flushes the mixer. Chat-box turns run in their own task so the session's receive loop stays responsive and keeps accepting interrupt messages while a reply is being generated.
• Failure handling — request errors are reported to the client as {"type":"error","text":…} and the turn ends cleanly.

Runtime system prompt (GET / POST /api/prompt)

The system prompt is normally the process-wide default (the built-in one, or LLM_SYSTEM_PROMPT). Because it is injected identically into every session, two clients cannot run with different instructions. GET/POST /api/prompt adds a runtime, in-memory override — per session or process-wide.

Each WebSocket session is assigned a server id (e.g. s-1729170000000-0001, the same id used for its audit log) and the id is pushed to the client right after connect, before the cues message:

{"type":"session","id":"s-1729170000000-0001"}

That id is the key of the REST surface:

# Effective prompt for a session (+ where it comes from)
curl 'localhost:8787/api/prompt?session_id=s-1729170000000-0001'

# Override it for that session
curl -X POST localhost:8787/api/prompt -H 'Content-Type: application/json' \
  -d '{"session_id":"s-1729170000000-0001","prompt":"You are a terse assistant. One sentence max."}'

# Clear the override (falls back to the configured default)
curl -X POST localhost:8787/api/prompt -H 'Content-Type: application/json' \
  -d '{"session_id":"s-1729170000000-0001","prompt":null}'

• Scope — with a session_id the override applies to that session only; omit it (null) to set a process-wide fallback used by every session without its own override.
• Precedence — session override → process-wide override → LLM_SYSTEM_PROMPT → built-in default. The response's source field (session / global / env / default / none) reports which one won, and has_override whether a runtime override is in effect.
• In-memory only — a restart restores the configured default, and a session override is dropped when that session disconnects. An unknown/disconnected session_id returns 404.
• prompt: null (or "") clears the override.
• The response also carries direct_answers_applied (whether the LLM_DIRECT_ANSWERS directive is currently appended).
• See src/prompt.rs and src/prompt_api.rs.

---

Tool calling

With LLM_TOOLS_FILE pointing at a JSON file, the model can call external HTTP services before answering. Tools are plain descriptions of an HTTP request, so any service (weather API, webhook, home automation, internal REST endpoint) can be exposed without touching the code:

[
  {
    "name": "get_weather",
    "description": "Current weather and temperature for a city.",
    "parameters": {
      "type": "object",
      "properties": { "city": { "type": "string" } },
      "required": ["city"]
    },
    "url": "https://wttr.in/{city}?format=3",
    "method": "GET",
    "headers": { "X-Api-Key": "…" }
  }
]

• url may contain {arg} placeholders substituted from the call arguments.
• method takes any HTTP verb, case-insensitive (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE, custom verbs); default POST. The arguments left after the URL placeholders are sent as query parameters for GET/HEAD/OPTIONS/TRACE/DELETE and as a JSON body for every other verb — so a full REST surface is covered: list/look up (GET), add (POST), update (PUT/PATCH), remove (DELETE).
• headers are optional and sent verbatim (API keys, etc.).

See tools.example.json for a ready-to-use file.

Runtime definitions (GET / POST /api/tools)

The file is read once at startup, so POST /api/tools installs a different set without a restart — the body is the same JSON array the file holds:

curl -X POST localhost:8787/api/tools -H 'Content-Type: application/json' \
  -d '[{"name":"get_weather","description":"…","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]},"url":"https://wttr.in/{city}?format=3","method":"GET"}]'

• The override lives in memory only: it replaces the LLM_TOOLS_FILE definitions from the next turn on, and a restart brings the file ones back. When the endpoint is never called, the file definitions stay in use.
• GET /api/tools reports the active definitions and their origin (source: file, request or none).
• 400 on an empty/invalid body or on a JSON array with no tool; 503 when the LLM is disabled. See src/tools_api.rs.

See TOOL_CALLING.md for the full guide.

Client-side parsing (and the streaming fallback)

A deployment may or may not run a server-side tool parser. When it does, the flag has to match the model's native format (--enable-auto-tool-choice --tool-call-parser …). A mismatch is silent and costly: an hermes parser (which expects <tool_call>{"name":…,"arguments":{…}}</tool_call>) against a Qwen3 model that emits the XML form makes the parser throw, so the call is dropped. In streaming the round then ends with an empty reply (finish_reason: "stop", no content), while the very same request without stream returns the call intact.

The engine therefore never relies on the server parser:

• Every request advertises tools + tool_choice: "auto".
• src/tools.rs parses both shapes — the standard structured message.tool_calls array and the model's native <tool_call> text:

  <tool_call>
  <function=get_weather>
  <parameter=city>
  Paris
  </parameter>
  </function>
  </tool_call>
  

• A streaming round that comes back empty is re-asked once without streaming (LlmClient::chat_reply), where the dropped call reappears and is parsed from either shape.

Fixing the server (--tool-call-parser qwen3_coder handles the Qwen3 XML format) removes the extra request — the fallback then simply never fires.

Turn loop

Each turn runs a bounded loop (LLM_TOOL_MAX_ITERATIONS, default 4):

1. Stream one reply. A ToolCallStripper removes <tool_call> blocks from the streamed text — a block that is still incomplete is held back, so it is never spoken by the TTS nor shown in the chat box even when split across deltas. The visible text is spoken as usual, so the assistant can say something short ("Let me check…") before calling. A round that streams back empty is re-asked once without streaming to recover a call the server dropped (see above).
2. If the round produced no tool call, the turn ends.
3. Otherwise the assistant message (with its native call) plus one tool role message per result are appended, and the loop re-prompts the model, which can then answer using the results.

Tool errors (unknown tool, transport failure, non-2xx status) are returned to the model as text, so it can recover and still produce an answer; the turn never fails because of a tool. Only the spoken text is written to the rolling history — the tool exchange stays local to the turn.

The tool_call event is also forwarded to the client for observability:

{ "type": "tool_call", "name": "get_weather", "arguments": { "city": "Paris" } }

---

External audio injection

The mixer has a dedicated external channel (external_queue) that plays an audio source injected from the client — a direct audio URL, an Icecast/Shoutcast radio stream, an HLS playlist, or a YouTube link — mixed together with the agent's voice. It is implemented by src/media.rs (MediaSource) and driven over the WebSocket by the audio_source message (see WebSocket Protocol). The Mivocero client exposes it in its footer bar.

Why ffmpeg / yt-dlp

The channel does not decode audio itself: it spawns an external decoder process whose raw PCM output feeds the ring buffer.

• ffmpeg (URL / radio / HLS) — reads the source and writes -f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1, i.e. exactly the mixer's format (44.1 kHz, 16-bit LE, mono). URL inputs use -reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5 so a dropped radio connection is retried instead of ending the stream.
• yt-dlp (YouTube) — used only to resolve a YouTube page to a direct media URL (-f bestaudio/best -g --no-playlist); that URL is then handed to ffmpeg. YouTube-specific cookies/extractor tweaks go in MEDIA_YTDLP_ARGS (e.g. --cookies /path/cookies.txt if the media server asks to "confirm you're not a bot").

Both binaries are configurable (MEDIA_FFMPEG_BIN, MEDIA_YTDLP_BIN) and are only required when the external channel is actually used.

Backpressure

A live radio always produces audio at real time, but a bounded VOD source (a YouTube video, a file) decodes far faster than real time. The decode task respects the channel's remaining_capacity() and sleeps 20 ms whenever the queue is full, so a fast source is throttled to the mixer's drain rate instead of overflowing the ring buffer (which would drop the oldest samples).

Independence from the agent

The external channel is deliberately not cleared by AudioMixer::flush(): a barge-in stops the agent (TTS + fillers), not the background music. It also does not count towards AudioMixer::is_active() — the agent is not considered to be "speaking" because the music is playing. Its volume is a runtime atomic (AudioMixer::set_external_gain) so the client slider takes effect on the next mixed frame.

Lifecycle

Step	What happens
play	Any current source is stopped; state loading is emitted. For a YouTube URL yt-dlp resolves the direct URL first.
ready	ffmpeg is spawned and reading begins; state playing is emitted.
streaming	Decoded PCM is pushed into external_queue under backpressure.
stop / new play / disconnect	The CancellationToken is cancelled and the child processes are killed (kill_on_drop(true)); stopped (or error) is emitted.

A monotonic generation counter guards against a stale task clearing the state of a newer source.

---

Mivocero chat client

A ready-to-use speech-to-speech example client is served at /mivocero (embedded from src/web/mivocero.html):

• Microphone — the same AudioWorklet capture path as the playground (44.1 kHz, 16-bit LE mono, 20 ms frames).
• Chat box — shows the user transcript (from STT) and the assistant reply as it streams, plus messages typed in the composer.
• Karaoke highlighting — both the assistant and the user bubbles are rendered word by word. The assistant's words light up as they are spoken, driven by the audio playhead (the playback worklet reports the samples it has actually emitted): words already spoken are bright, the word currently sounding is accented, and the rest stay dim. The rate is an estimate (KARAOKE_CPS, ~15 chars/s) — it only affects the visual highlight, never the audio. The user's words cannot follow the audio (the STT returns the whole transcript only once the utterance is over), so they are lit by a short left-to-right sweep (USER_REVEAL_MS_PER_WORD, ~90 ms/word) when the bubble appears.
• Text composer — typing and sending a message runs a full LLM → TTS turn without using the microphone ({"type":"user_text","text":…}).
• Interrupt button — flushes audio and cancels the in-flight reply ({"type":"interrupt"}).
• Barge-in indicator — a red Barge-in pill in the header that flashes and bumps a session counter whenever the agent is interrupted: either by the server-side VAD onset (via the {"type":"barge_in"} event) or by the local Stop button. The counter resets on each connect.
• External audio bar — footer controls to inject a background source by URL (direct audio / radio / YouTube), with Play / Stop buttons, a volume slider (0–150 %, sent as EXTERNAL_AUDIO_GAIN × slider) and a live state label driven by the server's audio_source events. Demonstrates the external audio channel.
• Cue indicators — a row of three tiles (Filler, Start filler, Backchannel) that flash and bump a session counter each time the server emits the matching cue, driven by the {"type":"cue"} events. The detail line shows the backchannel category (or the clip name), so the emission cadence of the latency masks is visible live. Counters reset on each connect.
• Cue methods panel — a collapsible Cue methods section below the tiles that documents the method used by each cue (when it fires, how a clip is chosen) and shows the live parameters reported by the server in the {"type":"cues"} event sent at session start: filler delay / gains / clip count, start-filler go-delay, and the backchannel gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap and categories — all read from the effective runtime configuration (no hard-coded values).
• Side menu — the ☰ button in the header opens a drawer that navigates between the chat and the custom voice builder.
• Idle auto-disconnect — the session closes itself after 2 minutes (IDLE_TIMEOUT_MS) without any traffic in either direction. The timer is reset by every server event and audio frame received, and by every user action sent (typed message, Stop, media controls) — so an ongoing reply or a playing external source counts as activity. On expiry the WebSocket is closed, the microphone is released, an explanatory message is appended to the chat and the Connect button becomes available again.

Open http://localhost:8787/mivocero (or https://… for remote microphone access — see HTTPS).

---

Custom voice builder

A dedicated page served at /customvoice (embedded from src/web/customvoice.html, linked from the Mivocero side menu) lets anyone build a voice-cloning reference from their own voice, without touching the filesystem or the .env. Two sources are offered:

• Record now — read a predefined phrase out loud (the microphone flow).
• Upload a file — pick an audio file and type its transcript (any container ffmpeg can read: wav, mp3, m4a, ogg, flac…).

The flow:

1. Language — a selector populated from GET /api/customvoice/phrases (the languages of crate::lang; the default is LANG_DEFAULT).
2. Source — pick Record now or Upload a file.
   • Record now — the page shows the phrase to read for the selected language. The phrases live server-side (src/customvoice.rs), so the transcript attached to the reference always matches the text the user was asked to read. The browser captures the microphone with MediaRecorder (Opus/WebM where supported); the Stop button stays disabled until 10 seconds have elapsed (a progress bar and a timer show the way to the minimum), and recording auto-stops at 20 s. The take can be played back and discarded before use.
   • Upload a file — choose an audio file (accept="audio/*") and type (or paste) the exact text spoken in it. Here the transcript is required: it is what drives the clone, so it must match the audio.
3. Use it — POST /api/customvoice?lang=<code>[&text=<transcript>] with the recording or the uploaded file as the body. The text parameter carries the upload flow's typed transcript and overrides the predefined phrase; when it is absent or blank the language's predefined phrase is used (so a plain recording needs no text). The server post-processes the take (see below), transcodes it to mono 16-bit PCM WAV at OUTPUT_SAMPLE_RATE (44.1 kHz) with ffmpeg, writes it next to the configured TTS_REFERENCE_AUDIO (the directory the Fish Audio backend can read — hence "the path on the TTS server" note in Configuration), writes the transcript as a sibling <name>.txt (the convention Config::tts_reference already falls back to), and installs the voice through TtsEngine::set_reference — effective immediately for new replies, no restart.
4. Reset — POST /api/customvoice/clear restores the reference configured at startup (TTS_REFERENCE_AUDIO), or clears it when none is configured.

Post-processing

A raw browser recording is uneven: it carries rumble and room noise, and its level depends on how far the user sat from the microphone. The take therefore goes through a cleanup chain before it is installed (src/customvoice.rs):

Stage	Filter	Why
Mono downmix	aformat=channel_layouts=mono	Everything downstream (and the reference itself) is mono. Done first so loudnorm measures the signal that is actually written.
Rumble / DC removal	highpass=f=80	Kills desk thumps, plosives and DC offset below the voice band.
Noise reduction	afftdn=nr=12:nf=-40	FFT denoise for steady room/hiss noise.
De-essing	deesser	Tames harsh sibilance that a clone would otherwise exaggerate.
Silence trim	silenceremove (+ areverse)	Drops leading/trailing silence so the reference starts on speech.
Loudness	loudnorm, two-pass, linear	Matches every take to the same programme level.

The loudness stage is deliberately two-pass: ffmpeg measures the cleaned signal first, then applies the measurement as one constant gain (linear=true). A single gain cannot pump or ride the level frame by frame — which is exactly what makes takes differ in volume.

The default target is -26 LUFS integrated with a -5 dBTP true-peak ceiling. Both sit below the shipped reference (voice_profiles/solis.wav measures ≈-23.4 LUFS at -7.8 dBTP): a user take is far more dynamic than that mastered reference, so at equal LUFS its much hotter peaks make the cloned voice sound louder than the default one. The lower target and the extra peak headroom buy that gap back, so a custom voice sits a touch under the default instead of on top of it. A take recorded 30 dB too quiet and one recorded hot both land on the same level.

If any stage fails (a filter quirk, an unreadable take, an all-silence recording) the endpoint logs a warning and falls back to the plain transcode — a take is never lost to post-processing.

Env var	Default	Description
CUSTOMVOICE_POSTPROCESS	true	Master switch; false stores the plain transcode.
CUSTOMVOICE_DENOISE	true	The afftdn + deesser stage — turn off for a clean studio take.
CUSTOMVOICE_TARGET_LUFS	-26	Integrated loudness target, below the shipped reference. Lower it for a quieter custom voice.
CUSTOMVOICE_TRUE_PEAK	-5	True-peak ceiling (dBTP).

Method	Route	Description
GET	/customvoice	The builder page.
GET	/api/customvoice/phrases	Reading phrases + the default language (JSON).
POST	/api/customvoice?lang=<code>[&text=<transcript>]	Body = recording or uploaded file; transcodes, persists (with the transcript) and installs the reference.
POST	/api/customvoice/clear	Restore the configured reference.

The body cap for the upload route is raised to 32 MB (the axum default is 2 MB) so an uncompressed WAV is accepted as well as a compressed capture. Microphone access requires a secure context: use https://… when the page is not opened from localhost (see HTTPS).

---

Session audit log

Every WebSocket session can be recorded to its own structured JSONL file — one JSON object per line — under SESSION_AUDIT_DIR (default logs/sessions), named s-<epoch_ms>-<seq>.jsonl. It is meant for post-mortem analysis and for tuning the latency knobs (why a filler fired, when a cue landed, how long the first audio took).

Each record carries a wall-clock timestamp, the elapsed time since the session started, an event discriminator and a payload:

{"ts":1729170000000,"elapsed_ms":12,"event":"session_start","data":{"config":{…},"runtime":{…}}}
{"ts":1729170000123,"elapsed_ms":135,"event":"control","data":{"type":"user_text","text":"…"}}
{"ts":1729170000456,"elapsed_ms":468,"event":"out","data":{"type":"assistant_delta","text":"Hola,"}}
{"ts":1729170000500,"elapsed_ms":512,"event":"out","data":{"type":"cue","kind":"start_filler","name":"sf02"}}
{"ts":1729170009000,"elapsed_ms":9012,"event":"session_end","data":{"reason":"client_disconnected","counts":{…}}}

event	When	data
session_start	Session opens	config — the full non-secret configuration snapshot; runtime — which of STT / VAD / tokenizer / LLM are actually active (plus the LLM model).
control	user_text control message received	The text turn.
out	Every event sent to the client	The exact WS payload — user, assistant_delta, assistant_end, tool_call, cue, barge_in, error, audio_source state, … Recorded at the single sender tap, so nothing sent to the browser is missed.
interrupt	Client barge-in (interrupt JSON or legacy INTERRUPT)	source (client / client_legacy).
barge_in	VAD-driven barge-in	prob (onset speech probability).
vad_speech_start / vad_speech_end	Utterance boundaries	prob; duration_ms, avg_prob.
utterance_dropped	VAD segment filtered by the utterance gate (no STT, no LLM turn)	duration_ms, avg_prob, too_short, too_soft.
media	External audio control	action (play/stop/volume) with url / gain.
reference	Voice-cloning reference change	action (set/clear) and the audio path.
speak	Direct TTS request (playground / fallback text)	text.
language_switch	Detected language changed	from, to, generated (whether clip banks were synthesized).
error	Transport / receiver error	where, text.
session_end	Session closes	reason, duration_ms, session_id, file, counts (per-event totals).

The sink is best-effort: if the file cannot be opened, or a write fails, the session runs unchanged (the failure is logged once in the tracing output and the sink then goes quiet). Files are appended and flushed after every record, so the log survives a crash or a pm2 restart. The directory is git-ignored.

---

WebSocket Protocol

Connect to ws://<host>:8787/ws.

Client → Server

Message	Description
Binary	Raw 16-bit LE mono PCM microphone frames (44.1 kHz). Resampled to 16 kHz server-side for STT.
Text — {"type":"user_text","text":"…"}	Text turn from the chat box: runs the same LLM → TTS path as a spoken utterance (no STT).
Text — {"type":"interrupt"}	Barge-in: cancel the in-flight LLM reply / TTS and flush all queued audio.
Text — {"type":"audio_source","action":"play","url":"…"}	Inject an external audio source (direct URL / radio / HLS / YouTube) into the mixer's external channel, replacing any current one.
Text — {"type":"audio_source","action":"stop"}	Stop the external source and clear the channel.
Text — {"type":"audio_source","action":"volume","gain":0.6}	Set the external channel gain (0..1) at runtime.
Text — INTERRUPT (also interrupt, BAARGEIN, bargein)	Legacy barge-in form (kept for the playground).
Text — SET_REFERENCE:<path>|<text>	Set/change the voice-cloning reference. <text> may be omitted to re-use the sibling .txt.
Text — CLEAR_REFERENCE	Disable voice cloning.
Text — anything else	Synthesize the text directly (useful for testing TTS without STT).

Server → Client

Message	Description
Binary	One 20 ms mixed frame of 16-bit LE mono PCM @ 44.1 kHz (1764 bytes). Silence is sent when no audio is queued, keeping the stream alive.
Text — {"type":"user","text":"…"}	The user's turn (STT transcript or chat-box message) — rendered in the chat box.
Text — {"type":"assistant_delta","text":"…"}	Streaming chunk of the assistant reply (chain-of-thought already removed).
Text — {"type":"assistant_end"}	The assistant turn is complete.
Text — {"type":"tool_call","name":"…","arguments":{…}}	The model called a tool; the call has been executed (result is not echoed in the chat box, only in the model's next round). Purely informational.
Text — {"type":"error","text":"…"}	LLM request failed; the turn was aborted.
Text — {"type":"audio_source","state":"…","url":"…"}	External audio channel state change: loading (decoding is being set up / YouTube URL resolving), playing (PCM is streaming), stopped, or error (with a text message).
Text — {"type":"cue","kind":"…","name":"…","category":"…"}	A latency-mask / backchannel cue was actually emitted into the mixer: kind is filler (end-of-turn), start_filler (turn start) or backchannel (during user speech); name is the clip name and category (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators.
Text — {"type":"cues","lang":"…","config":{…}}	Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (delay_ms, gains, clip count), the start-filler go_delay_ms and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's Cue methods panel so it shows the live values instead of hard-coded ones.
Text — {"type":"session","id":"…"}	Sent once right after connect: the server-assigned session id (also the audit-log file stem). It is the key used by GET/POST /api/prompt to read/override the system prompt of this session (Runtime system prompt).
Text — {"type":"barge_in","source":"vad","prob":…}	The agent was interrupted. Sent for the server-side VAD barge-in (the browser has no other way to know, since the onset is detected in the session task); the client-side interrupt does not echo one. Drives the demo's barge-in indicator.

---

Audio Pipeline

Constants are defined in src/config.rs:

Constant	Value	Meaning
OUTPUT_SAMPLE_RATE	44_100 Hz	Server → browser rate (matches Fish Audio Drama 3 native output).
STT_SAMPLE_RATE	16_000 Hz	Rate expected by Parakeet.
FRAME_DURATION_MS	20 ms	Mixer ticker period.
OUTPUT_FRAME_SAMPLES	882	Samples per frame (44.1 kHz × 20 ms).
OUTPUT_FRAME_BYTES	1764	Bytes per frame (16-bit mono).
EOU_SILENCE_MS	300 ms	Silence window that marks end-of-utterance.
VAD_RMS_THRESHOLD	0.01	RMS above which audio counts as voice.

Mixer & jitter buffering (src/audio/mixer.rs)

• Mixing formula: mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) + (external × external_gain). The filler gain is 1.0 when TTS is idle; when the reply's playback starts a still-queued filler/start-filler is instead faded out over 80 ms (FILLER_FADEOUT_FRAMES) and cut, so it hands off to the answer (a cue queued after playback has already begun falls back to filler_gain_duck, default 0.15). The backchannel gain is fixed (BACKCHANNEL_GAIN, default 0.35). The start-filler channel is summed with the filler, so it shares the same gain. The external channel gain starts at EXTERNAL_AUDIO_GAIN (default 0.6) and is adjustable at runtime (AudioMixer::set_external_gain). Output is clamped to i16.
• Pre-buffer: TTS playback does not start until the queue holds ≥ TTS_PREBUFFER_MS (default 550 ms) of synthesized audio. It absorbs uneven TTS chunk arrival; lower it to start speaking sooner, raise it for fewer drop-outs. Note: Drama-3's granularity is set by the server's vocoder window (--vocoder.factory.stream_stride / stream_followup_stride, in codec frames of 2048 samples = 46.4 ms). With the deployment used here (stride 4, followup 8, overlap 16) the first window is 4 frames = 0.186 s and the steady windows are 8 frames = 0.372 s, delivered ~0.35 s apart (the server runs at ~0.95× real time). Because TTFA ≈ TTFB + (prebuffer − window), any pre-buffer in (186, 557] ms primes on the same second window → TTFA ≈ 0.73 s; the default 550 ms takes that whole cushion at no extra latency. A pre-buffer below 186 ms starts playback on the first window (TTFA ≈ 0.38 s) but leaves almost no margin: under concurrent TTS requests one GPU serves several requests and the per-request throughput dips below real time (1.5× vs ~0.94× alone), so every chunk seam starves the queue and pauses playback → an audible micro-gap at the start of each sentence. With the upstream default (64, a 2.97 s burst every ~2.7 s) gapless playback would instead require a ~3 s pre-buffer; a large constant stride (16) both raises the TTFB (~1.2 s) and bursts, and a growing ramp gaps the tail while the small early windows drain.
• Resilient resume: the full pre-buffer is only required at the first playback of a reply. If the queue empties mid-reply (after the 200 ms drain grace) playback pauses, but resumes as soon as any audio is queued again — it does not wait to refill the whole jitter buffer. mark_stream_start re-arms the pre-buffer at the beginning of every turn.
• Drain grace: brief inter-chunk empties (< 200 ms) emit silence without pausing playback, avoiding audible drop-outs on ordinary arrival jitter.
• Ticker: MissedTickBehavior::Burst — missed ticks are caught up in a burst rather than dropped, so no 20 ms frames are lost.

---

Project Layout

audio-agent-engine/
├── Cargo.toml
├── .env.example              # documented configuration template
├── server.sh                 # start / stop / restart / status / logs / cert
├── certs/                    # self-signed TLS cert+key (generated)
├── UPDATES.md                # changelog (English, newest-first)
├── tools.example.json        # tool definitions for LLM tool calling (copy/edit)
├── sdk/                      # dependency-free JavaScript client SDK (+ .d.ts, served at /sdk/)
├── assets/fillers/<lang>/    # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin), one dir per language
├── assets/start_filler/<lang>/  # sf01.raw … turn-start start-filler clips + texts.txt index
├── assets/backchannel/<lang>/   # bc01.raw … backchannel clips + texts.txt index
├── models/                   # Parakeet ONNX + Silero VAD + tokenizer
└── src/
    ├── main.rs               # Axum server, routes, graceful shutdown
    ├── config.rs             # env-var config + audio constants
    ├── assets.rs             # startup clip bootstrap (auto-generate fillers/backchannel/start filler)
    ├── tls.rs                # optional HTTPS/WSS termination (rustls)
    ├── onnx.rs               # shared process-wide ONNX Runtime environment
    ├── audio/                # ring_buffer, mixer
    ├── tts/                  # ← plugin system
    │   ├── mod.rs            # TtsEngine trait + create_tts_engine factory
    │   ├── fish_audio.rs     # Fish Audio / SGLang Drama 3 plugin
    │   └── kokoro.rs         # Kokoro plugin (stub)
    ├── stt/                  # Parakeet ONNX + tokenizer + worker
    ├── vad/                  # Silero VAD v5 + onset/offset state machine
    ├── llm.rs                # OpenAI-compatible chat-completions client (streaming + think stripping)
    ├── tools.rs              # configurable HTTP tool calling (native <tool_call> parser + executor)
    ├── tools_api.rs          # GET/POST /api/tools — runtime tool definitions (in-memory override)
    ├── prompt.rs             # per-session system-prompt registry (overrides + resolution)
    ├── prompt_api.rs         # GET/POST /api/prompt — read/override the LLM system prompt per session
    ├── auth_api.rs           # POST /api/auth/phone-login — server-side proxy to the AquaChile auth API
    ├── media.rs              # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
    ├── customvoice.rs        # custom-voice builder endpoints (record → WAV reference → set_reference)
    ├── transcribe.rs         # POST /api/transcribe — Parakeet transcription of an uploaded audio file
    ├── apidoc.rs             # OpenAPI document (utoipa) rendered by the Swagger UI at /docs
    ├── audit.rs              # per-session JSONL audit log (config snapshot, events, summary)
    ├── orchestrator/         # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
    ├── ws/                   # handler, session (per-connection audio loop)
    └── web/                  # embedded playground.html + mivocero.html + customvoice.html

HTTP routes

Route	Description
GET /	Playground single-page client (embedded at compile time).
GET /mivocero	Mivocero speech-to-speech example client with chat box (embedded at compile time).
GET /customvoice	Custom-voice builder page (record or upload a reference — see Custom voice builder).
GET /api/customvoice/phrases	Reading phrases per language + the default language (JSON).
POST /api/customvoice?lang=<code>[&text=<transcript>]	Upload a recording or an audio file (with its transcript); transcode it to a WAV reference and install it as the active voice.
POST /api/customvoice/clear	Restore the reference configured at startup.
POST /api/transcribe	Upload a raw audio file (any ffmpeg-readable container); returns the Parakeet transcript of the whole file (text, duration_ms, tokens).
GET /api/tools	The active LLM tool definitions and their origin (source: file / request / none).
POST /api/tools	Install LLM tool definitions at runtime (same JSON array as LLM_TOOLS_FILE) — in-memory, no restart (Tool calling).
GET /api/prompt	The effective LLM system prompt, with its origin (source: session / global / env / default / none). Optional ?session_id=…; 404 for an unknown session (Runtime system prompt).
POST /api/prompt	Override the LLM system prompt at runtime — for one session ({"session_id":"…","prompt":"…"}) or process-wide (prompt only); prompt: null clears it. In-memory, no restart.
POST /api/auth/phone-login	Authenticate a phone number ({"phoneNumber":"+44…","session_id":"s-…"}) through a server-side proxy to the AquaChile auth API (AUTH_API_BASE + AUTH_PHONE_LOGIN_PATH) and return its payload (sessionToken + user). When session_id is provided, the engine stores a sanitized user profile as hidden LLM context for that live session (Phone login).
GET /health	Liveness probe (ok).
GET /sdk/audio-agent-client.js	The dependency-free JavaScript client SDK (Client SDK (JavaScript), source in sdk/).
GET /sdk/demo.html	Demo client built on the SDK (source in sdk/demo.html).
GET /ws	WebSocket upgrade → per-session audio loop.
GET /docs	Swagger UI for the REST surface (API documentation).
GET /api-docs/openapi.json	The raw OpenAPI 3 document the Swagger UI renders.

API documentation (Swagger UI)

The REST endpoints are documented with an OpenAPI 3 document derived from the handlers (utoipa) and browsable in a Swagger UI served by the engine itself (utoipa-swagger-ui):

• UI — GET /docs (→ /docs/) renders the interactive Swagger UI.
• JSON — GET /api-docs/openapi.json serves the raw OpenAPI document.

The document is built in src/apidoc.rs (the ApiDoc #[derive(OpenApi)], the response schemas) with per-handler #[utoipa::path] annotations on the endpoints. Only the JSON/REST endpoints are covered — the WebSocket endpoint (/ws) and the HTML client pages are streaming / static assets and are intentionally left out.

The utoipa-swagger-ui build script embeds the UI assets from a zip at compile time; the zip crate is therefore pinned to the 2.2.x line in Cargo.toml (the 2.3+ API no longer matches the 8.1 build script).

Transcription endpoint

POST /api/transcribe (src/transcribe.rs) turns an uploaded audio file into text with the same Parakeet TDT model the live session uses:

curl -s -X POST http://localhost:8787/api/transcribe \
     -H 'Content-Type: application/octet-stream' \
     --data-binary @clip.wav
# {"ok":true,"text":"…","duration_ms":14918,"tokens":83}

• Input — the raw audio bytes in the request body (any container ffmpeg can read: wav, mp3, m4a, ogg, flac…); no multipart, no lang parameter — Parakeet TDT v3 is multilingual and the language is auto-detected.
• Processing — ffmpeg decodes to mono 16 kHz s16le, and the engine's already-loaded STT model (AppState::stt) runs one inference over the whole file (spawn_blocking: the inference is CPU-bound and holds the ONNX session lock). No second ONNX environment is created.
• Output — a single transcript for the entire file (the audio is not segmented): { "ok": true, "text": …, "duration_ms": …, "tokens": … }.
• Errors — 400 empty body, 503 STT model not loaded, 502 ffmpeg decode failed, 500 temp-file / inference failure. Uploads are capped at 32 MB (same as /api/customvoice).

---

Phone login (POST /api/auth/phone-login)

The SDK demo can authenticate the user by phone number, read from the page URL (?phoneNumber=+44…), at the start of the session. The call is made through the engine, not directly from the browser:

curl -X POST localhost:8787/api/auth/phone-login \
  -H 'Content-Type: application/json' \
  -d '{"phoneNumber":"+447432477433"}'

The engine forwards the same JSON to the upstream AquaChile auth API (AUTH_API_BASE + AUTH_PHONE_LOGIN_PATH) and returns its status and body unchanged — on success a payload like {"sessionToken":"…","user":{"id":…,"name":…,"email":…,"permissions":[…]}}.

If the request also carries a live WebSocket session_id, the engine extracts a sanitized user profile (name, email, businessRoles, permissionSet, centers, permissions) and injects it into that session's LLM context as a hidden system block. The SDK demo does this automatically right after connect.

The proxy exists because the upstream API rejects browser calls from the client origin: its CORS preflight answers 400 Disallowed CORS origin and its response carries no Access-Control-Allow-Origin, so a direct fetch from the demo is blocked. Routing the call through the engine makes it same-origin for the browser. Errors: 400 for an empty/invalid body or a missing phoneNumber, 502 when the upstream is unreachable. See src/auth_api.rs.

---

Client SDK (JavaScript)

sdk/audio-agent-client.js is a dependency-free, single-file ESM module (with sdk/audio-agent-client.d.ts) for writing clients against the engine. It is deliberately decoupled from the server: copy the file to any host, no build step, no npm install. The engine also serves it at GET /sdk/audio-agent-client.js.

A ready-made demo client built on the SDK is served at GET /sdk/demo.html — open http://localhost:8787/sdk/demo.html and press Connect.

It covers:

• Realtime session — AudioAgentClient: connects to /ws, sends PCM (sendAudio) and typed controls (sendUserText, interrupt, playAudioSource, setReference, speak, …), and emits one event per server message (assistant_delta, cue, barge_in, tool_call, audio_source, error, …) plus a local audio event carrying the decoded PCM frames.
• REST — AudioAgentRest: health, getTools / setTools, getCustomVoicePhrases / setCustomVoice / clearCustomVoice, transcribe, openapi; failures throw an error carrying status and the parsed body.
• Browser audio helpers — MicCapture (mic → 882-sample Int16Array frames, via an inline AudioWorklet), PcmPlayer (jitter-buffered playback, flush() for barge-in) and AudioSession, which wires the three together.
• PCM utilities — int16ArrayFromBytes, float32ToInt16, int16ToFloat32, toInt16, plus the AUDIO constants (44.1 kHz, 20 ms = 882 samples).

Browser

<script type="module">
  import { AudioSession } from 'http://localhost:8787/sdk/audio-agent-client.js';

  const session = new AudioSession({ baseUrl: 'http://localhost:8787' });
  session.on('user', (e) => console.log('you:', e.text));
  session.on('assistant_delta', (e) => (log.textContent += e.text));
  session.on('error', (e) => console.error(e));
  await session.start();              // connect + microphone + playback
  session.sendUserText('hello');      // text turn
  session.interrupt();                // barge-in
</script>

Serve the page from a secure context (https:// or localhost) or the browser will refuse microphone access. Cross-origin pages must have CORS enabled on the engine (CORS_ALLOW_ORIGIN=https://your.app); /ws needs no CORS.

Node

The core (client + REST + PCM helpers) is browser-agnostic. Node 22+ has a global WebSocket; on older versions pass one in (e.g. the ws package). The audio helpers (MicCapture, PcmPlayer, AudioSession) require the Web Audio API and are browser-only.

import { AudioAgentClient, AudioAgentRest, toInt16 } from './sdk/audio-agent-client.js';
import WebSocket from 'ws';

const client = new AudioAgentClient({ baseUrl: 'http://localhost:8787', WebSocket });
client.on('assistant_delta', (e) => process.stdout.write(e.text));
await client.connect();
client.sendUserText("what's the weather in Rome?");
client.sendAudio(toInt16(new Float32Array(882)));   // PCM 16-bit LE, 44.1 kHz

const rest = new AudioAgentRest('http://localhost:8787');
console.log(await rest.getTools());

See sdk/README.md for the full reference.

---

Development

# Build
cargo build

# Run tests (tokenizer unit tests)
cargo test

# Run with debug logging
RUST_LOG=debug cargo run

# Release build (LTO + thin codegen)
cargo run --release

---

Related Documentation

• UPDATES.md — full changelog (English, newest-first).
• src/tts/mod.rs — trait + factory source of truth.