Audio Agent Engine

A production-ready real-time, full-duplex voice agent engine written in Rust (Tokio + Axum + ONNX Runtime). It streams microphone audio from the browser, runs speech recognition with end-of-utterance detection, injects a low-latency backchannel filler, and streams synthesized speech back to the browser — all mixed on a strict 20 ms ticker.

Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
                                            │
                                            ▼
                                   TurnManager (orchestrator)
                                            │
                                            ▼
                            LLM reasoning (OpenAI-compatible)
                                            │
                     ┌──────────────────────┼──────────────────────┐
                     ▼                      ▼                      ▼
            Filler / backchannel     TTS engine plugin       Mixer (20 ms ticker)
          (instant "uhm" + cues)   (Fish Audio / Kokoro)              │
                     └──────────────────────┴──────────────────────┘
                                            │
                                            ▼
                              Mixed 44.1 kHz / 16-bit mono PCM ──► Browser

---

Features

• Full-duplex WebSocket audio — 44.1 kHz / 16-bit mono PCM, 20 ms frames.
• Pluggable TTS engines — switch engines (Fish Audio, Kokoro, …) via .env without changing orchestration code. See TTS Plugin System.
• Latency filler — a preloaded filler is played on end-of-utterance only if the reply is late: it is scheduled FILLER_DELAY_MS (default 2500 ms) after the turn starts and dropped if the response audio has already begun. Clips are played round-robin from FILLERS_DIR; FILLER_ENABLED=false disables it.
• During-utterance backchannel — short acknowledgments ("mm-hmm", "ajá") are mixed in at a reduced gain while the user is speaking, so the agent signals it is listening without taking the turn. Plugin-style .env config.
• Barge-in / interrupt — cancels in-flight synthesis and flushes audio, triggered client-side or server-side by the VAD.
• Voice cloning — reference audio + transcript (Fish Audio references).
• Pipelined TTS — reply chunks are synthesized ahead of playback (up to TTS_PIPELINE_DEPTH at once) and replayed in order, hiding each request's start-up latency behind the previous chunk's audio.
• Double jitter buffer — server pre-buffer + browser playback buffer absorb uneven TTS chunk arrival, eliminating drop-outs.
• Silero VAD (ONNX) — server-side speech segmentation and fast barge-in.
• Parakeet STT (ONNX) with tokenizer decoding and EOU detection.
• LLM reasoning — the user transcript is sent to any OpenAI-compatible chat-completions server (vLLM, SGLang, llama.cpp, Ollama, OpenAI) and the reply is spoken by the TTS. Streaming (SSE) chunks the reply as it arrives (first chunk after a few words, tunable) for the lowest time-to-first-audio; chain-of-thought (reasoning_content / thinking) is stripped from both the audio and the chat box.
• Embedded playground — a single-page HTML client served at /.
• Mivocero speech-to-speech client — an example agent client at /mivocero with a microphone, a live chat box (user transcript + streaming assistant reply) and a text composer, useful to drive a full STT → LLM → TTS turn.

---

Requirements

• Rust 1.75+ (async traits are provided via the async-trait crate).
• A TTS backend reachable over HTTP (e.g. Fish Audio / SGLang Drama 3).
• (Optional) A Parakeet TDT ONNX model directory (models/parakeet/) for speech recognition. The server runs without STT (TTS still works via the playground text box) if the model is absent.
• (Optional) An OpenAI-compatible chat-completions endpoint (e.g. a local vLLM / SGLang server) for the LLM reasoning stage. Without it the pipeline is a plain STT → TTS echo.

---

Quick Start

cd audio-agent-engine

# 1. Configure (optional — defaults target Fish Audio Drama 3)
cp .env.example .env
$EDITOR .env

# 2. Build and run
cargo run --release

# 3. Open the clients
#    http://localhost:8787/          playground (TTS / audio test)
#    http://localhost:8787/mivocero  speech-to-speech agent with chat box

Environment variables can also be exported directly; a .env file is loaded automatically at startup (existing env vars take precedence).

Server control script

server.sh wraps start/stop/restart so you don't have to manage the process by hand:

./server.sh start            # start (builds automatically if the binary is missing)
./server.sh start --build    # rebuild, then start
./server.sh stop             # graceful stop (SIGKILL after 10 s)
./server.sh restart          # stop + start
./server.sh status           # process / port / scheme / HTTP status
./server.sh logs 100         # tail the last 100 lines (follows)
./server.sh cert             # generate a self-signed HTTPS certificate

It writes the PID to run/server.pid and appends stdout/stderr to run/server.log. Overridable env vars: PROFILE (release|debug, default release), RUST_LOG (default info; audio_agent_engine::tts=debug is always appended so the TTS streaming/latency diagnostics show up), PORT (default 8787).

HTTPS (microphone access)

Browsers only expose the microphone (getUserMedia) in a secure context — HTTPS or localhost. If you open the playground over plain HTTP on a non-localhost origin the "Connect & Start Mic" button fails with mic error.

Set both variables to serve HTTPS/WSS instead of HTTP:

TLS_CERT=certs/cert.pem
TLS_KEY=certs/key.pem

./server.sh cert generates a self-signed pair (SAN covers localhost, 127.0.0.1, ::1 and this host's IPs; use EXTRA_SAN=IP:1.2.3.4 to add more). Then open https://localhost:8787/ and accept the certificate warning — the page becomes a secure context and the mic works. Comment both variables out to fall back to plain HTTP. The server logs listening (HTTPS) or listening (HTTP) at startup accordingly.

---

Configuration

All configuration is via environment variables (or a .env file). See .env.example for a copy-paste template.

Variable	Default	Description
SERVER_PORT	8787	TCP port the Axum server listens on.
TTS_ENGINE	fish	TTS engine plugin — fish or kokoro. See below.
SGLANG_URL	http://10.8.0.3:8000	Base URL of the Fish Audio / SGLang server (fish engine).
TTS_MODEL	fishaudio/drama-3	Model identifier sent to /v1/audio/speech.
TTS_VOICE	default	Voice identifier.
TTS_REFERENCE_AUDIO	…/voice_profiles/solis.wav	Voice-cloning reference WAV (path on the TTS server).
TTS_REFERENCE_TEXT	(empty)	Reference transcript. If it names an existing .txt, its contents are loaded; otherwise the raw string is used. Empty → auto-load <audio-stem>.txt.
KOKORO_URL	http://localhost:8880	Kokoro HTTP endpoint (kokoro engine).
KOKORO_VOICE	af_heart	Kokoro voice name.
STT_MODEL_PATH	models/parakeet	Directory of the Parakeet TDT ONNX pipeline (see below).
STT_TOKENIZER_PATH	(empty)	Optional external tokenizer. Parakeet decodes text internally; leave empty.
VAD_ENABLED	true	Master switch for the Silero VAD. When false, the STT worker falls back to its internal RMS segmentation.
VAD_MODEL_PATH	models/silero_vad.onnx	Silero VAD v5 ONNX model (16 kHz, 512-sample windows).
VAD_THRESHOLD	0.5	Speech probability above which a window counts as voice.
VAD_NEG_THRESHOLD	0.35	Probability below which a window counts as silence (hysteresis; clamped to VAD_THRESHOLD).
VAD_MIN_SPEECH_MS	250	Speech required to confirm an utterance start.
VAD_MIN_SILENCE_MS	300	Silence required to close an utterance (end-of-utterance window).
VAD_SPEECH_PAD_MS	30	Audio kept before the detected onset so the first phoneme is not clipped.
VAD_BARGE_IN_MIN_SPEECH_MS	150	Speech required to fire the fast barge-in onset (clamped to VAD_MIN_SPEECH_MS).
VAD_SEGMENT_STT	true	Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation.
VAD_BARGE_IN	true	Interrupt the agent when the user starts speaking over it.
VAD_DEBUG	false	Log the speech probability of every analysis window.
FILLERS_DIR	assets/fillers	Directory of *.raw filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn.
FILLER_ENABLED	true	Master switch for the end-of-turn filler. false => no filler is ever queued.
FILLER_DELAY_MS	2500	How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer.
FILLER_GAIN_IDLE	1.0	Filler gain when TTS is idle.
FILLER_GAIN_DUCK	0.15	Filler gain while TTS is active (ducking).
TTS_PREBUFFER_MS	450	Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again.
TTS_PIPELINE_DEPTH	3	How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. 1 restores strictly serial synthesis.
BACKCHANNEL_ENABLED	true	Master switch for during-utterance acknowledgments.
BACKCHANNEL_DIR	assets/backchannel	Directory of *.raw backchannel PCM clips (16-bit LE mono, 44.1 kHz).
BACKCHANNEL_GAIN	0.35	Mix gain applied to the backchannel channel (0..1).
BACKCHANNEL_MIN_SPEECH_MS	1500	Continuous speech required before the first backchannel of a turn.
BACKCHANNEL_INTERVAL_MS	3000	Minimum spacing between two backchannels (clamped to ≥ BACKCHANNEL_MIN_SPEECH_MS).
BACKCHANNEL_MAX_PER_TURN	2	Maximum backchannels emitted per user turn.
LLM_ENABLED	(auto)	Master switch for the LLM reasoning stage. Default: enabled when LLM_BASE_URL and LLM_MODEL are both set. When false, the pipeline is a plain STT → TTS echo.
LLM_BASE_URL	(empty)	Base URL of the OpenAI-compatible endpoint; the client posts to {LLM_BASE_URL}/chat/completions.
LLM_API_KEY	EMPTY	Bearer token sent in the Authorization header (local servers usually ignore it).
LLM_MODEL	(empty)	Model identifier sent in the request body.
LLM_MAX_TOKENS	1024	max_tokens for the completion.
LLM_TEMPERATURE	0.7	Sampling temperature.
LLM_STREAM	true	true → SSE token streaming, TTS starts on the first words; false → one blocking request, the whole reply is synthesized at once.
LLM_STREAM_FIRST_CHUNK_CHARS	20	Characters buffered before the first chunk is sent to the TTS, cut on a word boundary. Small = the agent starts speaking on the first few words. 0 disables the early flush.
LLM_STREAM_CHUNK_CHARS	120	Same threshold for every following chunk (larger chunks give better prosody). 0 disables.
LLM_HISTORY_MESSAGES	16	Rolling conversation window (user+assistant messages) kept across turns.
LLM_ENABLE_THINKING	(unset)	false sends chat_template_kwargs: {"enable_thinking": false} — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers).
LLM_STRIP_REASONING	true	Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores reasoning_content deltas and removes inline think blocks. false passes the raw output through.
LLM_DIRECT_ANSWERS	true	Append an "answer directly, never narrate your reasoning" directive to the system prompt.
LLM_SYSTEM_PROMPT	(built-in)	Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points).
TLS_CERT	(empty)	PEM certificate chain. Set together with TLS_KEY to serve HTTPS/WSS.
TLS_KEY	(empty)	PEM private key matching TLS_CERT.

---

TTS Plugin System

The TTS layer is engine-agnostic. Every engine is a module under src/tts/ that implements the TtsEngine trait. The active engine is selected at runtime by the TTS_ENGINE environment variable — the orchestration layer (TurnManager, AppState) only ever sees an Arc<dyn TtsEngine>.

The TtsEngine trait

#[async_trait]
pub trait TtsEngine: Send + Sync {
    /// Human-readable engine name (used for logging).
    fn name(&self) -> &str;

    /// Rate (Hz) at which this engine pushes samples into the mixer queue.
    /// Must equal `config::OUTPUT_SAMPLE_RATE` (44_100).
    fn sample_rate(&self) -> u32;

    /// Synthesize `text` and stream PCM into `tts_queue`.
    /// Must honor `cancel` for barge-in and return promptly.
    async fn synthesize(
        &self,
        text: &str,
        tts_queue: &SampleRingBuffer,
        cancel: &CancellationToken,
    ) -> Result<()>;

    /// Runtime voice-cloning reference update (default: no-op).
    fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}
}

Contract for implementors:

1. Push 16-bit mono PCM at OUTPUT_SAMPLE_RATE (44 100 Hz) into tts_queue. If the engine's native rate differs, resample internally — the mixer runs at a fixed rate and does not resample.
2. Honor the CancellationToken: on cancellation, stop pushing and return Ok(()) promptly (barge-in / interruption).
3. Override set_reference() only if the engine supports voice cloning.

Engine selection (factory)

create_tts_engine() maps TTS_ENGINE to a concrete plugin. Values are case-insensitive:

TTS_ENGINE	Plugin type	Notes
fish · fishaudio · fish_audio · sglang	FishAudioTts	Fish Audio / SGLang Drama 3, OpenAI-compatible streaming. Production-ready.
kokoro	KokoroTts	Kokoro TTS. Stub — complete the HTTP call (see below).

An unknown value panics at startup with a clear message.

Bundled plugins

fish — Fish Audio / SGLang Drama 3

src/tts/fish_audio.rs

• POST {SGLANG_URL}/v1/audio/speech with response_format: "pcm", stream: true.
• Reads the actual output rate from the response header x-sample-rate (Drama 3 emits 44.1 kHz, which matches the mixer — no resampling needed).
• Supports voice cloning via the references field.
• Honors barge-in cancellation.

Starting the Drama 3 backend (SGLang-Omni)

The fish engine talks to an SGLang-Omni server running Drama 3. Start it with the vocoder streaming window narrowed, otherwise the vocoder emits ~3 s bursts and the client is forced into a large pre-buffer:

sgl-omni serve \
    --model-path fishaudio/drama-3 \
    --host 0.0.0.0 --port 8000 \
    --tts_engine.gpu_memory_fraction 0.45 \
    --tts_engine.factory.max_running_requests 4 \
    --vocoder.factory.stream_followup_stride 16 \
    --vocoder.factory.stream_followup_stride_ramp "[10,12,16]"

In this deployment the launcher is /root/drama3/start-drama3.sh and the server is kept alive by pm2 as the process start-drama3:

pm2 restart start-drama3 --update-env   # after editing the launcher
pm2 logs start-drama3 --lines 100       # CUDA-graph rebuild takes ~2 min at start

The two --vocoder.factory.* overrides are the key part:

Flag	Meaning
stream_followup_stride	Steady-state window, in codec frames (2048 samples = 46.4 ms each). 16 ⇒ 0.743 s bursts; the upstream default 64 ⇒ 2.97 s.
stream_followup_stride_ramp	Window sizes for the first windows, as a JSON tuple. [10,12,16] emits a 0.464 s then a 0.557 s window first, so playback can start sooner. Upstream default (10,12,32).

Two gotchas when writing the command:

• The flags are dotted keys into the pydantic config, so they go without a stages. prefix — --vocoder.factory.stream_followup_stride, not --stages.vocoder.factory.stream_followup_stride (the latter is rejected).
• The ramp tuple must be passed as a JSON string — "[10,12,16]" — because the config manager parses non-scalar paths with JSON.

Smaller stride = finer delivery granularity but more compute: each window re-decodes stream_overlap_tokens (default 32) of context, so stride 16 decodes 48 frames per 16 emitted. Keep the stride at or below the client's TTS_PREBUFFER_MS budget (see Mixer & jitter buffering) — with the defaults above, TTS_PREBUFFER_MS=450 starts playback on the first window.

kokoro — Kokoro TTS (stub)

src/tts/kokoro.rs

• Config via KOKORO_URL and KOKORO_VOICE.
• synthesize() currently returns an error — uncomment and adapt the provided example to call your Kokoro HTTP server.
• Kokoro typically outputs 24 kHz; the stub includes a resample_24k_to_output() helper (linear interpolation) that converts to the mixer rate before pushing. Replace it with a windowed-sinc/polyphase resampler for higher fidelity if needed.

Adding a new engine

1. Create src/tts/my_engine.rs and implement TtsEngine:

   use crate::audio::SampleRingBuffer;
   use crate::config::{Config, OUTPUT_SAMPLE_RATE};
   use crate::tts::TtsEngine;
   use anyhow::Result;
   use async_trait::async_trait;
   use tokio_util::sync::CancellationToken;
   
   pub struct MyEngine { /* http client, url, voice, … */ }
   
   impl MyEngine {
       pub fn new(config: &Config) -> Self { /* read env / config */ }
   }
   
   #[async_trait]
   impl TtsEngine for MyEngine {
       fn name(&self) -> &str { "my_engine" }
       fn sample_rate(&self) -> u32 { OUTPUT_SAMPLE_RATE }
   
       async fn synthesize(
           &self,
           text: &str,
           tts_queue: &SampleRingBuffer,
           cancel: &CancellationToken,
       ) -> Result<()> {
           // stream PCM, resample to OUTPUT_SAMPLE_RATE, push into tts_queue,
           // and stop when `cancel.is_cancelled()`.
           Ok(())
       }
   }
   

2. Register the module and the factory arm in src/tts/mod.rs:

   pub mod my_engine;
   // …
   "my_engine" => Arc::new(MyEngine::new(config)),
   

3. Select it at runtime:

   TTS_ENGINE=my_engine cargo run --release
   

---

STT — Parakeet TDT (pure Rust)

Speech recognition is implemented entirely in Rust via ONNX Runtime — no Python or NeMo dependency. src/stt/parakeet.rs runs the full NVIDIA Parakeet TDT 0.6B v3 pipeline:

16 kHz mono f32
  └─ nemo128.onnx             → mel features   [1, 128, T]
       └─ encoder-model*.onnx → encoder out    [1, 1024, T']
            └─ decoder_joint-model.onnx (greedy TDT) → token IDs
                 └─ vocab.txt  → text

The decoder is a transducer (TDT): at each encoder frame it predicts a token and a duration step. A positive step advances several frames at once; the blank token (<blk>, id 8192) means "nothing emitted here". This matches the reference implementation in istupakov/onnx-asr.

Model directory

STT_MODEL_PATH must point to a directory containing:

File	Purpose
nemo128.onnx	Log-mel feature extractor.
encoder-model.int8.onnx (preferred) or encoder-model.onnx	Conformer encoder. The int8 encoder (~650 MB, self-contained) is used automatically when present; otherwise the fp32 encoder (~42 MB + a .onnx.data sidecar) is loaded.
decoder_joint-model.onnx	TDT prediction network + joint.
vocab.txt	<piece> <id> token table (8193 entries).
config.json	Optional; supplies max_tokens_per_step (default 10).

Model files can be fetched from istupakov/parakeet-tdt-0.6b-v3-onnx into models/parakeet/.

Runtime behaviour

• The worker (SttWorker) consumes 16 kHz frames. By default the session's Silero VAD segments the audio and hands whole utterances to push_utterance; without VAD it falls back to RMS-based segmentation and triggers inference (spawn_blocking) after a EOU_SILENCE_MS (300 ms) silence window.
• Text is decoded inside the engine (SentencePiece ▁ → space, spacing normalized), so no external tokenizer is required — transcript.text is used directly by the TurnManager.

Testing

An opt-in end-to-end test decodes a 16 kHz mono WAV through the real model:

PARAKEET_TEST_WAV=/path/to/16k_mono.wav \
  cargo test --release -- --ignored test_infer_real_audio --nocapture

---

VAD — Silero V5 (ONNX)

Voice Activity Detection runs entirely on the server, in Rust, on the same 16 kHz mono stream that feeds the STT engine. It is implemented in src/vad/ and detects when the user starts and stops speaking.

Model

VAD_MODEL_PATH points to silero_vad.onnx (Silero VAD v5). The model consumes fixed 32 ms windows (512 samples @ 16 kHz) preceded by a 64-sample context carried over from the previous window, plus a recurrent state tensor [2, 1, 128] and a scalar sample-rate input. Each window yields a speech probability. The graph's tensor shapes and the sample-rate input rank are introspected at load time, so the wrapper adapts to the exported model rather than hard-coding shapes.

Events and state machine

VadDetector turns per-window probabilities into three events:

Event	Trigger	Used for
SpeechOnset	VAD_BARGE_IN_MIN_SPEECH_MS (150 ms) of speech	Fast barge-in: interrupt the agent as soon as the user starts talking.
SpeechStart	VAD_MIN_SPEECH_MS (250 ms) of speech	Confirms the utterance start.
SpeechEnd	VAD_MIN_SILENCE_MS (300 ms) of silence	Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine.

Onset/offset use hysteresis: a window counts as voice above VAD_THRESHOLD and as silence below VAD_NEG_THRESHOLD, which prevents flapping around a single threshold. A pre-roll ring (VAD_SPEECH_PAD_MS + VAD_MIN_SPEECH_MS + one window) is kept ahead of the detected onset so the first phonemes are never clipped.

Barge-in

When VAD_BARGE_IN=true and the agent is currently speaking (TurnManager::is_speaking, backed by AudioMixer::is_active), a SpeechOnset calls TurnManager::interrupt() — cancelling the in-flight TTS via its CancellationToken and flushing the mixer queues.

Wiring

The session task (src/ws/session.rs) owns one VadDetector per WebSocket session (the model itself is shared) and:

• with VAD_SEGMENT_STT=true (default): forwards SpeechEnd utterances to SttWorker::push_utterance, and does not forward raw frames (avoids double segmentation);
• with VAD_SEGMENT_STT=false: falls back to the STT worker's RMS segmentation, using the VAD only for barge-in;
• if VAD_ENABLED=false or the model fails to load: legacy behaviour (RMS segmentation only).

All VAD logs live under the audio_agent_engine::vad target; ./server.sh appends audio_agent_engine::vad=debug to RUST_LOG, and VAD_DEBUG=true additionally logs every window's probability.

---

Startup clip bootstrap

The filler and backchannel banks are plain *.raw PCM files. At startup assets::ensure_clips checks each expected clip and, if a clip is missing, synthesizes it with the configured TTS engine and writes it to disk (the directory is created if needed). Existing clips are never overwritten, so a fresh checkout works without a manual generation step and a partial/customized bank is left untouched.

• Defaults live in assets::FILLER_CLIPS (7 clips: uhm, eh, mmm, aver, bueno, pues, vaya) and assets::BACKCHANNEL_CLIPS (4 clips: mmhmm, aja, claro, yaveo). The generated voice matches the configured TTS voice-cloning reference.
• Backchannel clips are only generated when BACKCHANNEL_ENABLED=true.
• Generation is best-effort: if the TTS backend is unreachable each clip logs a warning and the server still starts (that bank simply stays silent).

INFO audio_agent_engine::assets: generating missing clips at startup label="filler" dir=assets/fillers missing=7 engine="fish-audio"
INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/uhm.raw
...
INFO audio_agent_engine: clip banks bootstrapped generated=11

---

Fillers — end-of-utterance acknowledgments

A filler is a short acknowledgment ("uhm", "eh") used as a latency mask while the reply is being generated. Because it exists to cover a gap, it is not played unconditionally: when the turn is dispatched (TurnManager::schedule_filler) a timer of FILLER_DELAY_MS (default 2500) is started, and the clip is queued only if the response audio has not started playing by then — talking over the first words of the answer is worse than a moment of silence. The check uses the mixer's tts_playback_started flag, which is set when the TTS pre-buffer is reached and reset at every mark_stream_start / flush.

• At most one filler per turn, and a barge-in (interrupt, VAD onset, new turn, disconnect) cancels the pending timer.
• FILLER_ENABLED=false disables the filler entirely (no clip is ever queued).
• Raise FILLER_DELAY_MS for slower LLM/TTS setups, lower it if the filler rarely fires.

Once queued, the mixer plays it at FILLER_GAIN_IDLE (1.0) and ducks it to FILLER_GAIN_DUCK (0.15) if TTS starts mid-clip.

FILLERS_DIR (default assets/fillers) holds *.raw clips — 16-bit LE mono PCM at 44.1 kHz — selected round-robin (TurnManager::next_filler → FillerBank::nth), so consecutive turns use different fillers. With a single clip the behaviour is unchanged. Missing clips are synthesized at startup (Startup clip bootstrap).

---

Backchannel — during-utterance acknowledgments

A backchannel is a short acknowledgment ("mm-hmm", "ajá", "claro") played while the user is still speaking, at a low gain, to signal that the agent is listening — without taking the turn. It is distinct from the filler — the instant acknowledgment injected at end-of-utterance by the TurnManager from FILLERS_DIR (played round-robin; see Fillers).

It is implemented in src/orchestrator/backchannel.rs.

Clips

BACKCHANNEL_DIR (default assets/backchannel) holds *.raw clips — 16-bit LE mono PCM at 44.1 kHz — played round-robin. The bundled set is synthesized with the same cloned voice as the TTS (mmhmm, aja, claro, yaveo). The clips are loaded once at startup into a FillerBank and shared across sessions. Missing clips are synthesized at startup (Startup clip bootstrap).

Trigger policy

BackchannelScheduler is fed by the VAD:

Signal	Action
VadEvent::SpeechStart	begin a turn (on_speech_start)
VadEvent::SpeechEnd	close the turn, reset counters (on_speech_end)
every frame while speaking	maybe_emit() attempts a cue

maybe_emit() emits a clip only when all of the following hold:

• BACKCHANNEL_ENABLED=true and the clip bank is non-empty;
• the user has been speaking for at least BACKCHANNEL_MIN_SPEECH_MS;
• at least BACKCHANNEL_INTERVAL_MS elapsed since the previous cue;
• fewer than BACKCHANNEL_MAX_PER_TURN cues were emitted this turn;
• the agent is not speaking (!TurnManager::is_speaking()) — cuing over the agent's own TTS would be an interruption, not a backchannel.

Mixing

The cue is pushed into the mixer's dedicated backchannel_queue and summed at a fixed low gain (BACKCHANNEL_GAIN, default 0.35):

mixed = tts + (filler × filler_gain) + (backchannel × BACKCHANNEL_GAIN)

The queue is included in AudioMixer::flush(), so a barge-in clears any pending cue along with the TTS and filler audio. Logs live under the audio_agent_engine::orchestrator::backchannel target (INFO when a cue is emitted).

---

LLM reasoning

When LLM_ENABLED=true (the default once LLM_BASE_URL + LLM_MODEL are set), TurnManager inserts a reasoning stage between STT and TTS:

microphone → VAD → STT ─┐
                        ├─→ LLM (chat completions) → chunker → TTS → mixer → browser
chat box  ──────────────┘

• Client — src/llm.rs. A minimal OpenAI-compatible client (POST {LLM_BASE_URL}/chat/completions) with two modes, selectable at runtime via LLM_STREAM:
  • chat_stream — SSE token streaming (stream: true); deltas are handled as they arrive.
  • chat — a single blocking request (stream: false).
• Chain-of-thought handling — reasoning models (Qwen3, …) may leak their plan in three different ways. Each is handled by an independent, .env- switchable mechanism, because a single one does not cover every deployment:
  1. LLM_ENABLE_THINKING=false — server-side switch: the request carries chat_template_kwargs: {"enable_thinking": false} (vLLM/SGLang), so the model answers directly and generates no reasoning at all. Left unset, the field is not sent, keeping the body valid for strict OpenAI-compatible servers.
  2. LLM_STRIP_REASONING=true (default) — client-side filter: separate reasoning_content deltas (vLLM with a reasoning parser) are dropped and inline think-tag blocks are removed by an incremental stripper that is stable across delta boundaries.
  3. LLM_DIRECT_ANSWERS=true (default) — prompt-level guard: appends DIRECT_ANSWER_DIRECTIVE to the system prompt, telling the model not to narrate its reasoning. This also covers servers with no reasoning parser at all, where the chain-of-thought is written straight into content as ordinary prose.
• Streaming chunking — the assistant text is split on ., !, ?, ; and newlines as it streams, and each chunk is dispatched to the TTS as soon as it is ready. A word-boundary flush bounds the latency for punctuation-free runs: the first chunk is cut once LLM_STREAM_FIRST_CHUNK_CHARS (default 20) characters are buffered — so the agent starts speaking on the first few words instead of waiting for a whole sentence — and every following chunk at LLM_STREAM_CHUNK_CHARS (default 120). Set either to 0 to disable that flush (chunks then end on sentence punctuation only).
• Pipelined synthesis — chunks are not synthesized one after another. Each chunk is requested as soon as the LLM produces it (up to TTS_PIPELINE_DEPTH, default 3, in flight at once) and streams into its own staging buffer; a relay forwards those buffers to the mixer in submission order. This overlaps each request's start-up latency with the previous chunk's playback, so raising the pre-buffer is no longer needed to avoid inter-chunk gaps.
• History — a rolling window of the last LLM_HISTORY_MESSAGES user/assistant messages is kept per connection and prepended with LLM_SYSTEM_PROMPT.
• Barge-in — an interrupt (client JSON, VAD onset, or disconnect) cancels both the in-flight HTTP request ([CancellationToken] in the reqwest tokio::select!, including while waiting for the next SSE chunk) and the TTS tasks, then flushes the mixer. Chat-box turns run in their own task so the session's receive loop stays responsive and keeps accepting interrupt messages while a reply is being generated.
• Failure handling — request errors are reported to the client as {"type":"error","text":…} and the turn ends cleanly.

---

Mivocero chat client

A ready-to-use speech-to-speech example client is served at /mivocero (embedded from src/web/mivocero.html):

• Microphone — the same AudioWorklet capture path as the playground (44.1 kHz, 16-bit LE mono, 20 ms frames).
• Chat box — shows the user transcript (from STT) and the assistant reply as it streams, plus messages typed in the composer.
• Karaoke highlighting — the assistant bubble is rendered word by word: each word lights up as it is spoken, driven by the audio playhead (the playback worklet reports the samples it has actually emitted). Words already spoken are bright, the word currently sounding is accented, and the rest stay dim. The rate is an estimate (KARAOKE_CPS, ~15 chars/s) — it only affects the visual highlight, never the audio.
• Text composer — typing and sending a message runs a full LLM → TTS turn without using the microphone ({"type":"user_text","text":…}).
• Interrupt button — flushes audio and cancels the in-flight reply ({"type":"interrupt"}).

Open http://localhost:8787/mivocero (or https://… for remote microphone access — see HTTPS).

---

WebSocket Protocol

Connect to ws://<host>:8787/ws.

Client → Server

Message	Description
Binary	Raw 16-bit LE mono PCM microphone frames (44.1 kHz). Resampled to 16 kHz server-side for STT.
Text — {"type":"user_text","text":"…"}	Text turn from the chat box: runs the same LLM → TTS path as a spoken utterance (no STT).
Text — {"type":"interrupt"}	Barge-in: cancel the in-flight LLM reply / TTS and flush all queued audio.
Text — INTERRUPT (also interrupt, BAARGEIN, bargein)	Legacy barge-in form (kept for the playground).
Text — SET_REFERENCE:<path>|<text>	Set/change the voice-cloning reference. <text> may be omitted to re-use the sibling .txt.
Text — CLEAR_REFERENCE	Disable voice cloning.
Text — anything else	Synthesize the text directly (useful for testing TTS without STT).

Server → Client

Message	Description
Binary	One 20 ms mixed frame of 16-bit LE mono PCM @ 44.1 kHz (1764 bytes). Silence is sent when no audio is queued, keeping the stream alive.
Text — {"type":"user","text":"…"}	The user's turn (STT transcript or chat-box message) — rendered in the chat box.
Text — {"type":"assistant_delta","text":"…"}	Streaming chunk of the assistant reply (chain-of-thought already removed).
Text — {"type":"assistant_end"}	The assistant turn is complete.
Text — {"type":"error","text":"…"}	LLM request failed; the turn was aborted.

---

Audio Pipeline

Constants are defined in src/config.rs:

Constant	Value	Meaning
OUTPUT_SAMPLE_RATE	44_100 Hz	Server → browser rate (matches Fish Audio Drama 3 native output).
STT_SAMPLE_RATE	16_000 Hz	Rate expected by Parakeet.
FRAME_DURATION_MS	20 ms	Mixer ticker period.
OUTPUT_FRAME_SAMPLES	882	Samples per frame (44.1 kHz × 20 ms).
OUTPUT_FRAME_BYTES	1764	Bytes per frame (16-bit mono).
EOU_SILENCE_MS	300 ms	Silence window that marks end-of-utterance.
VAD_RMS_THRESHOLD	0.01	RMS above which audio counts as voice.

Mixer & jitter buffering (src/audio/mixer.rs)

• Mixing formula: mixed = tts + (filler × filler_gain) + (backchannel × BACKCHANNEL_GAIN). The filler gain is 1.0 when TTS is idle, else filler_gain_duck (default 0.15); the backchannel gain is fixed (BACKCHANNEL_GAIN, default 0.35). Output is clamped to i16.
• Pre-buffer: TTS playback does not start until the queue holds ≥ TTS_PREBUFFER_MS (default 450 ms) of synthesized audio. It absorbs uneven TTS chunk arrival; lower it to start speaking sooner, raise it for fewer drop-outs. With pipelined synthesis (above) the pre-buffer only has to cover arrival jitter, not the TTS engine's per-request start-up, so it can stay small. Note: Drama-3 delivers ~1.04× real time, and its granularity is set by the server's vocoder window (--vocoder.factory.stream_followup_stride, in codec frames of 2048 samples = 46.4 ms). With stride 16 and ramp "[10,12,16]" the first window is 10 frames = 0.464 s, so 450 ms starts playback on that very first window (the steady state then delivers a 0.743 s burst every ~0.69 s). With the upstream default (64, a 2.97 s burst every ~2.7 s) gapless playback would instead require a ~3 s pre-buffer.
• Resilient resume: the full pre-buffer is only required at the first playback of a reply. If the queue empties mid-reply (after the 200 ms drain grace) playback pauses, but resumes as soon as any audio is queued again — it does not wait to refill the whole jitter buffer. mark_stream_start re-arms the pre-buffer at the beginning of every turn.
• Drain grace: brief inter-chunk empties (< 200 ms) emit silence without pausing playback, avoiding audible drop-outs on ordinary arrival jitter.
• Ticker: MissedTickBehavior::Burst — missed ticks are caught up in a burst rather than dropped, so no 20 ms frames are lost.

---

Project Layout

audio-agent-engine/
├── Cargo.toml
├── .env.example              # documented configuration template
├── server.sh                 # start / stop / restart / status / logs / cert
├── certs/                    # self-signed TLS cert+key (generated)
├── UPDATES.md                # changelog (English, newest-first)
├── assets/fillers/           # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin)
├── assets/backchannel/       # *.raw backchannel PCM (16-bit LE mono, 44.1 kHz)
├── models/                   # Parakeet ONNX + Silero VAD + tokenizer
└── src/
    ├── main.rs               # Axum server, routes, graceful shutdown
    ├── config.rs             # env-var config + audio constants
    ├── assets.rs             # startup clip bootstrap (auto-generate fillers/backchannel)
    ├── tls.rs                # optional HTTPS/WSS termination (rustls)
    ├── onnx.rs               # shared process-wide ONNX Runtime environment
    ├── audio/                # ring_buffer, mixer
    ├── tts/                  # ← plugin system
    │   ├── mod.rs            # TtsEngine trait + create_tts_engine factory
    │   ├── fish_audio.rs     # Fish Audio / SGLang Drama 3 plugin
    │   └── kokoro.rs         # Kokoro plugin (stub)
    ├── stt/                  # Parakeet ONNX + tokenizer + worker
    ├── vad/                  # Silero VAD v5 + onset/offset state machine
    ├── llm.rs                # OpenAI-compatible chat-completions client (streaming + think stripping)
    ├── orchestrator/         # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
    ├── ws/                   # handler, session (per-connection audio loop)
    └── web/                  # embedded playground.html + mivocero.html

HTTP routes

Route	Description
GET /	Playground single-page client (embedded at compile time).
GET /mivocero	Mivocero speech-to-speech example client with chat box (embedded at compile time).
GET /health	Liveness probe (ok).
GET /ws	WebSocket upgrade → per-session audio loop.

---

Development

# Build
cargo build

# Run tests (tokenizer unit tests)
cargo test

# Run with debug logging
RUST_LOG=debug cargo run

# Release build (LTO + thin codegen)
cargo run --release

---

Related Documentation

• UPDATES.md — full changelog (English, newest-first).
• src/tts/mod.rs — trait + factory source of truth.