776 lines
38 KiB
Markdown
776 lines
38 KiB
Markdown
# Audio Agent Engine
|
||
|
||
A production-ready **real-time, full-duplex voice agent engine** written in Rust
|
||
(Tokio + Axum + ONNX Runtime). It streams microphone audio from the browser,
|
||
runs speech recognition with end-of-utterance detection, injects a low-latency
|
||
backchannel filler, and streams synthesized speech back to the browser — all
|
||
mixed on a strict 20 ms ticker.
|
||
|
||
```
|
||
Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
|
||
│
|
||
▼
|
||
TurnManager (orchestrator)
|
||
│
|
||
▼
|
||
LLM reasoning (OpenAI-compatible)
|
||
│
|
||
┌──────────────────────┼──────────────────────┐
|
||
▼ ▼ ▼
|
||
Filler / backchannel TTS engine plugin Mixer (20 ms ticker)
|
||
(instant "uhm" + cues) (Fish Audio / Kokoro) │
|
||
└──────────────────────┴──────────────────────┘
|
||
│
|
||
▼
|
||
Mixed 44.1 kHz / 16-bit mono PCM ──► Browser
|
||
```
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
- **Full-duplex WebSocket audio** — 44.1 kHz / 16-bit mono PCM, 20 ms frames.
|
||
- **Pluggable TTS engines** — switch engines (Fish Audio, Kokoro, …) via `.env`
|
||
without changing orchestration code. See [TTS Plugin System](#tts-plugin-system).
|
||
- **Latency filler** — a preloaded filler is played on end-of-utterance *only if
|
||
the reply is late*: it is scheduled `FILLER_DELAY_MS` (default 2500 ms) after
|
||
the turn starts and dropped if the response audio has already begun. Clips are
|
||
played round-robin from `FILLERS_DIR`; `FILLER_ENABLED=false` disables it.
|
||
- **During-utterance backchannel** — short acknowledgments ("mm-hmm", "ajá")
|
||
are mixed in at a reduced gain *while the user is speaking*, so the agent
|
||
signals it is listening without taking the turn. Plugin-style `.env` config.
|
||
- **Barge-in / interrupt** — cancels in-flight synthesis and flushes audio,
|
||
triggered client-side or server-side by the VAD.
|
||
- **Voice cloning** — reference audio + transcript (Fish Audio `references`).
|
||
- **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to
|
||
`TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's
|
||
start-up latency behind the previous chunk's audio.
|
||
- **Double jitter buffer** — server pre-buffer + browser playback buffer absorb
|
||
uneven TTS chunk arrival, eliminating drop-outs.
|
||
- **Silero VAD (ONNX)** — server-side speech segmentation and fast barge-in.
|
||
- **Parakeet STT (ONNX)** with tokenizer decoding and EOU detection.
|
||
- **LLM reasoning** — the user transcript is sent to any OpenAI-compatible
|
||
chat-completions server (vLLM, SGLang, llama.cpp, Ollama, OpenAI) and the
|
||
reply is spoken by the TTS. Streaming (SSE) chunks the reply as it arrives
|
||
(first chunk after a few words, tunable) for the lowest time-to-first-audio;
|
||
chain-of-thought (`reasoning_content` / ` thinking`) is stripped from both the
|
||
audio and the chat box.
|
||
- **Embedded playground** — a single-page HTML client served at `/`.
|
||
- **Mivocero speech-to-speech client** — an example agent client at `/mivocero`
|
||
with a microphone, a live chat box (user transcript + streaming assistant
|
||
reply) and a text composer, useful to drive a full STT → LLM → TTS turn.
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
- Rust **1.75+** (async traits are provided via the `async-trait` crate).
|
||
- A TTS backend reachable over HTTP (e.g. Fish Audio / SGLang Drama 3).
|
||
- *(Optional)* A Parakeet TDT ONNX model directory (`models/parakeet/`) for
|
||
speech recognition. The server runs without STT (TTS still works via the
|
||
playground text box) if the model is absent.
|
||
- *(Optional)* An OpenAI-compatible chat-completions endpoint (e.g. a local
|
||
vLLM / SGLang server) for the LLM reasoning stage. Without it the pipeline is
|
||
a plain STT → TTS echo.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
cd audio-agent-engine
|
||
|
||
# 1. Configure (optional — defaults target Fish Audio Drama 3)
|
||
cp .env.example .env
|
||
$EDITOR .env
|
||
|
||
# 2. Build and run
|
||
cargo run --release
|
||
|
||
# 3. Open the clients
|
||
# http://localhost:8787/ playground (TTS / audio test)
|
||
# http://localhost:8787/mivocero speech-to-speech agent with chat box
|
||
```
|
||
|
||
Environment variables can also be `export`ed directly; a `.env` file is loaded
|
||
automatically at startup (existing env vars take precedence).
|
||
|
||
### Server control script
|
||
|
||
[`server.sh`](server.sh) wraps start/stop/restart so you don't have to manage the
|
||
process by hand:
|
||
|
||
```bash
|
||
./server.sh start # start (builds automatically if the binary is missing)
|
||
./server.sh start --build # rebuild, then start
|
||
./server.sh stop # graceful stop (SIGKILL after 10 s)
|
||
./server.sh restart # stop + start
|
||
./server.sh status # process / port / scheme / HTTP status
|
||
./server.sh logs 100 # tail the last 100 lines (follows)
|
||
./server.sh cert # generate a self-signed HTTPS certificate
|
||
```
|
||
|
||
It writes the PID to `run/server.pid` and appends stdout/stderr to
|
||
`run/server.log`. Overridable env vars: `PROFILE` (`release`|`debug`, default
|
||
`release`), `RUST_LOG` (default `info`; `audio_agent_engine::tts=debug` is always
|
||
appended so the TTS streaming/latency diagnostics show up), `PORT` (default
|
||
`8787`).
|
||
|
||
### HTTPS (microphone access)
|
||
|
||
Browsers only expose the microphone (`getUserMedia`) in a **secure context** —
|
||
HTTPS or `localhost`. If you open the playground over plain HTTP on a
|
||
non-localhost origin the "Connect & Start Mic" button fails with `mic error`.
|
||
|
||
Set both variables to serve HTTPS/WSS instead of HTTP:
|
||
|
||
```dotenv
|
||
TLS_CERT=certs/cert.pem
|
||
TLS_KEY=certs/key.pem
|
||
```
|
||
|
||
`./server.sh cert` generates a self-signed pair (SAN covers `localhost`,
|
||
`127.0.0.1`, `::1` and this host's IPs; use `EXTRA_SAN=IP:1.2.3.4` to add
|
||
more). Then open `https://localhost:8787/` and accept the certificate warning —
|
||
the page becomes a secure context and the mic works. Comment both variables out
|
||
to fall back to plain HTTP. The server logs `listening (HTTPS)` or
|
||
`listening (HTTP)` at startup accordingly.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All configuration is via environment variables (or a `.env` file).
|
||
See [`.env.example`](.env.example) for a copy-paste template.
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `SERVER_PORT` | `8787` | TCP port the Axum server listens on. |
|
||
| `TTS_ENGINE` | `fish` | **TTS engine plugin** — `fish` or `kokoro`. See below. |
|
||
| `SGLANG_URL` | `http://10.8.0.3:8000` | Base URL of the Fish Audio / SGLang server (`fish` engine). |
|
||
| `TTS_MODEL` | `fishaudio/drama-3` | Model identifier sent to `/v1/audio/speech`. |
|
||
| `TTS_VOICE` | `default` | Voice identifier. |
|
||
| `TTS_REFERENCE_AUDIO` | `…/voice_profiles/solis.wav` | Voice-cloning reference WAV (path on the TTS server). |
|
||
| `TTS_REFERENCE_TEXT` | *(empty)* | Reference transcript. If it names an existing `.txt`, its contents are loaded; otherwise the raw string is used. Empty → auto-load `<audio-stem>.txt`. |
|
||
| `KOKORO_URL` | `http://localhost:8880` | Kokoro HTTP endpoint (`kokoro` engine). |
|
||
| `KOKORO_VOICE` | `af_heart` | Kokoro voice name. |
|
||
| `STT_MODEL_PATH` | `models/parakeet` | Directory of the Parakeet TDT ONNX pipeline (see below). |
|
||
| `STT_TOKENIZER_PATH` | *(empty)* | Optional external tokenizer. Parakeet decodes text internally; leave empty. |
|
||
| `VAD_ENABLED` | `true` | Master switch for the Silero VAD. When `false`, the STT worker falls back to its internal RMS segmentation. |
|
||
| `VAD_MODEL_PATH` | `models/silero_vad.onnx` | Silero VAD v5 ONNX model (16 kHz, 512-sample windows). |
|
||
| `VAD_THRESHOLD` | `0.5` | Speech probability above which a window counts as voice. |
|
||
| `VAD_NEG_THRESHOLD` | `0.35` | Probability below which a window counts as silence (hysteresis; clamped to `VAD_THRESHOLD`). |
|
||
| `VAD_MIN_SPEECH_MS` | `250` | Speech required to confirm an utterance start. |
|
||
| `VAD_MIN_SILENCE_MS` | `300` | Silence required to close an utterance (end-of-utterance window). |
|
||
| `VAD_SPEECH_PAD_MS` | `30` | Audio kept before the detected onset so the first phoneme is not clipped. |
|
||
| `VAD_BARGE_IN_MIN_SPEECH_MS` | `150` | Speech required to fire the fast barge-in onset (clamped to `VAD_MIN_SPEECH_MS`). |
|
||
| `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. |
|
||
| `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. |
|
||
| `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. |
|
||
| `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. |
|
||
| `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. |
|
||
| `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. |
|
||
| `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. |
|
||
| `FILLER_GAIN_DUCK` | `0.15` | Filler gain while TTS is active (ducking). |
|
||
| `TTS_PREBUFFER_MS` | `900` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
|
||
| `TTS_PIPELINE_DEPTH` | `3` | How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. `1` restores strictly serial synthesis. |
|
||
| `BACKCHANNEL_ENABLED` | `true` | Master switch for during-utterance acknowledgments. |
|
||
| `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz). |
|
||
| `BACKCHANNEL_GAIN` | `0.35` | Mix gain applied to the backchannel channel (0..1). |
|
||
| `BACKCHANNEL_MIN_SPEECH_MS` | `1500` | Continuous speech required before the first backchannel of a turn. |
|
||
| `BACKCHANNEL_INTERVAL_MS` | `3000` | Minimum spacing between two backchannels (clamped to ≥ `BACKCHANNEL_MIN_SPEECH_MS`). |
|
||
| `BACKCHANNEL_MAX_PER_TURN` | `2` | Maximum backchannels emitted per user turn. |
|
||
| `LLM_ENABLED` | *(auto)* | Master switch for the LLM reasoning stage. Default: enabled when `LLM_BASE_URL` and `LLM_MODEL` are both set. When `false`, the pipeline is a plain STT → TTS echo. |
|
||
| `LLM_BASE_URL` | *(empty)* | Base URL of the OpenAI-compatible endpoint; the client posts to `{LLM_BASE_URL}/chat/completions`. |
|
||
| `LLM_API_KEY` | `EMPTY` | Bearer token sent in the `Authorization` header (local servers usually ignore it). |
|
||
| `LLM_MODEL` | *(empty)* | Model identifier sent in the request body. |
|
||
| `LLM_MAX_TOKENS` | `1024` | `max_tokens` for the completion. |
|
||
| `LLM_TEMPERATURE` | `0.7` | Sampling temperature. |
|
||
| `LLM_STREAM` | `true` | `true` → SSE token streaming, TTS starts on the first words; `false` → one blocking request, the whole reply is synthesized at once. |
|
||
| `LLM_STREAM_FIRST_CHUNK_CHARS` | `20` | Characters buffered before the **first** chunk is sent to the TTS, cut on a word boundary. Small = the agent starts speaking on the first few words. `0` disables the early flush. |
|
||
| `LLM_STREAM_CHUNK_CHARS` | `120` | Same threshold for every following chunk (larger chunks give better prosody). `0` disables. |
|
||
| `LLM_HISTORY_MESSAGES` | `16` | Rolling conversation window (user+assistant messages) kept across turns. |
|
||
| `LLM_ENABLE_THINKING` | *(unset)* | `false` sends `chat_template_kwargs: {"enable_thinking": false}` — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers). |
|
||
| `LLM_STRIP_REASONING` | `true` | Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores `reasoning_content` deltas and removes inline think blocks. `false` passes the raw output through. |
|
||
| `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. |
|
||
| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). |
|
||
| `TLS_CERT` | *(empty)* | PEM certificate chain. Set together with `TLS_KEY` to serve HTTPS/WSS. |
|
||
| `TLS_KEY` | *(empty)* | PEM private key matching `TLS_CERT`. |
|
||
|
||
---
|
||
|
||
## TTS Plugin System
|
||
|
||
The TTS layer is **engine-agnostic**. Every engine is a module under
|
||
[`src/tts/`](src/tts) that implements the [`TtsEngine`](src/tts/mod.rs) trait.
|
||
The active engine is selected at runtime by the `TTS_ENGINE` environment
|
||
variable — the orchestration layer (`TurnManager`, `AppState`) only ever sees an
|
||
`Arc<dyn TtsEngine>`.
|
||
|
||
### The `TtsEngine` trait
|
||
|
||
```rust
|
||
#[async_trait]
|
||
pub trait TtsEngine: Send + Sync {
|
||
/// Human-readable engine name (used for logging).
|
||
fn name(&self) -> &str;
|
||
|
||
/// Rate (Hz) at which this engine pushes samples into the mixer queue.
|
||
/// Must equal `config::OUTPUT_SAMPLE_RATE` (44_100).
|
||
fn sample_rate(&self) -> u32;
|
||
|
||
/// Synthesize `text` and stream PCM into `tts_queue`.
|
||
/// Must honor `cancel` for barge-in and return promptly.
|
||
async fn synthesize(
|
||
&self,
|
||
text: &str,
|
||
tts_queue: &SampleRingBuffer,
|
||
cancel: &CancellationToken,
|
||
) -> Result<()>;
|
||
|
||
/// Runtime voice-cloning reference update (default: no-op).
|
||
fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}
|
||
}
|
||
```
|
||
|
||
**Contract for implementors:**
|
||
|
||
1. Push **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44 100 Hz)** into `tts_queue`.
|
||
If the engine's native rate differs, **resample internally** — the mixer runs
|
||
at a fixed rate and does not resample.
|
||
2. Honor the `CancellationToken`: on cancellation, stop pushing and return
|
||
`Ok(())` promptly (barge-in / interruption).
|
||
3. Override `set_reference()` only if the engine supports voice cloning.
|
||
|
||
### Engine selection (factory)
|
||
|
||
[`create_tts_engine()`](src/tts/mod.rs) maps `TTS_ENGINE` to a concrete plugin.
|
||
Values are case-insensitive:
|
||
|
||
| `TTS_ENGINE` | Plugin type | Notes |
|
||
|---|---|---|
|
||
| `fish` · `fishaudio` · `fish_audio` · `sglang` | [`FishAudioTts`](src/tts/fish_audio.rs) | Fish Audio / SGLang Drama 3, OpenAI-compatible streaming. Production-ready. |
|
||
| `kokoro` | [`KokoroTts`](src/tts/kokoro.rs) | Kokoro TTS. **Stub** — complete the HTTP call (see below). |
|
||
|
||
An unknown value panics at startup with a clear message.
|
||
|
||
### Bundled plugins
|
||
|
||
#### `fish` — Fish Audio / SGLang Drama 3
|
||
|
||
[`src/tts/fish_audio.rs`](src/tts/fish_audio.rs)
|
||
|
||
- `POST {SGLANG_URL}/v1/audio/speech` with `response_format: "pcm"`, `stream: true`.
|
||
- Reads the actual output rate from the response header `x-sample-rate`
|
||
(Drama 3 emits **44.1 kHz**, which matches the mixer — no resampling needed).
|
||
- Supports voice cloning via the `references` field.
|
||
- Honors barge-in cancellation.
|
||
|
||
#### `kokoro` — Kokoro TTS *(stub)*
|
||
|
||
[`src/tts/kokoro.rs`](src/tts/kokoro.rs)
|
||
|
||
- Config via `KOKORO_URL` and `KOKORO_VOICE`.
|
||
- `synthesize()` currently returns an error — uncomment and adapt the provided
|
||
example to call your Kokoro HTTP server.
|
||
- Kokoro typically outputs **24 kHz**; the stub includes a
|
||
`resample_24k_to_output()` helper (linear interpolation) that converts to the
|
||
mixer rate before pushing. Replace it with a windowed-sinc/polyphase resampler
|
||
for higher fidelity if needed.
|
||
|
||
### Adding a new engine
|
||
|
||
1. Create `src/tts/my_engine.rs` and implement [`TtsEngine`](src/tts/mod.rs):
|
||
|
||
```rust
|
||
use crate::audio::SampleRingBuffer;
|
||
use crate::config::{Config, OUTPUT_SAMPLE_RATE};
|
||
use crate::tts::TtsEngine;
|
||
use anyhow::Result;
|
||
use async_trait::async_trait;
|
||
use tokio_util::sync::CancellationToken;
|
||
|
||
pub struct MyEngine { /* http client, url, voice, … */ }
|
||
|
||
impl MyEngine {
|
||
pub fn new(config: &Config) -> Self { /* read env / config */ }
|
||
}
|
||
|
||
#[async_trait]
|
||
impl TtsEngine for MyEngine {
|
||
fn name(&self) -> &str { "my_engine" }
|
||
fn sample_rate(&self) -> u32 { OUTPUT_SAMPLE_RATE }
|
||
|
||
async fn synthesize(
|
||
&self,
|
||
text: &str,
|
||
tts_queue: &SampleRingBuffer,
|
||
cancel: &CancellationToken,
|
||
) -> Result<()> {
|
||
// stream PCM, resample to OUTPUT_SAMPLE_RATE, push into tts_queue,
|
||
// and stop when `cancel.is_cancelled()`.
|
||
Ok(())
|
||
}
|
||
}
|
||
```
|
||
|
||
2. Register the module and the factory arm in [`src/tts/mod.rs`](src/tts/mod.rs):
|
||
|
||
```rust
|
||
pub mod my_engine;
|
||
// …
|
||
"my_engine" => Arc::new(MyEngine::new(config)),
|
||
```
|
||
|
||
3. Select it at runtime:
|
||
|
||
```bash
|
||
TTS_ENGINE=my_engine cargo run --release
|
||
```
|
||
|
||
---
|
||
|
||
## STT — Parakeet TDT (pure Rust)
|
||
|
||
Speech recognition is implemented entirely in Rust via ONNX Runtime — no Python
|
||
or NeMo dependency. [`src/stt/parakeet.rs`](src/stt/parakeet.rs) runs the full
|
||
NVIDIA **Parakeet TDT 0.6B v3** pipeline:
|
||
|
||
```text
|
||
16 kHz mono f32
|
||
└─ nemo128.onnx → mel features [1, 128, T]
|
||
└─ encoder-model*.onnx → encoder out [1, 1024, T']
|
||
└─ decoder_joint-model.onnx (greedy TDT) → token IDs
|
||
└─ vocab.txt → text
|
||
```
|
||
|
||
The decoder is a **transducer (TDT)**: at each encoder frame it predicts a token
|
||
*and* a duration step. A positive step advances several frames at once; the
|
||
blank token (`<blk>`, id 8192) means "nothing emitted here". This matches the
|
||
reference implementation in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr).
|
||
|
||
### Model directory
|
||
|
||
`STT_MODEL_PATH` must point to a directory containing:
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `nemo128.onnx` | Log-mel feature extractor. |
|
||
| `encoder-model.int8.onnx` *(preferred)* or `encoder-model.onnx` | Conformer encoder. The int8 encoder (~650 MB, self-contained) is used automatically when present; otherwise the fp32 encoder (~42 MB + a `.onnx.data` sidecar) is loaded. |
|
||
| `decoder_joint-model.onnx` | TDT prediction network + joint. |
|
||
| `vocab.txt` | `<piece> <id>` token table (8193 entries). |
|
||
| `config.json` | Optional; supplies `max_tokens_per_step` (default 10). |
|
||
|
||
Model files can be fetched from
|
||
[`istupakov/parakeet-tdt-0.6b-v3-onnx`](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx)
|
||
into `models/parakeet/`.
|
||
|
||
### Runtime behaviour
|
||
|
||
- The worker ([`SttWorker`](src/stt/parakeet.rs)) consumes 16 kHz frames. By
|
||
default the session's **Silero VAD** segments the audio and hands whole
|
||
utterances to `push_utterance`; without VAD it falls back to RMS-based
|
||
segmentation and triggers inference (`spawn_blocking`) after a
|
||
`EOU_SILENCE_MS` (300 ms) silence window.
|
||
- Text is decoded **inside** the engine (SentencePiece `▁` → space, spacing
|
||
normalized), so no external tokenizer is required — `transcript.text` is used
|
||
directly by the [`TurnManager`](src/orchestrator/turn_manager.rs).
|
||
|
||
### Testing
|
||
|
||
An opt-in end-to-end test decodes a 16 kHz mono WAV through the real model:
|
||
|
||
```bash
|
||
PARAKEET_TEST_WAV=/path/to/16k_mono.wav \
|
||
cargo test --release -- --ignored test_infer_real_audio --nocapture
|
||
```
|
||
|
||
---
|
||
|
||
## VAD — Silero V5 (ONNX)
|
||
|
||
Voice Activity Detection runs entirely on the server, in Rust, on the same
|
||
16 kHz mono stream that feeds the STT engine. It is implemented in
|
||
[`src/vad/`](src/vad) and detects when the user starts and stops speaking.
|
||
|
||
### Model
|
||
|
||
`VAD_MODEL_PATH` points to `silero_vad.onnx` (Silero VAD **v5**). The model
|
||
consumes fixed **32 ms** windows (512 samples @ 16 kHz) preceded by a 64-sample
|
||
context carried over from the previous window, plus a recurrent state tensor
|
||
`[2, 1, 128]` and a scalar sample-rate input. Each window yields a speech
|
||
probability. The graph's tensor shapes and the sample-rate input rank are
|
||
introspected at load time, so the wrapper adapts to the exported model rather
|
||
than hard-coding shapes.
|
||
|
||
### Events and state machine
|
||
|
||
[`VadDetector`](src/vad/silero.rs) turns per-window probabilities into three
|
||
events:
|
||
|
||
| Event | Trigger | Used for |
|
||
|---|---|---|
|
||
| `SpeechOnset` | `VAD_BARGE_IN_MIN_SPEECH_MS` (150 ms) of speech | Fast **barge-in**: interrupt the agent as soon as the user starts talking. |
|
||
| `SpeechStart` | `VAD_MIN_SPEECH_MS` (250 ms) of speech | Confirms the utterance start. |
|
||
| `SpeechEnd` | `VAD_MIN_SILENCE_MS` (300 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. |
|
||
|
||
Onset/offset use hysteresis: a window counts as voice above `VAD_THRESHOLD`
|
||
and as silence below `VAD_NEG_THRESHOLD`, which prevents flapping around a
|
||
single threshold. A **pre-roll ring**
|
||
(`VAD_SPEECH_PAD_MS + VAD_MIN_SPEECH_MS + one window`) is kept ahead of the
|
||
detected onset so the first phonemes are never clipped.
|
||
|
||
### Barge-in
|
||
|
||
When `VAD_BARGE_IN=true` and the agent is currently speaking
|
||
([`TurnManager::is_speaking`](src/orchestrator/turn_manager.rs), backed by
|
||
[`AudioMixer::is_active`](src/audio/mixer.rs)), a `SpeechOnset` calls
|
||
`TurnManager::interrupt()` — cancelling the in-flight TTS via its
|
||
`CancellationToken` and flushing the mixer queues.
|
||
|
||
### Wiring
|
||
|
||
The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one
|
||
`VadDetector` per WebSocket session (the model itself is shared) and:
|
||
|
||
- with `VAD_SEGMENT_STT=true` (default): forwards `SpeechEnd` utterances to
|
||
`SttWorker::push_utterance`, and does **not** forward raw frames (avoids
|
||
double segmentation);
|
||
- with `VAD_SEGMENT_STT=false`: falls back to the STT worker's RMS
|
||
segmentation, using the VAD only for barge-in;
|
||
- if `VAD_ENABLED=false` or the model fails to load: legacy behaviour (RMS
|
||
segmentation only).
|
||
|
||
All VAD logs live under the `audio_agent_engine::vad` target;
|
||
`./server.sh` appends `audio_agent_engine::vad=debug` to `RUST_LOG`, and
|
||
`VAD_DEBUG=true` additionally logs every window's probability.
|
||
|
||
---
|
||
|
||
## Startup clip bootstrap
|
||
|
||
The filler and backchannel banks are plain `*.raw` PCM files. At startup
|
||
[`assets::ensure_clips`](src/assets.rs) checks each expected clip and, **if a
|
||
clip is missing, synthesizes it with the configured TTS engine** and writes it
|
||
to disk (the directory is created if needed). Existing clips are never
|
||
overwritten, so a fresh checkout works without a manual generation step and a
|
||
partial/customized bank is left untouched.
|
||
|
||
- Defaults live in [`assets::FILLER_CLIPS`](src/assets.rs) (7 clips: `uhm`,
|
||
`eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya`) and
|
||
[`assets::BACKCHANNEL_CLIPS`](src/assets.rs) (4 clips: `mmhmm`, `aja`, `claro`,
|
||
`yaveo`). The generated voice matches the configured TTS voice-cloning
|
||
reference.
|
||
- Backchannel clips are only generated when `BACKCHANNEL_ENABLED=true`.
|
||
- Generation is best-effort: if the TTS backend is unreachable each clip logs a
|
||
warning and the server still starts (that bank simply stays silent).
|
||
|
||
```text
|
||
INFO audio_agent_engine::assets: generating missing clips at startup label="filler" dir=assets/fillers missing=7 engine="fish-audio"
|
||
INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/uhm.raw
|
||
...
|
||
INFO audio_agent_engine: clip banks bootstrapped generated=11
|
||
```
|
||
|
||
---
|
||
|
||
## Fillers — end-of-utterance acknowledgments
|
||
|
||
A **filler** is a short acknowledgment ("uhm", "eh") used as a **latency mask**
|
||
while the reply is being generated. Because it exists to cover a *gap*, it is
|
||
not played unconditionally: when the turn is dispatched
|
||
([`TurnManager::schedule_filler`](src/orchestrator/turn_manager.rs)) a timer of
|
||
`FILLER_DELAY_MS` (default `2500`) is started, and the clip is queued **only if
|
||
the response audio has not started playing by then** — talking over the first
|
||
words of the answer is worse than a moment of silence. The check uses the
|
||
mixer's `tts_playback_started` flag, which is set when the TTS pre-buffer is
|
||
reached and reset at every `mark_stream_start` / `flush`.
|
||
|
||
- **At most one filler per turn**, and a barge-in (`interrupt`, VAD onset, new
|
||
turn, disconnect) cancels the pending timer.
|
||
- `FILLER_ENABLED=false` disables the filler entirely (no clip is ever queued).
|
||
- Raise `FILLER_DELAY_MS` for slower LLM/TTS setups, lower it if the filler
|
||
rarely fires.
|
||
|
||
Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`) and ducks it to
|
||
`FILLER_GAIN_DUCK` (`0.15`) if TTS starts mid-clip.
|
||
|
||
`FILLERS_DIR` (default `assets/fillers`) holds `*.raw` clips — **16-bit LE mono
|
||
PCM at 44.1 kHz** — selected **round-robin** (`TurnManager::next_filler` →
|
||
[`FillerBank::nth`](src/orchestrator/turn_manager.rs)), so consecutive turns use
|
||
different fillers. With a single clip the behaviour is unchanged. Missing clips
|
||
are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)).
|
||
|
||
---
|
||
|
||
## Backchannel — during-utterance acknowledgments
|
||
|
||
A **backchannel** is a short acknowledgment ("mm-hmm", "ajá", "claro") played
|
||
*while the user is still speaking*, at a low gain, to signal that the agent is
|
||
listening — without taking the turn. It is distinct from the **filler** — the
|
||
instant acknowledgment injected at end-of-utterance by the
|
||
[`TurnManager`](src/orchestrator/turn_manager.rs) from `FILLERS_DIR` (played
|
||
round-robin; see [Fillers](#fillers--end-of-utterance-acknowledgments)).
|
||
|
||
It is implemented in [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs).
|
||
|
||
### Clips
|
||
|
||
`BACKCHANNEL_DIR` (default `assets/backchannel`) holds `*.raw` clips — **16-bit
|
||
LE mono PCM at 44.1 kHz** — played round-robin. The bundled set is synthesized
|
||
with the same cloned voice as the TTS (`mmhmm`, `aja`, `claro`, `yaveo`). The
|
||
clips are loaded once at startup into a
|
||
[`FillerBank`](src/orchestrator/turn_manager.rs) and shared across sessions.
|
||
Missing clips are synthesized at startup
|
||
([Startup clip bootstrap](#startup-clip-bootstrap)).
|
||
|
||
### Trigger policy
|
||
|
||
[`BackchannelScheduler`](src/orchestrator/backchannel.rs) is fed by the VAD:
|
||
|
||
| Signal | Action |
|
||
|---|---|
|
||
| `VadEvent::SpeechStart` | begin a turn (`on_speech_start`) |
|
||
| `VadEvent::SpeechEnd` | close the turn, reset counters (`on_speech_end`) |
|
||
| every frame while speaking | `maybe_emit()` attempts a cue |
|
||
|
||
`maybe_emit()` emits a clip **only when all** of the following hold:
|
||
|
||
- `BACKCHANNEL_ENABLED=true` and the clip bank is non-empty;
|
||
- the user has been speaking for at least `BACKCHANNEL_MIN_SPEECH_MS`;
|
||
- at least `BACKCHANNEL_INTERVAL_MS` elapsed since the previous cue;
|
||
- fewer than `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn;
|
||
- the agent is **not** speaking (`!TurnManager::is_speaking()`) — cuing over the
|
||
agent's own TTS would be an interruption, not a backchannel.
|
||
|
||
### Mixing
|
||
|
||
The cue is pushed into the mixer's dedicated `backchannel_queue` and summed at a
|
||
fixed low gain (`BACKCHANNEL_GAIN`, default `0.35`):
|
||
|
||
```text
|
||
mixed = tts + (filler × filler_gain) + (backchannel × BACKCHANNEL_GAIN)
|
||
```
|
||
|
||
The queue is included in `AudioMixer::flush()`, so a barge-in clears any pending
|
||
cue along with the TTS and filler audio. Logs live under the
|
||
`audio_agent_engine::orchestrator::backchannel` target (`INFO` when a cue is
|
||
emitted).
|
||
|
||
---
|
||
|
||
## LLM reasoning
|
||
|
||
When `LLM_ENABLED=true` (the default once `LLM_BASE_URL` + `LLM_MODEL` are set),
|
||
[`TurnManager`](src/orchestrator/turn_manager.rs) inserts a reasoning stage
|
||
between STT and TTS:
|
||
|
||
```
|
||
microphone → VAD → STT ─┐
|
||
├─→ LLM (chat completions) → chunker → TTS → mixer → browser
|
||
chat box ──────────────┘
|
||
```
|
||
|
||
- **Client** — [`src/llm.rs`](src/llm.rs). A minimal OpenAI-compatible client
|
||
(`POST {LLM_BASE_URL}/chat/completions`) with two modes, selectable at runtime
|
||
via `LLM_STREAM`:
|
||
- `chat_stream` — SSE token streaming (`stream: true`); deltas are handled as
|
||
they arrive.
|
||
- `chat` — a single blocking request (`stream: false`).
|
||
- **Chain-of-thought handling** — reasoning models (Qwen3, …) may leak their
|
||
plan in three different ways. Each is handled by an independent, `.env`-
|
||
switchable mechanism, because a single one does not cover every deployment:
|
||
1. `LLM_ENABLE_THINKING=false` — server-side switch: the request carries
|
||
`chat_template_kwargs: {"enable_thinking": false}` (vLLM/SGLang), so the
|
||
model answers directly and generates no reasoning at all. Left unset, the
|
||
field is not sent, keeping the body valid for strict OpenAI-compatible
|
||
servers.
|
||
2. `LLM_STRIP_REASONING=true` (default) — client-side filter: separate
|
||
`reasoning_content` deltas (vLLM with a reasoning parser) are dropped and
|
||
inline think-tag blocks are removed by an incremental stripper that is
|
||
stable across delta boundaries.
|
||
3. `LLM_DIRECT_ANSWERS=true` (default) — prompt-level guard: appends
|
||
[`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt, telling
|
||
the model not to narrate its reasoning. This also covers servers with no
|
||
reasoning parser at all, where the chain-of-thought is written straight
|
||
into `content` as ordinary prose.
|
||
- **Streaming chunking** — the assistant text is split on `.`, `!`, `?`, `;` and
|
||
newlines as it streams, and each chunk is dispatched to the TTS as soon as it
|
||
is ready. A word-boundary flush bounds the latency for punctuation-free runs:
|
||
the **first** chunk is cut once `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`)
|
||
characters are buffered — so the agent starts speaking on the first few words
|
||
instead of waiting for a whole sentence — and every following chunk at
|
||
`LLM_STREAM_CHUNK_CHARS` (default `120`). Set either to `0` to disable that
|
||
flush (chunks then end on sentence punctuation only).
|
||
- **Pipelined synthesis** — chunks are *not* synthesized one after another. Each
|
||
chunk is requested as soon as the LLM produces it (up to `TTS_PIPELINE_DEPTH`,
|
||
default `3`, in flight at once) and streams into its own staging buffer; a
|
||
relay forwards those buffers to the mixer **in submission order**. This
|
||
overlaps each request's start-up latency with the previous chunk's playback,
|
||
so raising the pre-buffer is no longer needed to avoid inter-chunk gaps.
|
||
- **History** — a rolling window of the last `LLM_HISTORY_MESSAGES` user/assistant
|
||
messages is kept per connection and prepended with `LLM_SYSTEM_PROMPT`.
|
||
- **Barge-in** — an interrupt (client JSON, VAD onset, or disconnect) cancels
|
||
both the in-flight HTTP request ([`CancellationToken`] in the `reqwest`
|
||
`tokio::select!`, including while waiting for the next SSE chunk) and the TTS
|
||
tasks, then flushes the mixer. Chat-box turns run in their own task so the
|
||
session's receive loop stays responsive and keeps accepting `interrupt`
|
||
messages while a reply is being generated.
|
||
- **Failure handling** — request errors are reported to the client as
|
||
`{"type":"error","text":…}` and the turn ends cleanly.
|
||
|
||
---
|
||
|
||
## Mivocero chat client
|
||
|
||
A ready-to-use speech-to-speech example client is served at **`/mivocero`**
|
||
(embedded from [`src/web/mivocero.html`](src/web/mivocero.html)):
|
||
|
||
- **Microphone** — the same AudioWorklet capture path as the playground
|
||
(44.1 kHz, 16-bit LE mono, 20 ms frames).
|
||
- **Chat box** — shows the user transcript (from STT) and the assistant reply
|
||
**as it streams**, plus messages typed in the composer.
|
||
- **Text composer** — typing and sending a message runs a full LLM → TTS turn
|
||
without using the microphone (`{"type":"user_text","text":…}`).
|
||
- **Interrupt button** — flushes audio and cancels the in-flight reply
|
||
(`{"type":"interrupt"}`).
|
||
|
||
Open `http://localhost:8787/mivocero` (or `https://…` for remote microphone
|
||
access — see [HTTPS](#https-microphone-access)).
|
||
|
||
---
|
||
|
||
## WebSocket Protocol
|
||
|
||
Connect to `ws://<host>:8787/ws`.
|
||
|
||
### Client → Server
|
||
|
||
| Message | Description |
|
||
|---|---|
|
||
| **Binary** | Raw 16-bit LE mono PCM microphone frames (44.1 kHz). Resampled to 16 kHz server-side for STT. |
|
||
| **Text — `{"type":"user_text","text":"…"}`** | Text turn from the chat box: runs the same LLM → TTS path as a spoken utterance (no STT). |
|
||
| **Text — `{"type":"interrupt"}`** | Barge-in: cancel the in-flight LLM reply / TTS and flush all queued audio. |
|
||
| **Text — `INTERRUPT`** *(also `interrupt`, `BAARGEIN`, `bargein`)* | Legacy barge-in form (kept for the playground). |
|
||
| **Text — `SET_REFERENCE:<path>\|<text>`** | Set/change the voice-cloning reference. `<text>` may be omitted to re-use the sibling `.txt`. |
|
||
| **Text — `CLEAR_REFERENCE`** | Disable voice cloning. |
|
||
| **Text — anything else** | Synthesize the text directly (useful for testing TTS without STT). |
|
||
|
||
### Server → Client
|
||
|
||
| Message | Description |
|
||
|---|---|
|
||
| **Binary** | One 20 ms mixed frame of 16-bit LE mono PCM @ 44.1 kHz (`1764` bytes). Silence is sent when no audio is queued, keeping the stream alive. |
|
||
| **Text — `{"type":"user","text":"…"}`** | The user's turn (STT transcript or chat-box message) — rendered in the chat box. |
|
||
| **Text — `{"type":"assistant_delta","text":"…"}`** | Streaming chunk of the assistant reply (chain-of-thought already removed). |
|
||
| **Text — `{"type":"assistant_end"}`** | The assistant turn is complete. |
|
||
| **Text — `{"type":"error","text":"…"}`** | LLM request failed; the turn was aborted. |
|
||
|
||
---
|
||
|
||
## Audio Pipeline
|
||
|
||
Constants are defined in [`src/config.rs`](src/config.rs):
|
||
|
||
| Constant | Value | Meaning |
|
||
|---|---|---|
|
||
| `OUTPUT_SAMPLE_RATE` | `44_100` Hz | Server → browser rate (matches Fish Audio Drama 3 native output). |
|
||
| `STT_SAMPLE_RATE` | `16_000` Hz | Rate expected by Parakeet. |
|
||
| `FRAME_DURATION_MS` | `20` ms | Mixer ticker period. |
|
||
| `OUTPUT_FRAME_SAMPLES` | `882` | Samples per frame (44.1 kHz × 20 ms). |
|
||
| `OUTPUT_FRAME_BYTES` | `1764` | Bytes per frame (16-bit mono). |
|
||
| `EOU_SILENCE_MS` | `300` ms | Silence window that marks end-of-utterance. |
|
||
| `VAD_RMS_THRESHOLD` | `0.01` | RMS above which audio counts as voice. |
|
||
|
||
### Mixer & jitter buffering ([`src/audio/mixer.rs`](src/audio/mixer.rs))
|
||
|
||
- Mixing formula:
|
||
`mixed = tts + (filler × filler_gain) + (backchannel × BACKCHANNEL_GAIN)`.
|
||
The filler gain is `1.0` when TTS is idle, else `filler_gain_duck` (default
|
||
`0.15`); the backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
|
||
Output is clamped to `i16`.
|
||
- **Pre-buffer:** TTS playback does not start until the queue holds
|
||
≥ `TTS_PREBUFFER_MS` (default **900 ms**) of synthesized audio. It absorbs
|
||
uneven TTS chunk arrival; lower it to start speaking sooner, raise it for fewer
|
||
drop-outs. With pipelined synthesis (above) the pre-buffer only has to cover
|
||
arrival jitter, not the TTS engine's per-request start-up, so it can stay
|
||
small. Note: Drama-3 delivers ~1.08× real time in bursts every ~2.7 s, so a
|
||
~900 ms buffer favours latency over perfect gaplessness on very long replies —
|
||
raising it towards ~3 s is what actually removes the remaining drop-outs.
|
||
- **Resilient resume:** the full pre-buffer is only required at the *first*
|
||
playback of a reply. If the queue empties mid-reply (after the 200 ms drain
|
||
grace) playback pauses, but resumes as soon as *any* audio is queued again —
|
||
it does not wait to refill the whole jitter buffer. `mark_stream_start` re-arms
|
||
the pre-buffer at the beginning of every turn.
|
||
- **Drain grace:** brief inter-chunk empties (< 200 ms) emit silence without
|
||
pausing playback, avoiding audible drop-outs on ordinary arrival jitter.
|
||
- **Ticker:** `MissedTickBehavior::Burst` — missed ticks are caught up in a burst
|
||
rather than dropped, so no 20 ms frames are lost.
|
||
|
||
---
|
||
|
||
## Project Layout
|
||
|
||
```
|
||
audio-agent-engine/
|
||
├── Cargo.toml
|
||
├── .env.example # documented configuration template
|
||
├── server.sh # start / stop / restart / status / logs / cert
|
||
├── certs/ # self-signed TLS cert+key (generated)
|
||
├── UPDATES.md # changelog (English, newest-first)
|
||
├── assets/fillers/ # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin)
|
||
├── assets/backchannel/ # *.raw backchannel PCM (16-bit LE mono, 44.1 kHz)
|
||
├── models/ # Parakeet ONNX + Silero VAD + tokenizer
|
||
└── src/
|
||
├── main.rs # Axum server, routes, graceful shutdown
|
||
├── config.rs # env-var config + audio constants
|
||
├── assets.rs # startup clip bootstrap (auto-generate fillers/backchannel)
|
||
├── tls.rs # optional HTTPS/WSS termination (rustls)
|
||
├── onnx.rs # shared process-wide ONNX Runtime environment
|
||
├── audio/ # ring_buffer, mixer
|
||
├── tts/ # ← plugin system
|
||
│ ├── mod.rs # TtsEngine trait + create_tts_engine factory
|
||
│ ├── fish_audio.rs # Fish Audio / SGLang Drama 3 plugin
|
||
│ └── kokoro.rs # Kokoro plugin (stub)
|
||
├── stt/ # Parakeet ONNX + tokenizer + worker
|
||
├── vad/ # Silero VAD v5 + onset/offset state machine
|
||
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
|
||
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
|
||
├── ws/ # handler, session (per-connection audio loop)
|
||
└── web/ # embedded playground.html + mivocero.html
|
||
```
|
||
|
||
### HTTP routes
|
||
|
||
| Route | Description |
|
||
|---|---|
|
||
| `GET /` | Playground single-page client (embedded at compile time). |
|
||
| `GET /mivocero` | Mivocero speech-to-speech example client with chat box (embedded at compile time). |
|
||
| `GET /health` | Liveness probe (`ok`). |
|
||
| `GET /ws` | WebSocket upgrade → per-session audio loop. |
|
||
|
||
---
|
||
|
||
## Development
|
||
|
||
```bash
|
||
# Build
|
||
cargo build
|
||
|
||
# Run tests (tokenizer unit tests)
|
||
cargo test
|
||
|
||
# Run with debug logging
|
||
RUST_LOG=debug cargo run
|
||
|
||
# Release build (LTO + thin codegen)
|
||
cargo run --release
|
||
```
|
||
|
||
---
|
||
|
||
## Related Documentation
|
||
|
||
- [`UPDATES.md`](UPDATES.md) — full changelog (English, newest-first).
|
||
- [`src/tts/mod.rs`](src/tts/mod.rs) — trait + factory source of truth.
|