roomote_PRE/piperust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
piperust/README.md
T
valenti d6bc2437c9 client sdk
2026-10-21 17:56:04 +01:00

1790 lines
105 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Audio Agent Engine
A production-ready **real-time, full-duplex voice agent engine** written in Rust
(Tokio + Axum + ONNX Runtime). It streams microphone audio from the browser,
runs speech recognition with end-of-utterance detection, injects a low-latency
backchannel filler, and streams synthesized speech back to the browser — all
mixed on a strict 20 ms ticker.
```
Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
│
▼
TurnManager (orchestrator)
│
▼
LLM reasoning (OpenAI-compatible)
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Filler / backchannel TTS engine plugin Mixer (20 ms ticker)
(instant "uhm" + cues) (Fish Audio / Kokoro) │
└──────────────────────┴──────────────────────┘
│
▼
Mixed 44.1 kHz / 16-bit mono PCM ──► Browser
```
---
## Features
- **Full-duplex WebSocket audio** — 44.1 kHz / 16-bit mono PCM, 20 ms frames.
- **JavaScript client SDK** — a dependency-free, single-file ESM module (browser
+ Node) wrapping the WebSocket session and the REST surface, so the client can
live on another server. See [Client SDK (JavaScript)](#client-sdk-javascript).
- **Pluggable TTS engines** — switch engines (Fish Audio, Kokoro, …) via `.env`
without changing orchestration code. See [TTS Plugin System](#tts-plugin-system).
- **Latency filler** — a preloaded filler is played on end-of-utterance *only if
the reply is late*: it is scheduled `FILLER_DELAY_MS` (default 2500 ms) after
the turn starts and dropped if the response audio has already begun. Clips are
played round-robin from `FILLERS_DIR`; `FILLER_ENABLED=false` disables it.
- **Start filler** — a second latency mask on its own mixer channel: if the first
audio chunk of the reply has not reached the mixer within
`START_FILLER_GO_DELAY` (default 1800 ms) the agent plays a short
acknowledgment ("Un momento...") while the LLM/TTS are still working, and drops
it as soon as audio arrives in time. One phrase is picked **at random** from
`START_FILLER_TEXTS`, so the cue varies turn to turn. Independent of the delayed
latency filler above; `START_FILLER_ENABLED=false` disables it. See
[Fillers](#fillers--end-of-utterance-acknowledgments).
- **During-utterance backchannel** — short acknowledgments ("mm-hmm", "ajá")
are mixed in at a reduced gain *while the user is speaking*, so the agent
signals it is listening without taking the turn. Like the start filler the
bank is a phrase list (`BACKCHANNEL_TEXTS`) with Drama-3 tags, one clip per
phrase synthesized at startup. To sound natural the scheduler cues only on a
VAD **micro-pause**, randomizes the timing, gates each pause probabilistically,
avoids repeating the recent clips, drifts the volume slightly and follows a
**category** progression through the turn (`category::phrase` lines).
- **External audio injection** — a dedicated mixer channel can play a background
source (direct audio URL, Icecast/Shoutcast radio, HLS, or a YouTube link)
alongside the agent's voice, with an independent runtime volume and its own
decoder process (`ffmpeg` for URLs/radio, `yt-dlp` for YouTube). See
[External audio injection](#external-audio-injection).
- **Barge-in / interrupt** — cancels in-flight synthesis and flushes audio,
triggered client-side or server-side by the VAD.
- **Voice cloning** — reference audio + transcript (Fish Audio `references`).
- **Custom voice builder** — a dedicated `/customvoice` page (linked from the
Mivocero side menu) asks the user to pick a language, then either read a
predefined phrase for at least 10 s **or upload an audio file with its typed
transcript**; the take is transcoded to a WAV reference and installed as the
active voice at runtime. See
[Custom voice builder](#custom-voice-builder).
- **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to
`TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's
start-up latency behind the previous chunk's audio.
- **OpenAPI / Swagger UI** — the REST endpoints are documented by an `utoipa`
OpenAPI document rendered as a Swagger UI at `/docs` (raw JSON at
`/api-docs/openapi.json`). See
[API documentation](#api-documentation-swagger-ui).
- **File transcription endpoint** — `POST /api/transcribe` decodes an uploaded
audio file with `ffmpeg` and returns the Parakeet transcript of the whole take
(reusing the live session's STT model). See
[Transcription endpoint](#transcription-endpoint).
- **Double jitter buffer** — server pre-buffer + browser playback buffer absorb
uneven TTS chunk arrival, eliminating drop-outs.
- **Silero VAD (ONNX)** — server-side speech segmentation and fast barge-in.
- **Parakeet STT (ONNX)** with tokenizer decoding and EOU detection.
- **LLM reasoning** — the user transcript is sent to any OpenAI-compatible
chat-completions server (vLLM, SGLang, llama.cpp, Ollama, OpenAI) and the
reply is spoken by the TTS. Streaming (SSE) chunks the reply as it arrives
(first chunk after a few words, tunable) for the lowest time-to-first-audio;
chain-of-thought (`reasoning_content` / ` thinking`) is stripped from both the
audio and the chat box.
- **Tool calling** — the model can call external HTTP tools described in a JSON
file (`LLM_TOOLS_FILE`) before answering, or installed at runtime with
`POST /api/tools` (in-memory override; without it the file definitions are
used). Calls are parsed client-side from
either the structured `tool_calls` array or the model's native `<tool_call>`
format (a round that streams back empty is re-asked without streaming),
executed over HTTP and fed back to the model; the tool-call block is never
spoken. See [Tool calling](#tool-calling).
- **Per-session system prompt** — each WebSocket session gets a server-assigned
id (delivered to the client in a `{"type":"session","id":"…"}` message) and the
LLM system prompt can be read and overridden **for that session** — or
process-wide — at runtime through `GET`/`POST /api/prompt`, without a restart.
See [Runtime system prompt](#runtime-system-prompt-get--post-apiprompt).
- **Embedded playground** — a single-page HTML client served at `/`.
- **Mivocero speech-to-speech client** — an example agent client at `/mivocero`
with a microphone, a live chat box (user transcript + streaming assistant
reply) and a text composer, useful to drive a full STT → LLM → TTS turn.
- **Per-session audit log** — each WebSocket session can be recorded to a
structured JSONL file: the configuration snapshot, every inbound control
message and outbound event, the cue emissions, the language switches, the
barge-ins and an end-of-session summary. Best-effort and off the audio hot
path. See [Session audit log](#session-audit-log).
---
## Requirements
- Rust **1.75+** (async traits are provided via the `async-trait` crate).
- A TTS backend reachable over HTTP (e.g. Fish Audio / SGLang Drama 3).
- *(Optional)* A Parakeet TDT ONNX model directory (`models/parakeet/`) for
speech recognition. The server runs without STT (TTS still works via the
playground text box) if the model is absent.
- *(Optional)* An OpenAI-compatible chat-completions endpoint (e.g. a local
vLLM / SGLang server) for the LLM reasoning stage. Without it the pipeline is
a plain STT → TTS echo.
- *(Optional)* `ffmpeg` on `PATH` (or `MEDIA_FFMPEG_BIN`) to inject external
audio from a URL / radio stream, and `yt-dlp` (or `MEDIA_YTDLP_BIN`) to inject
a YouTube link. Both are only needed for the external audio channel.
---
## Quick Start
```bash
cd audio-agent-engine
# 1. Configure (optional — defaults target Fish Audio Drama 3)
cp .env.example .env
$EDITOR .env
# 2. Build and run
cargo run --release
# 3. Open the clients
# http://localhost:8787/ playground (TTS / audio test)
# http://localhost:8787/mivocero speech-to-speech agent with chat box
# http://localhost:8787/customvoice custom-voice builder (record / upload a reference)
# http://localhost:8787/docs Swagger UI (REST API documentation)
```
Environment variables can also be `export`ed directly; a `.env` file is loaded
automatically at startup (existing env vars take precedence).
### Server control script
[`server.sh`](server.sh) wraps start/stop/restart so you don't have to manage the
process by hand:
```bash
./server.sh start # start (builds automatically if the binary is missing)
./server.sh start --build # rebuild, then start
./server.sh stop # graceful stop (SIGKILL after 10 s)
./server.sh restart # stop + start
./server.sh status # process / port / scheme / HTTP status
./server.sh logs 100 # tail the last 100 lines (follows)
./server.sh cert # generate a self-signed HTTPS certificate
```
It writes the PID to `run/server.pid` and appends stdout/stderr to
`run/server.log`. Overridable env vars: `PROFILE` (`release`|`debug`, default
`release`), `RUST_LOG` (default `info`; `audio_agent_engine::tts=debug` is always
appended so the TTS streaming/latency diagnostics show up), `PORT` (default
`8787`).
### HTTPS (microphone access)
Browsers only expose the microphone (`getUserMedia`) in a **secure context** —
HTTPS or `localhost`. If you open the playground over plain HTTP on a
non-localhost origin the "Connect & Start Mic" button fails with `mic error`.
Set both variables to serve HTTPS/WSS instead of HTTP:
```dotenv
TLS_CERT=certs/cert.pem
TLS_KEY=certs/key.pem
```
`./server.sh cert` generates a self-signed pair (SAN covers `localhost`,
`127.0.0.1`, `::1` and this host's IPs; use `EXTRA_SAN=IP:1.2.3.4` to add
more). Then open `https://localhost:8787/` and accept the certificate warning —
the page becomes a secure context and the mic works. Comment both variables out
to fall back to plain HTTP. The server logs `listening (HTTPS)` or
`listening (HTTP)` at startup accordingly.
---
## Configuration
All configuration is via environment variables (or a `.env` file).
See [`.env.example`](.env.example) for a copy-paste template.
| Variable | Default | Description |
|---|---|---|
| `SERVER_PORT` | `8787` | TCP port the Axum server listens on. |
| `TTS_ENGINE` | `fish` | **TTS engine plugin** — `fish` or `kokoro`. See below. |
| `SGLANG_URL` | `http://10.8.0.3:8000` | Base URL of the Fish Audio / SGLang server (`fish` engine). |
| `TTS_MODEL` | `fishaudio/drama-3` | Model identifier sent to `/v1/audio/speech`. |
| `TTS_VOICE` | `default` | Voice identifier. |
| `TTS_REFERENCE_AUDIO` | `…/voice_profiles/solis.wav` | Voice-cloning reference WAV (path on the TTS server). |
| `TTS_REFERENCE_TEXT` | *(empty)* | Reference transcript. If it names an existing `.txt`, its contents are loaded; otherwise the raw string is used. Empty → auto-load `<audio-stem>.txt`. |
| `TTS_TEMPERATURE` | `0.45` | Sampling temperature sent to the SGLang speech endpoint (`fish` engine). Lower values flatten the delivery — the recitative "radio drama" register lives in the distribution tails. Empty omits the field (server default). |
| `TTS_TOP_P` | `0.75` | Nucleus sampling `top_p` sent to SGLang. Trims the tail of the acoustic token distribution, where exaggerated intonations live. Empty omits the field. |
| `TTS_TOP_K` | *(unset)* | Optional `top_k` sampling. Unset/empty omits the field (server default). |
| `TTS_STYLE_TAG` | `[conversational, casual]` | Neutral style tag prepended to every synthesis request that does not already start with a `[...]` tag. LLM replies carry no tag (the system prompt forbids stage directions), so they get this register; clip banks with their own tag are left untouched. Empty disables the prefix. **Quote the value in `.env`** (it contains a space). |
| `KOKORO_URL` | `http://localhost:8880` | Kokoro HTTP endpoint (`kokoro` engine). |
| `KOKORO_VOICE` | `af_heart` | Kokoro voice name. |
| `STT_MODEL_PATH` | `models/parakeet` | Directory of the Parakeet TDT ONNX pipeline (see below). |
| `STT_TOKENIZER_PATH` | *(empty)* | Optional external tokenizer. Parakeet decodes text internally; leave empty. |
| `VAD_ENABLED` | `true` | Master switch for the Silero VAD. When `false`, the STT worker falls back to its internal RMS segmentation. |
| `VAD_MODEL_PATH` | `models/silero_vad.onnx` | Silero VAD v5 ONNX model (16 kHz, 512-sample windows). |
| `VAD_THRESHOLD` | `0.5` | Speech probability above which a window counts as voice. |
| `VAD_NEG_THRESHOLD` | `0.35` | Probability below which a window counts as silence (hysteresis; clamped to `VAD_THRESHOLD`). |
| `VAD_MIN_SPEECH_MS` | `250` | Speech required to confirm an utterance start. |
| `VAD_MIN_SILENCE_MS` | `700` | Silence required to close an utterance (end-of-utterance window). |
| `VAD_SPEECH_PAD_MS` | `30` | Audio kept before the detected onset so the first phoneme is not clipped. |
| `VAD_BARGE_IN_MIN_SPEECH_MS` | `150` | Speech required to fire the fast barge-in onset (clamped to `VAD_MIN_SPEECH_MS`). |
| `VAD_MIN_UTTERANCE_MS` | `300` | Utterance gate: drop a segment shorter than this before transcription (`0` disables). |
| `VAD_MIN_AVG_PROB` | `0.25` | Utterance gate: drop a segment whose average speech probability is below this before transcription (`0.0` disables). The average is taken over the **voiced windows only** (probability ≥ `VAD_NEG_THRESHOLD`), so the trailing hangover silence does not dilute short words like "uno" / "due". Kept well below `VAD_THRESHOLD`. |
| `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. |
| `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. |
| `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. |
| `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. Clips live in a per-language, per-voice-profile subdirectory `<dir>/<lang>/<voice_profile>` (e.g. `assets/fillers/es/maricel/`). |
| `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. |
| `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. |
| `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. |
| `FILLER_GAIN_DUCK` | `0.15` | Fallback filler gain while TTS is already active. Normally a still-queued filler is faded out (80 ms) and cut when the reply's playback starts, so this only applies to a cue queued *after* playback has begun. Also applies to the start-filler channel. |
| `START_FILLER_ENABLED` | `true` | Master switch for the **start filler** — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. `false` => no cue is ever queued. |
| `START_FILLER_GO_DELAY` | `1800` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. If the cue does fire and the reply then starts, it is faded out (80 ms) and cut so it does not overlap the answer. |
| `START_FILLER_TEXTS` | `[calm] Un momento...` | Phrases spoken by the start filler, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and a **random** one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, `[sigh]`, …) are passed through to the TTS. |
| `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language, per-voice-profile subdirectory `<dir>/<lang>/<voice_profile>` (e.g. `assets/start_filler/es/maricel/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. |
| `ASSET_VOICE_PROFILE` | *(auto)* | Optional explicit cue-asset profile name. When unset, the engine derives it from the active voice setup (prefer `TTS_VOICE` when it is not `default`, otherwise fall back to the reference-audio file stem, then to `default`). |
| `TTS_PREBUFFER_MS` | `550` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first `185.8` ms, then `371.6` ms), any value in `(186, 557]` ms primes on the *same* second window, so `550` takes the whole cushion at no extra TTFA (~`0.73 s`). Below the first window (~`186` ms) playback starts on that window (TTFA ≈ server TTFB, ~`0.38 s`) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
| `TTS_PIPELINE_DEPTH` | `3` | How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. `1` restores strictly serial synthesis. |
| `BACKCHANNEL_ENABLED` | `true` | Master switch for during-utterance acknowledgments. |
| `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of the generated `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz; `bc01.raw`, `bc02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language, per-voice-profile subdirectory `<dir>/<lang>/<voice_profile>` (e.g. `assets/backchannel/es/maricel/`). |
| `BACKCHANNEL_TEXTS` | `ack::[calm] Mm-hmm...` … | Phrases spoken by the backchannel, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and one is played per cue. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, …) are passed through to the TTS. A line may carry an optional **`category::`** prefix (e.g. `agree::[agreeing] Claro...`) used to progress through the turn; a line without it belongs to the default `ack` category. |
| `BACKCHANNEL_GAIN` | `0.35` | Mix gain applied to the backchannel channel (0..1). |
| `BACKCHANNEL_MIN_SPEECH_MS` | `1500` | Confirmed speech required before the first cue may be considered. |
| `BACKCHANNEL_INTERVAL_MS` | `3000` | Base spacing between two cues of the same turn; randomized by ±`BACKCHANNEL_JITTER_MS`/2. |
| `BACKCHANNEL_JITTER_MS` | `1200` | Timing jitter: delays the first cue by `0..JITTER` after `BACKCHANNEL_MIN_SPEECH_MS` and spreads the spacing. |
| `BACKCHANNEL_PAUSE_MIN_MS` | `96` | Lower bound of the VAD micro-pause window (ms). The reaction **depth** inside a pause is drawn at random in `[MIN, MAX]`, so cues don't all land at exactly `MIN`. Below this the user is likely still mid-word. |
| `BACKCHANNEL_PAUSE_MAX_MS` | `260` | Upper bound of the micro-pause window, capped below `VAD_MIN_SILENCE_MS` so a cue never lands on the pause that closes the turn. |
| `BACKCHANNEL_PROB` | `0.65` | Probability (0..1) of cueing on a qualifying micro-pause — keeps the agent from acknowledging every pause. |
| `BACKCHANNEL_VOLUME_JITTER` | `0.2` | Random per-cue volume scaling (± fraction), so cues don't sit at a fixed level. |
| `BACKCHANNEL_MAX_PER_TURN` | `3` | Maximum backchannels emitted per user turn. |
| `LANG_DEFAULT` | `es` | Base language of the **filler, start-filler and backchannel** banks — its clips live in the `<dir>/<lang>/<voice_profile>` subdirectory (e.g. `assets/backchannel/es/maricel/`). Other languages are generated on demand into their own language/voice-profile subdirectories. See [Multilingual clip banks](#multilingual-clip-banks). |
| `LANG_DETECT_ENABLED` | `true` | Master switch for LLM-based language detection. Requires the LLM (`LLM_ENABLED=true`). |
| `LANG_MIN_CHARS` | `12` | Minimum transcript length (chars) before a sample is sent to the detector. |
| `LANG_REDETECT_EVERY` | `5` | The first user turn is always checked; afterwards the language is re-checked once every N turns. |
| `LANG_LLM_FALLBACK_ENABLED` | `true` | Phrase tables cover `es/en/it/fr/de/pt`; for any other language the phrase list is generated with the LLM. `false` keeps only the tables. |
| `EXTERNAL_AUDIO_GAIN` | `0.6` | Mix gain applied to the external audio channel (0..1) at session start. Overridable at runtime from the Mivocero volume slider. |
| `MEDIA_FFMPEG_BIN` | `ffmpeg` | `ffmpeg` binary used to decode a URL / radio / HLS stream into PCM. |
| `MEDIA_YTDLP_BIN` | `yt-dlp` | `yt-dlp` binary used to resolve a YouTube link to a direct stream URL. |
| `MEDIA_YTDLP_ARGS` | *(empty)* | Extra whitespace-separated args appended to the `yt-dlp` invocation (e.g. `--cookies /path/cookies.txt` when YouTube asks for bot verification, or a custom `--extractor-args`). |
| `LLM_ENABLED` | *(auto)* | Master switch for the LLM reasoning stage. Default: enabled when `LLM_BASE_URL` and `LLM_MODEL` are both set. When `false`, the pipeline is a plain STT → TTS echo. |
| `LLM_BASE_URL` | *(empty)* | Base URL of the OpenAI-compatible endpoint; the client posts to `{LLM_BASE_URL}/chat/completions`. |
| `LLM_API_KEY` | `EMPTY` | Bearer token sent in the `Authorization` header (local servers usually ignore it). |
| `LLM_MODEL` | *(empty)* | Model identifier sent in the request body. |
| `LLM_MAX_TOKENS` | `1024` | `max_tokens` for the completion. |
| `LLM_TEMPERATURE` | `0.7` | Sampling temperature. |
| `LLM_STREAM` | `true` | `true` → SSE token streaming, TTS starts on the first words; `false` → one blocking request, the whole reply is synthesized at once. |
| `LLM_STREAM_FIRST_CHUNK_CHARS` | `20` | Characters buffered before the **first** chunk is sent to the TTS, cut on a word boundary. Small = the agent starts speaking on the first few words. `0` disables the early flush. |
| `LLM_STREAM_CHUNK_CHARS` | `240` | Same threshold for every following chunk (larger chunks give better prosody and fewer TTS seams). `0` disables. |
| `LLM_HISTORY_MESSAGES` | `16` | Rolling conversation window (user+assistant messages) kept across turns. |
| `LLM_ENABLE_THINKING` | *(unset)* | `false` sends `chat_template_kwargs: {"enable_thinking": false}` — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers). |
| `LLM_STRIP_REASONING` | `true` | Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores `reasoning_content` deltas and removes inline think blocks. `false` passes the raw output through. |
| `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. |
| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). Can also be overridden at runtime per session via `GET`/`POST /api/prompt` ([Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)). |
| `LLM_TOOLS_ENABLED` | *(auto)* | Master switch for tool calling. Default: enabled when `LLM_TOOLS_FILE` is a non-empty path. |
| `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method` — any HTTP verb, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
| `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops. |
| `LLM_TOOL_TIMEOUT_SECS` | `15` | Timeout of a single tool HTTP request. |
| `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | Tool responses are truncated to this many characters before being handed back to the model. |
| `TLS_CERT` | *(empty)* | PEM certificate chain. Set together with `TLS_KEY` to serve HTTPS/WSS. |
| `TLS_KEY` | *(empty)* | PEM private key matching `TLS_CERT`. |
| `CORS_ALLOW_ORIGIN` | *(empty)* | Comma-separated browser origins allowed to call the REST surface cross-origin (single `*` = any). Empty adds no CORS layer (default behaviour). Needed when the [client SDK](#client-sdk-javascript) is served from another origin. Does not affect `/ws` (a WebSocket handshake is not a CORS request). |
| `AUTH_API_BASE` | `https://aquachile.azurianlab.com` | Base URL of the auth API proxied by `POST /api/auth/phone-login` ([Phone login](#phone-login-post-apiauthphone-login)). |
| `AUTH_PHONE_LOGIN_PATH` | `/api/v1/auth/phone-login` | Login path on `AUTH_API_BASE`. |
| `SESSION_AUDIT_ENABLED` | `true` | Write a structured JSONL audit file per WebSocket session (config snapshot, inbound control messages, every outbound event, cues, language switches, barge-ins, end-of-session summary). Best-effort — a write failure never affects the session. See [Session audit log](#session-audit-log). |
| `SESSION_AUDIT_DIR` | `logs/sessions` | Directory holding the per-session audit files (`<id>.jsonl`). Relative to the process working directory; git-ignored. |
---
## TTS Plugin System
The TTS layer is **engine-agnostic**. Every engine is a module under
[`src/tts/`](src/tts) that implements the [`TtsEngine`](src/tts/mod.rs) trait.
The active engine is selected at runtime by the `TTS_ENGINE` environment
variable — the orchestration layer (`TurnManager`, `AppState`) only ever sees an
`Arc<dyn TtsEngine>`.
### The `TtsEngine` trait
```rust
#[async_trait]
pub trait TtsEngine: Send + Sync {
/// Human-readable engine name (used for logging).
fn name(&self) -> &str;
/// Rate (Hz) at which this engine pushes samples into the mixer queue.
/// Must equal `config::OUTPUT_SAMPLE_RATE` (44_100).
fn sample_rate(&self) -> u32;
/// Synthesize `text` and stream PCM into `tts_queue`.
/// Must honor `cancel` for barge-in and return promptly.
async fn synthesize(
&self,
text: &str,
tts_queue: &SampleRingBuffer,
cancel: &CancellationToken,
) -> Result<()>;
/// Runtime voice-cloning reference update (default: no-op).
fn set_reference(&self, _audio_path: Option<String>, _text: Option<String>) {}
}
```
**Contract for implementors:**
1. Push **16-bit mono PCM at `OUTPUT_SAMPLE_RATE` (44 100 Hz)** into `tts_queue`.
If the engine's native rate differs, **resample internally** — the mixer runs
at a fixed rate and does not resample.
2. Honor the `CancellationToken`: on cancellation, stop pushing and return
`Ok(())` promptly (barge-in / interruption).
3. Override `set_reference()` only if the engine supports voice cloning.
### Engine selection (factory)
[`create_tts_engine()`](src/tts/mod.rs) maps `TTS_ENGINE` to a concrete plugin.
Values are case-insensitive:
| `TTS_ENGINE` | Plugin type | Notes |
|---|---|---|
| `fish` · `fishaudio` · `fish_audio` · `sglang` | [`FishAudioTts`](src/tts/fish_audio.rs) | Fish Audio / SGLang Drama 3, OpenAI-compatible streaming. Production-ready. |
| `kokoro` | [`KokoroTts`](src/tts/kokoro.rs) | Kokoro TTS. **Stub** — complete the HTTP call (see below). |
An unknown value panics at startup with a clear message.
### Bundled plugins
#### `fish` — Fish Audio / SGLang Drama 3
[`src/tts/fish_audio.rs`](src/tts/fish_audio.rs)
- `POST {SGLANG_URL}/v1/audio/speech` with `response_format: "pcm"`, `stream: true`.
- Reads the actual output rate from the response header `x-sample-rate`
(Drama 3 emits **44.1 kHz**, which matches the mixer — no resampling needed).
- Supports voice cloning via the `references` field.
- Honors barge-in cancellation.
##### Starting the Drama 3 backend (SGLang-Omni)
The `fish` engine talks to an **SGLang-Omni** server running Drama 3. The
vocoder emits audio in **windows** of codec frames (2048 samples = 46.4 ms each);
the window sizes are the single biggest lever on latency, so the server must be
started with them tuned:
```bash
sgl-omni serve \
--model-path fishaudio/drama-3 \
--host 0.0.0.0 --port 8000 \
--tts_engine.gpu_memory_fraction 0.45 \
--tts_engine.factory.max_running_requests 4 \
--tts_engine.factory.enable_async_decode true \
--vocoder.factory.stream_stride 4 \
--vocoder.factory.stream_followup_stride_ramp "[8,8,8]" \
--vocoder.factory.stream_followup_stride 8 \
--vocoder.factory.stream_overlap_tokens 16 \
--vocoder.factory.stream_crossfade_samples 512
```
In this deployment the launcher is `/root/drama3/start-drama3.sh` and the server
is kept alive by **pm2** as the process `start-drama3`:
```bash
pm2 restart start-drama3 --update-env # after editing the launcher
pm2 logs start-drama3 --lines 100 # CUDA-graph rebuild takes ~2 min at start
```
| Flag | Value | Meaning |
|---|---|---|
| `stream_stride` | `4` | **First** window, in codec frames (decoded with *no* overlap context). Sets the server TTFB floor: at `4` it is ~`0.38 s`, at `16` it is ~`1.2 s`, because the TTS engine must emit that many codec tokens before the vocoder can decode anything. |
| `stream_followup_stride` | `8` | Steady-state window: `8` frames = `0.372 s`, arriving ~`0.35 s` apart (the server runs at ~`0.95×` real time), so the client queue grows and never starves. Larger windows mean **fewer seams** per second, which is what removes the audible artifacts. Upstream default `64` ⇒ `2.97 s` bursts. |
| `stream_followup_stride_ramp` | `"[8,8,8]"` | Windows 2..4, as a JSON tuple. Keep it flat: a *rising* ramp (e.g. `[4,6,10]`, or the upstream `(10,12,32)`) makes later windows arrive after the small ones have drained, which gaps playback at the start of a reply. |
| `stream_overlap_tokens` | `16` | Decoder warm-up context. It is decoded and **discarded** on output, so it only costs compute. This is the seam-quality lever: with too little context the streaming decoder wobbles in timbre/pitch at each seam (an audible "detune"/falsetto on long replies). `16` with stride `8` is a `(16+8)/8 = 3×` real-time decode load and stays sustainable (~`0.95×` ratio); the same `16` with stride `4` would be 5× and is **not** sustainable. Upstream default `32`. |
| `stream_crossfade_samples` | `512` | ~`11.6 ms` blend at each window seam, smoothing the hand-off between windows. |
Measured on this deployment (with the voice-cloning reference), server-side:
| Config | TTFB | Real-time ratio |
|---|---|---|
| `stream_stride 4`, followup `8`, overlap `16`, crossfade `512` *(this deployment)* | **~0.38 s** | ~`0.94–0.98×` |
| `stream_stride 4`, followup `4`, overlap `8`, crossfade `256` | ~`0.38 s` | ~`0.95×` (but audible detune / micro-gaps on long replies) |
| `stream_stride 16`, overlap `16` | ~`1.2 s` | ~`1.0×` |
The engine's Time-to-First-Audio is essentially `TTFB + (TTS_PREBUFFER_MS − first
window)`, quantized to the window the pre-buffer lands on. With this config the
first window is `185.8` ms and the second `371.6` ms, so any `TTS_PREBUFFER_MS`
in `(186, 557]` ms primes on the **same** second window → TTFA ≈ `0.73 s`. The
deployment ships `550 ms`: the whole cushion, no extra latency. Dropping to a
value below the first window (~`186` ms) starts playback on the first window
(TTFA ≈ `0.38 s`) but leaves little margin, and under concurrent TTS requests the
per-request throughput dips below real time (measured ~`1.5×` vs ~`0.94×` alone),
so each chunk seam starves the queue and pauses playback — an audible micro-gap at
the start of each sentence.
Two gotchas when writing the command:
- The flags are **dotted keys into the pydantic config**, so they go *without* a
`stages.` prefix — `--vocoder.factory.stream_followup_stride`, not
`--stages.vocoder.factory.stream_followup_stride` (the latter is rejected).
- The ramp tuple must be passed as a **JSON string** — `"[8,8,8]"` — because
the config manager parses non-scalar paths with JSON.
##### De-emphasis — reducing Drama 3's recitative register
Drama 3 tends to over-articulate: an LLM reply can come back with a theatrical,
radio-drama cadence rather than a conversational tone. The engine applies two
configurable mitigations, both in [`src/tts/fish_audio.rs`](src/tts/fish_audio.rs):
1. **Sampling knobs** — `temperature` / `top_p` / `top_k` are sent in the
`/v1/audio/speech` payload (`TTS_TEMPERATURE` default `0.45`, `TTS_TOP_P`
default `0.75`). Exaggerated intonations live in the tails of the acoustic
token distribution, so trimming them flattens the delivery. Lowering
`temperature` toward `0.3` flattens it further; `0.8+` makes it unstable and
theatrical again. An empty value omits the field (server default). Note that
this server **rejects `repetition_penalty`** with anything other than `1.0`,
so that knob cannot be used.
2. **Neutral style tag** — `TTS_STYLE_TAG` (default `[conversational, casual]`)
is prepended to every request whose text does not already start with a `[...]`
tag. LLM replies carry no tag (the system prompt forbids stage directions), so
they get this register; the filler/backchannel banks carry their own tags and
are left untouched. Empty disables the prefix. Quote the value in `.env`, as
it contains a space.
> **`.env` parsing note.** The file is read with `dotenvy`, which **fails on
> unquoted values containing spaces** — and a single bad line makes it discard the
> *entire* file (the process then silently falls back to built-in defaults: no
> LLM, plain HTTP). Always quote values that contain spaces, e.g.
> `TTS_STYLE_TAG="[conversational, casual]"`.
The remaining lever — replacing the voice-cloning reference with a more
conversational one — is not applied here: the default `solis.wav` is a
"locutor profesional" reading, whose prosody is part of the recitative tone.
#### `kokoro` — Kokoro TTS *(stub)*
[`src/tts/kokoro.rs`](src/tts/kokoro.rs)
- Config via `KOKORO_URL` and `KOKORO_VOICE`.
- `synthesize()` currently returns an error — uncomment and adapt the provided
example to call your Kokoro HTTP server.
- Kokoro typically outputs **24 kHz**; the stub includes a
`resample_24k_to_output()` helper (linear interpolation) that converts to the
mixer rate before pushing. Replace it with a windowed-sinc/polyphase resampler
for higher fidelity if needed.
### Adding a new engine
1. Create `src/tts/my_engine.rs` and implement [`TtsEngine`](src/tts/mod.rs):
```rust
use crate::audio::SampleRingBuffer;
use crate::config::{Config, OUTPUT_SAMPLE_RATE};
use crate::tts::TtsEngine;
use anyhow::Result;
use async_trait::async_trait;
use tokio_util::sync::CancellationToken;
pub struct MyEngine { /* http client, url, voice, … */ }
impl MyEngine {
pub fn new(config: &Config) -> Self { /* read env / config */ }
}
#[async_trait]
impl TtsEngine for MyEngine {
fn name(&self) -> &str { "my_engine" }
fn sample_rate(&self) -> u32 { OUTPUT_SAMPLE_RATE }
async fn synthesize(
&self,
text: &str,
tts_queue: &SampleRingBuffer,
cancel: &CancellationToken,
) -> Result<()> {
// stream PCM, resample to OUTPUT_SAMPLE_RATE, push into tts_queue,
// and stop when `cancel.is_cancelled()`.
Ok(())
}
}
```
2. Register the module and the factory arm in [`src/tts/mod.rs`](src/tts/mod.rs):
```rust
pub mod my_engine;
// …
"my_engine" => Arc::new(MyEngine::new(config)),
```
3. Select it at runtime:
```bash
TTS_ENGINE=my_engine cargo run --release
```
---
## STT — Parakeet TDT (pure Rust)
Speech recognition is implemented entirely in Rust via ONNX Runtime — no Python
or NeMo dependency. [`src/stt/parakeet.rs`](src/stt/parakeet.rs) runs the full
NVIDIA **Parakeet TDT 0.6B v3** pipeline:
```text
16 kHz mono f32
└─ nemo128.onnx → mel features [1, 128, T]
└─ encoder-model*.onnx → encoder out [1, 1024, T']
└─ decoder_joint-model.onnx (greedy TDT) → token IDs
└─ vocab.txt → text
```
The decoder is a **transducer (TDT)**: at each encoder frame it predicts a token
*and* a duration step. A positive step advances several frames at once; the
blank token (`<blk>`, id 8192) means "nothing emitted here". This matches the
reference implementation in [`istupakov/onnx-asr`](https://github.com/istupakov/onnx-asr).
### Model directory
`STT_MODEL_PATH` must point to a directory containing:
| File | Purpose |
|---|---|
| `nemo128.onnx` | Log-mel feature extractor. |
| `encoder-model.int8.onnx` *(preferred)* or `encoder-model.onnx` | Conformer encoder. The int8 encoder (~650 MB, self-contained) is used automatically when present; otherwise the fp32 encoder (~42 MB + a `.onnx.data` sidecar) is loaded. |
| `decoder_joint-model.onnx` | TDT prediction network + joint. |
| `vocab.txt` | `<piece> <id>` token table (8193 entries). |
| `config.json` | Optional; supplies `max_tokens_per_step` (default 10). |
Model files can be fetched from
[`istupakov/parakeet-tdt-0.6b-v3-onnx`](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx)
into `models/parakeet/`.
### Runtime behaviour
- The worker ([`SttWorker`](src/stt/parakeet.rs)) consumes 16 kHz frames. By
default the session's **Silero VAD** segments the audio and hands whole
utterances to `push_utterance`; without VAD it falls back to RMS-based
segmentation and triggers inference (`spawn_blocking`) after a
`EOU_SILENCE_MS` (300 ms) silence window.
- Text is decoded **inside** the engine (SentencePiece `▁` → space, spacing
normalized), so no external tokenizer is required — `transcript.text` is used
directly by the [`TurnManager`](src/orchestrator/turn_manager.rs).
### Testing
An opt-in end-to-end test decodes a 16 kHz mono WAV through the real model:
```bash
PARAKEET_TEST_WAV=/path/to/16k_mono.wav \
cargo test --release -- --ignored test_infer_real_audio --nocapture
```
---
## VAD — Silero V5 (ONNX)
Voice Activity Detection runs entirely on the server, in Rust, on the same
16 kHz mono stream that feeds the STT engine. It is implemented in
[`src/vad/`](src/vad) and detects when the user starts and stops speaking.
### Model
`VAD_MODEL_PATH` points to `silero_vad.onnx` (Silero VAD **v5**). The model
consumes fixed **32 ms** windows (512 samples @ 16 kHz) preceded by a 64-sample
context carried over from the previous window, plus a recurrent state tensor
`[2, 1, 128]` and a scalar sample-rate input. Each window yields a speech
probability. The graph's tensor shapes and the sample-rate input rank are
introspected at load time, so the wrapper adapts to the exported model rather
than hard-coding shapes.
### Events and state machine
[`VadDetector`](src/vad/silero.rs) turns per-window probabilities into three
events:
| Event | Trigger | Used for |
|---|---|---|
| `SpeechOnset` | `VAD_BARGE_IN_MIN_SPEECH_MS` (150 ms) of speech | Fast **barge-in**: interrupt the agent as soon as the user starts talking. |
| `SpeechStart` | `VAD_MIN_SPEECH_MS` (250 ms) of speech | Confirms the utterance start. |
| `SpeechEnd` | `VAD_MIN_SILENCE_MS` (700 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. |
Onset/offset use hysteresis: a window counts as voice above `VAD_THRESHOLD`
and as silence below `VAD_NEG_THRESHOLD`, which prevents flapping around a
single threshold. A **pre-roll ring**
(`VAD_SPEECH_PAD_MS + VAD_MIN_SPEECH_MS + one window`) is kept ahead of the
detected onset so the first phonemes are never clipped.
### Barge-in
When `VAD_BARGE_IN=true` and the agent has **audible** output
([`TurnManager::is_audible`](src/orchestrator/turn_manager.rs), backed by
[`AudioMixer::is_audible`](src/audio/mixer.rs)), a `SpeechOnset` calls
`TurnManager::interrupt()` — cancelling the in-flight TTS via its
`CancellationToken` and flushing the mixer queues.
The gate is on *audible* output, not on "speaking" in general: `is_audible` is
`false` while the reply is still being synthesised or is sitting in the TTS
pre-buffer, i.e. before anything has reached the user's ears. This stops a VAD
transient (a breath, a click, or the tail of the user's own utterance) from
cancelling a reply the user has not heard any of yet. Such a reply is not left
orphaned: the next turn's `on_eou` already cancels the previous in-flight
synthesis and restarts the stream. `TurnManager::is_speaking`
([`AudioMixer::is_active`](src/audio/mixer.rs)) is still used for the
backchannel gate, where cueing on top of *any* in-flight TTS would be an
interruption rather than an acknowledgment.
### Wiring
The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one
`VadDetector` per WebSocket session (the model itself is shared) and:
- with `VAD_SEGMENT_STT=true` (default): forwards `SpeechEnd` utterances to
`SttWorker::push_utterance`, and does **not** forward raw frames (avoids
double segmentation). Before forwarding, each segment passes the **utterance
gate** (`VAD_MIN_UTTERANCE_MS` / `VAD_MIN_AVG_PROB`): segments below either
threshold are dropped without transcription, which filters the VAD
false-triggers (breaths, clicks, acoustic echo, noise) that the STT would
otherwise turn into hallucinated text. The probability average is computed
over the **voiced windows only** (probability ≥ `VAD_NEG_THRESHOLD`), so the
`VAD_MIN_SILENCE_MS` hangover that closes the segment does not drag it down —
a short word like "uno" would otherwise average far below any usable gate. A
transcript that still decodes to no tokens is skipped too, so empty turns are
never sent to the LLM.
- with `VAD_SEGMENT_STT=false`: falls back to the STT worker's RMS
segmentation, using the VAD only for barge-in;
- if `VAD_ENABLED=false` or the model fails to load: legacy behaviour (RMS
segmentation only).
All VAD logs live under the `audio_agent_engine::vad` target;
`./server.sh` appends `audio_agent_engine::vad=debug` to `RUST_LOG`, and
`VAD_DEBUG=true` additionally logs every window's probability.
---
## Startup clip bootstrap
The filler, start-filler and backchannel banks are plain `*.raw` PCM files
(16-bit LE mono PCM at 44.1 kHz). At startup the clip banks are bootstrapped with
the configured TTS engine and the files are written to disk (the directory is
created if needed), so a fresh checkout works without a manual generation step.
Every bank lives in a per-language subdirectory `<dir>/<lang>` (see
[Multilingual clip banks](#multilingual-clip-banks)); the startup bootstrap
targets the `LANG_DEFAULT` subdirectory (e.g. `assets/fillers/es/`).
- Filler uses [`assets::ensure_clips`](src/assets.rs): each expected clip is
checked and **synthesized only if missing**, so an existing bank is left
untouched. Defaults live in [`assets::FILLER_CLIPS`](src/assets.rs) (7 clips:
`uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya`). The generated voice
matches the configured TTS voice-cloning reference.
- The **start filler** and the **backchannel** use the generic
[`assets::ensure_text_bank_clips`](src/assets.rs) (through
[`ensure_start_filler_clips`](src/assets.rs) / [`ensure_backchannel_clips`](src/assets.rs)):
both are driven by their phrase list (`START_FILLER_TEXTS` /
`BACKCHANNEL_TEXTS`) instead of a fixed clip set, so editing the list from
`.env` is enough to change what is spoken. The list the current clips were made
from is recorded in `<dir>/texts.txt`; when it is unchanged and every clip is
still on disk the synthesis is **skipped** (startup stays at ~2 s). Editing,
adding or dropping a phrase — or deleting a clip — triggers a **full refresh**
at the next restart, and clips left over from a previous list are removed.
Clips are named by position (`sf01`, `sf02`, … for the start filler; `bc01`,
`bc02`, … for the backchannel) and generated a few at a time
(`TEXT_BANK_CONCURRENCY`); a failed synthesis keeps the previous clip of that
position, and the index is only written once every clip succeeded, so a partial
failure is retried next time.
- Backchannel clips are only generated when `BACKCHANNEL_ENABLED=true`; its
default phrase list lives in
[`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs).
- Generation is best-effort: if the TTS backend is unreachable each clip logs a
warning and the server still starts (that bank simply stays silent).
```text
INFO audio_agent_engine::assets: generating clips at startup label="filler" dir=assets/fillers/es clips=7 engine="fish-audio"
INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/es/uhm.raw
...
INFO audio_agent_engine::assets: refreshing backchannel clips at startup label="backchannel" dir=assets/backchannel/es clips=8 engine="fish-audio"
INFO audio_agent_engine::assets: generated backchannel clip label="backchannel" name="bc01" samples=28672 path=assets/backchannel/es/bc01.raw
...
INFO audio_agent_engine::assets: removed stale backchannel clip label="backchannel" path=assets/backchannel/es/mmhmm.raw
INFO audio_agent_engine::assets: refreshing start-filler clips at startup label="start-filler" dir=assets/start_filler/es clips=22 engine="fish-audio"
INFO audio_agent_engine::assets: generated start-filler clip label="start-filler" name="sf01" samples=36864 path=assets/start_filler/es/sf01.raw
...
INFO audio_agent_engine::assets: removed stale start-filler clip label="start-filler" path=assets/start_filler/es/start_filler.raw
INFO audio_agent_engine: clip banks bootstrapped generated=30
```
On a later startup with unchanged lists:
```text
INFO audio_agent_engine::assets: backchannel clips up to date — skipping synthesis label="backchannel" dir=assets/backchannel/es clips=8
INFO audio_agent_engine::assets: start-filler clips up to date — skipping synthesis label="start-filler" dir=assets/start_filler/es clips=22
```
---
## Fillers — end-of-utterance acknowledgments
A **filler** is a short acknowledgment ("uhm", "eh") used as a **latency mask**
while the reply is being generated. Because it exists to cover a *gap*, it is
not played unconditionally: when the turn is dispatched
([`TurnManager::schedule_filler`](src/orchestrator/turn_manager.rs)) a timer of
`FILLER_DELAY_MS` (default `2500`) is started, and the clip is queued **only if
the response audio has not started playing by then** — talking over the first
words of the answer is worse than a moment of silence. The check uses the
mixer's `tts_playback_started` flag, which is set when the TTS pre-buffer is
reached and reset at every `mark_stream_start` / `flush`.
- **At most one filler per turn**, and a barge-in (`interrupt`, VAD onset, new
turn, disconnect) cancels the pending timer.
- `FILLER_ENABLED=false` disables the filler entirely (no clip is ever queued).
- Raise `FILLER_DELAY_MS` for slower LLM/TTS setups, lower it if the filler
rarely fires.
Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`). If TTS playback
starts mid-clip, the cue is **faded out** over `FILLER_FADEOUT_FRAMES`
(4 frames = 80 ms) and then cut — it is a latency mask, so it must hand off
cleanly to the reply instead of playing underneath it. (As a fallback, a cue
queued *after* playback has already started is ducked to `FILLER_GAIN_DUCK`.)
`FILLERS_DIR` (default `assets/fillers`) holds `*.raw` clips — **16-bit LE mono
PCM at 44.1 kHz** — selected **round-robin** (`TurnManager::next_filler` →
[`FillerBank::nth`](src/orchestrator/turn_manager.rs)), so consecutive turns use
different fillers. With a single clip the behaviour is unchanged. Missing clips
are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)).
### Start filler (turn-start acknowledgment when the reply is late)
A second latency mask, on its own mixer channel. It works exactly like the filler
above, but with its own — shorter — delay
([`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)):
`START_FILLER_GO_DELAY` (default **1800 ms**) after the turn is dispatched, if the
first audio chunk of the reply has **not** reached the mixer yet
(`AudioMixer::tts_audio_arrived` is still `false`) the cue is queued; if audio
arrived within that window, the cue is dropped instead of talking over the
beginning of the answer.
The gate is latched on chunk *arrival* (at the moment anything is pushed on the
TTS queue, checked on the 20 ms ticker), not on playback start: unlike
`tts_playback_started`, which only flips once the `TTS_PREBUFFER_MS` jitter buffer
is full, it does not wait out the pre-buffer. With the reference deployment the
first chunk lands ~`0.7`–`1.0 s` after the dispatch while playback starts
~`1.9 s` later, so gating on playback made the cue fire on every turn; gating on
arrival lets a reply that produces audio within `START_FILLER_GO_DELAY` suppress
it. A pending cue is also dropped on barge-in / new turn via the turn's
`CancellationToken`. This means a fast reply is never preceded by an
acknowledgment — only a late one is masked.
It lives on its own mixer channel (`start_filler_queue`), so it plays
independently of the round-robin latency filler above, but it shares the filler
gain: when the reply's playback starts, the still-queued cue is faded out
(80 ms) and cut rather than left running under the answer. Disabled by
`START_FILLER_ENABLED=false`.
Its clips live in `START_FILLER_DIR` (default `assets/start_filler`) and are
**not** part of the round-robin `FILLERS_DIR` bank. One clip is synthesized per
phrase of `START_FILLER_TEXTS` (default `"[calm] Un momento..."`) with the
configured voice, and [`FillerBank::random`](src/orchestrator/turn_manager.rs)
picks one **uniformly at random**, so the acknowledgment varies instead of
repeating the same sentence. Editing the phrase list re-synthesizes the bank at
the next restart (see
[Startup clip bootstrap](#startup-clip-bootstrap)); a failed synthesis leaves the
previous clip in place.
```text
START_FILLER_TEXTS="[calm] Un momento...
[thoughtful] A ver, veamos...
[sigh] Vale, ya voy..."
```
---
## Backchannel — during-utterance acknowledgments
A **backchannel** is a short acknowledgment ("mm-hmm", "ajá", "claro") played
*while the user is still speaking*, at a low gain, to signal that the agent is
listening — without taking the turn. It is distinct from the **filler** and the
**start filler** injected around end-of-utterance by the
[`TurnManager`](src/orchestrator/turn_manager.rs) (delayed mask from
`FILLERS_DIR` played round-robin, and a shorter-delay cue from `START_FILLER_DIR`;
see [Fillers](#fillers--end-of-utterance-acknowledgments)).
It is implemented in [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs).
### Clips
`BACKCHANNEL_DIR` (default `assets/backchannel`) holds the generated `*.raw`
clips — **16-bit LE mono PCM at 44.1 kHz** — one per phrase of
`BACKCHANNEL_TEXTS`, named by position (`bc01`, `bc02`, …). Each phrase may carry
an optional `category::` prefix; clips keep the list order, so the i-th clip
inherits the i-th phrase's category. The default list
([`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs), Spanish with
Drama-3 tags) is synthesized with the same cloned voice as the TTS. The clips are
loaded once at startup into a
[`FillerBank`](src/orchestrator/turn_manager.rs) and shared across sessions.
Missing clips are synthesized at startup
([Startup clip bootstrap](#startup-clip-bootstrap)).
### Trigger policy
[`BackchannelScheduler`](src/orchestrator/backchannel.rs) is fed by the VAD:
| Signal | Action |
|---|---|
| `VadEvent::SpeechStart` | begin a turn (`on_speech_start`, randomizes the first-cue instant) |
| `VadEvent::SpeechEnd` | close the turn, reset counters (`on_speech_end`) |
| every frame while speaking | `maybe_emit(pause_ms)` attempts a cue |
`maybe_emit()` emits a clip **only when all** of the following hold:
- `BACKCHANNEL_ENABLED=true` and the clip bank is non-empty;
- the agent is **not** speaking (`!TurnManager::is_speaking()`) — cuing over the
agent's own TTS would be an interruption, not a backchannel (checked by the
caller);
- the user is in a **micro-pause**: `VadDetector::silence_ms()` reached the
per-pause reaction depth, a random value drawn in `[BACKCHANNEL_PAUSE_MIN_MS,
BACKCHANNEL_PAUSE_MAX_MS]`, so the cue never lands mid-word *and* doesn't
always react at the same depth. A zero-length pause (voice ongoing) re-arms
the gate;
- the first cue is due (a per-turn randomized instant,
`BACKCHANNEL_MIN_SPEECH_MS + 0..BACKCHANNEL_JITTER_MS` after the speech start);
- a randomized `BACKCHANNEL_INTERVAL_MS ± JITTER/2` gap elapsed since the
previous cue;
- fewer than `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn;
- the **probabilistic gate** (`BACKCHANNEL_PROB`) does not skip this pause — so
the agent acknowledges some pauses but not all.
### Naturalness
Six mechanisms keep the cues from sounding like a metronome:
| Mechanism | How |
|---|---|
| Cue on micro-pauses | gated on the VAD `silence_ms()` (a `PAUSE_MIN..PAUSE_MAX` window, 32 ms steps, capped below `VAD_MIN_SILENCE_MS`), not on a timer |
| Randomized reaction depth | per pause, the depth to react at is drawn at random in `[PAUSE_MIN, PAUSE_MAX]`, so the cue doesn't always land at exactly `PAUSE_MIN` |
| Randomized timing | the first cue and every following gap are jittered by `BACKCHANNEL_JITTER_MS` |
| Probabilistic gate | `BACKCHANNEL_PROB` skips a share of the qualifying pauses |
| Anti-repetition | the last two clip names are excluded from the pick, across turns |
| Volume drift | each cue is scaled by a random ±`BACKCHANNEL_VOLUME_JITTER` factor |
The phrase list is further grouped in **categories**: the scheduler prefers a
category other than the previous cue's, then a category not yet used this turn,
so a turn tends to start with an acknowledgment (`ack`) and drift toward
agreement (`agree`) instead of repeating the same kind of nod.
### Mixing
The cue is pushed into the mixer's dedicated `backchannel_queue` and summed at a
fixed low gain (`BACKCHANNEL_GAIN`, default `0.35`):
```text
mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN)
```
The queue is included in `AudioMixer::flush()`, so a barge-in clears any pending
cue along with the TTS and filler audio. At session start the effective policy is
logged under `audio_agent_engine::orchestrator::backchannel`
(`backchannel scheduler ready …`, with the parsed categories), and every emitted
cue is logged under `audio_agent_engine::ws::session`
(`backchannel emitted during user speech`, with the clip name and the pause
length).
---
## Multilingual clip banks
The filler, start-filler and backchannel banks are pre-synthesized in a single
language — the base language `LANG_DEFAULT` (default `es`). When the user is
detected speaking a different language, the engine generates that language's
banks on demand and switches the session to them, **without touching** the clips
of any other language.
### Layout
Every language — **including the base `LANG_DEFAULT`** — keeps a separate bank
for each cue-asset voice profile under `FILLERS_DIR` / `START_FILLER_DIR` /
`BACKCHANNEL_DIR`:
```text
assets/backchannel/es/maricel/ <- base language (LANG_DEFAULT), profile maricel
assets/fillers/es/maricel/
assets/start_filler/es/maricel/
assets/backchannel/es/solis/ <- same language, different voice profile
assets/backchannel/en/maricel/ <- English, generated on demand
assets/fillers/en/maricel/
assets/start_filler/en/maricel/
```
On startup, if a legacy single-voice bank still exists directly under
`<dir>/<lang>/` (for example `assets/backchannel/es/bc01.raw`), the engine
copies those flat files into the active `<dir>/<lang>/<voice_profile>/`
directory when the target files are missing. The legacy files are left in
place, so the migration is non-destructive.
Because the refresh pass (`remove_stale_clips`) only prunes `*.raw` files inside
the directory it is given, scoping each language — base included — to its own
subdirectory is what guarantees a regeneration never deletes another language's
clips. No language is special-cased to the top level.
### Phrase source
Phrases come from built-in tables (Spanish, English, Italian, French, German and
Portuguese) in [`src/lang.rs`](src/lang.rs) — one table for the backchannel, one
for the end-of-turn filler and one for the start filler; for any other language
they are generated with the LLM (`LANG_LLM_FALLBACK_ENABLED=true`). Each
language's list is then synthesized through the same
[`ensure_text_bank_clips`](src/assets.rs) pipeline used at startup, so an
already-generated language is only reloaded (its `texts.txt` index matches) and
never re-synthesized.
### Detection & switching
- A dedicated, short LLM call returns only the ISO 639-1 code
([`lang::detect_language`](src/lang.rs)); it runs on the first user turn and
then once every `LANG_REDETECT_EVERY` turns (default `5`), so a
mid-conversation switch is picked up without paying for a detection call on
every turn.
- Detection only runs when `LANG_DETECT_ENABLED=true`, the LLM is present, at
least one language-aware cue (backchannel / filler / start filler) is enabled
and the transcript is at least `LANG_MIN_CHARS` long (shorter samples are too
noisy to classify).
- If the detected code equals the base language the session falls back to the
startup banks; otherwise the language's clips are ensured/loaded into a new
bank and the session switches to it.
- The active language is shared between the consumers through an
`Arc<LanguageState>` ([`src/lang.rs`](src/lang.rs)): the end-of-turn and start
fillers ([`TurnManager`](src/orchestrator/turn_manager.rs)) read the active
bank per turn, while the backchannel scheduler is re-pointed via
[`BackchannelScheduler::retarget`](src/orchestrator/backchannel.rs) whenever
the language generation counter changes. Detection runs in the background, so
it never blocks the reply — it takes effect from the following turn.
### Configuration
| Variable | Default | Purpose |
|---|---|---|
| `LANG_DEFAULT` | `es` | Language of the startup filler / start-filler / backchannel banks (each in its own `<dir>/<lang>` subdirectory) |
| `LANG_DETECT_ENABLED` | `true` | Master switch for LLM language detection |
| `LANG_MIN_CHARS` | `12` | Minimum transcript length before detecting |
| `LANG_REDETECT_EVERY` | `5` | Re-check the language every N user turns |
| `LANG_LLM_FALLBACK_ENABLED` | `true` | Generate phrases via LLM for languages absent from the tables |
---
## LLM reasoning
When `LLM_ENABLED=true` (the default once `LLM_BASE_URL` + `LLM_MODEL` are set),
[`TurnManager`](src/orchestrator/turn_manager.rs) inserts a reasoning stage
between STT and TTS:
```
microphone → VAD → STT ─┐
├─→ LLM (chat completions) → chunker → TTS → mixer → browser
chat box ──────────────┘
```
- **Client** — [`src/llm.rs`](src/llm.rs). A minimal OpenAI-compatible client
(`POST {LLM_BASE_URL}/chat/completions`) with two modes, selectable at runtime
via `LLM_STREAM`:
- `chat_stream` — SSE token streaming (`stream: true`); deltas are handled as
they arrive.
- `chat` — a single blocking request (`stream: false`).
- **Chain-of-thought handling** — reasoning models (Qwen3, …) may leak their
plan in three different ways. Each is handled by an independent, `.env`-
switchable mechanism, because a single one does not cover every deployment:
1. `LLM_ENABLE_THINKING=false` — server-side switch: the request carries
`chat_template_kwargs: {"enable_thinking": false}` (vLLM/SGLang), so the
model answers directly and generates no reasoning at all. Left unset, the
field is not sent, keeping the body valid for strict OpenAI-compatible
servers.
2. `LLM_STRIP_REASONING=true` (default) — client-side filter: separate
`reasoning_content` deltas (vLLM with a reasoning parser) are dropped and
inline think-tag blocks are removed by an incremental stripper that is
stable across delta boundaries.
3. `LLM_DIRECT_ANSWERS=true` (default) — prompt-level guard: appends
[`DIRECT_ANSWER_DIRECTIVE`](src/config.rs) to the system prompt, telling
the model not to narrate its reasoning. This also covers servers with no
reasoning parser at all, where the chain-of-thought is written straight
into `content` as ordinary prose.
- **Streaming chunking** — the assistant text is split on `.`, `!`, `?`, `;` and
newlines as it streams, and each chunk is dispatched to the TTS as soon as it
is ready. A word-boundary flush bounds the latency for punctuation-free runs:
the **first** chunk is cut once `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`)
characters are buffered — so the agent starts speaking on the first few words
instead of waiting for a whole sentence — and every following chunk at
`LLM_STREAM_CHUNK_CHARS` (default `240`). Set either to `0` to disable that
flush (chunks then end on sentence punctuation only). Larger follow-up chunks
mean fewer TTS requests, hence fewer chunk seams to starve the queue.
- **Pipelined synthesis** — chunks are *not* synthesized one after another. Each
chunk is requested as soon as the LLM produces it (up to `TTS_PIPELINE_DEPTH`,
default `3`, in flight at once) and streams into its own staging buffer; a
relay forwards those buffers to the mixer **in submission order**. This
overlaps each request's start-up latency with the previous chunk's playback,
so raising the pre-buffer is no longer needed to avoid inter-chunk gaps.
- **History** — a rolling window of the last `LLM_HISTORY_MESSAGES` user/assistant
messages is kept per connection and prepended with the effective system prompt
(the session override if any, else `LLM_SYSTEM_PROMPT`, else the built-in
default — see [Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)).
- **Barge-in** — an interrupt (client JSON, VAD onset, or disconnect) cancels
both the in-flight HTTP request ([`CancellationToken`] in the `reqwest`
`tokio::select!`, including while waiting for the next SSE chunk) and the TTS
tasks, then flushes the mixer. Chat-box turns run in their own task so the
session's receive loop stays responsive and keeps accepting `interrupt`
messages while a reply is being generated.
- **Failure handling** — request errors are reported to the client as
`{"type":"error","text":…}` and the turn ends cleanly.
### Runtime system prompt (`GET` / `POST /api/prompt`)
The system prompt is normally the process-wide default (the built-in one, or
`LLM_SYSTEM_PROMPT`). Because it is injected identically into every session, two
clients cannot run with different instructions. `GET`/`POST /api/prompt` adds a
**runtime, in-memory** override — per session or process-wide.
Each WebSocket session is assigned a server id (e.g. `s-1729170000000-0001`, the
same id used for its audit log) and the id is pushed to the client right after
connect, before the `cues` message:
```json
{"type":"session","id":"s-1729170000000-0001"}
```
That id is the key of the REST surface:
```bash
# Effective prompt for a session (+ where it comes from)
curl 'localhost:8787/api/prompt?session_id=s-1729170000000-0001'
# Override it for that session
curl -X POST localhost:8787/api/prompt -H 'Content-Type: application/json' \
-d '{"session_id":"s-1729170000000-0001","prompt":"You are a terse assistant. One sentence max."}'
# Clear the override (falls back to the configured default)
curl -X POST localhost:8787/api/prompt -H 'Content-Type: application/json' \
-d '{"session_id":"s-1729170000000-0001","prompt":null}'
```
- **Scope** — with a `session_id` the override applies to that session only; omit
it (`null`) to set a process-wide fallback used by every session without its
own override.
- **Precedence** — session override → process-wide override → `LLM_SYSTEM_PROMPT`
→ built-in default. The response's `source` field (`session` / `global` /
`env` / `default` / `none`) reports which one won, and `has_override` whether a
runtime override is in effect.
- **In-memory only** — a restart restores the configured default, and a session
override is dropped when that session disconnects. An unknown/disconnected
`session_id` returns `404`.
- `prompt: null` (or `""`) clears the override.
- The response also carries `direct_answers_applied` (whether the
`LLM_DIRECT_ANSWERS` directive is currently appended).
- See [`src/prompt.rs`](src/prompt.rs) and [`src/prompt_api.rs`](src/prompt_api.rs).
---
## Tool calling
With `LLM_TOOLS_FILE` pointing at a JSON file, the model can call external HTTP
services before answering. Tools are plain descriptions of an HTTP request, so
any service (weather API, webhook, home automation, internal REST endpoint) can
be exposed without touching the code:
```json
[
{
"name": "get_weather",
"description": "Current weather and temperature for a city.",
"parameters": {
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
},
"url": "https://wttr.in/{city}?format=3",
"method": "GET",
"headers": { "X-Api-Key": "…" }
}
]
```
- `url` may contain `{arg}` placeholders substituted from the call arguments.
- `method` takes **any** HTTP verb, case-insensitive (`GET`, `POST`, `PUT`,
`PATCH`, `DELETE`, `HEAD`, `OPTIONS`, `TRACE`, custom verbs); default `POST`.
The arguments left after the URL placeholders are sent as **query parameters**
for `GET`/`HEAD`/`OPTIONS`/`TRACE`/`DELETE` and as a **JSON body** for every
other verb — so a full REST surface is covered: list/look up (`GET`), add
(`POST`), update (`PUT`/`PATCH`), remove (`DELETE`).
- `headers` are optional and sent verbatim (API keys, etc.).
See [`tools.example.json`](tools.example.json) for a ready-to-use file.
### Runtime definitions (`GET` / `POST /api/tools`)
The file is read once at startup, so `POST /api/tools` installs a different set
without a restart — the body is **the same JSON array** the file holds:
```bash
curl -X POST localhost:8787/api/tools -H 'Content-Type: application/json' \
-d '[{"name":"get_weather","description":"…","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]},"url":"https://wttr.in/{city}?format=3","method":"GET"}]'
```
- The override lives **in memory only**: it replaces the `LLM_TOOLS_FILE`
definitions from the next turn on, and a restart brings the file ones back.
When the endpoint is never called, the file definitions stay in use.
- `GET /api/tools` reports the active definitions and their origin
(`source`: `file`, `request` or `none`).
- `400` on an empty/invalid body or on a JSON array with no tool; `503` when the
LLM is disabled. See [`src/tools_api.rs`](src/tools_api.rs).
See [TOOL_CALLING.md](TOOL_CALLING.md) for the full guide.
### Client-side parsing (and the streaming fallback)
A deployment may or may not run a server-side tool parser. When it does, the
flag has to match the model's native format (`--enable-auto-tool-choice
--tool-call-parser …`). A mismatch is silent and costly: an `hermes` parser
(which expects `<tool_call>{"name":…,"arguments":{…}}</tool_call>`) against a
Qwen3 model that emits the XML form makes the parser throw, so the call is
dropped. In **streaming** the round then ends with an empty reply
(`finish_reason: "stop"`, no content), while the very same request **without**
`stream` returns the call intact.
The engine therefore never relies on the server parser:
- Every request advertises `tools` + `tool_choice: "auto"`.
- [`src/tools.rs`](src/tools.rs) parses **both** shapes — the standard
structured `message.tool_calls` array **and** the model's native
`<tool_call>` text:
```text
<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>
```
- A streaming round that comes back **empty** is re-asked once **without
streaming** ([`LlmClient::chat_reply`](src/llm.rs)), where the dropped call
reappears and is parsed from either shape.
Fixing the server (`--tool-call-parser qwen3_coder` handles the Qwen3 XML
format) removes the extra request — the fallback then simply never fires.
### Turn loop
Each turn runs a bounded loop (`LLM_TOOL_MAX_ITERATIONS`, default `4`):
1. Stream one reply. A `ToolCallStripper` removes `<tool_call>` blocks from the
streamed text — a block that is still incomplete is **held back**, so it is
never spoken by the TTS nor shown in the chat box even when split across
deltas. The visible text is spoken as usual, so the assistant can say
something short ("Let me check…") before calling. A round that streams back
**empty** is re-asked once without streaming to recover a call the server
dropped (see above).
2. If the round produced no tool call, the turn ends.
3. Otherwise the assistant message (with its native call) plus one `tool` role
message per result are appended, and the loop re-prompts the model, which can
then answer using the results.
Tool errors (unknown tool, transport failure, non-2xx status) are returned to
the model as text, so it can recover and still produce an answer; the turn never
fails because of a tool. Only the *spoken* text is written to the rolling
history — the tool exchange stays local to the turn.
The `tool_call` event is also forwarded to the client for observability:
```json
{ "type": "tool_call", "name": "get_weather", "arguments": { "city": "Paris" } }
```
---
## External audio injection
The mixer has a dedicated **external channel** (`external_queue`) that plays an
audio source injected from the client — a direct audio URL, an Icecast/Shoutcast
radio stream, an HLS playlist, or a YouTube link — mixed together with the
agent's voice. It is implemented by [`src/media.rs`](src/media.rs)
(`MediaSource`) and driven over the WebSocket by the `audio_source` message
(see [WebSocket Protocol](#websocket-protocol)). The Mivocero client exposes it
in its footer bar.
### Why `ffmpeg` / `yt-dlp`
The channel does not decode audio itself: it spawns an external decoder process
whose raw PCM output feeds the ring buffer.
- **`ffmpeg`** (URL / radio / HLS) — reads the source and writes
`-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1`, i.e. exactly the mixer's
format (44.1 kHz, 16-bit LE, mono). URL inputs use `-reconnect 1
-reconnect_streamed 1 -reconnect_delay_max 5` so a dropped radio connection is
retried instead of ending the stream.
- **`yt-dlp`** (YouTube) — used only to *resolve* a YouTube page to a direct
media URL (`-f bestaudio/best -g --no-playlist`); that URL is then handed to
`ffmpeg`. YouTube-specific cookies/extractor tweaks go in `MEDIA_YTDLP_ARGS`
(e.g. `--cookies /path/cookies.txt` if the media server asks to "confirm
you're not a bot").
Both binaries are configurable (`MEDIA_FFMPEG_BIN`, `MEDIA_YTDLP_BIN`) and are
only required when the external channel is actually used.
### Backpressure
A live radio always produces audio at real time, but a bounded VOD source (a
YouTube video, a file) decodes far faster than real time. The decode task
respects the channel's `remaining_capacity()` and sleeps 20 ms whenever the
queue is full, so a fast source is throttled to the mixer's drain rate instead
of overflowing the ring buffer (which would drop the oldest samples).
### Independence from the agent
The external channel is deliberately **not** cleared by `AudioMixer::flush()`:
a barge-in stops the *agent* (TTS + fillers), not the background music. It also
does not count towards `AudioMixer::is_active()` — the agent is not considered
to be "speaking" because the music is playing. Its volume is a runtime atomic
(`AudioMixer::set_external_gain`) so the client slider takes effect on the next
mixed frame.
### Lifecycle
| Step | What happens |
|---|---|
| `play` | Any current source is stopped; state `loading` is emitted. For a YouTube URL `yt-dlp` resolves the direct URL first. |
| ready | `ffmpeg` is spawned and reading begins; state `playing` is emitted. |
| streaming | Decoded PCM is pushed into `external_queue` under backpressure. |
| `stop` / new `play` / disconnect | The `CancellationToken` is cancelled and the child processes are killed (`kill_on_drop(true)`); `stopped` (or `error`) is emitted. |
A monotonic generation counter guards against a stale task clearing the state of
a newer source.
---
## Mivocero chat client
A ready-to-use speech-to-speech example client is served at **`/mivocero`**
(embedded from [`src/web/mivocero.html`](src/web/mivocero.html)):
- **Microphone** — the same AudioWorklet capture path as the playground
(44.1 kHz, 16-bit LE mono, 20 ms frames).
- **Chat box** — shows the user transcript (from STT) and the assistant reply
**as it streams**, plus messages typed in the composer.
- **Karaoke highlighting** — both the assistant and the user bubbles are
rendered word by word. The assistant's words light up as they are *spoken*,
driven by the audio playhead (the playback worklet reports the samples it has
actually emitted): words already spoken are bright, the word currently
sounding is accented, and the rest stay dim. The rate is an estimate
(`KARAOKE_CPS`, ~15 chars/s) — it only affects the visual highlight, never the
audio. The user's words cannot follow the audio (the STT returns the whole
transcript only once the utterance is over), so they are lit by a short
left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word) when the bubble
appears.
- **Text composer** — typing and sending a message runs a full LLM → TTS turn
without using the microphone (`{"type":"user_text","text":…}`).
- **Interrupt button** — flushes audio and cancels the in-flight reply
(`{"type":"interrupt"}`).
- **Barge-in indicator** — a red **Barge-in** pill in the header that flashes and
bumps a session counter whenever the agent is interrupted: either by the
server-side VAD onset (via the `{"type":"barge_in"}` event) or by the local
**Stop** button. The counter resets on each connect.
- **External audio bar** — footer controls to inject a background source by URL
(direct audio / radio / YouTube), with **Play** / **Stop** buttons, a volume
slider (0–150 %, sent as `EXTERNAL_AUDIO_GAIN` × slider) and a live state
label driven by the server's `audio_source` events. Demonstrates the
[external audio channel](#external-audio-injection).
- **Cue indicators** — a row of three tiles (**Filler**, **Start filler**,
**Backchannel**) that flash and bump a session counter each time the server
emits the matching cue, driven by the `{"type":"cue"}` events. The detail line
shows the backchannel category (or the clip name), so the emission cadence of
the latency masks is visible live. Counters reset on each connect.
- **Cue methods panel** — a collapsible **Cue methods** section below the tiles
that documents the method used by each cue (when it fires, how a clip is
chosen) and shows the **live parameters** reported by the server in the
`{"type":"cues"}` event sent at session start: filler delay / gains / clip
count, start-filler go-delay, and the backchannel gain, min-speech, interval,
jitter, pause window, probability, volume jitter, per-turn cap and categories
— all read from the effective runtime configuration (no hard-coded values).
- **Side menu** — the ☰ button in the header opens a drawer that navigates
between the chat and the [custom voice builder](#custom-voice-builder).
- **Idle auto-disconnect** — the session closes itself after **2 minutes**
(`IDLE_TIMEOUT_MS`) without any traffic in either direction. The timer is
reset by every server event *and* audio frame received, and by every user
action sent (typed message, Stop, media controls) — so an ongoing reply or a
playing external source counts as activity. On expiry the WebSocket is closed,
the microphone is released, an explanatory message is appended to the chat and
the **Connect** button becomes available again.
Open `http://localhost:8787/mivocero` (or `https://…` for remote microphone
access — see [HTTPS](#https-microphone-access)).
---
## Custom voice builder
A dedicated page served at **`/customvoice`** (embedded from
[`src/web/customvoice.html`](src/web/customvoice.html), linked from the Mivocero
side menu) lets anyone build a voice-cloning reference from their own voice,
without touching the filesystem or the `.env`. Two sources are offered:
- **Record now** — read a predefined phrase out loud (the microphone flow).
- **Upload a file** — pick an audio file and **type its transcript** (any
container `ffmpeg` can read: wav, mp3, m4a, ogg, flac…).
The flow:
1. **Language** — a selector populated from `GET /api/customvoice/phrases`
(the languages of [`crate::lang`](src/lang.rs); the default is
`LANG_DEFAULT`).
2. **Source** — pick **Record now** or **Upload a file**.
- **Record now** — the page shows the phrase to read for the selected
language. The phrases live **server-side**
([`src/customvoice.rs`](src/customvoice.rs)), so the transcript attached to
the reference always matches the text the user was asked to read. The
browser captures the microphone with `MediaRecorder` (Opus/WebM where
supported); the **Stop** button stays disabled until **10 seconds** have
elapsed (a progress bar and a timer show the way to the minimum), and
recording auto-stops at 20 s. The take can be played back and discarded
before use.
- **Upload a file** — choose an audio file (`accept="audio/*"`) and type (or
paste) the exact text spoken in it. Here the transcript is **required**: it
is what drives the clone, so it must match the audio.
3. **Use it** — `POST /api/customvoice?lang=<code>[&text=<transcript>]` with the
recording or the uploaded file as the body. The `text` parameter carries the
upload flow's typed transcript and overrides the predefined phrase; when it
is absent or blank the language's predefined phrase is used (so a plain
recording needs no text). The server **post-processes** the take (see below),
transcodes it to **mono 16-bit PCM WAV at `OUTPUT_SAMPLE_RATE`** (44.1 kHz)
with `ffmpeg`, writes it next to the configured `TTS_REFERENCE_AUDIO` (the
directory the Fish Audio backend can read — hence "the path on the TTS
server" note in [Configuration](#configuration)), writes the transcript as a
sibling `<name>.txt` (the convention [`Config::tts_reference`](src/config.rs)
already falls back to), and installs the voice through
[`TtsEngine::set_reference`](src/tts/mod.rs) — effective immediately for new
replies, no restart.
4. **Reset** — `POST /api/customvoice/clear` restores the reference configured
at startup (`TTS_REFERENCE_AUDIO`), or clears it when none is configured.
### Post-processing
A raw browser recording is uneven: it carries rumble and room noise, and its
level depends on how far the user sat from the microphone. The take therefore
goes through a cleanup chain before it is installed
([`src/customvoice.rs`](src/customvoice.rs)):
| Stage | Filter | Why |
|---|---|---|
| Mono downmix | `aformat=channel_layouts=mono` | Everything downstream (and the reference itself) is mono. Done **first** so loudnorm measures the signal that is actually written. |
| Rumble / DC removal | `highpass=f=80` | Kills desk thumps, plosives and DC offset below the voice band. |
| Noise reduction | `afftdn=nr=12:nf=-40` | FFT denoise for steady room/hiss noise. |
| De-essing | `deesser` | Tames harsh sibilance that a clone would otherwise exaggerate. |
| Silence trim | `silenceremove` (+ `areverse`) | Drops leading/trailing silence so the reference starts on speech. |
| Loudness | `loudnorm`, **two-pass, linear** | Matches every take to the same programme level. |
The loudness stage is deliberately **two-pass**: ffmpeg measures the cleaned
signal first, then applies the measurement as **one constant gain**
(`linear=true`). A single gain cannot pump or ride the level frame by frame —
which is exactly what makes takes differ in volume.
The default target is **-26 LUFS** integrated with a **-5 dBTP** true-peak
ceiling. Both sit *below* the shipped reference (`voice_profiles/solis.wav`
measures ≈-23.4 LUFS at -7.8 dBTP): a user take is far more dynamic than that
mastered reference, so at equal LUFS its much hotter peaks make the cloned
voice sound louder than the default one. The lower target and the extra peak
headroom buy that gap back, so a custom voice sits a touch under the default
instead of on top of it. A take recorded 30 dB too quiet and one recorded hot
both land on the same level.
If any stage fails (a filter quirk, an unreadable take, an all-silence
recording) the endpoint logs a warning and **falls back to the plain
transcode** — a take is never lost to post-processing.
| Env var | Default | Description |
|---|---|---|
| `CUSTOMVOICE_POSTPROCESS` | `true` | Master switch; `false` stores the plain transcode. |
| `CUSTOMVOICE_DENOISE` | `true` | The `afftdn` + `deesser` stage — turn off for a clean studio take. |
| `CUSTOMVOICE_TARGET_LUFS` | `-26` | Integrated loudness target, below the shipped reference. Lower it for a quieter custom voice. |
| `CUSTOMVOICE_TRUE_PEAK` | `-5` | True-peak ceiling (dBTP). |
| Method | Route | Description |
|---|---|---|
| `GET` | `/customvoice` | The builder page. |
| `GET` | `/api/customvoice/phrases` | Reading phrases + the default language (JSON). |
| `POST` | `/api/customvoice?lang=<code>[&text=<transcript>]` | Body = recording or uploaded file; transcodes, persists (with the transcript) and installs the reference. |
| `POST` | `/api/customvoice/clear` | Restore the configured reference. |
The body cap for the upload route is raised to 32 MB (the axum default is 2 MB)
so an uncompressed WAV is accepted as well as a compressed capture. Microphone
access requires a secure context: use `https://…` when the page is not opened
from `localhost` (see [HTTPS](#https-microphone-access)).
---
## Session audit log
Every WebSocket session can be recorded to its own structured **JSONL** file —
one JSON object per line — under `SESSION_AUDIT_DIR` (default `logs/sessions`),
named `s-<epoch_ms>-<seq>.jsonl`. It is meant for post-mortem analysis and for
tuning the latency knobs (why a filler fired, when a cue landed, how long the
first audio took).
Each record carries a wall-clock timestamp, the elapsed time since the session
started, an event discriminator and a payload:
```json
{"ts":1729170000000,"elapsed_ms":12,"event":"session_start","data":{"config":{…},"runtime":{…}}}
{"ts":1729170000123,"elapsed_ms":135,"event":"control","data":{"type":"user_text","text":"…"}}
{"ts":1729170000456,"elapsed_ms":468,"event":"out","data":{"type":"assistant_delta","text":"Hola,"}}
{"ts":1729170000500,"elapsed_ms":512,"event":"out","data":{"type":"cue","kind":"start_filler","name":"sf02"}}
{"ts":1729170009000,"elapsed_ms":9012,"event":"session_end","data":{"reason":"client_disconnected","counts":{…}}}
```
| `event` | When | `data` |
|---|---|---|
| `session_start` | Session opens | `config` — the full non-secret configuration snapshot; `runtime` — which of STT / VAD / tokenizer / LLM are actually active (plus the LLM model). |
| `control` | `user_text` control message received | The text turn. |
| `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `barge_in`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. |
| `interrupt` | Client barge-in (`interrupt` JSON or legacy `INTERRUPT`) | `source` (`client` / `client_legacy`). |
| `barge_in` | VAD-driven barge-in | `prob` (onset speech probability). |
| `vad_speech_start` / `vad_speech_end` | Utterance boundaries | `prob`; `duration_ms`, `avg_prob`. |
| `utterance_dropped` | VAD segment filtered by the utterance gate (no STT, no LLM turn) | `duration_ms`, `avg_prob`, `too_short`, `too_soft`. |
| `media` | External audio control | `action` (`play`/`stop`/`volume`) with `url` / `gain`. |
| `reference` | Voice-cloning reference change | `action` (`set`/`clear`) and the audio path. |
| `speak` | Direct TTS request (playground / fallback text) | `text`. |
| `language_switch` | Detected language changed | `from`, `to`, `generated` (whether clip banks were synthesized). |
| `error` | Transport / receiver error | `where`, `text`. |
| `session_end` | Session closes | `reason`, `duration_ms`, `session_id`, `file`, `counts` (per-event totals). |
The sink is **best-effort**: if the file cannot be opened, or a write fails, the
session runs unchanged (the failure is logged once in the tracing output and the
sink then goes quiet). Files are appended and flushed after every record, so the
log survives a crash or a `pm2 restart`. The directory is git-ignored.
---
## WebSocket Protocol
Connect to `ws://<host>:8787/ws`.
### Client → Server
| Message | Description |
|---|---|
| **Binary** | Raw 16-bit LE mono PCM microphone frames (44.1 kHz). Resampled to 16 kHz server-side for STT. |
| **Text — `{"type":"user_text","text":"…"}`** | Text turn from the chat box: runs the same LLM → TTS path as a spoken utterance (no STT). |
| **Text — `{"type":"interrupt"}`** | Barge-in: cancel the in-flight LLM reply / TTS and flush all queued audio. |
| **Text — `{"type":"audio_source","action":"play","url":"…"}`** | Inject an external audio source (direct URL / radio / HLS / YouTube) into the mixer's external channel, replacing any current one. |
| **Text — `{"type":"audio_source","action":"stop"}`** | Stop the external source and clear the channel. |
| **Text — `{"type":"audio_source","action":"volume","gain":0.6}`** | Set the external channel gain (0..1) at runtime. |
| **Text — `INTERRUPT`** *(also `interrupt`, `BAARGEIN`, `bargein`)* | Legacy barge-in form (kept for the playground). |
| **Text — `SET_REFERENCE:<path>\|<text>`** | Set/change the voice-cloning reference. `<text>` may be omitted to re-use the sibling `.txt`. |
| **Text — `CLEAR_REFERENCE`** | Disable voice cloning. |
| **Text — anything else** | Synthesize the text directly (useful for testing TTS without STT). |
### Server → Client
| Message | Description |
|---|---|
| **Binary** | One 20 ms mixed frame of 16-bit LE mono PCM @ 44.1 kHz (`1764` bytes). Silence is sent when no audio is queued, keeping the stream alive. |
| **Text — `{"type":"user","text":"…"}`** | The user's turn (STT transcript or chat-box message) — rendered in the chat box. |
| **Text — `{"type":"assistant_delta","text":"…"}`** | Streaming chunk of the assistant reply (chain-of-thought already removed). |
| **Text — `{"type":"assistant_end"}`** | The assistant turn is complete. |
| **Text — `{"type":"tool_call","name":"…","arguments":{…}}`** | The model called a tool; the call has been executed (result is not echoed in the chat box, only in the model's next round). Purely informational. |
| **Text — `{"type":"error","text":"…"}`** | LLM request failed; the turn was aborted. |
| **Text — `{"type":"audio_source","state":"…","url":"…"}`** | External audio channel state change: `loading` (decoding is being set up / YouTube URL resolving), `playing` (PCM is streaming), `stopped`, or `error` (with a `text` message). |
| **Text — `{"type":"cue","kind":"…","name":"…","category":"…"}`** | A latency-mask / backchannel cue was actually emitted into the mixer: `kind` is `filler` (end-of-turn), `start_filler` (turn start) or `backchannel` (during user speech); `name` is the clip name and `category` (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators. |
| **Text — `{"type":"cues","lang":"…","config":{…}}`** | Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (`delay_ms`, gains, clip count), the start-filler `go_delay_ms` and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's **Cue methods** panel so it shows the live values instead of hard-coded ones. |
| **Text — `{"type":"session","id":"…"}`** | Sent once right after connect: the server-assigned session id (also the audit-log file stem). It is the key used by `GET`/`POST /api/prompt` to read/override the system prompt of this session ([Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)). |
| **Text — `{"type":"barge_in","source":"vad","prob":…}`** | The agent was interrupted. Sent for the server-side VAD barge-in (the browser has no other way to know, since the onset is detected in the session task); the client-side `interrupt` does not echo one. Drives the demo's barge-in indicator. |
---
## Audio Pipeline
Constants are defined in [`src/config.rs`](src/config.rs):
| Constant | Value | Meaning |
|---|---|---|
| `OUTPUT_SAMPLE_RATE` | `44_100` Hz | Server → browser rate (matches Fish Audio Drama 3 native output). |
| `STT_SAMPLE_RATE` | `16_000` Hz | Rate expected by Parakeet. |
| `FRAME_DURATION_MS` | `20` ms | Mixer ticker period. |
| `OUTPUT_FRAME_SAMPLES` | `882` | Samples per frame (44.1 kHz × 20 ms). |
| `OUTPUT_FRAME_BYTES` | `1764` | Bytes per frame (16-bit mono). |
| `EOU_SILENCE_MS` | `300` ms | Silence window that marks end-of-utterance. |
| `VAD_RMS_THRESHOLD` | `0.01` | RMS above which audio counts as voice. |
### Mixer & jitter buffering ([`src/audio/mixer.rs`](src/audio/mixer.rs))
- Mixing formula:
`mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) + (external × external_gain)`.
The filler gain is `1.0` when TTS is idle; when the reply's playback starts a
still-queued filler/start-filler is instead **faded out** over 80 ms
(`FILLER_FADEOUT_FRAMES`) and cut, so it hands off to the answer (a cue queued
after playback has already begun falls back to `filler_gain_duck`, default
`0.15`). The backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
The start-filler channel is summed with the filler, so it shares the same gain.
The external channel gain starts at `EXTERNAL_AUDIO_GAIN` (default `0.6`) and
is adjustable at runtime (`AudioMixer::set_external_gain`). Output is clamped
to `i16`.
- **Pre-buffer:** TTS playback does not start until the queue holds
≥ `TTS_PREBUFFER_MS` (default **550 ms**) of synthesized audio. It absorbs
uneven TTS chunk arrival; lower it to start speaking sooner, raise it for fewer
drop-outs. Note: Drama-3's *granularity* is set by the server's vocoder window
(`--vocoder.factory.stream_stride` / `stream_followup_stride`, in codec frames
of 2048 samples = 46.4 ms). With the deployment used here (stride `4`, followup
`8`, overlap `16`) the first window is 4 frames = 0.186 s and the steady windows
are 8 frames = 0.372 s, delivered ~0.35 s apart (the server runs at ~0.95× real
time). Because `TTFA ≈ TTFB + (prebuffer − window)`, any pre-buffer in
`(186, 557]` ms primes on the **same** second window → TTFA ≈ `0.73 s`; the
default `550 ms` takes that whole cushion at no extra latency. A pre-buffer
below ~`186 ms` starts playback on the first window (TTFA ≈ `0.38 s`) but leaves
almost no margin: under concurrent TTS requests one GPU serves several requests
and the per-request throughput dips below real time (~`1.5×` vs ~`0.94×` alone),
so every chunk seam starves the queue and pauses playback → an audible
micro-gap at the start of each sentence. With the upstream default (`64`, a
2.97 s burst every ~2.7 s) gapless playback would instead require a ~3 s
pre-buffer; a large constant stride (`16`) both raises the TTFB (~1.2 s) and
bursts, and a growing ramp gaps the tail while the small early windows drain.
- **Resilient resume:** the full pre-buffer is only required at the *first*
playback of a reply. If the queue empties mid-reply (after the 200 ms drain
grace) playback pauses, but resumes as soon as *any* audio is queued again —
it does not wait to refill the whole jitter buffer. `mark_stream_start` re-arms
the pre-buffer at the beginning of every turn.
- **Drain grace:** brief inter-chunk empties (< 200 ms) emit silence without
pausing playback, avoiding audible drop-outs on ordinary arrival jitter.
- **Ticker:** `MissedTickBehavior::Burst` — missed ticks are caught up in a burst
rather than dropped, so no 20 ms frames are lost.
---
## Project Layout
```
audio-agent-engine/
├── Cargo.toml
├── .env.example # documented configuration template
├── server.sh # start / stop / restart / status / logs / cert
├── certs/ # self-signed TLS cert+key (generated)
├── UPDATES.md # changelog (English, newest-first)
├── tools.example.json # tool definitions for LLM tool calling (copy/edit)
├── sdk/ # dependency-free JavaScript client SDK (+ .d.ts, served at /sdk/)
├── assets/fillers/<lang>/ # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin), one dir per language
├── assets/start_filler/<lang>/ # sf01.raw … turn-start start-filler clips + texts.txt index
├── assets/backchannel/<lang>/ # bc01.raw … backchannel clips + texts.txt index
├── models/ # Parakeet ONNX + Silero VAD + tokenizer
└── src/
├── main.rs # Axum server, routes, graceful shutdown
├── config.rs # env-var config + audio constants
├── assets.rs # startup clip bootstrap (auto-generate fillers/backchannel/start filler)
├── tls.rs # optional HTTPS/WSS termination (rustls)
├── onnx.rs # shared process-wide ONNX Runtime environment
├── audio/ # ring_buffer, mixer
├── tts/ # ← plugin system
│ ├── mod.rs # TtsEngine trait + create_tts_engine factory
│ ├── fish_audio.rs # Fish Audio / SGLang Drama 3 plugin
│ └── kokoro.rs # Kokoro plugin (stub)
├── stt/ # Parakeet ONNX + tokenizer + worker
├── vad/ # Silero VAD v5 + onset/offset state machine
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
├── tools.rs # configurable HTTP tool calling (native <tool_call> parser + executor)
├── tools_api.rs # GET/POST /api/tools — runtime tool definitions (in-memory override)
├── prompt.rs # per-session system-prompt registry (overrides + resolution)
├── prompt_api.rs # GET/POST /api/prompt — read/override the LLM system prompt per session
├── auth_api.rs # POST /api/auth/phone-login — server-side proxy to the AquaChile auth API
├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
├── customvoice.rs # custom-voice builder endpoints (record → WAV reference → set_reference)
├── transcribe.rs # POST /api/transcribe — Parakeet transcription of an uploaded audio file
├── apidoc.rs # OpenAPI document (utoipa) rendered by the Swagger UI at /docs
├── audit.rs # per-session JSONL audit log (config snapshot, events, summary)
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
├── ws/ # handler, session (per-connection audio loop)
└── web/ # embedded playground.html + mivocero.html + customvoice.html
```
### HTTP routes
| Route | Description |
|---|---|
| `GET /` | Playground single-page client (embedded at compile time). |
| `GET /mivocero` | Mivocero speech-to-speech example client with chat box (embedded at compile time). |
| `GET /customvoice` | Custom-voice builder page (record or upload a reference — see [Custom voice builder](#custom-voice-builder)). |
| `GET /api/customvoice/phrases` | Reading phrases per language + the default language (JSON). |
| `POST /api/customvoice?lang=<code>[&text=<transcript>]` | Upload a recording or an audio file (with its transcript); transcode it to a WAV reference and install it as the active voice. |
| `POST /api/customvoice/clear` | Restore the reference configured at startup. |
| `POST /api/transcribe` | Upload a raw audio file (any ffmpeg-readable container); returns the Parakeet transcript of the whole file (`text`, `duration_ms`, `tokens`). |
| `GET /api/tools` | The active LLM tool definitions and their origin (`source`: `file` / `request` / `none`). |
| `POST /api/tools` | Install LLM tool definitions at runtime (same JSON array as `LLM_TOOLS_FILE`) — in-memory, no restart ([Tool calling](#tool-calling)). |
| `GET /api/prompt` | The effective LLM system prompt, with its origin (`source`: `session` / `global` / `env` / `default` / `none`). Optional `?session_id=…`; `404` for an unknown session ([Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)). |
| `POST /api/prompt` | Override the LLM system prompt at runtime — for one session (`{"session_id":"…","prompt":"…"}`) or process-wide (`prompt` only); `prompt: null` clears it. In-memory, no restart. |
| `POST /api/auth/phone-login` | Authenticate a phone number (`{"phoneNumber":"+44…","session_id":"s-…"}`) through a **server-side proxy** to the AquaChile auth API (`AUTH_API_BASE` + `AUTH_PHONE_LOGIN_PATH`) and return its payload (`sessionToken` + `user`). When `session_id` is provided, the engine stores a sanitized user profile as hidden LLM context for that live session ([Phone login](#phone-login-post-apiauthphone-login)). |
| `GET /health` | Liveness probe (`ok`). |
| `GET /sdk/audio-agent-client.js` | The dependency-free JavaScript client SDK ([Client SDK (JavaScript)](#client-sdk-javascript), source in [`sdk/`](sdk)). |
| `GET /sdk/demo.html` | Demo client built on the SDK (source in [`sdk/demo.html`](sdk/demo.html)). |
| `GET /ws` | WebSocket upgrade → per-session audio loop. |
| `GET /docs` | Swagger UI for the REST surface ([API documentation](#api-documentation-swagger-ui)). |
| `GET /api-docs/openapi.json` | The raw OpenAPI 3 document the Swagger UI renders. |
### API documentation (Swagger UI)
The REST endpoints are documented with an **OpenAPI 3** document derived from
the handlers ([`utoipa`](https://docs.rs/utoipa)) and browsable in a **Swagger
UI** served by the engine itself ([`utoipa-swagger-ui`](https://docs.rs/utoipa-swagger-ui)):
- **UI** — `GET /docs` (→ `/docs/`) renders the interactive Swagger UI.
- **JSON** — `GET /api-docs/openapi.json` serves the raw OpenAPI document.
The document is built in [`src/apidoc.rs`](src/apidoc.rs) (the `ApiDoc`
`#[derive(OpenApi)]`, the response schemas) with per-handler
`#[utoipa::path]` annotations on the endpoints. Only the JSON/REST endpoints are
covered — the WebSocket endpoint (`/ws`) and the HTML client pages are streaming
/ static assets and are intentionally left out.
The `utoipa-swagger-ui` build script embeds the UI assets from a zip at compile
time; the `zip` crate is therefore pinned to the `2.2.x` line in `Cargo.toml`
(the 2.3+ API no longer matches the 8.1 build script).
### Transcription endpoint
`POST /api/transcribe` ([`src/transcribe.rs`](src/transcribe.rs)) turns an
uploaded audio file into text with the same Parakeet TDT model the live session
uses:
```bash
curl -s -X POST http://localhost:8787/api/transcribe \
-H 'Content-Type: application/octet-stream' \
--data-binary @clip.wav
# {"ok":true,"text":"…","duration_ms":14918,"tokens":83}
```
- **Input** — the raw audio bytes in the request body (any container `ffmpeg`
can read: wav, mp3, m4a, ogg, flac…); no multipart, no `lang` parameter —
Parakeet TDT v3 is multilingual and the language is auto-detected.
- **Processing** — `ffmpeg` decodes to mono 16 kHz `s16le`, and the engine's
**already-loaded** STT model ([`AppState::stt`](src/ws/handler.rs)) runs one
inference over the whole file (`spawn_blocking`: the inference is CPU-bound
and holds the ONNX session lock). No second ONNX environment is created.
- **Output** — a single transcript for the entire file (the audio is **not**
segmented): `{ "ok": true, "text": …, "duration_ms": …, "tokens": … }`.
- **Errors** — `400` empty body, `503` STT model not loaded, `502` ffmpeg
decode failed, `500` temp-file / inference failure. Uploads are capped at
32 MB (same as `/api/customvoice`).
---
## Phone login (`POST /api/auth/phone-login`)
The SDK demo can authenticate the user by phone number, read from the page URL
(`?phoneNumber=+44…`), at the start of the session. The call is made **through
the engine**, not directly from the browser:
```bash
curl -X POST localhost:8787/api/auth/phone-login \
-H 'Content-Type: application/json' \
-d '{"phoneNumber":"+447432477433"}'
```
The engine forwards the same JSON to the upstream AquaChile auth API
(`AUTH_API_BASE` + `AUTH_PHONE_LOGIN_PATH`) and returns its status and body
unchanged — on success a payload like
`{"sessionToken":"…","user":{"id":…,"name":…,"email":…,"permissions":[…]}}`.
If the request also carries a live WebSocket `session_id`, the engine extracts a
sanitized user profile (`name`, `email`, `businessRoles`, `permissionSet`,
`centers`, `permissions`) and injects it into that session's LLM context as a
hidden system block. The SDK demo does this automatically right after connect.
The proxy exists because the upstream API rejects browser calls from the client
origin: its CORS preflight answers `400 Disallowed CORS origin` and its response
carries no `Access-Control-Allow-Origin`, so a direct `fetch` from the demo is
blocked. Routing the call through the engine makes it same-origin for the
browser. Errors: `400` for an empty/invalid body or a missing `phoneNumber`,
`502` when the upstream is unreachable. See [`src/auth_api.rs`](src/auth_api.rs).
---
## Client SDK (JavaScript)
[`sdk/audio-agent-client.js`](sdk/audio-agent-client.js) is a **dependency-free,
single-file ESM module** (with [`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts))
for writing clients against the engine. It is deliberately decoupled from the
server: copy the file to any host, no build step, no npm install. The engine also
serves it at `GET /sdk/audio-agent-client.js`.
A ready-made **[demo client](sdk/demo.html)** built on the SDK is served at
`GET /sdk/demo.html` — open `http://localhost:8787/sdk/demo.html` and press
*Connect*.
It covers:
- **Realtime session** — `AudioAgentClient`: connects to `/ws`, sends PCM
(`sendAudio`) and typed controls (`sendUserText`, `interrupt`,
`playAudioSource`, `setReference`, `speak`, …), and emits one event per server
message (`assistant_delta`, `cue`, `barge_in`, `tool_call`, `audio_source`,
`error`, …) plus a local `audio` event carrying the decoded PCM frames.
- **REST** — `AudioAgentRest`: `health`, `getTools` / `setTools`,
`getCustomVoicePhrases` / `setCustomVoice` / `clearCustomVoice`, `transcribe`,
`openapi`; failures throw an error carrying `status` and the parsed `body`.
- **Browser audio helpers** — `MicCapture` (mic → 882-sample `Int16Array` frames,
via an inline AudioWorklet), `PcmPlayer` (jitter-buffered playback,
`flush()` for barge-in) and `AudioSession`, which wires the three together.
- **PCM utilities** — `int16ArrayFromBytes`, `float32ToInt16`, `int16ToFloat32`,
`toInt16`, plus the `AUDIO` constants (44.1 kHz, 20 ms = 882 samples).
### Browser
```html
<script type="module">
import { AudioSession } from 'http://localhost:8787/sdk/audio-agent-client.js';
const session = new AudioSession({ baseUrl: 'http://localhost:8787' });
session.on('user', (e) => console.log('you:', e.text));
session.on('assistant_delta', (e) => (log.textContent += e.text));
session.on('error', (e) => console.error(e));
await session.start(); // connect + microphone + playback
session.sendUserText('hello'); // text turn
session.interrupt(); // barge-in
</script>
```
Serve the page from a **secure context** (`https://` or `localhost`) or the
browser will refuse microphone access. Cross-origin pages must have CORS enabled
on the engine (`CORS_ALLOW_ORIGIN=https://your.app`); `/ws` needs no CORS.
### Node
The core (client + REST + PCM helpers) is browser-agnostic. Node 22+ has a global
`WebSocket`; on older versions pass one in (e.g. the `ws` package). The audio
helpers (`MicCapture`, `PcmPlayer`, `AudioSession`) require the Web Audio API and
are browser-only.
```js
import { AudioAgentClient, AudioAgentRest, toInt16 } from './sdk/audio-agent-client.js';
import WebSocket from 'ws';
const client = new AudioAgentClient({ baseUrl: 'http://localhost:8787', WebSocket });
client.on('assistant_delta', (e) => process.stdout.write(e.text));
await client.connect();
client.sendUserText("what's the weather in Rome?");
client.sendAudio(toInt16(new Float32Array(882))); // PCM 16-bit LE, 44.1 kHz
const rest = new AudioAgentRest('http://localhost:8787');
console.log(await rest.getTools());
```
See [`sdk/README.md`](sdk/README.md) for the full reference.
---
## Development
```bash
# Build
cargo build
# Run tests (tokenizer unit tests)
cargo test
# Run with debug logging
RUST_LOG=debug cargo run
# Release build (LTO + thin codegen)
cargo run --release
```
---
## Related Documentation
- [`UPDATES.md`](UPDATES.md) — full changelog (English, newest-first).
- [`src/tts/mod.rs`](src/tts/mod.rs) — trait + factory source of truth.
Reference in New Issue View Git Blame Copy Permalink