roomote_PRE/piperust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

step2

Browse Source
This commit is contained in:
valenti
2026-10-17 15:49:20 +01:00
parent 430997bd2e
commit bfed42c3b3
124 changed files with 5646 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+599 -76
View File
@@ -36,9 +36,27 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
the reply is late*: it is scheduled `FILLER_DELAY_MS` (default 2500 ms) after
the turn starts and dropped if the response audio has already begun. Clips are
played round-robin from `FILLERS_DIR`; `FILLER_ENABLED=false` disables it.
- **Start filler** — a second latency mask on its own mixer channel: if the first
audio chunk of the reply has not reached the mixer within
`START_FILLER_GO_DELAY` (default 1200 ms) the agent plays a short
acknowledgment ("Un momento...") while the LLM/TTS are still working, and drops
it as soon as audio arrives in time. One phrase is picked **at random** from
`START_FILLER_TEXTS`, so the cue varies turn to turn. Independent of the delayed
latency filler above; `START_FILLER_ENABLED=false` disables it. See
[Fillers](#fillers--end-of-utterance-acknowledgments).
- **During-utterance backchannel** — short acknowledgments ("mm-hmm", "ajá")
are mixed in at a reduced gain *while the user is speaking*, so the agent
signals it is listening without taking the turn. Plugin-style `.env` config.
signals it is listening without taking the turn. Like the start filler the
bank is a phrase list (`BACKCHANNEL_TEXTS`) with Drama-3 tags, one clip per
phrase synthesized at startup. To sound natural the scheduler cues only on a
VAD **micro-pause**, randomizes the timing, gates each pause probabilistically,
avoids repeating the recent clips, drifts the volume slightly and follows a
**category** progression through the turn (`category::phrase` lines).
- **External audio injection** — a dedicated mixer channel can play a background
source (direct audio URL, Icecast/Shoutcast radio, HLS, or a YouTube link)
alongside the agent's voice, with an independent runtime volume and its own
decoder process (`ffmpeg` for URLs/radio, `yt-dlp` for YouTube). See
[External audio injection](#external-audio-injection).
- **Barge-in / interrupt** — cancels in-flight synthesis and flushes audio,
triggered client-side or server-side by the VAD.
- **Voice cloning** — reference audio + transcript (Fish Audio `references`).
@@ -55,10 +73,20 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
(first chunk after a few words, tunable) for the lowest time-to-first-audio;
chain-of-thought (`reasoning_content` / ` thinking`) is stripped from both the
audio and the chat box.
- **Tool calling** — the model can call external HTTP tools described in a JSON
file (`LLM_TOOLS_FILE`) before answering. Calls are parsed client-side from
the model's native `<tool_call>` format, executed over HTTP and fed back to
the model; the tool-call block is never spoken. See
[Tool calling](#tool-calling).
- **Embedded playground** — a single-page HTML client served at `/`.
- **Mivocero speech-to-speech client** — an example agent client at `/mivocero`
with a microphone, a live chat box (user transcript + streaming assistant
reply) and a text composer, useful to drive a full STT → LLM → TTS turn.
- **Per-session audit log** — each WebSocket session can be recorded to a
structured JSONL file: the configuration snapshot, every inbound control
message and outbound event, the cue emissions, the language switches, the
barge-ins and an end-of-session summary. Best-effort and off the audio hot
path. See [Session audit log](#session-audit-log).
---
@@ -72,6 +100,9 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
- *(Optional)* An OpenAI-compatible chat-completions endpoint (e.g. a local
vLLM / SGLang server) for the LLM reasoning stage. Without it the pipeline is
a plain STT → TTS echo.
- *(Optional)* `ffmpeg` on `PATH` (or `MEDIA_FFMPEG_BIN`) to inject external
audio from a URL / radio stream, and `yt-dlp` (or `MEDIA_YTDLP_BIN`) to inject
a YouTube link. Both are only needed for the external audio channel.
---
@@ -152,6 +183,10 @@ See [`.env.example`](.env.example) for a copy-paste template.
| `TTS_VOICE` | `default` | Voice identifier. |
| `TTS_REFERENCE_AUDIO` | `…/voice_profiles/solis.wav` | Voice-cloning reference WAV (path on the TTS server). |
| `TTS_REFERENCE_TEXT` | *(empty)* | Reference transcript. If it names an existing `.txt`, its contents are loaded; otherwise the raw string is used. Empty → auto-load `<audio-stem>.txt`. |
| `TTS_TEMPERATURE` | `0.45` | Sampling temperature sent to the SGLang speech endpoint (`fish` engine). Lower values flatten the delivery — the recitative "radio drama" register lives in the distribution tails. Empty omits the field (server default). |
| `TTS_TOP_P` | `0.75` | Nucleus sampling `top_p` sent to SGLang. Trims the tail of the acoustic token distribution, where exaggerated intonations live. Empty omits the field. |
| `TTS_TOP_K` | *(unset)* | Optional `top_k` sampling. Unset/empty omits the field (server default). |
| `TTS_STYLE_TAG` | `[conversational, casual]` | Neutral style tag prepended to every synthesis request that does not already start with a `[...]` tag. LLM replies carry no tag (the system prompt forbids stage directions), so they get this register; clip banks with their own tag are left untouched. Empty disables the prefix. **Quote the value in `.env`** (it contains a space). |
| `KOKORO_URL` | `http://localhost:8880` | Kokoro HTTP endpoint (`kokoro` engine). |
| `KOKORO_VOICE` | `af_heart` | Kokoro voice name. |
| `STT_MODEL_PATH` | `models/parakeet` | Directory of the Parakeet TDT ONNX pipeline (see below). |
@@ -167,19 +202,38 @@ See [`.env.example`](.env.example) for a copy-paste template.
| `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. |
| `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. |
| `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. |
| `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. |
| `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/fillers/es/`). |
| `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. |
| `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. |
| `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. |
| `FILLER_GAIN_DUCK` | `0.15` | Filler gain while TTS is active (ducking). |
| `TTS_PREBUFFER_MS` | `450` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
| `FILLER_GAIN_DUCK` | `0.15` | Filler gain while TTS is active (ducking). Also applies to the start-filler channel. |
| `START_FILLER_ENABLED` | `true` | Master switch for the **start filler** — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. `false` => no cue is ever queued. |
| `START_FILLER_GO_DELAY` | `1200` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. |
| `START_FILLER_TEXTS` | `[calm] Un momento...` | Phrases spoken by the start filler, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and a **random** one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, `[sigh]`, …) are passed through to the TTS. |
| `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/start_filler/es/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. |
| `TTS_PREBUFFER_MS` | `550` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first `185.8` ms, then `371.6` ms), any value in `(186, 557]` ms primes on the *same* second window, so `550` takes the whole cushion at no extra TTFA (~`0.73 s`). Below the first window (~`186` ms) playback starts on that window (TTFA ≈ server TTFB, ~`0.38 s`) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
| `TTS_PIPELINE_DEPTH` | `3` | How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. `1` restores strictly serial synthesis. |
| `BACKCHANNEL_ENABLED` | `true` | Master switch for during-utterance acknowledgments. |
| `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz). |
| `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of the generated `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz; `bc01.raw`, `bc02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/backchannel/es/`). |
| `BACKCHANNEL_TEXTS` | `ack::[calm] Mm-hmm...` … | Phrases spoken by the backchannel, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and one is played per cue. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, …) are passed through to the TTS. A line may carry an optional **`category::`** prefix (e.g. `agree::[agreeing] Claro...`) used to progress through the turn; a line without it belongs to the default `ack` category. |
| `BACKCHANNEL_GAIN` | `0.35` | Mix gain applied to the backchannel channel (0..1). |
| `BACKCHANNEL_MIN_SPEECH_MS` | `1500` | Continuous speech required before the first backchannel of a turn. |
| `BACKCHANNEL_INTERVAL_MS` | `3000` | Minimum spacing between two backchannels (clamped to ≥ `BACKCHANNEL_MIN_SPEECH_MS`). |
| `BACKCHANNEL_MAX_PER_TURN` | `2` | Maximum backchannels emitted per user turn. |
| `BACKCHANNEL_MIN_SPEECH_MS` | `1500` | Confirmed speech required before the first cue may be considered. |
| `BACKCHANNEL_INTERVAL_MS` | `3000` | Base spacing between two cues of the same turn; randomized by ±`BACKCHANNEL_JITTER_MS`/2. |
| `BACKCHANNEL_JITTER_MS` | `1200` | Timing jitter: delays the first cue by `0..JITTER` after `BACKCHANNEL_MIN_SPEECH_MS` and spreads the spacing. |
| `BACKCHANNEL_PAUSE_MIN_MS` | `96` | Lower bound of the VAD micro-pause window (ms). The reaction **depth** inside a pause is drawn at random in `[MIN, MAX]`, so cues don't all land at exactly `MIN`. Below this the user is likely still mid-word. |
| `BACKCHANNEL_PAUSE_MAX_MS` | `260` | Upper bound of the micro-pause window, capped below `VAD_MIN_SILENCE_MS` so a cue never lands on the pause that closes the turn. |
| `BACKCHANNEL_PROB` | `0.65` | Probability (0..1) of cueing on a qualifying micro-pause — keeps the agent from acknowledging every pause. |
| `BACKCHANNEL_VOLUME_JITTER` | `0.2` | Random per-cue volume scaling (± fraction), so cues don't sit at a fixed level. |
| `BACKCHANNEL_MAX_PER_TURN` | `3` | Maximum backchannels emitted per user turn. |
| `LANG_DEFAULT` | `es` | Base language of the **filler, start-filler and backchannel** banks — its clips live in the `<dir>/<lang>` subdirectory (e.g. `assets/backchannel/es/`). Other languages are generated on demand into their own per-language subdirectories. See [Multilingual clip banks](#multilingual-clip-banks). |
| `LANG_DETECT_ENABLED` | `true` | Master switch for LLM-based language detection. Requires the LLM (`LLM_ENABLED=true`). |
| `LANG_MIN_CHARS` | `12` | Minimum transcript length (chars) before a sample is sent to the detector. |
| `LANG_REDETECT_EVERY` | `5` | The first user turn is always checked; afterwards the language is re-checked once every N turns. |
| `LANG_LLM_FALLBACK_ENABLED` | `true` | Phrase tables cover `es/en/it/fr/de/pt`; for any other language the phrase list is generated with the LLM. `false` keeps only the tables. |
| `EXTERNAL_AUDIO_GAIN` | `0.6` | Mix gain applied to the external audio channel (0..1) at session start. Overridable at runtime from the Mivocero volume slider. |
| `MEDIA_FFMPEG_BIN` | `ffmpeg` | `ffmpeg` binary used to decode a URL / radio / HLS stream into PCM. |
| `MEDIA_YTDLP_BIN` | `yt-dlp` | `yt-dlp` binary used to resolve a YouTube link to a direct stream URL. |
| `MEDIA_YTDLP_ARGS` | *(empty)* | Extra whitespace-separated args appended to the `yt-dlp` invocation (e.g. `--cookies /path/cookies.txt` when YouTube asks for bot verification, or a custom `--extractor-args`). |
| `LLM_ENABLED` | *(auto)* | Master switch for the LLM reasoning stage. Default: enabled when `LLM_BASE_URL` and `LLM_MODEL` are both set. When `false`, the pipeline is a plain STT → TTS echo. |
| `LLM_BASE_URL` | *(empty)* | Base URL of the OpenAI-compatible endpoint; the client posts to `{LLM_BASE_URL}/chat/completions`. |
| `LLM_API_KEY` | `EMPTY` | Bearer token sent in the `Authorization` header (local servers usually ignore it). |
@@ -188,14 +242,21 @@ See [`.env.example`](.env.example) for a copy-paste template.
| `LLM_TEMPERATURE` | `0.7` | Sampling temperature. |
| `LLM_STREAM` | `true` | `true` → SSE token streaming, TTS starts on the first words; `false` → one blocking request, the whole reply is synthesized at once. |
| `LLM_STREAM_FIRST_CHUNK_CHARS` | `20` | Characters buffered before the **first** chunk is sent to the TTS, cut on a word boundary. Small = the agent starts speaking on the first few words. `0` disables the early flush. |
| `LLM_STREAM_CHUNK_CHARS` | `120` | Same threshold for every following chunk (larger chunks give better prosody). `0` disables. |
| `LLM_STREAM_CHUNK_CHARS` | `240` | Same threshold for every following chunk (larger chunks give better prosody and fewer TTS seams). `0` disables. |
| `LLM_HISTORY_MESSAGES` | `16` | Rolling conversation window (user+assistant messages) kept across turns. |
| `LLM_ENABLE_THINKING` | *(unset)* | `false` sends `chat_template_kwargs: {"enable_thinking": false}` — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers). |
| `LLM_STRIP_REASONING` | `true` | Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores `reasoning_content` deltas and removes inline think blocks. `false` passes the raw output through. |
| `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. |
| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). |
| `LLM_TOOLS_ENABLED` | *(auto)* | Master switch for tool calling. Default: enabled when `LLM_TOOLS_FILE` is a non-empty path. |
| `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method`, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
| `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops. |
| `LLM_TOOL_TIMEOUT_SECS` | `15` | Timeout of a single tool HTTP request. |
| `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | Tool responses are truncated to this many characters before being handed back to the model. |
| `TLS_CERT` | *(empty)* | PEM certificate chain. Set together with `TLS_KEY` to serve HTTPS/WSS. |
| `TLS_KEY` | *(empty)* | PEM private key matching `TLS_CERT`. |
| `SESSION_AUDIT_ENABLED` | `true` | Write a structured JSONL audit file per WebSocket session (config snapshot, inbound control messages, every outbound event, cues, language switches, barge-ins, end-of-session summary). Best-effort — a write failure never affects the session. See [Session audit log](#session-audit-log). |
| `SESSION_AUDIT_DIR` | `logs/sessions` | Directory holding the per-session audit files (`<id>.jsonl`). Relative to the process working directory; git-ignored. |
---
@@ -268,9 +329,10 @@ An unknown value panics at startup with a clear message.
##### Starting the Drama 3 backend (SGLang-Omni)
The `fish` engine talks to an **SGLang-Omni** server running Drama 3. Start it
with the vocoder streaming window narrowed, otherwise the vocoder emits ~3 s
bursts and the client is forced into a large pre-buffer:
The `fish` engine talks to an **SGLang-Omni** server running Drama 3. The
vocoder emits audio in **windows** of codec frames (2048 samples = 46.4 ms each);
the window sizes are the single biggest lever on latency, so the server must be
started with them tuned:
```bash
sgl-omni serve \
@@ -278,8 +340,12 @@ sgl-omni serve \
--host 0.0.0.0 --port 8000 \
--tts_engine.gpu_memory_fraction 0.45 \
--tts_engine.factory.max_running_requests 4 \
--vocoder.factory.stream_followup_stride 16 \
--vocoder.factory.stream_followup_stride_ramp "[10,12,16]"
--tts_engine.factory.enable_async_decode true \
--vocoder.factory.stream_stride 4 \
--vocoder.factory.stream_followup_stride_ramp "[8,8,8]" \
--vocoder.factory.stream_followup_stride 8 \
--vocoder.factory.stream_overlap_tokens 16 \
--vocoder.factory.stream_crossfade_samples 512
```
In this deployment the launcher is `/root/drama3/start-drama3.sh` and the server
@@ -290,26 +356,71 @@ pm2 restart start-drama3 --update-env # after editing the launcher
pm2 logs start-drama3 --lines 100 # CUDA-graph rebuild takes ~2 min at start
```
The two `--vocoder.factory.*` overrides are the key part:
| Flag | Value | Meaning |
|---|---|---|
| `stream_stride` | `4` | **First** window, in codec frames (decoded with *no* overlap context). Sets the server TTFB floor: at `4` it is ~`0.38 s`, at `16` it is ~`1.2 s`, because the TTS engine must emit that many codec tokens before the vocoder can decode anything. |
| `stream_followup_stride` | `8` | Steady-state window: `8` frames = `0.372 s`, arriving ~`0.35 s` apart (the server runs at ~`0.95×` real time), so the client queue grows and never starves. Larger windows mean **fewer seams** per second, which is what removes the audible artifacts. Upstream default `64` ⇒ `2.97 s` bursts. |
| `stream_followup_stride_ramp` | `"[8,8,8]"` | Windows 2..4, as a JSON tuple. Keep it flat: a *rising* ramp (e.g. `[4,6,10]`, or the upstream `(10,12,32)`) makes later windows arrive after the small ones have drained, which gaps playback at the start of a reply. |
| `stream_overlap_tokens` | `16` | Decoder warm-up context. It is decoded and **discarded** on output, so it only costs compute. This is the seam-quality lever: with too little context the streaming decoder wobbles in timbre/pitch at each seam (an audible "detune"/falsetto on long replies). `16` with stride `8` is a `(16+8)/8 = 3×` real-time decode load and stays sustainable (~`0.95×` ratio); the same `16` with stride `4` would be 5× and is **not** sustainable. Upstream default `32`. |
| `stream_crossfade_samples` | `512` | ~`11.6 ms` blend at each window seam, smoothing the hand-off between windows. |
| Flag | Meaning |
|---|---|
| `stream_followup_stride` | Steady-state window, in codec frames (`2048` samples = `46.4 ms` each). `16` ⇒ `0.743 s` bursts; the upstream default `64` ⇒ `2.97 s`. |
| `stream_followup_stride_ramp` | Window sizes for the first windows, as a JSON tuple. `[10,12,16]` emits a `0.464 s` then a `0.557 s` window first, so playback can start sooner. Upstream default `(10,12,32)`. |
Measured on this deployment (with the voice-cloning reference), server-side:
| Config | TTFB | Real-time ratio |
|---|---|---|
| `stream_stride 4`, followup `8`, overlap `16`, crossfade `512` *(this deployment)* | **~0.38 s** | ~`0.94–0.98×` |
| `stream_stride 4`, followup `4`, overlap `8`, crossfade `256` | ~`0.38 s` | ~`0.95×` (but audible detune / micro-gaps on long replies) |
| `stream_stride 16`, overlap `16` | ~`1.2 s` | ~`1.0×` |
The engine's Time-to-First-Audio is essentially `TTFB + (TTS_PREBUFFER_MS − first
window)`, quantized to the window the pre-buffer lands on. With this config the
first window is `185.8` ms and the second `371.6` ms, so any `TTS_PREBUFFER_MS`
in `(186, 557]` ms primes on the **same** second window → TTFA ≈ `0.73 s`. The
deployment ships `550 ms`: the whole cushion, no extra latency. Dropping to a
value below the first window (~`186` ms) starts playback on the first window
(TTFA ≈ `0.38 s`) but leaves little margin, and under concurrent TTS requests the
per-request throughput dips below real time (measured ~`1.5×` vs ~`0.94×` alone),
so each chunk seam starves the queue and pauses playback — an audible micro-gap at
the start of each sentence.
Two gotchas when writing the command:
- The flags are **dotted keys into the pydantic config**, so they go *without* a
`stages.` prefix — `--vocoder.factory.stream_followup_stride`, not
`--stages.vocoder.factory.stream_followup_stride` (the latter is rejected).
- The ramp tuple must be passed as a **JSON string** — `"[10,12,16]"` — because
- The ramp tuple must be passed as a **JSON string** — `"[8,8,8]"` — because
the config manager parses non-scalar paths with JSON.
Smaller stride = finer delivery granularity but more compute: each window
re-decodes `stream_overlap_tokens` (default `32`) of context, so `stride 16`
decodes 48 frames per 16 emitted. Keep the stride at or below the client's
`TTS_PREBUFFER_MS` budget (see *Mixer & jitter buffering*) — with the defaults
above, `TTS_PREBUFFER_MS=450` starts playback on the first window.
##### De-emphasis — reducing Drama 3's recitative register
Drama 3 tends to over-articulate: an LLM reply can come back with a theatrical,
radio-drama cadence rather than a conversational tone. The engine applies two
configurable mitigations, both in [`src/tts/fish_audio.rs`](src/tts/fish_audio.rs):
1. **Sampling knobs** — `temperature` / `top_p` / `top_k` are sent in the
`/v1/audio/speech` payload (`TTS_TEMPERATURE` default `0.45`, `TTS_TOP_P`
default `0.75`). Exaggerated intonations live in the tails of the acoustic
token distribution, so trimming them flattens the delivery. Lowering
`temperature` toward `0.3` flattens it further; `0.8+` makes it unstable and
theatrical again. An empty value omits the field (server default). Note that
this server **rejects `repetition_penalty`** with anything other than `1.0`,
so that knob cannot be used.
2. **Neutral style tag** — `TTS_STYLE_TAG` (default `[conversational, casual]`)
is prepended to every request whose text does not already start with a `[...]`
tag. LLM replies carry no tag (the system prompt forbids stage directions), so
they get this register; the filler/backchannel banks carry their own tags and
are left untouched. Empty disables the prefix. Quote the value in `.env`, as
it contains a space.
> **`.env` parsing note.** The file is read with `dotenvy`, which **fails on
> unquoted values containing spaces** — and a single bad line makes it discard the
> *entire* file (the process then silently falls back to built-in defaults: no
> LLM, plain HTTP). Always quote values that contain spaces, e.g.
> `TTS_STYLE_TAG="[conversational, casual]"`.
The remaining lever — replacing the voice-cloning reference with a more
conversational one — is not applied here: the default `solis.wav` is a
"locutor profesional" reading, whose prosody is part of the recitative tone.
#### `kokoro` — Kokoro TTS *(stub)*
@@ -494,27 +605,60 @@ All VAD logs live under the `audio_agent_engine::vad` target;
## Startup clip bootstrap
The filler and backchannel banks are plain `*.raw` PCM files. At startup
[`assets::ensure_clips`](src/assets.rs) checks each expected clip and, **if a
clip is missing, synthesizes it with the configured TTS engine** and writes it
to disk (the directory is created if needed). Existing clips are never
overwritten, so a fresh checkout works without a manual generation step and a
partial/customized bank is left untouched.
The filler, start-filler and backchannel banks are plain `*.raw` PCM files
(16-bit LE mono PCM at 44.1 kHz). At startup the clip banks are bootstrapped with
the configured TTS engine and the files are written to disk (the directory is
created if needed), so a fresh checkout works without a manual generation step.
Every bank lives in a per-language subdirectory `<dir>/<lang>` (see
[Multilingual clip banks](#multilingual-clip-banks)); the startup bootstrap
targets the `LANG_DEFAULT` subdirectory (e.g. `assets/fillers/es/`).
- Defaults live in [`assets::FILLER_CLIPS`](src/assets.rs) (7 clips: `uhm`,
`eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya`) and
[`assets::BACKCHANNEL_CLIPS`](src/assets.rs) (4 clips: `mmhmm`, `aja`, `claro`,
`yaveo`). The generated voice matches the configured TTS voice-cloning
reference.
- Backchannel clips are only generated when `BACKCHANNEL_ENABLED=true`.
- Filler uses [`assets::ensure_clips`](src/assets.rs): each expected clip is
checked and **synthesized only if missing**, so an existing bank is left
untouched. Defaults live in [`assets::FILLER_CLIPS`](src/assets.rs) (7 clips:
`uhm`, `eh`, `mmm`, `aver`, `bueno`, `pues`, `vaya`). The generated voice
matches the configured TTS voice-cloning reference.
- The **start filler** and the **backchannel** use the generic
[`assets::ensure_text_bank_clips`](src/assets.rs) (through
[`ensure_start_filler_clips`](src/assets.rs) / [`ensure_backchannel_clips`](src/assets.rs)):
both are driven by their phrase list (`START_FILLER_TEXTS` /
`BACKCHANNEL_TEXTS`) instead of a fixed clip set, so editing the list from
`.env` is enough to change what is spoken. The list the current clips were made
from is recorded in `<dir>/texts.txt`; when it is unchanged and every clip is
still on disk the synthesis is **skipped** (startup stays at ~2 s). Editing,
adding or dropping a phrase — or deleting a clip — triggers a **full refresh**
at the next restart, and clips left over from a previous list are removed.
Clips are named by position (`sf01`, `sf02`, … for the start filler; `bc01`,
`bc02`, … for the backchannel) and generated a few at a time
(`TEXT_BANK_CONCURRENCY`); a failed synthesis keeps the previous clip of that
position, and the index is only written once every clip succeeded, so a partial
failure is retried next time.
- Backchannel clips are only generated when `BACKCHANNEL_ENABLED=true`; its
default phrase list lives in
[`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs).
- Generation is best-effort: if the TTS backend is unreachable each clip logs a
warning and the server still starts (that bank simply stays silent).
```text
INFO audio_agent_engine::assets: generating missing clips at startup label="filler" dir=assets/fillers missing=7 engine="fish-audio"
INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/uhm.raw
INFO audio_agent_engine::assets: generating clips at startup label="filler" dir=assets/fillers/es clips=7 engine="fish-audio"
INFO audio_agent_engine::assets: generated clip label="filler" name="uhm" samples=32768 path=assets/fillers/es/uhm.raw
...
INFO audio_agent_engine: clip banks bootstrapped generated=11
INFO audio_agent_engine::assets: refreshing backchannel clips at startup label="backchannel" dir=assets/backchannel/es clips=8 engine="fish-audio"
INFO audio_agent_engine::assets: generated backchannel clip label="backchannel" name="bc01" samples=28672 path=assets/backchannel/es/bc01.raw
...
INFO audio_agent_engine::assets: removed stale backchannel clip label="backchannel" path=assets/backchannel/es/mmhmm.raw
INFO audio_agent_engine::assets: refreshing start-filler clips at startup label="start-filler" dir=assets/start_filler/es clips=22 engine="fish-audio"
INFO audio_agent_engine::assets: generated start-filler clip label="start-filler" name="sf01" samples=36864 path=assets/start_filler/es/sf01.raw
...
INFO audio_agent_engine::assets: removed stale start-filler clip label="start-filler" path=assets/start_filler/es/start_filler.raw
INFO audio_agent_engine: clip banks bootstrapped generated=30
```
On a later startup with unchanged lists:
```text
INFO audio_agent_engine::assets: backchannel clips up to date — skipping synthesis label="backchannel" dir=assets/backchannel/es clips=8
INFO audio_agent_engine::assets: start-filler clips up to date — skipping synthesis label="start-filler" dir=assets/start_filler/es clips=22
```
---
@@ -546,25 +690,73 @@ PCM at 44.1 kHz** — selected **round-robin** (`TurnManager::next_filler` →
different fillers. With a single clip the behaviour is unchanged. Missing clips
are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)).
### Start filler (turn-start acknowledgment when the reply is late)
A second latency mask, on its own mixer channel. It works exactly like the filler
above, but with its own — shorter — delay
([`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)):
`START_FILLER_GO_DELAY` (default **1200 ms**) after the turn is dispatched, if the
first audio chunk of the reply has **not** reached the mixer yet
(`AudioMixer::tts_audio_arrived` is still `false`) the cue is queued; if audio
arrived within that window, the cue is dropped instead of talking over the
beginning of the answer.
The gate is latched on chunk *arrival* (at the moment anything is pushed on the
TTS queue, checked on the 20 ms ticker), not on playback start: unlike
`tts_playback_started`, which only flips once the `TTS_PREBUFFER_MS` jitter buffer
is full, it does not wait out the pre-buffer. With the reference deployment the
first chunk lands ~`0.7`–`1.0 s` after the dispatch while playback starts
~`1.9 s` later, so gating on playback made the cue fire on every turn; gating on
arrival lets a reply that produces audio within `START_FILLER_GO_DELAY` suppress
it. A pending cue is also dropped on barge-in / new turn via the turn's
`CancellationToken`. This means a fast reply is never preceded by an
acknowledgment — only a late one is masked.
It lives on its own mixer channel (`start_filler_queue`), so it plays
independently of the round-robin latency filler above, but it shares the filler
gain, so it ducks to `FILLER_GAIN_DUCK` once the reply starts. Disabled by
`START_FILLER_ENABLED=false`.
Its clips live in `START_FILLER_DIR` (default `assets/start_filler`) and are
**not** part of the round-robin `FILLERS_DIR` bank. One clip is synthesized per
phrase of `START_FILLER_TEXTS` (default `"[calm] Un momento..."`) with the
configured voice, and [`FillerBank::random`](src/orchestrator/turn_manager.rs)
picks one **uniformly at random**, so the acknowledgment varies instead of
repeating the same sentence. Editing the phrase list re-synthesizes the bank at
the next restart (see
[Startup clip bootstrap](#startup-clip-bootstrap)); a failed synthesis leaves the
previous clip in place.
```text
START_FILLER_TEXTS="[calm] Un momento...
[thoughtful] A ver, veamos...
[sigh] Vale, ya voy..."
```
---
## Backchannel — during-utterance acknowledgments
A **backchannel** is a short acknowledgment ("mm-hmm", "ajá", "claro") played
*while the user is still speaking*, at a low gain, to signal that the agent is
listening — without taking the turn. It is distinct from the **filler** — the
instant acknowledgment injected at end-of-utterance by the
[`TurnManager`](src/orchestrator/turn_manager.rs) from `FILLERS_DIR` (played
round-robin; see [Fillers](#fillers--end-of-utterance-acknowledgments)).
listening — without taking the turn. It is distinct from the **filler** and the
**start filler** injected around end-of-utterance by the
[`TurnManager`](src/orchestrator/turn_manager.rs) (delayed mask from
`FILLERS_DIR` played round-robin, and a shorter-delay cue from `START_FILLER_DIR`;
see [Fillers](#fillers--end-of-utterance-acknowledgments)).
It is implemented in [`src/orchestrator/backchannel.rs`](src/orchestrator/backchannel.rs).
### Clips
`BACKCHANNEL_DIR` (default `assets/backchannel`) holds `*.raw` clips — **16-bit
LE mono PCM at 44.1 kHz** — played round-robin. The bundled set is synthesized
with the same cloned voice as the TTS (`mmhmm`, `aja`, `claro`, `yaveo`). The
clips are loaded once at startup into a
`BACKCHANNEL_DIR` (default `assets/backchannel`) holds the generated `*.raw`
clips — **16-bit LE mono PCM at 44.1 kHz** — one per phrase of
`BACKCHANNEL_TEXTS`, named by position (`bc01`, `bc02`, …). Each phrase may carry
an optional `category::` prefix; clips keep the list order, so the i-th clip
inherits the i-th phrase's category. The default list
([`DEFAULT_BACKCHANNEL_TEXTS`](src/orchestrator/backchannel.rs), Spanish with
Drama-3 tags) is synthesized with the same cloned voice as the TTS. The clips are
loaded once at startup into a
[`FillerBank`](src/orchestrator/turn_manager.rs) and shared across sessions.
Missing clips are synthesized at startup
([Startup clip bootstrap](#startup-clip-bootstrap)).
@@ -575,18 +767,46 @@ Missing clips are synthesized at startup
| Signal | Action |
|---|---|
| `VadEvent::SpeechStart` | begin a turn (`on_speech_start`) |
| `VadEvent::SpeechStart` | begin a turn (`on_speech_start`, randomizes the first-cue instant) |
| `VadEvent::SpeechEnd` | close the turn, reset counters (`on_speech_end`) |
| every frame while speaking | `maybe_emit()` attempts a cue |
| every frame while speaking | `maybe_emit(pause_ms)` attempts a cue |
`maybe_emit()` emits a clip **only when all** of the following hold:
- `BACKCHANNEL_ENABLED=true` and the clip bank is non-empty;
- the user has been speaking for at least `BACKCHANNEL_MIN_SPEECH_MS`;
- at least `BACKCHANNEL_INTERVAL_MS` elapsed since the previous cue;
- fewer than `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn;
- the agent is **not** speaking (`!TurnManager::is_speaking()`) — cuing over the
agent's own TTS would be an interruption, not a backchannel.
agent's own TTS would be an interruption, not a backchannel (checked by the
caller);
- the user is in a **micro-pause**: `VadDetector::silence_ms()` reached the
per-pause reaction depth, a random value drawn in `[BACKCHANNEL_PAUSE_MIN_MS,
BACKCHANNEL_PAUSE_MAX_MS]`, so the cue never lands mid-word *and* doesn't
always react at the same depth. A zero-length pause (voice ongoing) re-arms
the gate;
- the first cue is due (a per-turn randomized instant,
`BACKCHANNEL_MIN_SPEECH_MS + 0..BACKCHANNEL_JITTER_MS` after the speech start);
- a randomized `BACKCHANNEL_INTERVAL_MS ± JITTER/2` gap elapsed since the
previous cue;
- fewer than `BACKCHANNEL_MAX_PER_TURN` cues were emitted this turn;
- the **probabilistic gate** (`BACKCHANNEL_PROB`) does not skip this pause — so
the agent acknowledges some pauses but not all.
### Naturalness
Six mechanisms keep the cues from sounding like a metronome:
| Mechanism | How |
|---|---|
| Cue on micro-pauses | gated on the VAD `silence_ms()` (a `PAUSE_MIN..PAUSE_MAX` window, 32 ms steps, capped below `VAD_MIN_SILENCE_MS`), not on a timer |
| Randomized reaction depth | per pause, the depth to react at is drawn at random in `[PAUSE_MIN, PAUSE_MAX]`, so the cue doesn't always land at exactly `PAUSE_MIN` |
| Randomized timing | the first cue and every following gap are jittered by `BACKCHANNEL_JITTER_MS` |
| Probabilistic gate | `BACKCHANNEL_PROB` skips a share of the qualifying pauses |
| Anti-repetition | the last two clip names are excluded from the pick, across turns |
| Volume drift | each cue is scaled by a random ±`BACKCHANNEL_VOLUME_JITTER` factor |
The phrase list is further grouped in **categories**: the scheduler prefers a
category other than the previous cue's, then a category not yet used this turn,
so a turn tends to start with an acknowledgment (`ack`) and drift toward
agreement (`agree`) instead of repeating the same kind of nod.
### Mixing
@@ -594,13 +814,90 @@ The cue is pushed into the mixer's dedicated `backchannel_queue` and summed at a
fixed low gain (`BACKCHANNEL_GAIN`, default `0.35`):
```text
mixed = tts + (filler × filler_gain) + (backchannel × BACKCHANNEL_GAIN)
mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN)
```
The queue is included in `AudioMixer::flush()`, so a barge-in clears any pending
cue along with the TTS and filler audio. Logs live under the
`audio_agent_engine::orchestrator::backchannel` target (`INFO` when a cue is
emitted).
cue along with the TTS and filler audio. At session start the effective policy is
logged under `audio_agent_engine::orchestrator::backchannel`
(`backchannel scheduler ready …`, with the parsed categories), and every emitted
cue is logged under `audio_agent_engine::ws::session`
(`backchannel emitted during user speech`, with the clip name and the pause
length).
---
## Multilingual clip banks
The filler, start-filler and backchannel banks are pre-synthesized in a single
language — the base language `LANG_DEFAULT` (default `es`). When the user is
detected speaking a different language, the engine generates that language's
banks on demand and switches the session to them, **without touching** the clips
of any other language.
### Layout
Every language — **including the base `LANG_DEFAULT`** — lives in its own
subdirectory named after its ISO 639-1 code under `FILLERS_DIR` /
`START_FILLER_DIR` / `BACKCHANNEL_DIR`:
```text
assets/backchannel/es/ <- base language (LANG_DEFAULT), pre-synthesized
assets/fillers/es/
assets/start_filler/es/
assets/backchannel/en/ <- English, generated on demand
assets/backchannel/it/ <- Italian, generated on demand
assets/fillers/en/
assets/start_filler/en/
```
Because the refresh pass (`remove_stale_clips`) only prunes `*.raw` files inside
the directory it is given, scoping each language — base included — to its own
subdirectory is what guarantees a regeneration never deletes another language's
clips. No language is special-cased to the top level.
### Phrase source
Phrases come from built-in tables (Spanish, English, Italian, French, German and
Portuguese) in [`src/lang.rs`](src/lang.rs) — one table for the backchannel, one
for the end-of-turn filler and one for the start filler; for any other language
they are generated with the LLM (`LANG_LLM_FALLBACK_ENABLED=true`). Each
language's list is then synthesized through the same
[`ensure_text_bank_clips`](src/assets.rs) pipeline used at startup, so an
already-generated language is only reloaded (its `texts.txt` index matches) and
never re-synthesized.
### Detection & switching
- A dedicated, short LLM call returns only the ISO 639-1 code
([`lang::detect_language`](src/lang.rs)); it runs on the first user turn and
then once every `LANG_REDETECT_EVERY` turns (default `5`), so a
mid-conversation switch is picked up without paying for a detection call on
every turn.
- Detection only runs when `LANG_DETECT_ENABLED=true`, the LLM is present, at
least one language-aware cue (backchannel / filler / start filler) is enabled
and the transcript is at least `LANG_MIN_CHARS` long (shorter samples are too
noisy to classify).
- If the detected code equals the base language the session falls back to the
startup banks; otherwise the language's clips are ensured/loaded into a new
bank and the session switches to it.
- The active language is shared between the consumers through an
`Arc<LanguageState>` ([`src/lang.rs`](src/lang.rs)): the end-of-turn and start
fillers ([`TurnManager`](src/orchestrator/turn_manager.rs)) read the active
bank per turn, while the backchannel scheduler is re-pointed via
[`BackchannelScheduler::retarget`](src/orchestrator/backchannel.rs) whenever
the language generation counter changes. Detection runs in the background, so
it never blocks the reply — it takes effect from the following turn.
### Configuration
| Variable | Default | Purpose |
|---|---|---|
| `LANG_DEFAULT` | `es` | Language of the startup filler / start-filler / backchannel banks (each in its own `<dir>/<lang>` subdirectory) |
| `LANG_DETECT_ENABLED` | `true` | Master switch for LLM language detection |
| `LANG_MIN_CHARS` | `12` | Minimum transcript length before detecting |
| `LANG_REDETECT_EVERY` | `5` | Re-check the language every N user turns |
| `LANG_LLM_FALLBACK_ENABLED` | `true` | Generate phrases via LLM for languages absent from the tables |
---
@@ -645,8 +942,9 @@ chat box ──────────────┘
the **first** chunk is cut once `LLM_STREAM_FIRST_CHUNK_CHARS` (default `20`)
characters are buffered — so the agent starts speaking on the first few words
instead of waiting for a whole sentence — and every following chunk at
`LLM_STREAM_CHUNK_CHARS` (default `120`). Set either to `0` to disable that
flush (chunks then end on sentence punctuation only).
`LLM_STREAM_CHUNK_CHARS` (default `240`). Set either to `0` to disable that
flush (chunks then end on sentence punctuation only). Larger follow-up chunks
mean fewer TTS requests, hence fewer chunk seams to starve the queue.
- **Pipelined synthesis** — chunks are *not* synthesized one after another. Each
chunk is requested as soon as the LLM produces it (up to `TTS_PIPELINE_DEPTH`,
default `3`, in flight at once) and streams into its own staging buffer; a
@@ -666,6 +964,145 @@ chat box ──────────────┘
---
## Tool calling
With `LLM_TOOLS_FILE` pointing at a JSON file, the model can call external HTTP
services before answering. Tools are plain descriptions of an HTTP request, so
any service (weather API, webhook, home automation, internal REST endpoint) can
be exposed without touching the code:
```json
[
{
"name": "get_weather",
"description": "Current weather and temperature for a city.",
"parameters": {
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
},
"url": "https://wttr.in/{city}?format=3",
"method": "GET",
"headers": { "X-Api-Key": "…" }
}
]
```
- `url` may contain `{arg}` placeholders substituted from the call arguments.
- `GET` sends the arguments that were **not** consumed by a placeholder as query
parameters; `POST` (the default) sends them as a JSON body.
- `headers` are optional and sent verbatim (API keys, etc.).
See [`tools.example.json`](tools.example.json) for a ready-to-use file.
### Why a client-side parser
The deployment's OpenAI-compatible server (vLLM) runs **without** a tool parser
(`--enable-auto-tool-choice --tool-call-parser …`), so it never returns
structured `tool_calls` or `finish_reason: "tool_calls"`. The tools are still
advertised in every request (`tools` + `tool_choice: "auto"`), so the Qwen model
knows about them and emits the call **as text** in its native format:
```text
<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>
```
[`src/tools.rs`](src/tools.rs) parses that format client-side and works with any
server, no server-side reconfiguration needed.
### Turn loop
Each turn runs a bounded loop (`LLM_TOOL_MAX_ITERATIONS`, default `4`):
1. Stream one reply. A `ToolCallStripper` removes `<tool_call>` blocks from the
streamed text — a block that is still incomplete is **held back**, so it is
never spoken by the TTS nor shown in the chat box even when split across
deltas. The visible text is spoken as usual, so the assistant can say
something short ("Let me check…") before calling.
2. If the round produced no tool call, the turn ends.
3. Otherwise the assistant message (with its native call) plus one `tool` role
message per result are appended, and the loop re-prompts the model, which can
then answer using the results.
Tool errors (unknown tool, transport failure, non-2xx status) are returned to
the model as text, so it can recover and still produce an answer; the turn never
fails because of a tool. Only the *spoken* text is written to the rolling
history — the tool exchange stays local to the turn.
The `tool_call` event is also forwarded to the client for observability:
```json
{ "type": "tool_call", "name": "get_weather", "arguments": { "city": "Paris" } }
```
---
## External audio injection
The mixer has a dedicated **external channel** (`external_queue`) that plays an
audio source injected from the client — a direct audio URL, an Icecast/Shoutcast
radio stream, an HLS playlist, or a YouTube link — mixed together with the
agent's voice. It is implemented by [`src/media.rs`](src/media.rs)
(`MediaSource`) and driven over the WebSocket by the `audio_source` message
(see [WebSocket Protocol](#websocket-protocol)). The Mivocero client exposes it
in its footer bar.
### Why `ffmpeg` / `yt-dlp`
The channel does not decode audio itself: it spawns an external decoder process
whose raw PCM output feeds the ring buffer.
- **`ffmpeg`** (URL / radio / HLS) — reads the source and writes
`-f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1`, i.e. exactly the mixer's
format (44.1 kHz, 16-bit LE, mono). URL inputs use `-reconnect 1
-reconnect_streamed 1 -reconnect_delay_max 5` so a dropped radio connection is
retried instead of ending the stream.
- **`yt-dlp`** (YouTube) — used only to *resolve* a YouTube page to a direct
media URL (`-f bestaudio/best -g --no-playlist`); that URL is then handed to
`ffmpeg`. YouTube-specific cookies/extractor tweaks go in `MEDIA_YTDLP_ARGS`
(e.g. `--cookies /path/cookies.txt` if the media server asks to "confirm
you're not a bot").
Both binaries are configurable (`MEDIA_FFMPEG_BIN`, `MEDIA_YTDLP_BIN`) and are
only required when the external channel is actually used.
### Backpressure
A live radio always produces audio at real time, but a bounded VOD source (a
YouTube video, a file) decodes far faster than real time. The decode task
respects the channel's `remaining_capacity()` and sleeps 20 ms whenever the
queue is full, so a fast source is throttled to the mixer's drain rate instead
of overflowing the ring buffer (which would drop the oldest samples).
### Independence from the agent
The external channel is deliberately **not** cleared by `AudioMixer::flush()`:
a barge-in stops the *agent* (TTS + fillers), not the background music. It also
does not count towards `AudioMixer::is_active()` — the agent is not considered
to be "speaking" because the music is playing. Its volume is a runtime atomic
(`AudioMixer::set_external_gain`) so the client slider takes effect on the next
mixed frame.
### Lifecycle
| Step | What happens |
|---|---|
| `play` | Any current source is stopped; state `loading` is emitted. For a YouTube URL `yt-dlp` resolves the direct URL first. |
| ready | `ffmpeg` is spawned and reading begins; state `playing` is emitted. |
| streaming | Decoded PCM is pushed into `external_queue` under backpressure. |
| `stop` / new `play` / disconnect | The `CancellationToken` is cancelled and the child processes are killed (`kill_on_drop(true)`); `stopped` (or `error`) is emitted. |
A monotonic generation counter guards against a stale task clearing the state of
a newer source.
---
## Mivocero chat client
A ready-to-use speech-to-speech example client is served at **`/mivocero`**
@@ -685,12 +1122,77 @@ A ready-to-use speech-to-speech example client is served at **`/mivocero`**
without using the microphone (`{"type":"user_text","text":…}`).
- **Interrupt button** — flushes audio and cancels the in-flight reply
(`{"type":"interrupt"}`).
- **External audio bar** — footer controls to inject a background source by URL
(direct audio / radio / YouTube), with **Play** / **Stop** buttons, a volume
slider (0–150 %, sent as `EXTERNAL_AUDIO_GAIN` × slider) and a live state
label driven by the server's `audio_source` events. Demonstrates the
[external audio channel](#external-audio-injection).
- **Cue indicators** — a row of three tiles (**Filler**, **Start filler**,
**Backchannel**) that flash and bump a session counter each time the server
emits the matching cue, driven by the `{"type":"cue"}` events. The detail line
shows the backchannel category (or the clip name), so the emission cadence of
the latency masks is visible live. Counters reset on each connect.
- **Cue methods panel** — a collapsible **Cue methods** section below the tiles
that documents the method used by each cue (when it fires, how a clip is
chosen) and shows the **live parameters** reported by the server in the
`{"type":"cues"}` event sent at session start: filler delay / gains / clip
count, start-filler go-delay, and the backchannel gain, min-speech, interval,
jitter, pause window, probability, volume jitter, per-turn cap and categories
— all read from the effective runtime configuration (no hard-coded values).
- **Idle auto-disconnect** — the session closes itself after **2 minutes**
(`IDLE_TIMEOUT_MS`) without any traffic in either direction. The timer is
reset by every server event *and* audio frame received, and by every user
action sent (typed message, Stop, media controls) — so an ongoing reply or a
playing external source counts as activity. On expiry the WebSocket is closed,
the microphone is released, an explanatory message is appended to the chat and
the **Connect** button becomes available again.
Open `http://localhost:8787/mivocero` (or `https://…` for remote microphone
access — see [HTTPS](#https-microphone-access)).
---
## Session audit log
Every WebSocket session can be recorded to its own structured **JSONL** file —
one JSON object per line — under `SESSION_AUDIT_DIR` (default `logs/sessions`),
named `s-<epoch_ms>-<seq>.jsonl`. It is meant for post-mortem analysis and for
tuning the latency knobs (why a filler fired, when a cue landed, how long the
first audio took).
Each record carries a wall-clock timestamp, the elapsed time since the session
started, an event discriminator and a payload:
```json
{"ts":1729170000000,"elapsed_ms":12,"event":"session_start","data":{"config":{…},"runtime":{…}}}
{"ts":1729170000123,"elapsed_ms":135,"event":"control","data":{"type":"user_text","text":"…"}}
{"ts":1729170000456,"elapsed_ms":468,"event":"out","data":{"type":"assistant_delta","text":"Hola,"}}
{"ts":1729170000500,"elapsed_ms":512,"event":"out","data":{"type":"cue","kind":"start_filler","name":"sf02"}}
{"ts":1729170009000,"elapsed_ms":9012,"event":"session_end","data":{"reason":"client_disconnected","counts":{…}}}
```
| `event` | When | `data` |
|---|---|---|
| `session_start` | Session opens | `config` — the full non-secret configuration snapshot; `runtime` — which of STT / VAD / tokenizer / LLM are actually active (plus the LLM model). |
| `control` | `user_text` control message received | The text turn. |
| `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. |
| `interrupt` | Client barge-in (`interrupt` JSON or legacy `INTERRUPT`) | `source` (`client` / `client_legacy`). |
| `barge_in` | VAD-driven barge-in | `prob` (onset speech probability). |
| `vad_speech_start` / `vad_speech_end` | Utterance boundaries | `prob`; `duration_ms`, `avg_prob`. |
| `media` | External audio control | `action` (`play`/`stop`/`volume`) with `url` / `gain`. |
| `reference` | Voice-cloning reference change | `action` (`set`/`clear`) and the audio path. |
| `speak` | Direct TTS request (playground / fallback text) | `text`. |
| `language_switch` | Detected language changed | `from`, `to`, `generated` (whether clip banks were synthesized). |
| `error` | Transport / receiver error | `where`, `text`. |
| `session_end` | Session closes | `reason`, `duration_ms`, `session_id`, `file`, `counts` (per-event totals). |
The sink is **best-effort**: if the file cannot be opened, or a write fails, the
session runs unchanged (the failure is logged once in the tracing output and the
sink then goes quiet). Files are appended and flushed after every record, so the
log survives a crash or a `pm2 restart`. The directory is git-ignored.
---
## WebSocket Protocol
Connect to `ws://<host>:8787/ws`.
@@ -702,6 +1204,9 @@ Connect to `ws://<host>:8787/ws`.
| **Binary** | Raw 16-bit LE mono PCM microphone frames (44.1 kHz). Resampled to 16 kHz server-side for STT. |
| **Text — `{"type":"user_text","text":"…"}`** | Text turn from the chat box: runs the same LLM → TTS path as a spoken utterance (no STT). |
| **Text — `{"type":"interrupt"}`** | Barge-in: cancel the in-flight LLM reply / TTS and flush all queued audio. |
| **Text — `{"type":"audio_source","action":"play","url":"…"}`** | Inject an external audio source (direct URL / radio / HLS / YouTube) into the mixer's external channel, replacing any current one. |
| **Text — `{"type":"audio_source","action":"stop"}`** | Stop the external source and clear the channel. |
| **Text — `{"type":"audio_source","action":"volume","gain":0.6}`** | Set the external channel gain (0..1) at runtime. |
| **Text — `INTERRUPT`** *(also `interrupt`, `BAARGEIN`, `bargein`)* | Legacy barge-in form (kept for the playground). |
| **Text — `SET_REFERENCE:<path>\|<text>`** | Set/change the voice-cloning reference. `<text>` may be omitted to re-use the sibling `.txt`. |
| **Text — `CLEAR_REFERENCE`** | Disable voice cloning. |
@@ -715,7 +1220,11 @@ Connect to `ws://<host>:8787/ws`.
| **Text — `{"type":"user","text":"…"}`** | The user's turn (STT transcript or chat-box message) — rendered in the chat box. |
| **Text — `{"type":"assistant_delta","text":"…"}`** | Streaming chunk of the assistant reply (chain-of-thought already removed). |
| **Text — `{"type":"assistant_end"}`** | The assistant turn is complete. |
| **Text — `{"type":"tool_call","name":"…","arguments":{…}}`** | The model called a tool; the call has been executed (result is not echoed in the chat box, only in the model's next round). Purely informational. |
| **Text — `{"type":"error","text":"…"}`** | LLM request failed; the turn was aborted. |
| **Text — `{"type":"audio_source","state":"…","url":"…"}`** | External audio channel state change: `loading` (decoding is being set up / YouTube URL resolving), `playing` (PCM is streaming), `stopped`, or `error` (with a `text` message). |
| **Text — `{"type":"cue","kind":"…","name":"…","category":"…"}`** | A latency-mask / backchannel cue was actually emitted into the mixer: `kind` is `filler` (end-of-turn), `start_filler` (turn start) or `backchannel` (during user speech); `name` is the clip name and `category` (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators. |
| **Text — `{"type":"cues","lang":"…","config":{…}}`** | Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (`delay_ms`, gains, clip count), the start-filler `go_delay_ms` and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's **Cue methods** panel so it shows the live values instead of hard-coded ones. |
---
@@ -736,23 +1245,32 @@ Constants are defined in [`src/config.rs`](src/config.rs):
### Mixer & jitter buffering ([`src/audio/mixer.rs`](src/audio/mixer.rs))
- Mixing formula:
`mixed = tts + (filler × filler_gain) + (backchannel × BACKCHANNEL_GAIN)`.
`mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) + (external × external_gain)`.
The filler gain is `1.0` when TTS is idle, else `filler_gain_duck` (default
`0.15`); the backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
Output is clamped to `i16`.
The start-filler channel is summed with the filler, so it shares the same gain.
The external channel gain starts at `EXTERNAL_AUDIO_GAIN` (default `0.6`) and
is adjustable at runtime (`AudioMixer::set_external_gain`). Output is clamped
to `i16`.
- **Pre-buffer:** TTS playback does not start until the queue holds
≥ `TTS_PREBUFFER_MS` (default **450 ms**) of synthesized audio. It absorbs
≥ `TTS_PREBUFFER_MS` (default **550 ms**) of synthesized audio. It absorbs
uneven TTS chunk arrival; lower it to start speaking sooner, raise it for fewer
drop-outs. With pipelined synthesis (above) the pre-buffer only has to cover
arrival jitter, not the TTS engine's per-request start-up, so it can stay
small. Note: Drama-3 delivers ~1.04× real time, and its *granularity* is set by
the server's vocoder window
(`--vocoder.factory.stream_followup_stride`, in codec frames of 2048 samples =
46.4 ms). With stride `16` and ramp `"[10,12,16]"` the first window is 10
frames = 0.464 s, so `450 ms` starts playback on that very first window (the
steady state then delivers a 0.743 s burst every ~0.69 s). With the upstream
default (`64`, a 2.97 s burst every ~2.7 s) gapless playback would instead
require a ~3 s pre-buffer.
drop-outs. Note: Drama-3's *granularity* is set by the server's vocoder window
(`--vocoder.factory.stream_stride` / `stream_followup_stride`, in codec frames
of 2048 samples = 46.4 ms). With the deployment used here (stride `4`, followup
`8`, overlap `16`) the first window is 4 frames = 0.186 s and the steady windows
are 8 frames = 0.372 s, delivered ~0.35 s apart (the server runs at ~0.95× real
time). Because `TTFA ≈ TTFB + (prebuffer − window)`, any pre-buffer in
`(186, 557]` ms primes on the **same** second window → TTFA ≈ `0.73 s`; the
default `550 ms` takes that whole cushion at no extra latency. A pre-buffer
below ~`186 ms` starts playback on the first window (TTFA ≈ `0.38 s`) but leaves
almost no margin: under concurrent TTS requests one GPU serves several requests
and the per-request throughput dips below real time (~`1.5×` vs ~`0.94×` alone),
so every chunk seam starves the queue and pauses playback → an audible
micro-gap at the start of each sentence. With the upstream default (`64`, a
2.97 s burst every ~2.7 s) gapless playback would instead require a ~3 s
pre-buffer; a large constant stride (`16`) both raises the TTFB (~1.2 s) and
bursts, and a growing ramp gaps the tail while the small early windows drain.
- **Resilient resume:** the full pre-buffer is only required at the *first*
playback of a reply. If the queue empties mid-reply (after the 200 ms drain
grace) playback pauses, but resumes as soon as *any* audio is queued again —
@@ -774,13 +1292,15 @@ audio-agent-engine/
├── server.sh # start / stop / restart / status / logs / cert
├── certs/ # self-signed TLS cert+key (generated)
├── UPDATES.md # changelog (English, newest-first)
├── assets/fillers/ # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin)
├── assets/backchannel/ # *.raw backchannel PCM (16-bit LE mono, 44.1 kHz)
├── tools.example.json # tool definitions for LLM tool calling (copy/edit)
├── assets/fillers/<lang>/ # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin), one dir per language
├── assets/start_filler/<lang>/ # sf01.raw … turn-start start-filler clips + texts.txt index
├── assets/backchannel/<lang>/ # bc01.raw … backchannel clips + texts.txt index
├── models/ # Parakeet ONNX + Silero VAD + tokenizer
└── src/
├── main.rs # Axum server, routes, graceful shutdown
├── config.rs # env-var config + audio constants
├── assets.rs # startup clip bootstrap (auto-generate fillers/backchannel)
├── assets.rs # startup clip bootstrap (auto-generate fillers/backchannel/start filler)
├── tls.rs # optional HTTPS/WSS termination (rustls)
├── onnx.rs # shared process-wide ONNX Runtime environment
├── audio/ # ring_buffer, mixer
@@ -791,6 +1311,9 @@ audio-agent-engine/
├── stt/ # Parakeet ONNX + tokenizer + worker
├── vad/ # Silero VAD v5 + onset/offset state machine
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
├── tools.rs # configurable HTTP tool calling (native <tool_call> parser + executor)
├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
├── audit.rs # per-session JSONL audit log (config snapshot, events, summary)
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
├── ws/ # handler, session (per-connection audio loop)
└── web/ # embedded playground.html + mivocero.html