roomote_PRE/piperust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

step3

Browse Source
This commit is contained in:
valenti
2026-10-17 22:01:31 +01:00
parent bfed42c3b3
commit 79f927d968
53 changed files with 2158 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+214 -46
View File
@@ -38,7 +38,7 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
played round-robin from `FILLERS_DIR`; `FILLER_ENABLED=false` disables it.
- **Start filler** — a second latency mask on its own mixer channel: if the first
audio chunk of the reply has not reached the mixer within
`START_FILLER_GO_DELAY` (default 1200 ms) the agent plays a short
`START_FILLER_GO_DELAY` (default 1800 ms) the agent plays a short
acknowledgment ("Un momento...") while the LLM/TTS are still working, and drops
it as soon as audio arrives in time. One phrase is picked **at random** from
`START_FILLER_TEXTS`, so the cue varies turn to turn. Independent of the delayed
@@ -60,6 +60,12 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
- **Barge-in / interrupt** — cancels in-flight synthesis and flushes audio,
triggered client-side or server-side by the VAD.
- **Voice cloning** — reference audio + transcript (Fish Audio `references`).
- **Custom voice builder** — a dedicated `/customvoice` page (linked from the
Mivocero side menu) asks the user to pick a language, then either read a
predefined phrase for at least 10 s **or upload an audio file with its typed
transcript**; the take is transcoded to a WAV reference and installed as the
active voice at runtime. See
[Custom voice builder](#custom-voice-builder).
- **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to
`TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's
start-up latency behind the previous chunk's audio.
@@ -75,9 +81,10 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
audio and the chat box.
- **Tool calling** — the model can call external HTTP tools described in a JSON
file (`LLM_TOOLS_FILE`) before answering. Calls are parsed client-side from
the model's native `<tool_call>` format, executed over HTTP and fed back to
the model; the tool-call block is never spoken. See
[Tool calling](#tool-calling).
either the structured `tool_calls` array or the model's native `<tool_call>`
format (a round that streams back empty is re-asked without streaming),
executed over HTTP and fed back to the model; the tool-call block is never
spoken. See [Tool calling](#tool-calling).
- **Embedded playground** — a single-page HTML client served at `/`.
- **Mivocero speech-to-speech client** — an example agent client at `/mivocero`
with a microphone, a live chat box (user transcript + streaming assistant
@@ -119,8 +126,9 @@ $EDITOR .env
cargo run --release
# 3. Open the clients
# http://localhost:8787/ playground (TTS / audio test)
# http://localhost:8787/mivocero speech-to-speech agent with chat box
# http://localhost:8787/ playground (TTS / audio test)
# http://localhost:8787/mivocero speech-to-speech agent with chat box
# http://localhost:8787/customvoice custom-voice builder (record / upload a reference)
```
Environment variables can also be `export`ed directly; a `.env` file is loaded
@@ -196,9 +204,11 @@ See [`.env.example`](.env.example) for a copy-paste template.
| `VAD_THRESHOLD` | `0.5` | Speech probability above which a window counts as voice. |
| `VAD_NEG_THRESHOLD` | `0.35` | Probability below which a window counts as silence (hysteresis; clamped to `VAD_THRESHOLD`). |
| `VAD_MIN_SPEECH_MS` | `250` | Speech required to confirm an utterance start. |
| `VAD_MIN_SILENCE_MS` | `300` | Silence required to close an utterance (end-of-utterance window). |
| `VAD_MIN_SILENCE_MS` | `700` | Silence required to close an utterance (end-of-utterance window). |
| `VAD_SPEECH_PAD_MS` | `30` | Audio kept before the detected onset so the first phoneme is not clipped. |
| `VAD_BARGE_IN_MIN_SPEECH_MS` | `150` | Speech required to fire the fast barge-in onset (clamped to `VAD_MIN_SPEECH_MS`). |
| `VAD_MIN_UTTERANCE_MS` | `300` | Utterance gate: drop a segment shorter than this before transcription (`0` disables). |
| `VAD_MIN_AVG_PROB` | `0.25` | Utterance gate: drop a segment whose average speech probability is below this before transcription (`0.0` disables). The average is taken over the **voiced windows only** (probability ≥ `VAD_NEG_THRESHOLD`), so the trailing hangover silence does not dilute short words like "uno" / "due". Kept well below `VAD_THRESHOLD`. |
| `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. |
| `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. |
| `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. |
@@ -206,9 +216,9 @@ See [`.env.example`](.env.example) for a copy-paste template.
| `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. |
| `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. |
| `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. |
| `FILLER_GAIN_DUCK` | `0.15` | Filler gain while TTS is active (ducking). Also applies to the start-filler channel. |
| `FILLER_GAIN_DUCK` | `0.15` | Fallback filler gain while TTS is already active. Normally a still-queued filler is faded out (80 ms) and cut when the reply's playback starts, so this only applies to a cue queued *after* playback has begun. Also applies to the start-filler channel. |
| `START_FILLER_ENABLED` | `true` | Master switch for the **start filler** — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. `false` => no cue is ever queued. |
| `START_FILLER_GO_DELAY` | `1200` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. |
| `START_FILLER_GO_DELAY` | `1800` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. If the cue does fire and the reply then starts, it is faded out (80 ms) and cut so it does not overlap the answer. |
| `START_FILLER_TEXTS` | `[calm] Un momento...` | Phrases spoken by the start filler, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and a **random** one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, `[sigh]`, …) are passed through to the TTS. |
| `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/start_filler/es/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. |
| `TTS_PREBUFFER_MS` | `550` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first `185.8` ms, then `371.6` ms), any value in `(186, 557]` ms primes on the *same* second window, so `550` takes the whole cushion at no extra TTFA (~`0.73 s`). Below the first window (~`186` ms) playback starts on that window (TTFA ≈ server TTFB, ~`0.38 s`) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
@@ -568,7 +578,7 @@ events:
|---|---|---|
| `SpeechOnset` | `VAD_BARGE_IN_MIN_SPEECH_MS` (150 ms) of speech | Fast **barge-in**: interrupt the agent as soon as the user starts talking. |
| `SpeechStart` | `VAD_MIN_SPEECH_MS` (250 ms) of speech | Confirms the utterance start. |
| `SpeechEnd` | `VAD_MIN_SILENCE_MS` (300 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. |
| `SpeechEnd` | `VAD_MIN_SILENCE_MS` (700 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. |
Onset/offset use hysteresis: a window counts as voice above `VAD_THRESHOLD`
and as silence below `VAD_NEG_THRESHOLD`, which prevents flapping around a
@@ -578,12 +588,23 @@ detected onset so the first phonemes are never clipped.
### Barge-in
When `VAD_BARGE_IN=true` and the agent is currently speaking
([`TurnManager::is_speaking`](src/orchestrator/turn_manager.rs), backed by
[`AudioMixer::is_active`](src/audio/mixer.rs)), a `SpeechOnset` calls
When `VAD_BARGE_IN=true` and the agent has **audible** output
([`TurnManager::is_audible`](src/orchestrator/turn_manager.rs), backed by
[`AudioMixer::is_audible`](src/audio/mixer.rs)), a `SpeechOnset` calls
`TurnManager::interrupt()` — cancelling the in-flight TTS via its
`CancellationToken` and flushing the mixer queues.
The gate is on *audible* output, not on "speaking" in general: `is_audible` is
`false` while the reply is still being synthesised or is sitting in the TTS
pre-buffer, i.e. before anything has reached the user's ears. This stops a VAD
transient (a breath, a click, or the tail of the user's own utterance) from
cancelling a reply the user has not heard any of yet. Such a reply is not left
orphaned: the next turn's `on_eou` already cancels the previous in-flight
synthesis and restarts the stream. `TurnManager::is_speaking`
([`AudioMixer::is_active`](src/audio/mixer.rs)) is still used for the
backchannel gate, where cueing on top of *any* in-flight TTS would be an
interruption rather than an acknowledgment.
### Wiring
The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one
@@ -591,7 +612,16 @@ The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one
- with `VAD_SEGMENT_STT=true` (default): forwards `SpeechEnd` utterances to
`SttWorker::push_utterance`, and does **not** forward raw frames (avoids
double segmentation);
double segmentation). Before forwarding, each segment passes the **utterance
gate** (`VAD_MIN_UTTERANCE_MS` / `VAD_MIN_AVG_PROB`): segments below either
threshold are dropped without transcription, which filters the VAD
false-triggers (breaths, clicks, acoustic echo, noise) that the STT would
otherwise turn into hallucinated text. The probability average is computed
over the **voiced windows only** (probability ≥ `VAD_NEG_THRESHOLD`), so the
`VAD_MIN_SILENCE_MS` hangover that closes the segment does not drag it down —
a short word like "uno" would otherwise average far below any usable gate. A
transcript that still decodes to no tokens is skipped too, so empty turns are
never sent to the LLM.
- with `VAD_SEGMENT_STT=false`: falls back to the STT worker's RMS
segmentation, using the VAD only for barge-in;
- if `VAD_ENABLED=false` or the model fails to load: legacy behaviour (RMS
@@ -681,8 +711,11 @@ reached and reset at every `mark_stream_start` / `flush`.
- Raise `FILLER_DELAY_MS` for slower LLM/TTS setups, lower it if the filler
rarely fires.
Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`) and ducks it to
`FILLER_GAIN_DUCK` (`0.15`) if TTS starts mid-clip.
Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`). If TTS playback
starts mid-clip, the cue is **faded out** over `FILLER_FADEOUT_FRAMES`
(4 frames = 80 ms) and then cut — it is a latency mask, so it must hand off
cleanly to the reply instead of playing underneath it. (As a fallback, a cue
queued *after* playback has already started is ducked to `FILLER_GAIN_DUCK`.)
`FILLERS_DIR` (default `assets/fillers`) holds `*.raw` clips — **16-bit LE mono
PCM at 44.1 kHz** — selected **round-robin** (`TurnManager::next_filler` →
@@ -695,7 +728,7 @@ are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)).
A second latency mask, on its own mixer channel. It works exactly like the filler
above, but with its own — shorter — delay
([`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)):
`START_FILLER_GO_DELAY` (default **1200 ms**) after the turn is dispatched, if the
`START_FILLER_GO_DELAY` (default **1800 ms**) after the turn is dispatched, if the
first audio chunk of the reply has **not** reached the mixer yet
(`AudioMixer::tts_audio_arrived` is still `false`) the cue is queued; if audio
arrived within that window, the cue is dropped instead of talking over the
@@ -714,7 +747,8 @@ acknowledgment — only a late one is masked.
It lives on its own mixer channel (`start_filler_queue`), so it plays
independently of the round-robin latency filler above, but it shares the filler
gain, so it ducks to `FILLER_GAIN_DUCK` once the reply starts. Disabled by
gain: when the reply's playback starts, the still-queued cue is faded out
(80 ms) and cut rather than left running under the answer. Disabled by
`START_FILLER_ENABLED=false`.
Its clips live in `START_FILLER_DIR` (default `assets/start_filler`) and are
@@ -995,26 +1029,38 @@ be exposed without touching the code:
See [`tools.example.json`](tools.example.json) for a ready-to-use file.
### Why a client-side parser
### Client-side parsing (and the streaming fallback)
The deployment's OpenAI-compatible server (vLLM) runs **without** a tool parser
(`--enable-auto-tool-choice --tool-call-parser …`), so it never returns
structured `tool_calls` or `finish_reason: "tool_calls"`. The tools are still
advertised in every request (`tools` + `tool_choice: "auto"`), so the Qwen model
knows about them and emits the call **as text** in its native format:
A deployment may or may not run a server-side tool parser. When it does, the
flag has to match the model's native format (`--enable-auto-tool-choice
--tool-call-parser …`). A mismatch is silent and costly: an `hermes` parser
(which expects `<tool_call>{"name":…,"arguments":{…}}</tool_call>`) against a
Qwen3 model that emits the XML form makes the parser throw, so the call is
dropped. In **streaming** the round then ends with an empty reply
(`finish_reason: "stop"`, no content), while the very same request **without**
`stream` returns the call intact.
```text
<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>
```
The engine therefore never relies on the server parser:
[`src/tools.rs`](src/tools.rs) parses that format client-side and works with any
server, no server-side reconfiguration needed.
- Every request advertises `tools` + `tool_choice: "auto"`.
- [`src/tools.rs`](src/tools.rs) parses **both** shapes — the standard
structured `message.tool_calls` array **and** the model's native
`<tool_call>` text:
```text
<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>
```
- A streaming round that comes back **empty** is re-asked once **without
streaming** ([`LlmClient::chat_reply`](src/llm.rs)), where the dropped call
reappears and is parsed from either shape.
Fixing the server (`--tool-call-parser qwen3_coder` handles the Qwen3 XML
format) removes the extra request — the fallback then simply never fires.
### Turn loop
@@ -1024,7 +1070,9 @@ Each turn runs a bounded loop (`LLM_TOOL_MAX_ITERATIONS`, default `4`):
streamed text — a block that is still incomplete is **held back**, so it is
never spoken by the TTS nor shown in the chat box even when split across
deltas. The visible text is spoken as usual, so the assistant can say
something short ("Let me check…") before calling.
something short ("Let me check…") before calling. A round that streams back
**empty** is re-asked once without streaming to recover a call the server
dropped (see above).
2. If the round produced no tool call, the turn ends.
3. Otherwise the assistant message (with its native call) plus one `tool` role
message per result are appended, and the loop re-prompts the model, which can
@@ -1112,16 +1160,24 @@ A ready-to-use speech-to-speech example client is served at **`/mivocero`**
(44.1 kHz, 16-bit LE mono, 20 ms frames).
- **Chat box** — shows the user transcript (from STT) and the assistant reply
**as it streams**, plus messages typed in the composer.
- **Karaoke highlighting** — the assistant bubble is rendered word by word: each
word lights up as it is *spoken*, driven by the audio playhead (the playback
worklet reports the samples it has actually emitted). Words already spoken are
bright, the word currently sounding is accented, and the rest stay dim. The
rate is an estimate (`KARAOKE_CPS`, ~15 chars/s) — it only affects the visual
highlight, never the audio.
- **Karaoke highlighting** — both the assistant and the user bubbles are
rendered word by word. The assistant's words light up as they are *spoken*,
driven by the audio playhead (the playback worklet reports the samples it has
actually emitted): words already spoken are bright, the word currently
sounding is accented, and the rest stay dim. The rate is an estimate
(`KARAOKE_CPS`, ~15 chars/s) — it only affects the visual highlight, never the
audio. The user's words cannot follow the audio (the STT returns the whole
transcript only once the utterance is over), so they are lit by a short
left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word) when the bubble
appears.
- **Text composer** — typing and sending a message runs a full LLM → TTS turn
without using the microphone (`{"type":"user_text","text":…}`).
- **Interrupt button** — flushes audio and cancels the in-flight reply
(`{"type":"interrupt"}`).
- **Barge-in indicator** — a red **Barge-in** pill in the header that flashes and
bumps a session counter whenever the agent is interrupted: either by the
server-side VAD onset (via the `{"type":"barge_in"}` event) or by the local
**Stop** button. The counter resets on each connect.
- **External audio bar** — footer controls to inject a background source by URL
(direct audio / radio / YouTube), with **Play** / **Stop** buttons, a volume
slider (0–150 %, sent as `EXTERNAL_AUDIO_GAIN` × slider) and a live state
@@ -1139,6 +1195,8 @@ A ready-to-use speech-to-speech example client is served at **`/mivocero`**
count, start-filler go-delay, and the backchannel gain, min-speech, interval,
jitter, pause window, probability, volume jitter, per-turn cap and categories
— all read from the effective runtime configuration (no hard-coded values).
- **Side menu** — the ☰ button in the header opens a drawer that navigates
between the chat and the [custom voice builder](#custom-voice-builder).
- **Idle auto-disconnect** — the session closes itself after **2 minutes**
(`IDLE_TIMEOUT_MS`) without any traffic in either direction. The timer is
reset by every server event *and* audio frame received, and by every user
@@ -1152,6 +1210,106 @@ access — see [HTTPS](#https-microphone-access)).
---
## Custom voice builder
A dedicated page served at **`/customvoice`** (embedded from
[`src/web/customvoice.html`](src/web/customvoice.html), linked from the Mivocero
side menu) lets anyone build a voice-cloning reference from their own voice,
without touching the filesystem or the `.env`. Two sources are offered:
- **Record now** — read a predefined phrase out loud (the microphone flow).
- **Upload a file** — pick an audio file and **type its transcript** (any
container `ffmpeg` can read: wav, mp3, m4a, ogg, flac…).
The flow:
1. **Language** — a selector populated from `GET /api/customvoice/phrases`
(the languages of [`crate::lang`](src/lang.rs); the default is
`LANG_DEFAULT`).
2. **Source** — pick **Record now** or **Upload a file**.
- **Record now** — the page shows the phrase to read for the selected
language. The phrases live **server-side**
([`src/customvoice.rs`](src/customvoice.rs)), so the transcript attached to
the reference always matches the text the user was asked to read. The
browser captures the microphone with `MediaRecorder` (Opus/WebM where
supported); the **Stop** button stays disabled until **10 seconds** have
elapsed (a progress bar and a timer show the way to the minimum), and
recording auto-stops at 20 s. The take can be played back and discarded
before use.
- **Upload a file** — choose an audio file (`accept="audio/*"`) and type (or
paste) the exact text spoken in it. Here the transcript is **required**: it
is what drives the clone, so it must match the audio.
3. **Use it** — `POST /api/customvoice?lang=<code>[&text=<transcript>]` with the
recording or the uploaded file as the body. The `text` parameter carries the
upload flow's typed transcript and overrides the predefined phrase; when it
is absent or blank the language's predefined phrase is used (so a plain
recording needs no text). The server **post-processes** the take (see below),
transcodes it to **mono 16-bit PCM WAV at `OUTPUT_SAMPLE_RATE`** (44.1 kHz)
with `ffmpeg`, writes it next to the configured `TTS_REFERENCE_AUDIO` (the
directory the Fish Audio backend can read — hence "the path on the TTS
server" note in [Configuration](#configuration)), writes the transcript as a
sibling `<name>.txt` (the convention [`Config::tts_reference`](src/config.rs)
already falls back to), and installs the voice through
[`TtsEngine::set_reference`](src/tts/mod.rs) — effective immediately for new
replies, no restart.
4. **Reset** — `POST /api/customvoice/clear` restores the reference configured
at startup (`TTS_REFERENCE_AUDIO`), or clears it when none is configured.
### Post-processing
A raw browser recording is uneven: it carries rumble and room noise, and its
level depends on how far the user sat from the microphone. The take therefore
goes through a cleanup chain before it is installed
([`src/customvoice.rs`](src/customvoice.rs)):
| Stage | Filter | Why |
|---|---|---|
| Mono downmix | `aformat=channel_layouts=mono` | Everything downstream (and the reference itself) is mono. Done **first** so loudnorm measures the signal that is actually written. |
| Rumble / DC removal | `highpass=f=80` | Kills desk thumps, plosives and DC offset below the voice band. |
| Noise reduction | `afftdn=nr=12:nf=-40` | FFT denoise for steady room/hiss noise. |
| De-essing | `deesser` | Tames harsh sibilance that a clone would otherwise exaggerate. |
| Silence trim | `silenceremove` (+ `areverse`) | Drops leading/trailing silence so the reference starts on speech. |
| Loudness | `loudnorm`, **two-pass, linear** | Matches every take to the same programme level. |
The loudness stage is deliberately **two-pass**: ffmpeg measures the cleaned
signal first, then applies the measurement as **one constant gain**
(`linear=true`). A single gain cannot pump or ride the level frame by frame —
which is exactly what makes takes differ in volume.
The default target is **-26 LUFS** integrated with a **-5 dBTP** true-peak
ceiling. Both sit *below* the shipped reference (`voice_profiles/solis.wav`
measures ≈-23.4 LUFS at -7.8 dBTP): a user take is far more dynamic than that
mastered reference, so at equal LUFS its much hotter peaks make the cloned
voice sound louder than the default one. The lower target and the extra peak
headroom buy that gap back, so a custom voice sits a touch under the default
instead of on top of it. A take recorded 30 dB too quiet and one recorded hot
both land on the same level.
If any stage fails (a filter quirk, an unreadable take, an all-silence
recording) the endpoint logs a warning and **falls back to the plain
transcode** — a take is never lost to post-processing.
| Env var | Default | Description |
|---|---|---|
| `CUSTOMVOICE_POSTPROCESS` | `true` | Master switch; `false` stores the plain transcode. |
| `CUSTOMVOICE_DENOISE` | `true` | The `afftdn` + `deesser` stage — turn off for a clean studio take. |
| `CUSTOMVOICE_TARGET_LUFS` | `-26` | Integrated loudness target, below the shipped reference. Lower it for a quieter custom voice. |
| `CUSTOMVOICE_TRUE_PEAK` | `-5` | True-peak ceiling (dBTP). |
| Method | Route | Description |
|---|---|---|
| `GET` | `/customvoice` | The builder page. |
| `GET` | `/api/customvoice/phrases` | Reading phrases + the default language (JSON). |
| `POST` | `/api/customvoice?lang=<code>[&text=<transcript>]` | Body = recording or uploaded file; transcodes, persists (with the transcript) and installs the reference. |
| `POST` | `/api/customvoice/clear` | Restore the configured reference. |
The body cap for the upload route is raised to 32 MB (the axum default is 2 MB)
so an uncompressed WAV is accepted as well as a compressed capture. Microphone
access requires a secure context: use `https://…` when the page is not opened
from `localhost` (see [HTTPS](#https-microphone-access)).
---
## Session audit log
Every WebSocket session can be recorded to its own structured **JSONL** file —
@@ -1175,10 +1333,11 @@ started, an event discriminator and a payload:
|---|---|---|
| `session_start` | Session opens | `config` — the full non-secret configuration snapshot; `runtime` — which of STT / VAD / tokenizer / LLM are actually active (plus the LLM model). |
| `control` | `user_text` control message received | The text turn. |
| `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. |
| `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `barge_in`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. |
| `interrupt` | Client barge-in (`interrupt` JSON or legacy `INTERRUPT`) | `source` (`client` / `client_legacy`). |
| `barge_in` | VAD-driven barge-in | `prob` (onset speech probability). |
| `vad_speech_start` / `vad_speech_end` | Utterance boundaries | `prob`; `duration_ms`, `avg_prob`. |
| `utterance_dropped` | VAD segment filtered by the utterance gate (no STT, no LLM turn) | `duration_ms`, `avg_prob`, `too_short`, `too_soft`. |
| `media` | External audio control | `action` (`play`/`stop`/`volume`) with `url` / `gain`. |
| `reference` | Voice-cloning reference change | `action` (`set`/`clear`) and the audio path. |
| `speak` | Direct TTS request (playground / fallback text) | `text`. |
@@ -1225,6 +1384,7 @@ Connect to `ws://<host>:8787/ws`.
| **Text — `{"type":"audio_source","state":"…","url":"…"}`** | External audio channel state change: `loading` (decoding is being set up / YouTube URL resolving), `playing` (PCM is streaming), `stopped`, or `error` (with a `text` message). |
| **Text — `{"type":"cue","kind":"…","name":"…","category":"…"}`** | A latency-mask / backchannel cue was actually emitted into the mixer: `kind` is `filler` (end-of-turn), `start_filler` (turn start) or `backchannel` (during user speech); `name` is the clip name and `category` (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators. |
| **Text — `{"type":"cues","lang":"…","config":{…}}`** | Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (`delay_ms`, gains, clip count), the start-filler `go_delay_ms` and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's **Cue methods** panel so it shows the live values instead of hard-coded ones. |
| **Text — `{"type":"barge_in","source":"vad","prob":…}`** | The agent was interrupted. Sent for the server-side VAD barge-in (the browser has no other way to know, since the onset is detected in the session task); the client-side `interrupt` does not echo one. Drives the demo's barge-in indicator. |
---
@@ -1246,8 +1406,11 @@ Constants are defined in [`src/config.rs`](src/config.rs):
- Mixing formula:
`mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) + (external × external_gain)`.
The filler gain is `1.0` when TTS is idle, else `filler_gain_duck` (default
`0.15`); the backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
The filler gain is `1.0` when TTS is idle; when the reply's playback starts a
still-queued filler/start-filler is instead **faded out** over 80 ms
(`FILLER_FADEOUT_FRAMES`) and cut, so it hands off to the answer (a cue queued
after playback has already begun falls back to `filler_gain_duck`, default
`0.15`). The backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
The start-filler channel is summed with the filler, so it shares the same gain.
The external channel gain starts at `EXTERNAL_AUDIO_GAIN` (default `0.6`) and
is adjustable at runtime (`AudioMixer::set_external_gain`). Output is clamped
@@ -1313,10 +1476,11 @@ audio-agent-engine/
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
├── tools.rs # configurable HTTP tool calling (native <tool_call> parser + executor)
├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
├── customvoice.rs # custom-voice builder endpoints (record → WAV reference → set_reference)
├── audit.rs # per-session JSONL audit log (config snapshot, events, summary)
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
├── ws/ # handler, session (per-connection audio loop)
└── web/ # embedded playground.html + mivocero.html
└── web/ # embedded playground.html + mivocero.html + customvoice.html
```
### HTTP routes
@@ -1325,6 +1489,10 @@ audio-agent-engine/
|---|---|
| `GET /` | Playground single-page client (embedded at compile time). |
| `GET /mivocero` | Mivocero speech-to-speech example client with chat box (embedded at compile time). |
| `GET /customvoice` | Custom-voice builder page (record or upload a reference — see [Custom voice builder](#custom-voice-builder)). |
| `GET /api/customvoice/phrases` | Reading phrases per language + the default language (JSON). |
| `POST /api/customvoice?lang=<code>[&text=<transcript>]` | Upload a recording or an audio file (with its transcript); transcode it to a WAV reference and install it as the active voice. |
| `POST /api/customvoice/clear` | Restore the reference configured at startup. |
| `GET /health` | Liveness probe (`ok`). |
| `GET /ws` | WebSocket upgrade → per-session audio loop. |