step3
This commit is contained in:
valenti
+30
-5
@@ -39,6 +39,19 @@ TTS_TOP_K=
|
||||
# are left untouched. Empty disables the prefix.
|
||||
TTS_STYLE_TAG="[conversational, casual]"
|
||||
|
||||
# --- Custom voice post-processing (/customvoice page) ---
|
||||
# A freshly recorded reference is cleaned and level-matched before it is
|
||||
# installed. Disable the whole chain to store the plain transcode.
|
||||
CUSTOMVOICE_POSTPROCESS=true
|
||||
# Noise reduction (FFT denoise) + de-essing. Turn off for a clean studio take.
|
||||
CUSTOMVOICE_DENOISE=true
|
||||
# Integrated loudness target (LUFS) and true-peak ceiling (dBTP). Every take
|
||||
# lands on the same level, so recordings never jump in volume. Kept below the
|
||||
# shipped reference (solis.wav measures ≈-23.4 LUFS): a user take is far more
|
||||
# dynamic, so at equal LUFS its hotter peaks make the cloned voice sound louder.
|
||||
CUSTOMVOICE_TARGET_LUFS=-26
|
||||
CUSTOMVOICE_TRUE_PEAK=-5
|
||||
|
||||
# --- Kokoro (TTS_ENGINE=kokoro) ---
|
||||
KOKORO_URL=http://localhost:8880
|
||||
KOKORO_VOICE=af_heart
|
||||
@@ -64,11 +77,22 @@ VAD_NEG_THRESHOLD=0.35
|
||||
# Speech required (ms) to confirm an utterance start.
|
||||
VAD_MIN_SPEECH_MS=250
|
||||
# Silence required (ms) to close an utterance (end-of-utterance window).
|
||||
VAD_MIN_SILENCE_MS=300
|
||||
VAD_MIN_SILENCE_MS=700
|
||||
# Audio kept (ms) before the detected onset, so the first phoneme is not clipped.
|
||||
VAD_SPEECH_PAD_MS=30
|
||||
# Speech required (ms) to fire the fast barge-in onset (must be <= VAD_MIN_SPEECH_MS).
|
||||
VAD_BARGE_IN_MIN_SPEECH_MS=150
|
||||
# Utterance gate applied before transcription. A VAD segment is dropped (no STT,
|
||||
# no LLM turn) when it is shorter than VAD_MIN_UTTERANCE_MS ms (0 disables) or
|
||||
# its average speech probability is below VAD_MIN_AVG_PROB (0.0 disables). The
|
||||
# average is taken over the voiced windows only (prob >= VAD_NEG_THRESHOLD), so
|
||||
# the min_silence hangover does not dilute short words ("uno", "due"). This
|
||||
# filters the VAD false-triggers (breaths, clicks, acoustic echo, noise) that the
|
||||
# STT turns into hallucinated text ("Bye.", "She", <transcript 0 tokens>) and
|
||||
# that would each otherwise cost a full LLM turn. Raise the probability to be
|
||||
# stricter; lower it if genuine soft speech is being dropped.
|
||||
VAD_MIN_UTTERANCE_MS=300
|
||||
VAD_MIN_AVG_PROB=0.25
|
||||
# Feed the STT engine with VAD-segmented utterances (true) instead of letting
|
||||
# the worker run its own RMS segmentation (false).
|
||||
VAD_SEGMENT_STT=true
|
||||
@@ -95,14 +119,15 @@ FILLER_GAIN_DUCK=0.15
|
||||
# the reply has not reached the mixer within START_FILLER_GO_DELAY ms, one of the
|
||||
# phrases below (picked at random) is played so the agent acknowledges the turn
|
||||
# while the LLM/TTS are still working. If audio arrives in time the cue is
|
||||
# dropped. It shares the filler gain, so it ducks out of the way once the reply
|
||||
# starts.
|
||||
# dropped. It shares the filler gain and is faded out (then cut) as soon as the
|
||||
# reply starts playing, so it does not overlap the answer.
|
||||
START_FILLER_ENABLED=true
|
||||
# How long (ms) to wait for the first audio chunk of the reply before playing the
|
||||
# start filler (default 1200). Measured from chunk ARRIVAL (not from playback
|
||||
# start, which would also include the TTS_PREBUFFER_MS jitter buffer). Same idea
|
||||
# as FILLER_DELAY_MS, but shorter.
|
||||
START_FILLER_GO_DELAY=500
|
||||
# as FILLER_DELAY_MS, but shorter. Raised above the default so the cue only fires
|
||||
# when the first audio is genuinely late.
|
||||
START_FILLER_GO_DELAY=1800
|
||||
# Phrases spoken by the start filler, ONE PER LINE (a single line with "|"
|
||||
# separators also works). One clip is synthesized per phrase and a RANDOM one is
|
||||
# played when the reply is late, so the cue varies turn to turn. The bank follows
|
||||
|
||||
@@ -38,7 +38,7 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
|
||||
played round-robin from `FILLERS_DIR`; `FILLER_ENABLED=false` disables it.
|
||||
- **Start filler** — a second latency mask on its own mixer channel: if the first
|
||||
audio chunk of the reply has not reached the mixer within
|
||||
`START_FILLER_GO_DELAY` (default 1200 ms) the agent plays a short
|
||||
`START_FILLER_GO_DELAY` (default 1800 ms) the agent plays a short
|
||||
acknowledgment ("Un momento...") while the LLM/TTS are still working, and drops
|
||||
it as soon as audio arrives in time. One phrase is picked **at random** from
|
||||
`START_FILLER_TEXTS`, so the cue varies turn to turn. Independent of the delayed
|
||||
@@ -60,6 +60,12 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
|
||||
- **Barge-in / interrupt** — cancels in-flight synthesis and flushes audio,
|
||||
triggered client-side or server-side by the VAD.
|
||||
- **Voice cloning** — reference audio + transcript (Fish Audio `references`).
|
||||
- **Custom voice builder** — a dedicated `/customvoice` page (linked from the
|
||||
Mivocero side menu) asks the user to pick a language, then either read a
|
||||
predefined phrase for at least 10 s **or upload an audio file with its typed
|
||||
transcript**; the take is transcoded to a WAV reference and installed as the
|
||||
active voice at runtime. See
|
||||
[Custom voice builder](#custom-voice-builder).
|
||||
- **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to
|
||||
`TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's
|
||||
start-up latency behind the previous chunk's audio.
|
||||
@@ -75,9 +81,10 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
|
||||
audio and the chat box.
|
||||
- **Tool calling** — the model can call external HTTP tools described in a JSON
|
||||
file (`LLM_TOOLS_FILE`) before answering. Calls are parsed client-side from
|
||||
the model's native `<tool_call>` format, executed over HTTP and fed back to
|
||||
the model; the tool-call block is never spoken. See
|
||||
[Tool calling](#tool-calling).
|
||||
either the structured `tool_calls` array or the model's native `<tool_call>`
|
||||
format (a round that streams back empty is re-asked without streaming),
|
||||
executed over HTTP and fed back to the model; the tool-call block is never
|
||||
spoken. See [Tool calling](#tool-calling).
|
||||
- **Embedded playground** — a single-page HTML client served at `/`.
|
||||
- **Mivocero speech-to-speech client** — an example agent client at `/mivocero`
|
||||
with a microphone, a live chat box (user transcript + streaming assistant
|
||||
@@ -121,6 +128,7 @@ cargo run --release
|
||||
# 3. Open the clients
|
||||
# http://localhost:8787/ playground (TTS / audio test)
|
||||
# http://localhost:8787/mivocero speech-to-speech agent with chat box
|
||||
# http://localhost:8787/customvoice custom-voice builder (record / upload a reference)
|
||||
```
|
||||
|
||||
Environment variables can also be `export`ed directly; a `.env` file is loaded
|
||||
@@ -196,9 +204,11 @@ See [`.env.example`](.env.example) for a copy-paste template.
|
||||
| `VAD_THRESHOLD` | `0.5` | Speech probability above which a window counts as voice. |
|
||||
| `VAD_NEG_THRESHOLD` | `0.35` | Probability below which a window counts as silence (hysteresis; clamped to `VAD_THRESHOLD`). |
|
||||
| `VAD_MIN_SPEECH_MS` | `250` | Speech required to confirm an utterance start. |
|
||||
| `VAD_MIN_SILENCE_MS` | `300` | Silence required to close an utterance (end-of-utterance window). |
|
||||
| `VAD_MIN_SILENCE_MS` | `700` | Silence required to close an utterance (end-of-utterance window). |
|
||||
| `VAD_SPEECH_PAD_MS` | `30` | Audio kept before the detected onset so the first phoneme is not clipped. |
|
||||
| `VAD_BARGE_IN_MIN_SPEECH_MS` | `150` | Speech required to fire the fast barge-in onset (clamped to `VAD_MIN_SPEECH_MS`). |
|
||||
| `VAD_MIN_UTTERANCE_MS` | `300` | Utterance gate: drop a segment shorter than this before transcription (`0` disables). |
|
||||
| `VAD_MIN_AVG_PROB` | `0.25` | Utterance gate: drop a segment whose average speech probability is below this before transcription (`0.0` disables). The average is taken over the **voiced windows only** (probability ≥ `VAD_NEG_THRESHOLD`), so the trailing hangover silence does not dilute short words like "uno" / "due". Kept well below `VAD_THRESHOLD`. |
|
||||
| `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. |
|
||||
| `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. |
|
||||
| `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. |
|
||||
@@ -206,9 +216,9 @@ See [`.env.example`](.env.example) for a copy-paste template.
|
||||
| `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. |
|
||||
| `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. |
|
||||
| `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. |
|
||||
| `FILLER_GAIN_DUCK` | `0.15` | Filler gain while TTS is active (ducking). Also applies to the start-filler channel. |
|
||||
| `FILLER_GAIN_DUCK` | `0.15` | Fallback filler gain while TTS is already active. Normally a still-queued filler is faded out (80 ms) and cut when the reply's playback starts, so this only applies to a cue queued *after* playback has begun. Also applies to the start-filler channel. |
|
||||
| `START_FILLER_ENABLED` | `true` | Master switch for the **start filler** — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. `false` => no cue is ever queued. |
|
||||
| `START_FILLER_GO_DELAY` | `1200` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. |
|
||||
| `START_FILLER_GO_DELAY` | `1800` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. If the cue does fire and the reply then starts, it is faded out (80 ms) and cut so it does not overlap the answer. |
|
||||
| `START_FILLER_TEXTS` | `[calm] Un momento...` | Phrases spoken by the start filler, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and a **random** one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, `[sigh]`, …) are passed through to the TTS. |
|
||||
| `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/start_filler/es/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. |
|
||||
| `TTS_PREBUFFER_MS` | `550` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first `185.8` ms, then `371.6` ms), any value in `(186, 557]` ms primes on the *same* second window, so `550` takes the whole cushion at no extra TTFA (~`0.73 s`). Below the first window (~`186` ms) playback starts on that window (TTFA ≈ server TTFB, ~`0.38 s`) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
|
||||
@@ -568,7 +578,7 @@ events:
|
||||
|---|---|---|
|
||||
| `SpeechOnset` | `VAD_BARGE_IN_MIN_SPEECH_MS` (150 ms) of speech | Fast **barge-in**: interrupt the agent as soon as the user starts talking. |
|
||||
| `SpeechStart` | `VAD_MIN_SPEECH_MS` (250 ms) of speech | Confirms the utterance start. |
|
||||
| `SpeechEnd` | `VAD_MIN_SILENCE_MS` (300 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. |
|
||||
| `SpeechEnd` | `VAD_MIN_SILENCE_MS` (700 ms) of silence | Closes the utterance; the captured PCM (with a pre-roll pad) is handed to the STT engine. |
|
||||
|
||||
Onset/offset use hysteresis: a window counts as voice above `VAD_THRESHOLD`
|
||||
and as silence below `VAD_NEG_THRESHOLD`, which prevents flapping around a
|
||||
@@ -578,12 +588,23 @@ detected onset so the first phonemes are never clipped.
|
||||
|
||||
### Barge-in
|
||||
|
||||
When `VAD_BARGE_IN=true` and the agent is currently speaking
|
||||
([`TurnManager::is_speaking`](src/orchestrator/turn_manager.rs), backed by
|
||||
[`AudioMixer::is_active`](src/audio/mixer.rs)), a `SpeechOnset` calls
|
||||
When `VAD_BARGE_IN=true` and the agent has **audible** output
|
||||
([`TurnManager::is_audible`](src/orchestrator/turn_manager.rs), backed by
|
||||
[`AudioMixer::is_audible`](src/audio/mixer.rs)), a `SpeechOnset` calls
|
||||
`TurnManager::interrupt()` — cancelling the in-flight TTS via its
|
||||
`CancellationToken` and flushing the mixer queues.
|
||||
|
||||
The gate is on *audible* output, not on "speaking" in general: `is_audible` is
|
||||
`false` while the reply is still being synthesised or is sitting in the TTS
|
||||
pre-buffer, i.e. before anything has reached the user's ears. This stops a VAD
|
||||
transient (a breath, a click, or the tail of the user's own utterance) from
|
||||
cancelling a reply the user has not heard any of yet. Such a reply is not left
|
||||
orphaned: the next turn's `on_eou` already cancels the previous in-flight
|
||||
synthesis and restarts the stream. `TurnManager::is_speaking`
|
||||
([`AudioMixer::is_active`](src/audio/mixer.rs)) is still used for the
|
||||
backchannel gate, where cueing on top of *any* in-flight TTS would be an
|
||||
interruption rather than an acknowledgment.
|
||||
|
||||
### Wiring
|
||||
|
||||
The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one
|
||||
@@ -591,7 +612,16 @@ The session task ([`src/ws/session.rs`](src/ws/session.rs)) owns one
|
||||
|
||||
- with `VAD_SEGMENT_STT=true` (default): forwards `SpeechEnd` utterances to
|
||||
`SttWorker::push_utterance`, and does **not** forward raw frames (avoids
|
||||
double segmentation);
|
||||
double segmentation). Before forwarding, each segment passes the **utterance
|
||||
gate** (`VAD_MIN_UTTERANCE_MS` / `VAD_MIN_AVG_PROB`): segments below either
|
||||
threshold are dropped without transcription, which filters the VAD
|
||||
false-triggers (breaths, clicks, acoustic echo, noise) that the STT would
|
||||
otherwise turn into hallucinated text. The probability average is computed
|
||||
over the **voiced windows only** (probability ≥ `VAD_NEG_THRESHOLD`), so the
|
||||
`VAD_MIN_SILENCE_MS` hangover that closes the segment does not drag it down —
|
||||
a short word like "uno" would otherwise average far below any usable gate. A
|
||||
transcript that still decodes to no tokens is skipped too, so empty turns are
|
||||
never sent to the LLM.
|
||||
- with `VAD_SEGMENT_STT=false`: falls back to the STT worker's RMS
|
||||
segmentation, using the VAD only for barge-in;
|
||||
- if `VAD_ENABLED=false` or the model fails to load: legacy behaviour (RMS
|
||||
@@ -681,8 +711,11 @@ reached and reset at every `mark_stream_start` / `flush`.
|
||||
- Raise `FILLER_DELAY_MS` for slower LLM/TTS setups, lower it if the filler
|
||||
rarely fires.
|
||||
|
||||
Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`) and ducks it to
|
||||
`FILLER_GAIN_DUCK` (`0.15`) if TTS starts mid-clip.
|
||||
Once queued, the mixer plays it at `FILLER_GAIN_IDLE` (`1.0`). If TTS playback
|
||||
starts mid-clip, the cue is **faded out** over `FILLER_FADEOUT_FRAMES`
|
||||
(4 frames = 80 ms) and then cut — it is a latency mask, so it must hand off
|
||||
cleanly to the reply instead of playing underneath it. (As a fallback, a cue
|
||||
queued *after* playback has already started is ducked to `FILLER_GAIN_DUCK`.)
|
||||
|
||||
`FILLERS_DIR` (default `assets/fillers`) holds `*.raw` clips — **16-bit LE mono
|
||||
PCM at 44.1 kHz** — selected **round-robin** (`TurnManager::next_filler` →
|
||||
@@ -695,7 +728,7 @@ are synthesized at startup ([Startup clip bootstrap](#startup-clip-bootstrap)).
|
||||
A second latency mask, on its own mixer channel. It works exactly like the filler
|
||||
above, but with its own — shorter — delay
|
||||
([`TurnManager::schedule_start_filler`](src/orchestrator/turn_manager.rs)):
|
||||
`START_FILLER_GO_DELAY` (default **1200 ms**) after the turn is dispatched, if the
|
||||
`START_FILLER_GO_DELAY` (default **1800 ms**) after the turn is dispatched, if the
|
||||
first audio chunk of the reply has **not** reached the mixer yet
|
||||
(`AudioMixer::tts_audio_arrived` is still `false`) the cue is queued; if audio
|
||||
arrived within that window, the cue is dropped instead of talking over the
|
||||
@@ -714,7 +747,8 @@ acknowledgment — only a late one is masked.
|
||||
|
||||
It lives on its own mixer channel (`start_filler_queue`), so it plays
|
||||
independently of the round-robin latency filler above, but it shares the filler
|
||||
gain, so it ducks to `FILLER_GAIN_DUCK` once the reply starts. Disabled by
|
||||
gain: when the reply's playback starts, the still-queued cue is faded out
|
||||
(80 ms) and cut rather than left running under the answer. Disabled by
|
||||
`START_FILLER_ENABLED=false`.
|
||||
|
||||
Its clips live in `START_FILLER_DIR` (default `assets/start_filler`) and are
|
||||
@@ -995,14 +1029,23 @@ be exposed without touching the code:
|
||||
|
||||
See [`tools.example.json`](tools.example.json) for a ready-to-use file.
|
||||
|
||||
### Why a client-side parser
|
||||
### Client-side parsing (and the streaming fallback)
|
||||
|
||||
The deployment's OpenAI-compatible server (vLLM) runs **without** a tool parser
|
||||
(`--enable-auto-tool-choice --tool-call-parser …`), so it never returns
|
||||
structured `tool_calls` or `finish_reason: "tool_calls"`. The tools are still
|
||||
advertised in every request (`tools` + `tool_choice: "auto"`), so the Qwen model
|
||||
knows about them and emits the call **as text** in its native format:
|
||||
A deployment may or may not run a server-side tool parser. When it does, the
|
||||
flag has to match the model's native format (`--enable-auto-tool-choice
|
||||
--tool-call-parser …`). A mismatch is silent and costly: an `hermes` parser
|
||||
(which expects `<tool_call>{"name":…,"arguments":{…}}</tool_call>`) against a
|
||||
Qwen3 model that emits the XML form makes the parser throw, so the call is
|
||||
dropped. In **streaming** the round then ends with an empty reply
|
||||
(`finish_reason: "stop"`, no content), while the very same request **without**
|
||||
`stream` returns the call intact.
|
||||
|
||||
The engine therefore never relies on the server parser:
|
||||
|
||||
- Every request advertises `tools` + `tool_choice: "auto"`.
|
||||
- [`src/tools.rs`](src/tools.rs) parses **both** shapes — the standard
|
||||
structured `message.tool_calls` array **and** the model's native
|
||||
`<tool_call>` text:
|
||||
```text
|
||||
<tool_call>
|
||||
<function=get_weather>
|
||||
@@ -1012,9 +1055,12 @@ Paris
|
||||
</function>
|
||||
</tool_call>
|
||||
```
|
||||
- A streaming round that comes back **empty** is re-asked once **without
|
||||
streaming** ([`LlmClient::chat_reply`](src/llm.rs)), where the dropped call
|
||||
reappears and is parsed from either shape.
|
||||
|
||||
[`src/tools.rs`](src/tools.rs) parses that format client-side and works with any
|
||||
server, no server-side reconfiguration needed.
|
||||
Fixing the server (`--tool-call-parser qwen3_coder` handles the Qwen3 XML
|
||||
format) removes the extra request — the fallback then simply never fires.
|
||||
|
||||
### Turn loop
|
||||
|
||||
@@ -1024,7 +1070,9 @@ Each turn runs a bounded loop (`LLM_TOOL_MAX_ITERATIONS`, default `4`):
|
||||
streamed text — a block that is still incomplete is **held back**, so it is
|
||||
never spoken by the TTS nor shown in the chat box even when split across
|
||||
deltas. The visible text is spoken as usual, so the assistant can say
|
||||
something short ("Let me check…") before calling.
|
||||
something short ("Let me check…") before calling. A round that streams back
|
||||
**empty** is re-asked once without streaming to recover a call the server
|
||||
dropped (see above).
|
||||
2. If the round produced no tool call, the turn ends.
|
||||
3. Otherwise the assistant message (with its native call) plus one `tool` role
|
||||
message per result are appended, and the loop re-prompts the model, which can
|
||||
@@ -1112,16 +1160,24 @@ A ready-to-use speech-to-speech example client is served at **`/mivocero`**
|
||||
(44.1 kHz, 16-bit LE mono, 20 ms frames).
|
||||
- **Chat box** — shows the user transcript (from STT) and the assistant reply
|
||||
**as it streams**, plus messages typed in the composer.
|
||||
- **Karaoke highlighting** — the assistant bubble is rendered word by word: each
|
||||
word lights up as it is *spoken*, driven by the audio playhead (the playback
|
||||
worklet reports the samples it has actually emitted). Words already spoken are
|
||||
bright, the word currently sounding is accented, and the rest stay dim. The
|
||||
rate is an estimate (`KARAOKE_CPS`, ~15 chars/s) — it only affects the visual
|
||||
highlight, never the audio.
|
||||
- **Karaoke highlighting** — both the assistant and the user bubbles are
|
||||
rendered word by word. The assistant's words light up as they are *spoken*,
|
||||
driven by the audio playhead (the playback worklet reports the samples it has
|
||||
actually emitted): words already spoken are bright, the word currently
|
||||
sounding is accented, and the rest stay dim. The rate is an estimate
|
||||
(`KARAOKE_CPS`, ~15 chars/s) — it only affects the visual highlight, never the
|
||||
audio. The user's words cannot follow the audio (the STT returns the whole
|
||||
transcript only once the utterance is over), so they are lit by a short
|
||||
left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word) when the bubble
|
||||
appears.
|
||||
- **Text composer** — typing and sending a message runs a full LLM → TTS turn
|
||||
without using the microphone (`{"type":"user_text","text":…}`).
|
||||
- **Interrupt button** — flushes audio and cancels the in-flight reply
|
||||
(`{"type":"interrupt"}`).
|
||||
- **Barge-in indicator** — a red **Barge-in** pill in the header that flashes and
|
||||
bumps a session counter whenever the agent is interrupted: either by the
|
||||
server-side VAD onset (via the `{"type":"barge_in"}` event) or by the local
|
||||
**Stop** button. The counter resets on each connect.
|
||||
- **External audio bar** — footer controls to inject a background source by URL
|
||||
(direct audio / radio / YouTube), with **Play** / **Stop** buttons, a volume
|
||||
slider (0–150 %, sent as `EXTERNAL_AUDIO_GAIN` × slider) and a live state
|
||||
@@ -1139,6 +1195,8 @@ A ready-to-use speech-to-speech example client is served at **`/mivocero`**
|
||||
count, start-filler go-delay, and the backchannel gain, min-speech, interval,
|
||||
jitter, pause window, probability, volume jitter, per-turn cap and categories
|
||||
— all read from the effective runtime configuration (no hard-coded values).
|
||||
- **Side menu** — the ☰ button in the header opens a drawer that navigates
|
||||
between the chat and the [custom voice builder](#custom-voice-builder).
|
||||
- **Idle auto-disconnect** — the session closes itself after **2 minutes**
|
||||
(`IDLE_TIMEOUT_MS`) without any traffic in either direction. The timer is
|
||||
reset by every server event *and* audio frame received, and by every user
|
||||
@@ -1152,6 +1210,106 @@ access — see [HTTPS](#https-microphone-access)).
|
||||
|
||||
---
|
||||
|
||||
## Custom voice builder
|
||||
|
||||
A dedicated page served at **`/customvoice`** (embedded from
|
||||
[`src/web/customvoice.html`](src/web/customvoice.html), linked from the Mivocero
|
||||
side menu) lets anyone build a voice-cloning reference from their own voice,
|
||||
without touching the filesystem or the `.env`. Two sources are offered:
|
||||
|
||||
- **Record now** — read a predefined phrase out loud (the microphone flow).
|
||||
- **Upload a file** — pick an audio file and **type its transcript** (any
|
||||
container `ffmpeg` can read: wav, mp3, m4a, ogg, flac…).
|
||||
|
||||
The flow:
|
||||
|
||||
1. **Language** — a selector populated from `GET /api/customvoice/phrases`
|
||||
(the languages of [`crate::lang`](src/lang.rs); the default is
|
||||
`LANG_DEFAULT`).
|
||||
2. **Source** — pick **Record now** or **Upload a file**.
|
||||
- **Record now** — the page shows the phrase to read for the selected
|
||||
language. The phrases live **server-side**
|
||||
([`src/customvoice.rs`](src/customvoice.rs)), so the transcript attached to
|
||||
the reference always matches the text the user was asked to read. The
|
||||
browser captures the microphone with `MediaRecorder` (Opus/WebM where
|
||||
supported); the **Stop** button stays disabled until **10 seconds** have
|
||||
elapsed (a progress bar and a timer show the way to the minimum), and
|
||||
recording auto-stops at 20 s. The take can be played back and discarded
|
||||
before use.
|
||||
- **Upload a file** — choose an audio file (`accept="audio/*"`) and type (or
|
||||
paste) the exact text spoken in it. Here the transcript is **required**: it
|
||||
is what drives the clone, so it must match the audio.
|
||||
3. **Use it** — `POST /api/customvoice?lang=<code>[&text=<transcript>]` with the
|
||||
recording or the uploaded file as the body. The `text` parameter carries the
|
||||
upload flow's typed transcript and overrides the predefined phrase; when it
|
||||
is absent or blank the language's predefined phrase is used (so a plain
|
||||
recording needs no text). The server **post-processes** the take (see below),
|
||||
transcodes it to **mono 16-bit PCM WAV at `OUTPUT_SAMPLE_RATE`** (44.1 kHz)
|
||||
with `ffmpeg`, writes it next to the configured `TTS_REFERENCE_AUDIO` (the
|
||||
directory the Fish Audio backend can read — hence "the path on the TTS
|
||||
server" note in [Configuration](#configuration)), writes the transcript as a
|
||||
sibling `<name>.txt` (the convention [`Config::tts_reference`](src/config.rs)
|
||||
already falls back to), and installs the voice through
|
||||
[`TtsEngine::set_reference`](src/tts/mod.rs) — effective immediately for new
|
||||
replies, no restart.
|
||||
4. **Reset** — `POST /api/customvoice/clear` restores the reference configured
|
||||
at startup (`TTS_REFERENCE_AUDIO`), or clears it when none is configured.
|
||||
|
||||
### Post-processing
|
||||
|
||||
A raw browser recording is uneven: it carries rumble and room noise, and its
|
||||
level depends on how far the user sat from the microphone. The take therefore
|
||||
goes through a cleanup chain before it is installed
|
||||
([`src/customvoice.rs`](src/customvoice.rs)):
|
||||
|
||||
| Stage | Filter | Why |
|
||||
|---|---|---|
|
||||
| Mono downmix | `aformat=channel_layouts=mono` | Everything downstream (and the reference itself) is mono. Done **first** so loudnorm measures the signal that is actually written. |
|
||||
| Rumble / DC removal | `highpass=f=80` | Kills desk thumps, plosives and DC offset below the voice band. |
|
||||
| Noise reduction | `afftdn=nr=12:nf=-40` | FFT denoise for steady room/hiss noise. |
|
||||
| De-essing | `deesser` | Tames harsh sibilance that a clone would otherwise exaggerate. |
|
||||
| Silence trim | `silenceremove` (+ `areverse`) | Drops leading/trailing silence so the reference starts on speech. |
|
||||
| Loudness | `loudnorm`, **two-pass, linear** | Matches every take to the same programme level. |
|
||||
|
||||
The loudness stage is deliberately **two-pass**: ffmpeg measures the cleaned
|
||||
signal first, then applies the measurement as **one constant gain**
|
||||
(`linear=true`). A single gain cannot pump or ride the level frame by frame —
|
||||
which is exactly what makes takes differ in volume.
|
||||
|
||||
The default target is **-26 LUFS** integrated with a **-5 dBTP** true-peak
|
||||
ceiling. Both sit *below* the shipped reference (`voice_profiles/solis.wav`
|
||||
measures ≈-23.4 LUFS at -7.8 dBTP): a user take is far more dynamic than that
|
||||
mastered reference, so at equal LUFS its much hotter peaks make the cloned
|
||||
voice sound louder than the default one. The lower target and the extra peak
|
||||
headroom buy that gap back, so a custom voice sits a touch under the default
|
||||
instead of on top of it. A take recorded 30 dB too quiet and one recorded hot
|
||||
both land on the same level.
|
||||
|
||||
If any stage fails (a filter quirk, an unreadable take, an all-silence
|
||||
recording) the endpoint logs a warning and **falls back to the plain
|
||||
transcode** — a take is never lost to post-processing.
|
||||
|
||||
| Env var | Default | Description |
|
||||
|---|---|---|
|
||||
| `CUSTOMVOICE_POSTPROCESS` | `true` | Master switch; `false` stores the plain transcode. |
|
||||
| `CUSTOMVOICE_DENOISE` | `true` | The `afftdn` + `deesser` stage — turn off for a clean studio take. |
|
||||
| `CUSTOMVOICE_TARGET_LUFS` | `-26` | Integrated loudness target, below the shipped reference. Lower it for a quieter custom voice. |
|
||||
| `CUSTOMVOICE_TRUE_PEAK` | `-5` | True-peak ceiling (dBTP). |
|
||||
|
||||
| Method | Route | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/customvoice` | The builder page. |
|
||||
| `GET` | `/api/customvoice/phrases` | Reading phrases + the default language (JSON). |
|
||||
| `POST` | `/api/customvoice?lang=<code>[&text=<transcript>]` | Body = recording or uploaded file; transcodes, persists (with the transcript) and installs the reference. |
|
||||
| `POST` | `/api/customvoice/clear` | Restore the configured reference. |
|
||||
|
||||
The body cap for the upload route is raised to 32 MB (the axum default is 2 MB)
|
||||
so an uncompressed WAV is accepted as well as a compressed capture. Microphone
|
||||
access requires a secure context: use `https://…` when the page is not opened
|
||||
from `localhost` (see [HTTPS](#https-microphone-access)).
|
||||
|
||||
---
|
||||
|
||||
## Session audit log
|
||||
|
||||
Every WebSocket session can be recorded to its own structured **JSONL** file —
|
||||
@@ -1175,10 +1333,11 @@ started, an event discriminator and a payload:
|
||||
|---|---|---|
|
||||
| `session_start` | Session opens | `config` — the full non-secret configuration snapshot; `runtime` — which of STT / VAD / tokenizer / LLM are actually active (plus the LLM model). |
|
||||
| `control` | `user_text` control message received | The text turn. |
|
||||
| `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. |
|
||||
| `out` | **Every** event sent to the client | The exact WS payload — `user`, `assistant_delta`, `assistant_end`, `tool_call`, `cue`, `barge_in`, `error`, `audio_source` state, … Recorded at the single sender tap, so nothing sent to the browser is missed. |
|
||||
| `interrupt` | Client barge-in (`interrupt` JSON or legacy `INTERRUPT`) | `source` (`client` / `client_legacy`). |
|
||||
| `barge_in` | VAD-driven barge-in | `prob` (onset speech probability). |
|
||||
| `vad_speech_start` / `vad_speech_end` | Utterance boundaries | `prob`; `duration_ms`, `avg_prob`. |
|
||||
| `utterance_dropped` | VAD segment filtered by the utterance gate (no STT, no LLM turn) | `duration_ms`, `avg_prob`, `too_short`, `too_soft`. |
|
||||
| `media` | External audio control | `action` (`play`/`stop`/`volume`) with `url` / `gain`. |
|
||||
| `reference` | Voice-cloning reference change | `action` (`set`/`clear`) and the audio path. |
|
||||
| `speak` | Direct TTS request (playground / fallback text) | `text`. |
|
||||
@@ -1225,6 +1384,7 @@ Connect to `ws://<host>:8787/ws`.
|
||||
| **Text — `{"type":"audio_source","state":"…","url":"…"}`** | External audio channel state change: `loading` (decoding is being set up / YouTube URL resolving), `playing` (PCM is streaming), `stopped`, or `error` (with a `text` message). |
|
||||
| **Text — `{"type":"cue","kind":"…","name":"…","category":"…"}`** | A latency-mask / backchannel cue was actually emitted into the mixer: `kind` is `filler` (end-of-turn), `start_filler` (turn start) or `backchannel` (during user speech); `name` is the clip name and `category` (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators. |
|
||||
| **Text — `{"type":"cues","lang":"…","config":{…}}`** | Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (`delay_ms`, gains, clip count), the start-filler `go_delay_ms` and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's **Cue methods** panel so it shows the live values instead of hard-coded ones. |
|
||||
| **Text — `{"type":"barge_in","source":"vad","prob":…}`** | The agent was interrupted. Sent for the server-side VAD barge-in (the browser has no other way to know, since the onset is detected in the session task); the client-side `interrupt` does not echo one. Drives the demo's barge-in indicator. |
|
||||
|
||||
---
|
||||
|
||||
@@ -1246,8 +1406,11 @@ Constants are defined in [`src/config.rs`](src/config.rs):
|
||||
|
||||
- Mixing formula:
|
||||
`mixed = tts + ((filler + start_filler) × filler_gain) + (backchannel × BACKCHANNEL_GAIN) + (external × external_gain)`.
|
||||
The filler gain is `1.0` when TTS is idle, else `filler_gain_duck` (default
|
||||
`0.15`); the backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
|
||||
The filler gain is `1.0` when TTS is idle; when the reply's playback starts a
|
||||
still-queued filler/start-filler is instead **faded out** over 80 ms
|
||||
(`FILLER_FADEOUT_FRAMES`) and cut, so it hands off to the answer (a cue queued
|
||||
after playback has already begun falls back to `filler_gain_duck`, default
|
||||
`0.15`). The backchannel gain is fixed (`BACKCHANNEL_GAIN`, default `0.35`).
|
||||
The start-filler channel is summed with the filler, so it shares the same gain.
|
||||
The external channel gain starts at `EXTERNAL_AUDIO_GAIN` (default `0.6`) and
|
||||
is adjustable at runtime (`AudioMixer::set_external_gain`). Output is clamped
|
||||
@@ -1313,10 +1476,11 @@ audio-agent-engine/
|
||||
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
|
||||
├── tools.rs # configurable HTTP tool calling (native <tool_call> parser + executor)
|
||||
├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
|
||||
├── customvoice.rs # custom-voice builder endpoints (record → WAV reference → set_reference)
|
||||
├── audit.rs # per-session JSONL audit log (config snapshot, events, summary)
|
||||
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
|
||||
├── ws/ # handler, session (per-connection audio loop)
|
||||
└── web/ # embedded playground.html + mivocero.html
|
||||
└── web/ # embedded playground.html + mivocero.html + customvoice.html
|
||||
```
|
||||
|
||||
### HTTP routes
|
||||
@@ -1325,6 +1489,10 @@ audio-agent-engine/
|
||||
|---|---|
|
||||
| `GET /` | Playground single-page client (embedded at compile time). |
|
||||
| `GET /mivocero` | Mivocero speech-to-speech example client with chat box (embedded at compile time). |
|
||||
| `GET /customvoice` | Custom-voice builder page (record or upload a reference — see [Custom voice builder](#custom-voice-builder)). |
|
||||
| `GET /api/customvoice/phrases` | Reading phrases per language + the default language (JSON). |
|
||||
| `POST /api/customvoice?lang=<code>[&text=<transcript>]` | Upload a recording or an audio file (with its transcript); transcode it to a WAV reference and install it as the active voice. |
|
||||
| `POST /api/customvoice/clear` | Restore the reference configured at startup. |
|
||||
| `GET /health` | Liveness probe (`ok`). |
|
||||
| `GET /ws` | WebSocket upgrade → per-session audio loop. |
|
||||
|
||||
|
||||
+372
@@ -5,6 +5,378 @@ Entries are written in English and ordered newest-first.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Feat: custom voice accepts an uploaded audio file + typed transcript
|
||||
|
||||
### Added
|
||||
|
||||
- **Upload source on the `/customvoice` page.** A **Source** toggle now offers
|
||||
two ways to build the reference: **Record now** (the existing microphone flow,
|
||||
read a predefined phrase for ≥ 10 s) or **Upload a file** — pick any audio
|
||||
file `ffmpeg` can read (`accept="audio/*"`: wav, mp3, m4a, ogg, flac…) and
|
||||
**type (or paste) its transcript**. In upload mode the transcript is required
|
||||
and gates the *Use it* button, since the typed text is what drives the clone.
|
||||
Both modes share the same *Use it* / *Reset* actions and preview player.
|
||||
- **`text` query parameter on `POST /api/customvoice`.** The endpoint is now
|
||||
`POST /api/customvoice?lang=<code>[&text=<transcript>]`. A non-blank `text`
|
||||
overrides the language's predefined reading phrase for the transcript written
|
||||
to the sibling `.txt` and passed to `set_reference`; when absent or blank the
|
||||
predefined phrase is used as before (so a plain recording is unchanged).
|
||||
Backend: `SetParams` gained `text: Option<String>` and `set_custom_voice`
|
||||
resolves the transcript from it (`src/customvoice.rs`).
|
||||
|
||||
### Notes
|
||||
|
||||
- No format constraints are imposed on the upload: the same ffmpeg probe used
|
||||
for recordings handles whatever container is sent. The 32 MB body cap already
|
||||
covers uncompressed WAVs.
|
||||
- README updated (feature bullet, usage comment, *Custom voice builder* flow and
|
||||
API/route tables, `text` parameter).
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Fix: short utterances ("uno", "due") dropped by the STT gate
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The probability average now counts voiced windows only.** The gate's
|
||||
`avg_prob` was the mean of the raw VAD probability over **every** window from
|
||||
speech start to segment close — including the `VAD_MIN_SILENCE_MS` (700 ms)
|
||||
hangover that ends the utterance. For a long phrase that silence is a small
|
||||
fraction, but for a short word it is most of the segment: "uno" (~350 ms of
|
||||
voice + ~700 ms of silence) averaged **0.065–0.199** and was discarded as
|
||||
`too_soft`, even though the user was plainly speaking. In the same session the
|
||||
long phrases passed at 0.32–0.60. The average is now taken only over windows
|
||||
whose probability is ≥ `VAD_NEG_THRESHOLD` (`src/vad/silero.rs`), so the voiced
|
||||
portion scores its true value and short real words clear the gate while noise
|
||||
still does not.
|
||||
|
||||
### Notes
|
||||
|
||||
- `VAD_MIN_AVG_PROB` stays at `0.25`; its meaning is now stricter (it no longer
|
||||
benefits from the silence dilution), but genuine short words score well above
|
||||
it. README / `.env` / `.env.example` / `src/vad/mod.rs` docs aligned.
|
||||
- Tune with `VAD_MIN_AVG_PROB`: raise it if noisy segments become hallucinated
|
||||
turns, lower it if genuinely soft speech is still dropped.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Fix: too many utterances dropped by the STT gate
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`VAD_MIN_AVG_PROB` lowered `0.45` → `0.25`.** Log analysis showed **62
|
||||
drops, all `too_soft`** (none for duration), still occurring after a restart.
|
||||
Accepted utterances measured `avg_prob` 0.47–0.76 while dropped ones measured
|
||||
0.08–0.45 — including plainly real speech cut by a hair (2136 ms @ 0.428,
|
||||
3576 ms @ 0.436). The gate sat almost on top of the per-frame speech threshold
|
||||
(`VAD_THRESHOLD=0.5`): a real utterance hovers around the threshold on its
|
||||
onset/release, so its *average* lands just under 0.45 and the segment is
|
||||
discarded. The `0.25` default leaves a proper margin — genuine soft speech
|
||||
(0.3–0.45) now passes, while clearly noisy segments (below 0.25) are still
|
||||
filtered. Update `VAD_MIN_AVG_PROB` in `.env` (default, `.env.example`,
|
||||
`src/vad/mod.rs` and README aligned).
|
||||
|
||||
### Notes
|
||||
|
||||
- Tune with `VAD_MIN_AVG_PROB`: lower it further if genuine speech is still
|
||||
dropped, raise it if noisy segments are turning into hallucinated turns.
|
||||
`0.0` disables the probability gate.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Fix: custom voice still louder than the default voice
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Custom voice level pushed below the reference.** Matching the shipped
|
||||
reference at the *integrated* level was not enough: a measured take lands at
|
||||
≈`-23.25 LUFS` against `solis.wav`'s ≈`-23.44 LUFS`, i.e. the same loudness,
|
||||
yet it still sounded louder. The difference is **dynamic range and peaks** —
|
||||
the user take measures ≈`6.2 LRA` / `-2.2 dBTP` versus `1.6 LRA` / `-7.8 dBTP`
|
||||
for the mastered reference, so its peaks sit ~5.5 dB hotter and read as
|
||||
"louder". The normalization target is now `-26 LUFS` with a `-5 dBTP`
|
||||
ceiling, which drops a take ~3 dB and pulls its peaks down to ≈`-5 dBTP` so a
|
||||
custom voice sits just under the default instead of on top of it.
|
||||
- Defaults changed: `CUSTOMVOICE_TARGET_LUFS` `-23` → **`-26`**,
|
||||
`CUSTOMVOICE_TRUE_PEAK` `-1.5` → **`-5`** (code, `.env`, `.env.example`,
|
||||
README).
|
||||
|
||||
### Notes
|
||||
|
||||
- Lower `CUSTOMVOICE_TARGET_LUFS` further for a quieter custom voice; the value
|
||||
is applied at record time, so a take must be re-recorded (or the endpoint
|
||||
re-posted) for a change to take effect.
|
||||
- The custom voice lives only in memory (`TtsEngine::set_reference`): a restart
|
||||
reverts the reference to the configured `TTS_REFERENCE_AUDIO` voice.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Fix: custom voice louder than the default voice
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Loudness target aligned with the shipped reference.** The custom-voice
|
||||
chain normalized to `-18 LUFS` while `voice_profiles/solis.wav` measures
|
||||
≈`-23.4 LUFS`, so a custom voice came out ~5.4 dB louder than the voice it
|
||||
replaced. The default is now **`-23`** (`CUSTOMVOICE_TARGET_LUFS`), matching
|
||||
the reference — swapping voices no longer changes how loud the agent sounds.
|
||||
- **Mono downmix moved inside the filter chain.** `-ac 1` used to be applied at
|
||||
the output stage, i.e. *after* loudnorm: a **stereo** upload was measured as
|
||||
stereo and then folded to mono, landing the written file ~3 dB below the
|
||||
target. `aformat=channel_layouts=mono` now runs first, so loudnorm measures
|
||||
the signal that is actually written.
|
||||
|
||||
### Notes
|
||||
|
||||
- Verified: `solis.wav` (stereo, -23.4 LUFS) and a quiet mono take (-48.6 LUFS)
|
||||
both render to exactly **-23.00 LUFS**, mono 44.1 kHz. `cargo test` → 25
|
||||
passed.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Post-processing for the custom voice (level match + cleanup)
|
||||
|
||||
### Added
|
||||
|
||||
- **Post-processing chain** for a recorded custom voice
|
||||
(`src/customvoice.rs`), applied before the reference is installed:
|
||||
`highpass=f=80` (rumble/DC), optional `afftdn` denoise + `deesser`,
|
||||
leading/trailing silence trim (`silenceremove` + `areverse`) and a
|
||||
**two-pass EBU R128 `loudnorm` with `linear=true`**.
|
||||
- **New env knobs**: `CUSTOMVOICE_POSTPROCESS` (default `true`),
|
||||
`CUSTOMVOICE_DENOISE` (default `true`), `CUSTOMVOICE_TARGET_LUFS`
|
||||
(default `-18`), `CUSTOMVOICE_TRUE_PEAK` (default `-1.5`).
|
||||
|
||||
### Notes
|
||||
|
||||
- The loudness stage measures the cleaned signal first, then applies the result
|
||||
as a **single constant gain** — a per-frame (single-pass) norm is what makes
|
||||
takes jump in volume, which is the problem this fixes. Verified: a take at
|
||||
**-48.6 LUFS** and one at **-17.5 LUFS** both render to **≈ -18 LUFS** with
|
||||
TP at **-1.5 dBTP**.
|
||||
- Any stage failing (unreadable take, all-silence clip, filter quirk) degrades
|
||||
to the plain transcode with a warning — a take is never lost.
|
||||
- Error handling verified: garbage input → `502` with a clean message, unknown
|
||||
language → `400`. `cargo test` → 25 passed.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Custom voice builder (`/customvoice`) + Mivocero side menu
|
||||
|
||||
### Added
|
||||
|
||||
- **Custom-voice builder page** at `GET /customvoice`
|
||||
(`src/web/customvoice.html`). It asks the user for a **language**, shows a
|
||||
predefined phrase to read (long enough for cloning, ~10 s), records it with
|
||||
`MediaRecorder` and installs the take as the active voice. The **Stop**
|
||||
button is disabled until **10 s** have elapsed (timer + progress bar), the
|
||||
take auto-stops at 20 s, and it can be played back and discarded before use.
|
||||
- **Backend endpoints** (`src/customvoice.rs`): `GET /api/customvoice/phrases`
|
||||
(reading phrases per language + the configured default), `POST
|
||||
/api/customvoice?lang=<code>` (raw recording body), `POST
|
||||
/api/customvoice/clear`. The body cap of the upload route is raised to 32 MB.
|
||||
- **Side menu** on `/mivocero` (☰ drawer) linking to the chat and to the
|
||||
custom-voice page; the same drawer is on `/customvoice`.
|
||||
|
||||
### Notes
|
||||
|
||||
- Phrases live server-side, so the reference transcript always matches the text
|
||||
the user read. `POST /api/customvoice` transcodes the upload to mono 16-bit
|
||||
PCM WAV at `OUTPUT_SAMPLE_RATE` (44.1 kHz) with `ffmpeg`, writes it next to
|
||||
`TTS_REFERENCE_AUDIO` (the directory the Fish Audio backend can read) with a
|
||||
sibling `<name>.txt`, and installs it via `TtsEngine::set_reference` — live
|
||||
for new replies, no restart.
|
||||
- `POST /api/customvoice/clear` restores the reference configured at startup
|
||||
(`TTS_REFERENCE_AUDIO`) rather than dropping it entirely.
|
||||
- Verified: uploading an audio file returns `{"ok":true,…}` and produces a
|
||||
44.1 kHz/mono `pcm_s16le` WAV with its `.txt`; `clear` logs the restore of
|
||||
`solis.wav`. Tests: `cargo test` → 25 passed.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Tool calling survives a mis-matched server-side tool parser
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Tool calls dropped on the streaming path are recovered.** The production
|
||||
vLLM server runs `--tool-call-parser hermes`, which expects the
|
||||
`<tool_call>{"name":…,"arguments":{…}}</tool_call>` JSON shape; the Qwen3
|
||||
model emits the XML shape instead, so the parser throws and the call is
|
||||
discarded. In streaming the turn then ended with an **empty** reply
|
||||
(`finish_reason: "stop"`) and no `tool_call` ever fired — the assistant either
|
||||
stayed silent or narrated a call it never made.
|
||||
- **Non-streaming fallback** in the turn loop
|
||||
(`src/orchestrator/turn_manager.rs`): a round that comes back empty while
|
||||
tools are configured is re-asked once **without streaming** via the new
|
||||
`LlmClient::chat_reply` (`src/llm.rs`), which returns both the visible text
|
||||
and the parsed tool calls.
|
||||
- **Structured calls are parsed too** — `tools::parse_openai_tool_calls`
|
||||
(`src/tools.rs`) reads the standard `message.tool_calls[]` array, so the
|
||||
engine keeps working when the server parser *does* match (and returns
|
||||
`finish_reason: "tool_calls"`). The native `<tool_call>` text path is
|
||||
unchanged.
|
||||
|
||||
### Notes
|
||||
|
||||
- End-to-end verified: "Che tempo fa a New York adesso?" now emits
|
||||
`tool_call {name: get_weather, arguments: {city: "New York"}}` and answers
|
||||
with live wttr.in data. A tool-free turn is unchanged (no extra request).
|
||||
- The clean fix remains server-side — `--tool-call-parser qwen3_coder` matches
|
||||
the Qwen3 XML format. The fallback makes the engine correct regardless, at the
|
||||
cost of one extra request only on an otherwise-empty round.
|
||||
- Tests: `cargo test` → 25 passed (2 new in `tools::tests`).
|
||||
|
||||
---
|
||||
|
||||
|
||||
## 2026-10-17 — Longer end-of-utterance window (`VAD_MIN_SILENCE_MS` 300 → 700 ms)
|
||||
|
||||
### Changed
|
||||
|
||||
- **`VAD_MIN_SILENCE_MS` raised** from `300` to `700` ms in `.env` /
|
||||
`.env.example`, and the fallback default in `src/vad/mod.rs` aligned to
|
||||
`700`. This is the silence window the VAD waits for before closing an
|
||||
utterance (see `SileroVad::step`, `silence_run_ms >= min_silence_ms`).
|
||||
- **README** — updated the config table and the `SpeechEnd` trigger row.
|
||||
|
||||
### Notes
|
||||
|
||||
- With `300` ms a brief thinking pause (~0,3 s) closed the turn and sent a
|
||||
partial sentence to the STT/LLM. `700` ms tolerates a natural mid-sentence
|
||||
pause before the agent answers. The trade-off is a slightly later reply (the
|
||||
agent now waits ~0,7 s of silence instead of ~0,3 s), which the filler /
|
||||
start-filler cues already mask.
|
||||
- The backchannel micro-pause window is unaffected: its `pause_max_ms` (`260`)
|
||||
is still below the new cap (`min_silence_ms - 32 = 668`).
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Mivocero: barge-in indicator + karaoke on the user bubbles
|
||||
|
||||
### Added
|
||||
|
||||
- **Server `{"type":"barge_in","source":"vad","prob":…}` event** in
|
||||
`src/ws/session.rs` — emitted when the server-side VAD onset interrupts the
|
||||
agent. The onset is detected in the session task, so the browser had no other
|
||||
way to know a barge-in had happened.
|
||||
- **Barge-in indicator** in `src/web/mivocero.html` — a red **Barge-in** pill in
|
||||
the header that flashes and bumps a session counter. Driven by the
|
||||
`barge_in` event (VAD onset) and by the local **Stop** button; the counter
|
||||
resets on each connect.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Karaoke on the user bubbles** in `src/web/mivocero.html` — the user's own
|
||||
messages (STT transcript and typed text) are now rendered word by word like
|
||||
the assistant's, with colours tuned for the blue bubble. Because the STT
|
||||
returns the whole transcript only once the utterance has ended, the words are
|
||||
lit by a short left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word)
|
||||
instead of following the audio playhead.
|
||||
- **README** — documented the `barge_in` event in both the server→client table
|
||||
and the audit `out` row, and updated the Mivocero section (karaoke, barge-in
|
||||
indicator).
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — VAD barge-in gated on audible output (no more silent cancelled replies)
|
||||
|
||||
### Added
|
||||
|
||||
- **`AudioMixer::is_audible()`** in `src/audio/mixer.rs` — reports whether the
|
||||
agent has *audible* output: the reply is playing out (`tts_primed`) or a
|
||||
latency-mask cue (filler / start filler) is queued. Unlike `is_active()` it
|
||||
is `false` while the reply is still being synthesised or sits in the TTS
|
||||
pre-buffer — nothing has reached the user's ears yet.
|
||||
- **`TurnManager::is_audible()`** in `src/orchestrator/turn_manager.rs` —
|
||||
forwards the mixer method for the barge-in gate.
|
||||
|
||||
### Changed
|
||||
|
||||
- **VAD-driven barge-in gate** in `src/ws/session.rs` — a `SpeechOnset` now
|
||||
interrupts only when `TurnManager::is_audible()` is true, instead of
|
||||
`is_speaking()` (`is_active()`). `is_speaking()` is retained for the
|
||||
backchannel gate, where cueing over any in-flight TTS would still be wrong.
|
||||
|
||||
### Notes
|
||||
|
||||
- Root cause: `is_speaking()` is already `true` as soon as the TTS request is
|
||||
in flight (`tts_stream_active`), so a VAD onset landing during the
|
||||
synthesis / pre-buffer window (~1.2–1.6 s) cancelled the reply before a
|
||||
single sample was played. On the analysed session
|
||||
(`s-1792251858371-0001`, ~210 s, 21 turns) three replies streamed their text
|
||||
to the client but produced **zero playback**, and three more were truncated
|
||||
after 0.1–0.7 s; several of the onsets were false triggers (the utterance
|
||||
was later dropped by the gate with `avg_prob` 0.28 / 0.21).
|
||||
- A reply cancelled this way is not orphaned: the next turn's `on_eou` already
|
||||
cancels the previous in-flight synthesis and restarts the stream, so the
|
||||
audio follows the newest turn instead of being lost.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Filler fade-out on reply start (no more overlap with the answer)
|
||||
|
||||
### Changed
|
||||
|
||||
- **Mixer filler fade-out** in `src/audio/mixer.rs` — when the reply's playback
|
||||
starts (TTS pre-buffer reached) while an end-of-turn filler or a start filler
|
||||
is still queued, the cue is now **faded out** over `FILLER_FADEOUT_FRAMES`
|
||||
(4 frames = 80 ms) and then cut, instead of being ducked to `filler_gain_duck`
|
||||
and left playing underneath the whole answer. The filler is only a latency
|
||||
mask, so it now hands off cleanly to the reply.
|
||||
- New `AudioMixer::filler_fade_frames` counter, armed on the priming
|
||||
transition and reset by `mark_stream_start` / `flush`.
|
||||
- **`START_FILLER_GO_DELAY` raised** from `1200` to `1800` ms in `.env` /
|
||||
`.env.example` — the cue now only fires when the first audio chunk is
|
||||
genuinely late, instead of on the boundary where it used to overlap the
|
||||
beginning of the reply.
|
||||
|
||||
### Notes
|
||||
|
||||
- Root cause: with `START_FILLER_GO_DELAY=1200` the first TTS audio arrived
|
||||
0.75–1.4 s after the cue, while the `sf03` clip is ~2.83 s long — so the cue
|
||||
played for its first second at full gain and its tail then overlapped the
|
||||
start of the answer. On the analysed session this happened on 5 of 6 start
|
||||
fillers (all `sf03`).
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Utterance gate: filter spurious VAD turns before STT
|
||||
|
||||
### Added
|
||||
|
||||
- **Utterance gate** in `src/ws/session.rs` — a VAD segment is now dropped
|
||||
**before** transcription when it is too short or too "soft", so the STT can no
|
||||
longer turn a VAD false-trigger (breath, click, acoustic echo, noise) into a
|
||||
hallucinated text turn that would each otherwise cost a full LLM turn.
|
||||
Two independent thresholds, both env-driven:
|
||||
- `VAD_MIN_UTTERANCE_MS` (`VadSettings::min_utterance_ms`, default `300`,
|
||||
`0` disables) — drop segments shorter than this.
|
||||
- `VAD_MIN_AVG_PROB` (`VadSettings::min_avg_prob`, default `0.45`, `0.0`
|
||||
disables) — drop segments whose average speech probability is below this.
|
||||
A dropped segment is logged and recorded in the audit as `utterance_dropped`
|
||||
(`duration_ms`, `avg_prob`, `too_short`, `too_soft`).
|
||||
- **Empty-transcript guard** in `src/orchestrator/turn_manager.rs::on_eou` — a
|
||||
transcript that decodes to an empty string (or has no tokens at all) no
|
||||
longer opens an LLM turn.
|
||||
|
||||
### Configuration
|
||||
|
||||
- New `.env` keys next to the other `VAD_*` options:
|
||||
|
||||
```
|
||||
VAD_MIN_UTTERANCE_MS=300
|
||||
VAD_MIN_AVG_PROB=0.45
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- The gate runs only when `VAD_SEGMENT_STT=true`. Raise the probability to be
|
||||
stricter; lower the duration to keep very short backchannels ("sì", "ok").
|
||||
- Documentation updated: README (VAD configuration table, VAD wiring section,
|
||||
audit-event table) and project memory.
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)
|
||||
|
||||
### Added
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,8 @@
|
||||
Sawa...
|
||||
Ndiyo...
|
||||
Uh-huh...
|
||||
Sijui...
|
||||
Vizuri...
|
||||
Sawa...
|
||||
Hapo...
|
||||
Ee...
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,7 @@
|
||||
uhm...
|
||||
well...
|
||||
let me see...
|
||||
eish...
|
||||
mwah...
|
||||
aha...
|
||||
eh...
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,3 @@
|
||||
siku mbili...
|
||||
hebu nikujulishe...
|
||||
nitaangalia...
|
||||
+63
-7
@@ -34,6 +34,12 @@ use tracing::{debug, info};
|
||||
/// re-introduces the pre-buffer latency mid-reply.
|
||||
const TTS_DRAIN_GRACE_FRAMES: usize = 10;
|
||||
|
||||
/// Number of 20 ms frames over which the latency-mask filler (end-of-turn
|
||||
/// filler and start filler) is faded out once the reply's playback starts,
|
||||
/// before being cut. The filler is only meant to mask latency, so it must hand
|
||||
/// off cleanly to the answer instead of running underneath it. 4 × 20 ms = 80 ms.
|
||||
const FILLER_FADEOUT_FRAMES: u32 = 4;
|
||||
|
||||
/// Shared audio mixer holding the TTS, filler, start-filler and backchannel
|
||||
/// sample queues.
|
||||
#[derive(Clone)]
|
||||
@@ -91,6 +97,11 @@ pub struct AudioMixer {
|
||||
/// `mark_stream_start` and `flush`. Used to decide whether a turn-start
|
||||
/// filler is still useful.
|
||||
tts_first_chunk: Arc<AtomicBool>,
|
||||
/// Remaining frames of the filler fade-out envelope. `0` means no fade is in
|
||||
/// progress. Armed to [`FILLER_FADEOUT_FRAMES`] when the reply's playback
|
||||
/// starts while a filler is still queued, so the latency-mask cue is faded
|
||||
/// out and cut instead of playing underneath the answer.
|
||||
filler_fade_frames: Arc<AtomicU32>,
|
||||
}
|
||||
|
||||
impl AudioMixer {
|
||||
@@ -121,6 +132,7 @@ impl AudioMixer {
|
||||
tts_stream_active: Arc::new(AtomicBool::new(false)),
|
||||
tts_playback_started: Arc::new(AtomicBool::new(false)),
|
||||
tts_first_chunk: Arc::new(AtomicBool::new(false)),
|
||||
filler_fade_frames: Arc::new(AtomicU32::new(0)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -133,11 +145,13 @@ impl AudioMixer {
|
||||
/// Mixing formula:
|
||||
/// `mixed = tts + (filler + start_filler) * filler_gain
|
||||
/// + backchannel * backchannel_gain + external * external_gain`.
|
||||
/// The filler gain is dynamically attenuated to `filler_gain_duck` while
|
||||
/// TTS is being played out, otherwise it stays at `filler_gain_idle`; the
|
||||
/// backchannel gain is fixed and low so the cue stays in the background.
|
||||
/// The start-filler channel shares the filler gain, so the immediate
|
||||
/// acknowledgment ducks out of the way once the reply starts. The external
|
||||
/// The filler gain is `filler_gain_idle` while TTS is silent and ducks to
|
||||
/// `filler_gain_duck` while TTS is being played out; the backchannel gain is
|
||||
/// fixed and low so the cue stays in the background. The start-filler
|
||||
/// channel shares the filler gain. When the reply's playback starts, a
|
||||
/// still-queued filler/start-filler is instead faded out over
|
||||
/// `FILLER_FADEOUT_FRAMES` and then cut, so the latency-mask cue hands off
|
||||
/// cleanly to the answer instead of running underneath it. The external
|
||||
/// channel is a user-injected background source with its own (runtime
|
||||
/// adjustable) gain.
|
||||
pub fn pull_frame(&self) -> Option<Vec<u8>> {
|
||||
@@ -176,6 +190,13 @@ impl AudioMixer {
|
||||
self.tts_primed.store(true, Ordering::Relaxed);
|
||||
self.tts_drain_frames.store(0, Ordering::Relaxed);
|
||||
self.tts_playback_started.store(true, Ordering::Relaxed);
|
||||
// The reply is about to play: arm the filler fade-out so any
|
||||
// latency-mask cue still queued hands off cleanly instead of
|
||||
// running underneath the answer.
|
||||
if self.filler_queue.len() > 0 || self.start_filler_queue.len() > 0 {
|
||||
self.filler_fade_frames
|
||||
.store(FILLER_FADEOUT_FRAMES, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
let tts_active = self.tts_primed.load(Ordering::Relaxed);
|
||||
@@ -201,8 +222,15 @@ impl AudioMixer {
|
||||
);
|
||||
}
|
||||
|
||||
// Determine filler gain based on whether TTS is actively playing.
|
||||
let filler_gain = if tts_active {
|
||||
// Determine filler gain. Normally the filler ducks to `filler_gain_duck`
|
||||
// while TTS plays; but once the reply's playback starts a still-queued
|
||||
// latency-mask filler is faded out (idle gain → 0) over
|
||||
// `FILLER_FADEOUT_FRAMES` and then cut, so it never runs underneath the
|
||||
// answer.
|
||||
let fade_remaining = self.filler_fade_frames.load(Ordering::Relaxed);
|
||||
let filler_gain = if fade_remaining > 0 {
|
||||
self.filler_gain_idle * (fade_remaining as f32 / FILLER_FADEOUT_FRAMES as f32)
|
||||
} else if tts_active {
|
||||
self.filler_gain_duck
|
||||
} else {
|
||||
self.filler_gain_idle
|
||||
@@ -227,6 +255,18 @@ impl AudioMixer {
|
||||
let backchannel_n = self.backchannel_queue.pop_into(&mut backchannel_buf);
|
||||
let external_n = self.external_queue.pop_into(&mut external_buf);
|
||||
|
||||
// Filler fade-out bookkeeping, done after the pops so this frame's
|
||||
// samples are already captured: decay one step per frame and drop the
|
||||
// remaining queued filler once the envelope reaches zero.
|
||||
if fade_remaining == 1 {
|
||||
self.filler_queue.clear();
|
||||
self.start_filler_queue.clear();
|
||||
self.filler_fade_frames.store(0, Ordering::Relaxed);
|
||||
} else if fade_remaining > 0 {
|
||||
self.filler_fade_frames
|
||||
.store(fade_remaining - 1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
// If no channel produced anything, skip sending a frame.
|
||||
if tts_n == 0
|
||||
&& filler_n == 0
|
||||
@@ -272,6 +312,7 @@ impl AudioMixer {
|
||||
self.tts_first_chunk.store(false, Ordering::Relaxed);
|
||||
self.tts_primed.store(false, Ordering::Relaxed);
|
||||
self.tts_drain_frames.store(0, Ordering::Relaxed);
|
||||
self.filler_fade_frames.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Signal that the TTS stream has finished producing audio. The mixer will
|
||||
@@ -294,6 +335,20 @@ impl AudioMixer {
|
||||
|| self.start_filler_queue.len() > 0
|
||||
}
|
||||
|
||||
/// Whether the agent has *audible* output: the reply is being played out, or
|
||||
/// a latency-mask cue (filler / start filler) is queued and about to play.
|
||||
///
|
||||
/// Unlike [`Self::is_active`] this is `false` while the reply is still being
|
||||
/// synthesised or is sitting in the pre-buffer (only the TTS stream is in
|
||||
/// flight) — nothing has reached the user's ears yet. Used to gate
|
||||
/// VAD-driven barge-in so a transient or the tail of the user's own utterance
|
||||
/// cannot cancel a reply before the user has heard any of it.
|
||||
pub fn is_audible(&self) -> bool {
|
||||
self.tts_primed.load(Ordering::Relaxed)
|
||||
|| self.filler_queue.len() > 0
|
||||
|| self.start_filler_queue.len() > 0
|
||||
}
|
||||
|
||||
/// Whether the response audio has started playing since the last
|
||||
/// `mark_stream_start`. `false` means the agent is still silent (TTS
|
||||
/// request in flight, pre-buffer not reached yet), i.e. the latency filler
|
||||
@@ -328,6 +383,7 @@ impl AudioMixer {
|
||||
self.tts_stream_active.store(false, Ordering::Relaxed);
|
||||
self.tts_playback_started.store(false, Ordering::Relaxed);
|
||||
self.tts_first_chunk.store(false, Ordering::Relaxed);
|
||||
self.filler_fade_frames.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Set the gain applied to the external audio channel. Used by the client's
|
||||
|
||||
+36
-2
@@ -84,6 +84,28 @@ pub struct Config {
|
||||
/// recitative register; clip banks that carry their own tag are left
|
||||
/// untouched.
|
||||
pub tts_style_tag: String,
|
||||
/// Post-process a freshly recorded custom voice before it is installed
|
||||
/// (`CUSTOMVOICE_POSTPROCESS`, default `true`): rumble removal, silence
|
||||
/// trimming and EBU R128 loudness normalization. Disabling it stores the
|
||||
/// plain transcode (see [`crate::customvoice`]).
|
||||
pub customvoice_postprocess: bool,
|
||||
/// Enable the noise-reduction / de-essing stage of the custom-voice
|
||||
/// post-processing (`CUSTOMVOICE_DENOISE`, default `true`). Turn it off for
|
||||
/// an already clean studio recording.
|
||||
pub customvoice_denoise: bool,
|
||||
/// Integrated loudness target in LUFS for the custom-voice normalization
|
||||
/// (`CUSTOMVOICE_TARGET_LUFS`, default `-26`). Every take lands on the same
|
||||
/// programme level, so recordings do not jump in volume. Set **below** the
|
||||
/// shipped reference (`voice_profiles/solis.wav` measures ≈-23.4 LUFS):
|
||||
/// a user take is far more dynamic than the mastered reference, so at equal
|
||||
/// LUFS its much hotter peaks make the cloned voice sound louder than the
|
||||
/// default one. The gap buys that headroom back.
|
||||
pub customvoice_target_lufs: f32,
|
||||
/// True-peak ceiling in dBTP for the custom-voice normalization
|
||||
/// (`CUSTOMVOICE_TRUE_PEAK`, default `-5`) — headroom against clipping.
|
||||
/// Kept well below 0 on purpose: a dynamic user take would otherwise peak
|
||||
/// several dB hotter than the mastered reference even at the same LUFS.
|
||||
pub customvoice_true_peak: f32,
|
||||
/// Filesystem directory containing the Parakeet TDT ONNX pipeline
|
||||
/// (`nemo128.onnx`, `encoder-model*.onnx`, `decoder_joint-model.onnx`,
|
||||
/// `vocab.txt`, `config.json`).
|
||||
@@ -148,7 +170,7 @@ pub struct Config {
|
||||
/// [`Config::start_filler_go_delay_ms`]).
|
||||
pub start_filler_enabled: bool,
|
||||
/// How long (ms) the agent waits for the response audio to start before
|
||||
/// playing the start filler (`START_FILLER_GO_DELAY`, default `1200`). Same
|
||||
/// playing the start filler (`START_FILLER_GO_DELAY`, default `1800`). Same
|
||||
/// idea as [`Config::filler_delay_ms`] — a late reply is masked with a
|
||||
/// short acknowledgment, an on-time one is not — but on the dedicated
|
||||
/// start-filler channel and with its own (shorter) delay. If playback has
|
||||
@@ -290,6 +312,10 @@ impl Default for Config {
|
||||
tts_top_p: Some(0.75),
|
||||
tts_top_k: None,
|
||||
tts_style_tag: "[conversational, casual]".to_string(),
|
||||
customvoice_postprocess: true,
|
||||
customvoice_denoise: true,
|
||||
customvoice_target_lufs: -26.0,
|
||||
customvoice_true_peak: -5.0,
|
||||
stt_model_path: PathBuf::from("models/parakeet"),
|
||||
stt_tokenizer_path: None,
|
||||
fillers_dir: PathBuf::from("assets/fillers"),
|
||||
@@ -304,7 +330,7 @@ impl Default for Config {
|
||||
media_ytdlp_bin: "yt-dlp".to_string(),
|
||||
media_ytdlp_args: Vec::new(),
|
||||
start_filler_enabled: true,
|
||||
start_filler_go_delay_ms: 1200,
|
||||
start_filler_go_delay_ms: 1800,
|
||||
start_filler_dir: PathBuf::from("assets/start_filler"),
|
||||
start_filler_texts: vec!["[calm] Un momento...".to_string()],
|
||||
tls_cert: None,
|
||||
@@ -383,6 +409,14 @@ impl Config {
|
||||
if let Ok(v) = std::env::var("TTS_STYLE_TAG") {
|
||||
cfg.tts_style_tag = v.trim().to_string();
|
||||
}
|
||||
// Custom-voice post-processing (see `crate::customvoice`).
|
||||
cfg.customvoice_postprocess =
|
||||
env_bool("CUSTOMVOICE_POSTPROCESS", cfg.customvoice_postprocess);
|
||||
cfg.customvoice_denoise = env_bool("CUSTOMVOICE_DENOISE", cfg.customvoice_denoise);
|
||||
cfg.customvoice_target_lufs =
|
||||
env_f32("CUSTOMVOICE_TARGET_LUFS", cfg.customvoice_target_lufs);
|
||||
cfg.customvoice_true_peak =
|
||||
env_f32("CUSTOMVOICE_TRUE_PEAK", cfg.customvoice_true_peak);
|
||||
if let Ok(p) = std::env::var("STT_MODEL_PATH") {
|
||||
cfg.stt_model_path = PathBuf::from(p);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
//! Custom-voice builder: turns a recording (or an uploaded audio file) into a
|
||||
//! Fish Audio voice-cloning reference.
|
||||
//!
|
||||
//! The `/customvoice` page records the user reading a phrase (at least 10 s
|
||||
//! long) **or** lets them upload an audio file and type the matching
|
||||
//! transcript. The bytes are POSTed here, transcoded to a mono PCM WAV in the
|
||||
//! directory that holds the configured reference — the same directory the Fish
|
||||
//! Audio backend already reads, so the new file is immediately usable — the
|
||||
//! transcript is written as a sibling `.txt` (the convention
|
||||
//! [`crate::config::Config::tts_reference`] falls back to), and the
|
||||
//! running TTS engine is pointed at the new file via
|
||||
//! [`crate::tts::TtsEngine::set_reference`].
|
||||
//!
|
||||
//! Before installation the take goes through a **post-processing chain**
|
||||
//! (see [`postprocess`]): high-pass to drop rumble/DC, optional FFT denoise and
|
||||
//! de-essing, leading/trailing silence trim, and a **two-pass EBU R128
|
||||
//! loudness normalization** (single linear gain, so takes never jump in
|
||||
//! volume). Any step failing falls back to the plain transcode, so a take is
|
||||
//! never lost to a filter quirk.
|
||||
//!
|
||||
//! The predefined phrases live server-side so a plain recording always carries
|
||||
//! a transcript matching the text the page asked the user to read; a
|
||||
//! caller-supplied transcript (the upload flow) overrides it.
|
||||
|
||||
use crate::config::{Config, OUTPUT_SAMPLE_RATE};
|
||||
use crate::ws::AppState;
|
||||
use axum::body::Bytes;
|
||||
use axum::extract::{Query, State};
|
||||
use axum::http::StatusCode;
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use axum::Json;
|
||||
use serde::Deserialize;
|
||||
use serde_json::{json, Value};
|
||||
use std::ffi::OsString;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Output, Stdio};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use tokio::process::Command;
|
||||
use tracing::{error, info, warn};
|
||||
|
||||
/// A predefined reading phrase, keyed by ISO 639-1 code. Long enough
|
||||
/// (~10 s at a natural pace) to give the cloning model a usable sample.
|
||||
struct Phrase {
|
||||
code: &'static str,
|
||||
label: &'static str,
|
||||
phrase: &'static str,
|
||||
}
|
||||
|
||||
/// Reading phrases for the languages supported by [`crate::lang`].
|
||||
const PHRASES: &[Phrase] = &[
|
||||
Phrase {
|
||||
code: "es",
|
||||
label: "Español",
|
||||
phrase: "Hola, esta es mi voz personal. Estoy leyendo esta frase para crear una voz \
|
||||
personalizada para mí. Hablo de forma natural y clara, para que el sistema pueda \
|
||||
aprender el timbre y el ritmo de mi voz.",
|
||||
},
|
||||
Phrase {
|
||||
code: "en",
|
||||
label: "English",
|
||||
phrase: "Hello, this is my personal voice. I am reading this sentence to create a custom \
|
||||
voice for me. I speak in a natural and clear way, so the system can learn the \
|
||||
tone and the rhythm of my voice.",
|
||||
},
|
||||
Phrase {
|
||||
code: "it",
|
||||
label: "Italiano",
|
||||
phrase: "Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una \
|
||||
voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può \
|
||||
imparare il timbro e il ritmo della mia voce.",
|
||||
},
|
||||
Phrase {
|
||||
code: "fr",
|
||||
label: "Français",
|
||||
phrase: "Bonjour, c'est ma voix personnelle. Je lis cette phrase pour créer une voix \
|
||||
personnalisée pour moi. Je parle de manière naturelle et claire, afin que le \
|
||||
système puisse apprendre le timbre et le rythme de ma voix.",
|
||||
},
|
||||
Phrase {
|
||||
code: "de",
|
||||
label: "Deutsch",
|
||||
phrase: "Hallo, das ist meine persönliche Stimme. Ich lese diesen Satz, um eine eigene \
|
||||
Stimme für mich zu erstellen. Ich spreche natürlich und deutlich, damit das \
|
||||
System die Klangfarbe und den Rhythmus meiner Stimme lernen kann.",
|
||||
},
|
||||
Phrase {
|
||||
code: "pt",
|
||||
label: "Português",
|
||||
phrase: "Olá, esta é a minha voz pessoal. Estou a ler esta frase para criar uma voz \
|
||||
personalizada para mim. Falo de forma natural e clara, para que o sistema possa \
|
||||
aprender o timbre e o ritmo da minha voz.",
|
||||
},
|
||||
];
|
||||
|
||||
/// `GET /api/customvoice/phrases` — the reading phrases and the configured
|
||||
/// default language, used to populate the page's language selector.
|
||||
pub async fn phrases(State(state): State<AppState>) -> Json<Value> {
|
||||
let languages: Vec<Value> = PHRASES
|
||||
.iter()
|
||||
.map(|p| json!({ "code": p.code, "label": p.label, "phrase": p.phrase }))
|
||||
.collect();
|
||||
Json(json!({
|
||||
"default": state.config.lang_default,
|
||||
"languages": languages,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Query string of `POST /api/customvoice`.
|
||||
#[derive(Deserialize)]
|
||||
pub struct SetParams {
|
||||
/// ISO 639-1 code of the language the user recorded in.
|
||||
pub lang: String,
|
||||
/// Optional transcript override: the text actually spoken in the audio.
|
||||
/// Used by the upload flow, where the user types (or pastes) it. When
|
||||
/// absent or blank, the language's predefined reading phrase is used.
|
||||
pub text: Option<String>,
|
||||
}
|
||||
|
||||
/// `POST /api/customvoice?lang=<code>[&text=<transcript>]` — body is the raw
|
||||
/// audio (a browser recording or an uploaded file; whatever container the
|
||||
/// client sent, ffmpeg probes the content). Transcodes it to a WAV reference,
|
||||
/// persists the transcript, and installs the voice for the running session.
|
||||
pub async fn set_custom_voice(
|
||||
State(state): State<AppState>,
|
||||
Query(params): Query<SetParams>,
|
||||
body: Bytes,
|
||||
) -> Response {
|
||||
let lang = params.lang.trim().to_ascii_lowercase();
|
||||
let Some(entry) = PHRASES.iter().find(|p| p.code == lang) else {
|
||||
return error_response(StatusCode::BAD_REQUEST, "unsupported language");
|
||||
};
|
||||
if body.is_empty() {
|
||||
return error_response(StatusCode::BAD_REQUEST, "empty recording");
|
||||
}
|
||||
|
||||
// Transcript attached to the reference: the caller-supplied text when it is
|
||||
// non-blank (the upload flow lets the user type it), otherwise the
|
||||
// language's predefined reading phrase.
|
||||
let transcript = params
|
||||
.text
|
||||
.as_deref()
|
||||
.map(str::trim)
|
||||
.filter(|t| !t.is_empty())
|
||||
.unwrap_or(entry.phrase)
|
||||
.to_string();
|
||||
|
||||
// Save next to the configured reference: that directory is already readable
|
||||
// by the Fish Audio backend, so no extra path plumbing is needed.
|
||||
let dir = reference_dir(&state.config);
|
||||
if let Err(e) = tokio::fs::create_dir_all(&dir).await {
|
||||
error!(error = %e, dir = %dir.display(), "failed to create reference directory");
|
||||
return error_response(StatusCode::INTERNAL_SERVER_ERROR, "cannot create reference directory");
|
||||
}
|
||||
|
||||
let stamp = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_millis())
|
||||
.unwrap_or(0);
|
||||
let base = format!("custom_{}_{}", entry.code, stamp);
|
||||
let raw_path = dir.join(format!("{base}.upload"));
|
||||
let wav_path = dir.join(format!("{base}.wav"));
|
||||
let txt_path = dir.join(format!("{base}.txt"));
|
||||
|
||||
if let Err(e) = tokio::fs::write(&raw_path, &body).await {
|
||||
error!(error = %e, path = %raw_path.display(), "failed to store upload");
|
||||
return error_response(StatusCode::INTERNAL_SERVER_ERROR, "cannot store upload");
|
||||
}
|
||||
|
||||
match render_reference(&state.config, &raw_path, &wav_path).await {
|
||||
Ok(()) => {}
|
||||
Err(msg) => {
|
||||
let _ = tokio::fs::remove_file(&raw_path).await;
|
||||
error!(error = %msg, "custom-voice transcode failed");
|
||||
return error_response(StatusCode::BAD_GATEWAY, &msg);
|
||||
}
|
||||
}
|
||||
let _ = tokio::fs::remove_file(&raw_path).await;
|
||||
|
||||
if let Err(e) = tokio::fs::write(&txt_path, &transcript).await {
|
||||
// Non-fatal: the in-memory reference below still carries the transcript.
|
||||
warn!(error = %e, path = %txt_path.display(), "failed to persist reference transcript");
|
||||
}
|
||||
|
||||
let audio_path = wav_path.to_string_lossy().into_owned();
|
||||
state
|
||||
.tts
|
||||
.set_reference(Some(audio_path.clone()), Some(transcript.clone()));
|
||||
info!(lang = %entry.code, audio = %audio_path, "custom voice installed");
|
||||
|
||||
(
|
||||
StatusCode::OK,
|
||||
Json(json!({
|
||||
"ok": true,
|
||||
"lang": entry.code,
|
||||
"audio": audio_path,
|
||||
"phrase": transcript,
|
||||
})),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
/// `POST /api/customvoice/clear` — drop the custom voice and restore the
|
||||
/// reference configured at startup (the `TTS_REFERENCE_AUDIO` voice), or no
|
||||
/// reference at all when none is configured.
|
||||
pub async fn clear_custom_voice(State(state): State<AppState>) -> Json<Value> {
|
||||
match state.config.tts_reference() {
|
||||
Some((audio_path, text)) => {
|
||||
info!(audio = %audio_path, "custom voice cleared — restored configured reference");
|
||||
state.tts.set_reference(Some(audio_path), Some(text));
|
||||
}
|
||||
None => {
|
||||
info!("custom voice cleared — no reference configured");
|
||||
state.tts.set_reference(None, None);
|
||||
}
|
||||
}
|
||||
Json(json!({ "ok": true }))
|
||||
}
|
||||
|
||||
/// The directory holding voice-cloning references: the parent of the
|
||||
/// configured reference WAV, or `voice_profiles` when nothing is configured.
|
||||
fn reference_dir(config: &Config) -> PathBuf {
|
||||
config
|
||||
.tts_reference_audio
|
||||
.as_ref()
|
||||
.and_then(|p| p.parent())
|
||||
.filter(|p| !p.as_os_str().is_empty())
|
||||
.map(PathBuf::from)
|
||||
.unwrap_or_else(|| PathBuf::from("voice_profiles"))
|
||||
}
|
||||
|
||||
/// Produce the final WAV reference: transcode the upload and, unless disabled,
|
||||
/// run the post-processing chain. Any post-processing failure degrades to the
|
||||
/// plain transcode rather than losing the take.
|
||||
async fn render_reference(config: &Config, input: &Path, output: &Path) -> Result<(), String> {
|
||||
if config.customvoice_postprocess {
|
||||
match postprocess(config, input, output).await {
|
||||
Ok(()) => return Ok(()),
|
||||
Err(e) => warn!(
|
||||
error = %e,
|
||||
"custom-voice post-processing failed — storing the plain transcode"
|
||||
),
|
||||
}
|
||||
}
|
||||
transcode_to_wav(config, input, output).await
|
||||
}
|
||||
|
||||
/// Two-pass EBU R128 normalization over the cleaned signal: measure, then apply
|
||||
/// the measurement as a **single linear gain**. One gain for the whole take is
|
||||
/// what keeps the level consistent instead of pumping frame by frame — the
|
||||
/// usual cause of uneven volume in a cloned reference.
|
||||
async fn postprocess(config: &Config, input: &Path, output: &Path) -> Result<(), String> {
|
||||
let chain = cleanup_chain(config.customvoice_denoise);
|
||||
let opts = format!(
|
||||
"I={}:TP={}:LRA=11",
|
||||
config.customvoice_target_lufs, config.customvoice_true_peak
|
||||
);
|
||||
|
||||
// Pass 1 — measure. loudnorm prints its JSON summary through the logger, so
|
||||
// this run keeps the default log level.
|
||||
let measure_args: Vec<OsString> = vec![
|
||||
"-hide_banner".into(),
|
||||
"-nostdin".into(),
|
||||
"-nostats".into(),
|
||||
"-i".into(),
|
||||
input.into(),
|
||||
"-af".into(),
|
||||
format!("{chain},loudnorm={opts}:print_format=json").into(),
|
||||
"-f".into(),
|
||||
"null".into(),
|
||||
"-".into(),
|
||||
];
|
||||
let measured = run_ffmpeg(config, &measure_args).await?;
|
||||
let measured_stderr = String::from_utf8_lossy(&measured.stderr);
|
||||
let m = parse_loudnorm(&measured_stderr)
|
||||
.ok_or_else(|| "could not read the loudness measurement".to_string())?;
|
||||
|
||||
// Pass 2 — apply the measurement as a linear gain and write the reference.
|
||||
let apply = format!(
|
||||
"{chain},loudnorm={opts}:measured_I={}:measured_TP={}:measured_LRA={}:\
|
||||
measured_thresh={}:offset={}:linear=true",
|
||||
m.input_i, m.input_tp, m.input_lra, m.input_thresh, m.target_offset
|
||||
);
|
||||
let args: Vec<OsString> = vec![
|
||||
"-hide_banner".into(),
|
||||
"-nostdin".into(),
|
||||
"-loglevel".into(),
|
||||
"error".into(),
|
||||
"-y".into(),
|
||||
"-i".into(),
|
||||
input.into(),
|
||||
"-af".into(),
|
||||
apply.into(),
|
||||
"-ar".into(),
|
||||
OUTPUT_SAMPLE_RATE.to_string().into(),
|
||||
"-ac".into(),
|
||||
"1".into(),
|
||||
"-c:a".into(),
|
||||
"pcm_s16le".into(),
|
||||
output.into(),
|
||||
];
|
||||
run_ffmpeg(config, &args).await.map(|_| ())
|
||||
}
|
||||
|
||||
/// The cleanup filters shared by both loudnorm passes: downmix to mono, then
|
||||
/// rumble/DC removal, optional FFT denoise and de-essing, and a
|
||||
/// leading/trailing silence trim (the clip is reversed to trim the tail with
|
||||
/// the same filter).
|
||||
///
|
||||
/// The mono downmix comes **first** on purpose: loudnorm measures whatever
|
||||
/// reaches it, so measuring a stereo input and only folding it to mono at the
|
||||
/// output stage would land the written file ~3 dB below the target (the energy
|
||||
/// of two channels collapses into one).
|
||||
fn cleanup_chain(denoise: bool) -> String {
|
||||
let trim = "silenceremove=start_periods=1:start_duration=0.1:\
|
||||
start_threshold=-50dB:start_silence=0.05";
|
||||
let mut filters: Vec<&str> = vec!["aformat=channel_layouts=mono", "highpass=f=80"];
|
||||
if denoise {
|
||||
filters.push("afftdn=nr=12:nf=-40");
|
||||
filters.push("deesser");
|
||||
}
|
||||
filters.extend([trim, "areverse", trim, "areverse"]);
|
||||
filters.join(",")
|
||||
}
|
||||
|
||||
/// Transcode `input` to a mono 16-bit PCM WAV at the mixer's sample rate.
|
||||
async fn transcode_to_wav(config: &Config, input: &Path, output: &Path) -> Result<(), String> {
|
||||
let args: Vec<OsString> = vec![
|
||||
"-hide_banner".into(),
|
||||
"-nostdin".into(),
|
||||
"-loglevel".into(),
|
||||
"error".into(),
|
||||
"-y".into(),
|
||||
"-i".into(),
|
||||
input.into(),
|
||||
"-ac".into(),
|
||||
"1".into(),
|
||||
"-ar".into(),
|
||||
OUTPUT_SAMPLE_RATE.to_string().into(),
|
||||
"-c:a".into(),
|
||||
"pcm_s16le".into(),
|
||||
output.into(),
|
||||
];
|
||||
run_ffmpeg(config, &args).await.map(|_| ())
|
||||
}
|
||||
|
||||
/// Run ffmpeg with `args`. Output is discarded; on a non-zero exit the last
|
||||
/// stderr line is surfaced as the error message.
|
||||
async fn run_ffmpeg(config: &Config, args: &[OsString]) -> Result<Output, String> {
|
||||
let out = Command::new(&config.media_ffmpeg_bin)
|
||||
.args(args)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::piped())
|
||||
.output()
|
||||
.await
|
||||
.map_err(|e| format!("failed to spawn ffmpeg ({}): {e}", config.media_ffmpeg_bin))?;
|
||||
|
||||
if !out.status.success() {
|
||||
let err = String::from_utf8_lossy(&out.stderr);
|
||||
let last = err.trim().lines().last().unwrap_or("").to_string();
|
||||
return Err(if last.is_empty() {
|
||||
format!("ffmpeg exited with {}", out.status)
|
||||
} else {
|
||||
format!("ffmpeg: {last}")
|
||||
});
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// The loudness figures loudnorm reports in its JSON summary.
|
||||
struct Loudness {
|
||||
input_i: f64,
|
||||
input_tp: f64,
|
||||
input_lra: f64,
|
||||
input_thresh: f64,
|
||||
target_offset: f64,
|
||||
}
|
||||
|
||||
/// Extract the loudnorm JSON summary from ffmpeg's log output.
|
||||
fn parse_loudnorm(stderr: &str) -> Option<Loudness> {
|
||||
let start = stderr.rfind('{')?;
|
||||
let end = start + stderr[start..].find('}')?;
|
||||
let value: Value = serde_json::from_str(&stderr[start..=end]).ok()?;
|
||||
let num = |key: &str| value.get(key)?.as_str()?.trim().parse::<f64>().ok();
|
||||
Some(Loudness {
|
||||
input_i: num("input_i")?,
|
||||
input_tp: num("input_tp")?,
|
||||
input_lra: num("input_lra")?,
|
||||
input_thresh: num("input_thresh")?,
|
||||
target_offset: num("target_offset")?,
|
||||
})
|
||||
}
|
||||
|
||||
/// Build a JSON error response.
|
||||
fn error_response(status: StatusCode, msg: &str) -> Response {
|
||||
(status, Json(json!({ "ok": false, "error": msg }))).into_response()
|
||||
}
|
||||
+59
@@ -61,6 +61,14 @@ impl ChatMessage {
|
||||
}
|
||||
}
|
||||
|
||||
/// A completed non-streaming reply: the visible text plus any tool calls the
|
||||
/// server returned, parsed either from the standard `tool_calls` array or from
|
||||
/// the model's native `<tool_call>` text (see [`crate::tools`]).
|
||||
pub struct ChatReply {
|
||||
pub text: String,
|
||||
pub tool_calls: Vec<crate::tools::ToolCall>,
|
||||
}
|
||||
|
||||
/// Client for an OpenAI-compatible `/chat/completions` endpoint.
|
||||
#[derive(Clone)]
|
||||
pub struct LlmClient {
|
||||
@@ -218,6 +226,57 @@ impl LlmClient {
|
||||
Ok(self.clean(&self.delta_text(message)))
|
||||
}
|
||||
|
||||
/// Non-streaming request that also surfaces tool calls.
|
||||
///
|
||||
/// Used as a fallback for a streaming round that came back empty. Several
|
||||
/// OpenAI-compatible servers (notably vLLM with a `--tool-call-parser` that
|
||||
/// does not match the model's native format) drop the call on the SSE path
|
||||
/// — the stream ends with no content — while the very same request without
|
||||
/// `stream` returns it intact. The call is parsed from the standard
|
||||
/// `tool_calls` field when present, otherwise from the `<tool_call>` text
|
||||
/// inside `content`.
|
||||
pub async fn chat_reply(
|
||||
&self,
|
||||
messages: &[ChatMessage],
|
||||
cancel: &CancellationToken,
|
||||
) -> Result<ChatReply> {
|
||||
let body = self.request_body(messages, false);
|
||||
let req = self
|
||||
.http
|
||||
.post(&self.endpoint)
|
||||
.header("Authorization", format!("Bearer {}", self.api_key))
|
||||
.json(&body)
|
||||
.send();
|
||||
|
||||
let resp = tokio::select! {
|
||||
r = req => r?,
|
||||
_ = cancel.cancelled() => {
|
||||
return Ok(ChatReply { text: String::new(), tool_calls: Vec::new() });
|
||||
}
|
||||
};
|
||||
|
||||
let status = resp.status();
|
||||
if !status.is_success() {
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
return Err(anyhow!("LLM HTTP {}: {}", status, text.trim()));
|
||||
}
|
||||
let value: serde_json::Value = resp.json().await?;
|
||||
let message = &value["choices"][0]["message"];
|
||||
|
||||
let mut tool_calls = crate::tools::parse_openai_tool_calls(message);
|
||||
let mut text = self.clean(&self.delta_text(message));
|
||||
if tool_calls.is_empty() {
|
||||
// No structured call: the server may have left the model's native
|
||||
// `<tool_call>` block in the text (mis-matched tool parser).
|
||||
let (visible, calls) = crate::tools::split_tool_calls(&text);
|
||||
if !calls.is_empty() {
|
||||
tool_calls = calls;
|
||||
text = visible;
|
||||
}
|
||||
}
|
||||
Ok(ChatReply { text, tool_calls })
|
||||
}
|
||||
|
||||
/// One-off non-streaming completion with an explicit system/user pair, its
|
||||
/// own token budget and temperature, and **no tools advertised**.
|
||||
///
|
||||
|
||||
+18
-2
@@ -7,6 +7,7 @@ mod assets;
|
||||
mod audio;
|
||||
mod audit;
|
||||
mod config;
|
||||
mod customvoice;
|
||||
mod lang;
|
||||
mod llm;
|
||||
mod media;
|
||||
@@ -25,11 +26,12 @@ use crate::orchestrator::FillerBank;
|
||||
use crate::stt::{ParakeetStt, Tokenizer};
|
||||
use crate::tts::create_tts_engine;
|
||||
use crate::vad::SileroVad;
|
||||
use crate::web::{MIVOCERO_HTML, PLAYGROUND_HTML};
|
||||
use crate::web::{CUSTOMVOICE_HTML, MIVOCERO_HTML, PLAYGROUND_HTML};
|
||||
use crate::ws::{ws_handler, AppState};
|
||||
use axum::extract::DefaultBodyLimit;
|
||||
use axum::http::header::{CONTENT_TYPE, CACHE_CONTROL};
|
||||
use axum::response::{Html, IntoResponse, Response};
|
||||
use axum::routing::get;
|
||||
use axum::routing::{get, post};
|
||||
use axum::Router;
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
@@ -232,6 +234,15 @@ async fn main() -> anyhow::Result<()> {
|
||||
let app = Router::new()
|
||||
.route("/", get(playground))
|
||||
.route("/mivocero", get(mivocero))
|
||||
.route("/customvoice", get(customvoice_page))
|
||||
// A recording reference is a few hundred KB, but the client may upload
|
||||
// an uncompressed WAV; lift the default 2 MB body cap for this route.
|
||||
.route(
|
||||
"/api/customvoice",
|
||||
post(customvoice::set_custom_voice).layer(DefaultBodyLimit::max(32 * 1024 * 1024)),
|
||||
)
|
||||
.route("/api/customvoice/clear", post(customvoice::clear_custom_voice))
|
||||
.route("/api/customvoice/phrases", get(customvoice::phrases))
|
||||
.route("/health", get(health))
|
||||
.route("/ws", get(ws_handler))
|
||||
.with_state(app_state);
|
||||
@@ -281,6 +292,11 @@ async fn mivocero() -> Response {
|
||||
html_response(MIVOCERO_HTML)
|
||||
}
|
||||
|
||||
/// `GET /customvoice` — serve the voice-cloning builder page.
|
||||
async fn customvoice_page() -> Response {
|
||||
html_response(CUSTOMVOICE_HTML)
|
||||
}
|
||||
|
||||
/// Build an HTML response with caching disabled.
|
||||
fn html_response(body: &'static str) -> Response {
|
||||
let mut resp = Html(body).into_response();
|
||||
|
||||
@@ -371,13 +371,19 @@ impl TurnManager {
|
||||
// external tokenizer only if no text was produced.
|
||||
let text = if !transcript.text.trim().is_empty() {
|
||||
transcript.text.trim().to_string()
|
||||
} else if transcript.token_ids.is_empty() {
|
||||
// No tokens at all: nothing was said (a VAD false-trigger that
|
||||
// slipped past the gate). Skip the turn instead of forwarding the
|
||||
// "<transcript 0 tokens>" placeholder to the LLM.
|
||||
debug!("no tokens decoded, skipping turn");
|
||||
return;
|
||||
} else {
|
||||
match &self.tokenizer {
|
||||
Some(tok) => tok.decode(&transcript.token_ids),
|
||||
None => format!("<transcript {} tokens>", transcript.token_ids.len()),
|
||||
}
|
||||
};
|
||||
if text.is_empty() {
|
||||
if text.trim().is_empty() {
|
||||
debug!("empty transcript after decode, skipping turn");
|
||||
return;
|
||||
}
|
||||
@@ -592,9 +598,10 @@ impl TurnManager {
|
||||
// Tool-calling loop. Each round streams one reply; tool call blocks are
|
||||
// stripped from the spoken/displayed text (the model emits them as
|
||||
// text — see `crate::tools`), the calls run over HTTP, and the results
|
||||
// are handed back for the next round. Bounded by
|
||||
// `LLM_TOOL_MAX_ITERATIONS`. With no tools configured the loop runs
|
||||
// exactly once.
|
||||
// are handed back for the next round. A round that streams back empty
|
||||
// is re-asked without streaming, where the server returns the call it
|
||||
// dropped from the SSE path. Bounded by `LLM_TOOL_MAX_ITERATIONS`. With
|
||||
// no tools configured the loop runs exactly once.
|
||||
let registry = llm.tools().cloned();
|
||||
let max_rounds = self.config.llm_tool_max_iterations.max(1);
|
||||
let mut spoken = String::new();
|
||||
@@ -678,7 +685,36 @@ impl TurnManager {
|
||||
break;
|
||||
}
|
||||
|
||||
let calls = stripper.take_calls();
|
||||
let mut calls = stripper.take_calls();
|
||||
|
||||
// Streaming rescue. Some OpenAI-compatible servers swallow the
|
||||
// model's native tool call on the SSE path — vLLM with a
|
||||
// `--tool-call-parser` that does not match the model's format ends
|
||||
// the stream with no content at all — while the same request
|
||||
// *without* streaming returns the call intact. When a round
|
||||
// produced nothing and tools are configured, re-ask once,
|
||||
// non-streaming, and take the call from there.
|
||||
if calls.is_empty()
|
||||
&& registry.is_some()
|
||||
&& !cancel.is_cancelled()
|
||||
&& spoken.trim().is_empty()
|
||||
&& round + 1 < max_rounds
|
||||
{
|
||||
match llm.chat_reply(&messages, &cancel).await {
|
||||
Ok(reply) => {
|
||||
calls = reply.tool_calls;
|
||||
if calls.is_empty() && !reply.text.trim().is_empty() {
|
||||
// No call after all: the blocking reply holds the
|
||||
// actual answer — speak it rather than go silent.
|
||||
self.emit(json!({ "type": "assistant_delta", "text": &reply.text }));
|
||||
spoken.push_str(&reply.text);
|
||||
let _ = sentence_tx.send(reply.text.trim().to_string());
|
||||
}
|
||||
}
|
||||
Err(e) => error!(error = %e, "tool-call fallback request failed"),
|
||||
}
|
||||
}
|
||||
|
||||
if calls.is_empty() {
|
||||
break;
|
||||
}
|
||||
@@ -730,11 +766,21 @@ impl TurnManager {
|
||||
}
|
||||
|
||||
/// Whether the agent is currently speaking (TTS in flight or queued audio).
|
||||
/// Used by the VAD-driven barge-in to avoid flushing an idle mixer.
|
||||
/// Used by the backchannel gate to avoid cuing on top of the agent's own
|
||||
/// TTS — see [`Self::is_audible`] for the barge-in gate.
|
||||
pub fn is_speaking(&self) -> bool {
|
||||
self.mixer.is_active()
|
||||
}
|
||||
|
||||
/// Whether the agent has *audible* output (reply playing out, or a
|
||||
/// latency-mask cue queued). Unlike [`Self::is_speaking`] this is `false`
|
||||
/// while the reply is still being synthesised or sits in the pre-buffer.
|
||||
/// Used by the VAD-driven barge-in: a transient must not cancel a reply
|
||||
/// before the user has heard any of it.
|
||||
pub fn is_audible(&self) -> bool {
|
||||
self.mixer.is_audible()
|
||||
}
|
||||
|
||||
/// Update the voice-cloning reference at runtime. Pass `None` for
|
||||
/// `audio_path` to clear the reference.
|
||||
pub fn set_reference(&self, audio_path: Option<String>, text: Option<String>) {
|
||||
|
||||
@@ -341,6 +341,32 @@ fn coerce_value(raw: &str) -> Value {
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse the structured `tool_calls` array of an OpenAI chat `message` object.
|
||||
///
|
||||
/// Used when the server *does* run a matching tool parser and returns the call
|
||||
/// in the standard shape — `message.tool_calls[].function.{name,arguments}` —
|
||||
/// as opposed to the native `<tool_call>` text handled by [`split_tool_calls`].
|
||||
/// Per the OpenAI spec `arguments` is a JSON-encoded string; a payload that does
|
||||
/// not decode to an object yields an empty argument set.
|
||||
pub fn parse_openai_tool_calls(message: &Value) -> Vec<ToolCall> {
|
||||
let Some(items) = message["tool_calls"].as_array() else {
|
||||
return Vec::new();
|
||||
};
|
||||
let mut calls = Vec::new();
|
||||
for item in items {
|
||||
let Some(name) = item["function"]["name"].as_str() else {
|
||||
continue;
|
||||
};
|
||||
let raw = item["function"]["arguments"].as_str().unwrap_or("");
|
||||
let arguments = serde_json::from_str::<Value>(raw.trim())
|
||||
.ok()
|
||||
.filter(Value::is_object)
|
||||
.unwrap_or_else(|| Value::Object(Map::new()));
|
||||
calls.push(ToolCall { name: name.to_string(), arguments });
|
||||
}
|
||||
calls
|
||||
}
|
||||
|
||||
// --- Streaming stripper -----------------------------------------------------
|
||||
|
||||
/// Incremental filter that turns a stream of raw model deltas into visible
|
||||
@@ -472,4 +498,32 @@ mod tests {
|
||||
assert_eq!(s.push("Nessun tool qui."), "Nessun tool qui.");
|
||||
assert!(s.take_calls().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_structured_tool_calls() {
|
||||
let message = serde_json::json!({
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"tool_calls": [{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
|
||||
}]
|
||||
});
|
||||
let calls = parse_openai_tool_calls(&message);
|
||||
assert_eq!(calls.len(), 1);
|
||||
assert_eq!(calls[0].name, "get_weather");
|
||||
assert_eq!(calls[0].arguments["city"], "Paris");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn structured_tool_calls_absent_or_malformed() {
|
||||
assert!(parse_openai_tool_calls(&serde_json::json!({ "content": "hi" })).is_empty());
|
||||
let message = serde_json::json!({
|
||||
"tool_calls": [{ "function": { "name": "f", "arguments": "not json" } }]
|
||||
});
|
||||
let calls = parse_openai_tool_calls(&message);
|
||||
assert_eq!(calls.len(), 1);
|
||||
assert!(calls[0].arguments.as_object().unwrap().is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
+19
-1
@@ -44,6 +44,20 @@ pub struct VadSettings {
|
||||
pub speech_pad_ms: u32,
|
||||
/// Speech required to fire the barge-in onset (`VAD_BARGE_IN_MIN_SPEECH_MS`).
|
||||
pub barge_in_min_speech_ms: u32,
|
||||
/// Drop an utterance shorter than this before transcription
|
||||
/// (`VAD_MIN_UTTERANCE_MS`, default 300). `0` disables the check. Filters
|
||||
/// the short VAD false-triggers (breaths, clicks, echo).
|
||||
pub min_utterance_ms: u32,
|
||||
/// Drop an utterance whose average speech probability is below this before
|
||||
/// transcription (`VAD_MIN_AVG_PROB`, default 0.25). `0.0` disables the
|
||||
/// check. Filters the soft/noisy segments that the STT would otherwise
|
||||
/// turn into hallucinated text. The average is taken over the **voiced
|
||||
/// windows only** (probability ≥ `neg_threshold`), so the trailing
|
||||
/// hangover silence (`min_silence_ms`) does not dilute short utterances
|
||||
/// like "uno" / "due". A genuine utterance still fluctuates around the
|
||||
/// speech threshold on its onset/release, so keep this well below
|
||||
/// `threshold` (default 0.5).
|
||||
pub min_avg_prob: f32,
|
||||
/// Feed the STT engine with VAD-segmented utterances instead of letting it
|
||||
/// run its own RMS segmentation (`VAD_SEGMENT_STT`).
|
||||
pub segment_stt: bool,
|
||||
@@ -62,9 +76,11 @@ impl Default for VadSettings {
|
||||
threshold: 0.5,
|
||||
neg_threshold: 0.35,
|
||||
min_speech_ms: 250,
|
||||
min_silence_ms: 300,
|
||||
min_silence_ms: 700,
|
||||
speech_pad_ms: 30,
|
||||
barge_in_min_speech_ms: 150,
|
||||
min_utterance_ms: 300,
|
||||
min_avg_prob: 0.25,
|
||||
segment_stt: true,
|
||||
barge_in: true,
|
||||
debug: false,
|
||||
@@ -91,6 +107,8 @@ impl VadSettings {
|
||||
s.min_silence_ms = env_u32("VAD_MIN_SILENCE_MS", s.min_silence_ms);
|
||||
s.speech_pad_ms = env_u32("VAD_SPEECH_PAD_MS", s.speech_pad_ms);
|
||||
s.barge_in_min_speech_ms = env_u32("VAD_BARGE_IN_MIN_SPEECH_MS", s.barge_in_min_speech_ms);
|
||||
s.min_utterance_ms = env_u32("VAD_MIN_UTTERANCE_MS", s.min_utterance_ms);
|
||||
s.min_avg_prob = env_f32("VAD_MIN_AVG_PROB", s.min_avg_prob);
|
||||
|
||||
// Hysteresis must stay below the speech threshold, and the fast
|
||||
// barge-in onset can never be slower than the utterance onset.
|
||||
|
||||
+9
-1
@@ -219,8 +219,10 @@ pub struct VadDetector {
|
||||
speech_run_ms: u32,
|
||||
/// Consecutive silence milliseconds while in speech.
|
||||
silence_run_ms: u32,
|
||||
/// Running probability average over the current utterance.
|
||||
/// Running sum of speech probabilities over the voiced windows of the
|
||||
/// current utterance.
|
||||
prob_sum: f32,
|
||||
/// Number of voiced windows contributing to `prob_sum`.
|
||||
speech_windows: u64,
|
||||
/// Windows analysed since the detector was created (for logging).
|
||||
window_count: u64,
|
||||
@@ -365,8 +367,14 @@ impl VadDetector {
|
||||
}
|
||||
|
||||
self.utterance.extend_from_slice(window);
|
||||
// Average only over the voiced windows. The trailing hangover (the
|
||||
// `min_silence_ms` of silence that closes the segment) must not dilute
|
||||
// the score: for a short word like "uno" it is most of the segment and
|
||||
// would drag the mean below any sane gate.
|
||||
if prob >= self.settings.neg_threshold {
|
||||
self.prob_sum += prob;
|
||||
self.speech_windows += 1;
|
||||
}
|
||||
|
||||
if prob < self.settings.neg_threshold {
|
||||
self.silence_run_ms = self.silence_run_ms.saturating_add(WINDOW_MS);
|
||||
|
||||
@@ -0,0 +1,568 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Custom Voice — Mivocero</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0b0f1a;
|
||||
--panel: #141b2d;
|
||||
--accent: #4f8cff;
|
||||
--accent-dim: #2a4a8a;
|
||||
--ok: #3ddc97;
|
||||
--warn: #ffb454;
|
||||
--err: #ff5470;
|
||||
--text: #e6edf7;
|
||||
--muted: #8a94a6;
|
||||
--border: #243049;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { height: 100%; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
padding: 18px;
|
||||
}
|
||||
.app {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
width: 100%;
|
||||
max-width: 820px;
|
||||
height: calc(100vh - 36px);
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 14px;
|
||||
overflow: hidden;
|
||||
}
|
||||
header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding: 14px 18px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
header h1 { font-size: 16px; margin: 0; flex: 1; }
|
||||
main { flex: 1; overflow-y: auto; padding: 20px 22px; }
|
||||
.step { margin-bottom: 22px; }
|
||||
.step h2 {
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: .7px;
|
||||
color: var(--muted);
|
||||
margin: 0 0 8px;
|
||||
}
|
||||
select {
|
||||
width: 100%;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 10px 12px;
|
||||
font-size: 14px;
|
||||
}
|
||||
.phrase {
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-left: 3px solid var(--accent);
|
||||
border-radius: 8px;
|
||||
padding: 14px 16px;
|
||||
font-size: 15px;
|
||||
line-height: 1.55;
|
||||
color: var(--text);
|
||||
}
|
||||
.actions { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; }
|
||||
button {
|
||||
background: var(--accent-dim);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--accent);
|
||||
border-radius: 8px;
|
||||
padding: 9px 16px;
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
transition: all .15s;
|
||||
}
|
||||
button:hover:not(:disabled) { background: var(--accent); }
|
||||
button:disabled { opacity: .4; cursor: not-allowed; }
|
||||
button.danger { border-color: var(--err); background: rgba(255,84,112,.15); }
|
||||
button.danger:hover:not(:disabled) { background: var(--err); }
|
||||
button.ghost { background: transparent; border-color: var(--border); }
|
||||
button.ghost:hover:not(:disabled) { background: var(--bg); border-color: var(--accent); }
|
||||
.timer {
|
||||
font-variant-numeric: tabular-nums;
|
||||
font-size: 15px;
|
||||
color: var(--muted);
|
||||
min-width: 74px;
|
||||
}
|
||||
.timer.ready { color: var(--ok); }
|
||||
.timer.rec { color: var(--err); }
|
||||
.rec-dot {
|
||||
width: 10px; height: 10px; border-radius: 50%;
|
||||
background: var(--err); display: none; flex: none;
|
||||
}
|
||||
.recording .rec-dot { display: inline-block; animation: pulse 1s infinite; }
|
||||
@keyframes pulse { 0%,100% { opacity: 1; } 50% { opacity: .25; } }
|
||||
.bar { height: 5px; background: var(--bg); border: 1px solid var(--border); border-radius: 999px; overflow: hidden; margin-top: 12px; }
|
||||
.bar > div { height: 100%; width: 0; background: var(--accent); transition: width .12s linear; }
|
||||
.bar.ready > div { background: var(--ok); }
|
||||
audio { width: 100%; margin-top: 12px; }
|
||||
.tabs { display: flex; gap: 8px; }
|
||||
.tab {
|
||||
flex: 1;
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
color: var(--muted);
|
||||
border-radius: 8px;
|
||||
padding: 10px 12px;
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
transition: all .15s;
|
||||
}
|
||||
.tab:hover { border-color: var(--accent); color: var(--text); }
|
||||
.tab.active { background: var(--accent-dim); border-color: var(--accent); color: var(--text); }
|
||||
input[type=file] {
|
||||
width: 100%;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
border: 1px dashed var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 12px;
|
||||
font-size: 13px;
|
||||
}
|
||||
input[type=file]::file-selector-button {
|
||||
background: var(--accent-dim);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--accent);
|
||||
border-radius: 6px;
|
||||
padding: 6px 12px;
|
||||
margin-right: 12px;
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
}
|
||||
textarea {
|
||||
width: 100%;
|
||||
min-height: 90px;
|
||||
resize: vertical;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 10px 12px;
|
||||
font-size: 14px;
|
||||
font-family: inherit;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.note { font-size: 13px; color: var(--muted); line-height: 1.5; margin: 8px 0 0; }
|
||||
.log {
|
||||
margin-top: 16px;
|
||||
padding: 10px 12px;
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
font-family: ui-monospace, Menlo, monospace;
|
||||
font-size: 12px;
|
||||
color: var(--muted);
|
||||
white-space: pre-wrap;
|
||||
min-height: 20px;
|
||||
}
|
||||
.log.ok { color: var(--ok); border-color: var(--ok); }
|
||||
.log.err { color: var(--err); border-color: var(--err); }
|
||||
|
||||
/* --- Shared side menu --- */
|
||||
.menu-btn {
|
||||
background: transparent;
|
||||
border: 1px solid var(--border);
|
||||
color: var(--text);
|
||||
border-radius: 8px;
|
||||
padding: 6px 11px;
|
||||
font-size: 16px;
|
||||
line-height: 1;
|
||||
cursor: pointer;
|
||||
}
|
||||
.menu-btn:hover { background: var(--bg); }
|
||||
.sidemenu {
|
||||
position: fixed;
|
||||
top: 0; left: 0;
|
||||
height: 100%;
|
||||
width: 240px;
|
||||
background: var(--panel);
|
||||
border-right: 1px solid var(--border);
|
||||
transform: translateX(-100%);
|
||||
transition: transform .2s ease;
|
||||
z-index: 40;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
padding: 18px 14px;
|
||||
gap: 6px;
|
||||
}
|
||||
.sidemenu.open { transform: translateX(0); }
|
||||
.sidemenu h2 {
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: .7px;
|
||||
color: var(--muted);
|
||||
margin: 0 0 10px;
|
||||
padding: 0 12px;
|
||||
}
|
||||
.sidemenu a {
|
||||
display: block;
|
||||
padding: 10px 12px;
|
||||
border-radius: 8px;
|
||||
color: var(--text);
|
||||
text-decoration: none;
|
||||
font-size: 14px;
|
||||
border: 1px solid transparent;
|
||||
}
|
||||
.sidemenu a:hover { background: var(--bg); }
|
||||
.sidemenu a.active { background: var(--accent-dim); border-color: var(--accent); }
|
||||
.backdrop {
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
background: rgba(0,0,0,.45);
|
||||
opacity: 0;
|
||||
pointer-events: none;
|
||||
transition: opacity .2s;
|
||||
z-index: 30;
|
||||
}
|
||||
.backdrop.open { opacity: 1; pointer-events: auto; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="backdrop" id="menuBackdrop"></div>
|
||||
<nav class="sidemenu" id="sidemenu">
|
||||
<h2>Menu</h2>
|
||||
<a href="/mivocero">🎙️ Mivocero</a>
|
||||
<a href="/customvoice" class="active">🎨 Custom Voice</a>
|
||||
</nav>
|
||||
|
||||
<div class="app">
|
||||
<header>
|
||||
<button class="menu-btn" id="menuBtn" title="Menu">☰</button>
|
||||
<h1>🎨 Custom Voice</h1>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<div class="step">
|
||||
<h2>1 · Language</h2>
|
||||
<select id="langSelect"></select>
|
||||
</div>
|
||||
|
||||
<div class="step">
|
||||
<h2>2 · Source</h2>
|
||||
<div class="tabs">
|
||||
<button type="button" class="tab active" id="tabRecord">🎙️ Record now</button>
|
||||
<button type="button" class="tab" id="tabUpload">📁 Upload a file</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="recordSection">
|
||||
<div class="step">
|
||||
<h2>3 · Read this phrase out loud</h2>
|
||||
<div class="phrase" id="phrase">Loading…</div>
|
||||
</div>
|
||||
|
||||
<div class="step">
|
||||
<h2>4 · Record (at least 10 seconds)</h2>
|
||||
<div class="actions" id="recordRow">
|
||||
<button id="btnRecord">● Start recording</button>
|
||||
<button id="btnStop" class="danger" disabled>■ Stop</button>
|
||||
<button id="btnReset" class="ghost" disabled>Discard</button>
|
||||
<span class="rec-dot"></span>
|
||||
<span class="timer" id="timer">0:00 / 0:10</span>
|
||||
</div>
|
||||
<div class="bar" id="bar"><div id="barFill"></div></div>
|
||||
<audio id="preview" controls hidden></audio>
|
||||
<p class="note">
|
||||
Speak naturally in a quiet room. The recording is transcoded to a WAV
|
||||
voice-cloning reference and installed as the active voice for the agent.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="uploadSection" hidden>
|
||||
<div class="step">
|
||||
<h2>3 · Upload an audio file</h2>
|
||||
<input type="file" id="fileInput" accept="audio/*" />
|
||||
<audio id="uploadPreview" controls hidden></audio>
|
||||
</div>
|
||||
|
||||
<div class="step">
|
||||
<h2>4 · Type the transcript</h2>
|
||||
<textarea id="uploadText" placeholder="Type (or paste) the exact text spoken in the audio…"></textarea>
|
||||
<p class="note">
|
||||
The transcript must match what is said in the file — it drives the
|
||||
voice-cloning reference, so accuracy matters. Any format ffmpeg can
|
||||
read (wav, mp3, m4a, ogg, flac…) works.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="step">
|
||||
<h2>5 · Use it</h2>
|
||||
<div class="actions">
|
||||
<button id="btnSave" disabled>✓ Use as custom voice</button>
|
||||
<button id="btnClear" class="ghost">↺ Reset to default voice</button>
|
||||
</div>
|
||||
<div class="log" id="log"></div>
|
||||
</div>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(() => {
|
||||
// --- Side menu -----------------------------------------------------------
|
||||
const menu = document.getElementById('sidemenu');
|
||||
const backdrop = document.getElementById('menuBackdrop');
|
||||
const setMenu = (open) => {
|
||||
menu.classList.toggle('open', open);
|
||||
backdrop.classList.toggle('open', open);
|
||||
};
|
||||
document.getElementById('menuBtn').addEventListener('click', () => setMenu(true));
|
||||
backdrop.addEventListener('click', () => setMenu(false));
|
||||
|
||||
// --- Elements ------------------------------------------------------------
|
||||
const langSelect = document.getElementById('langSelect');
|
||||
const phraseBox = document.getElementById('phrase');
|
||||
const btnRecord = document.getElementById('btnRecord');
|
||||
const btnStop = document.getElementById('btnStop');
|
||||
const btnReset = document.getElementById('btnReset');
|
||||
const btnSave = document.getElementById('btnSave');
|
||||
const btnClear = document.getElementById('btnClear');
|
||||
const timerEl = document.getElementById('timer');
|
||||
const bar = document.getElementById('bar');
|
||||
const barFill = document.getElementById('barFill');
|
||||
const preview = document.getElementById('preview');
|
||||
const logEl = document.getElementById('log');
|
||||
const tabRecord = document.getElementById('tabRecord');
|
||||
const tabUpload = document.getElementById('tabUpload');
|
||||
const recordSection = document.getElementById('recordSection');
|
||||
const uploadSection = document.getElementById('uploadSection');
|
||||
const fileInput = document.getElementById('fileInput');
|
||||
const uploadText = document.getElementById('uploadText');
|
||||
const uploadPreview = document.getElementById('uploadPreview');
|
||||
|
||||
const MIN_SECONDS = 10;
|
||||
const AUTO_STOP_SECONDS = 20;
|
||||
|
||||
let languages = [];
|
||||
let stream = null;
|
||||
let recorder = null;
|
||||
let chunks = [];
|
||||
let blob = null;
|
||||
let uploadFile = null;
|
||||
let mode = 'record';
|
||||
let elapsed = 0;
|
||||
let ticker = null;
|
||||
let startedAt = 0;
|
||||
|
||||
const fmt = (s) => `${Math.floor(s / 60)}:${String(Math.floor(s % 60)).padStart(2, '0')}`;
|
||||
const log = (msg, cls = '') => { logEl.textContent = msg; logEl.className = 'log ' + cls; };
|
||||
|
||||
const currentPhrase = () => {
|
||||
const entry = languages.find(l => l.code === langSelect.value);
|
||||
return entry ? entry.phrase : '';
|
||||
};
|
||||
|
||||
// The save button is enabled once the active source holds a usable take:
|
||||
// a recording (record mode) or a file plus a non-blank transcript (upload).
|
||||
const refreshSave = () => {
|
||||
btnSave.disabled = mode === 'record'
|
||||
? !blob
|
||||
: !(uploadFile && uploadText.value.trim());
|
||||
};
|
||||
|
||||
// --- Source toggle -------------------------------------------------------
|
||||
function setMode(next) {
|
||||
mode = next;
|
||||
const rec = next === 'record';
|
||||
tabRecord.classList.toggle('active', rec);
|
||||
tabUpload.classList.toggle('active', !rec);
|
||||
recordSection.hidden = !rec;
|
||||
uploadSection.hidden = rec;
|
||||
log('');
|
||||
refreshSave();
|
||||
}
|
||||
|
||||
tabRecord.addEventListener('click', () => setMode('record'));
|
||||
tabUpload.addEventListener('click', () => setMode('upload'));
|
||||
|
||||
// --- Load the phrases ----------------------------------------------------
|
||||
fetch('/api/customvoice/phrases')
|
||||
.then(r => r.json())
|
||||
.then(data => {
|
||||
languages = data.languages || [];
|
||||
langSelect.innerHTML = '';
|
||||
for (const l of languages) {
|
||||
const opt = document.createElement('option');
|
||||
opt.value = l.code;
|
||||
opt.textContent = l.label;
|
||||
langSelect.appendChild(opt);
|
||||
}
|
||||
if (languages.some(l => l.code === data.default)) langSelect.value = data.default;
|
||||
phraseBox.textContent = currentPhrase();
|
||||
})
|
||||
.catch(e => log('Failed to load phrases: ' + e.message, 'err'));
|
||||
|
||||
langSelect.addEventListener('change', () => { phraseBox.textContent = currentPhrase(); });
|
||||
|
||||
// --- Recording -----------------------------------------------------------
|
||||
function updateTimer() {
|
||||
elapsed = Math.min(AUTO_STOP_SECONDS, (performance.now() - startedAt) / 1000);
|
||||
timerEl.textContent = `${fmt(elapsed)} / ${fmt(MIN_SECONDS)}`;
|
||||
barFill.style.width = Math.min(100, (elapsed / MIN_SECONDS) * 100) + '%';
|
||||
if (elapsed >= MIN_SECONDS) {
|
||||
timerEl.classList.add('ready');
|
||||
bar.classList.add('ready');
|
||||
btnStop.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
btnRecord.addEventListener('click', async () => {
|
||||
log('');
|
||||
blob = null;
|
||||
refreshSave();
|
||||
preview.hidden = true;
|
||||
try {
|
||||
stream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
} catch (e) {
|
||||
log('Microphone access denied: ' + e.message, 'err');
|
||||
return;
|
||||
}
|
||||
const candidates = ['audio/webm;codecs=opus', 'audio/webm', 'audio/ogg;codecs=opus', 'audio/mp4'];
|
||||
const mime = candidates.find(m => window.MediaRecorder && MediaRecorder.isTypeSupported(m));
|
||||
try {
|
||||
recorder = mime ? new MediaRecorder(stream, { mimeType: mime }) : new MediaRecorder(stream);
|
||||
} catch (e) {
|
||||
log('MediaRecorder unavailable: ' + e.message, 'err');
|
||||
stopStream();
|
||||
return;
|
||||
}
|
||||
|
||||
chunks = [];
|
||||
recorder.ondataavailable = (e) => { if (e.data && e.data.size) chunks.push(e.data); };
|
||||
recorder.onstop = () => {
|
||||
clearInterval(ticker);
|
||||
document.getElementById('recordRow').classList.remove('recording');
|
||||
blob = new Blob(chunks, { type: recorder.mimeType || 'audio/webm' });
|
||||
stopStream();
|
||||
if (elapsed < MIN_SECONDS) {
|
||||
log(`Recording too short (${elapsed.toFixed(1)} s) — please read for at least ${MIN_SECONDS} s.`, 'err');
|
||||
return;
|
||||
}
|
||||
preview.src = URL.createObjectURL(blob);
|
||||
preview.hidden = false;
|
||||
refreshSave();
|
||||
btnReset.disabled = false;
|
||||
log(`Recording ready (${elapsed.toFixed(1)} s). Listen, then use it.`, 'ok');
|
||||
};
|
||||
|
||||
elapsed = 0;
|
||||
startedAt = performance.now();
|
||||
timerEl.textContent = `${fmt(0)} / ${fmt(MIN_SECONDS)}`;
|
||||
timerEl.classList.remove('ready');
|
||||
bar.classList.remove('ready');
|
||||
barFill.style.width = '0%';
|
||||
btnStop.disabled = true;
|
||||
btnReset.disabled = true;
|
||||
document.getElementById('recordRow').classList.add('recording');
|
||||
timerEl.classList.add('rec');
|
||||
|
||||
recorder.start();
|
||||
ticker = setInterval(() => {
|
||||
updateTimer();
|
||||
if (elapsed >= AUTO_STOP_SECONDS) recorder.stop();
|
||||
}, 100);
|
||||
btnRecord.disabled = true;
|
||||
});
|
||||
|
||||
btnStop.addEventListener('click', () => { if (recorder && recorder.state !== 'inactive') recorder.stop(); });
|
||||
|
||||
function stopStream() {
|
||||
btnRecord.disabled = false;
|
||||
timerEl.classList.remove('rec');
|
||||
if (stream) { stream.getTracks().forEach(t => t.stop()); stream = null; }
|
||||
}
|
||||
|
||||
btnReset.addEventListener('click', () => {
|
||||
blob = null;
|
||||
preview.hidden = true;
|
||||
preview.removeAttribute('src');
|
||||
refreshSave();
|
||||
btnReset.disabled = true;
|
||||
elapsed = 0;
|
||||
timerEl.textContent = `${fmt(0)} / ${fmt(MIN_SECONDS)}`;
|
||||
timerEl.classList.remove('ready');
|
||||
bar.classList.remove('ready');
|
||||
barFill.style.width = '0%';
|
||||
log('');
|
||||
});
|
||||
|
||||
// --- Upload --------------------------------------------------------------
|
||||
fileInput.addEventListener('change', () => {
|
||||
uploadFile = fileInput.files && fileInput.files[0] ? fileInput.files[0] : null;
|
||||
if (uploadFile) {
|
||||
uploadPreview.src = URL.createObjectURL(uploadFile);
|
||||
uploadPreview.hidden = false;
|
||||
} else {
|
||||
uploadPreview.hidden = true;
|
||||
uploadPreview.removeAttribute('src');
|
||||
}
|
||||
refreshSave();
|
||||
});
|
||||
|
||||
uploadText.addEventListener('input', refreshSave);
|
||||
|
||||
// --- Save / clear --------------------------------------------------------
|
||||
btnSave.addEventListener('click', async () => {
|
||||
// Pick the active source: the recording (record mode) or the chosen file
|
||||
// plus its typed transcript (upload mode).
|
||||
let body, text, type;
|
||||
if (mode === 'record') {
|
||||
if (!blob) return;
|
||||
body = blob;
|
||||
type = blob.type || 'application/octet-stream';
|
||||
} else {
|
||||
text = uploadText.value.trim();
|
||||
if (!uploadFile || !text) return;
|
||||
body = uploadFile;
|
||||
type = uploadFile.type || 'application/octet-stream';
|
||||
}
|
||||
|
||||
btnSave.disabled = true;
|
||||
log('Uploading…');
|
||||
try {
|
||||
let url = '/api/customvoice?lang=' + encodeURIComponent(langSelect.value);
|
||||
if (text) url += '&text=' + encodeURIComponent(text);
|
||||
const r = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': type },
|
||||
body,
|
||||
});
|
||||
const data = await r.json().catch(() => ({}));
|
||||
if (!r.ok || !data.ok) throw new Error(data.error || ('HTTP ' + r.status));
|
||||
log('Custom voice installed — ' + data.audio, 'ok');
|
||||
} catch (e) {
|
||||
log('Failed: ' + e.message, 'err');
|
||||
btnSave.disabled = false;
|
||||
}
|
||||
});
|
||||
|
||||
btnClear.addEventListener('click', async () => {
|
||||
log('Resetting…');
|
||||
try {
|
||||
const r = await fetch('/api/customvoice/clear', { method: 'POST' });
|
||||
const data = await r.json().catch(() => ({}));
|
||||
if (!r.ok || !data.ok) throw new Error(data.error || ('HTTP ' + r.status));
|
||||
log('Default voice restored.', 'ok');
|
||||
} catch (e) {
|
||||
log('Failed: ' + e.message, 'err');
|
||||
}
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
+192
-2
@@ -93,8 +93,29 @@
|
||||
.bubble.karaoke .kw { color: var(--muted); transition: color .18s ease; }
|
||||
.bubble.karaoke .kw.done { color: var(--text); }
|
||||
.bubble.karaoke .kw.now { color: var(--accent); text-shadow: 0 0 8px rgba(79,140,255,.55); }
|
||||
/* Same karaoke style on the user's own bubbles, tuned for the blue bubble. */
|
||||
.msg.user .bubble.karaoke .kw { color: rgba(230,237,247,.5); }
|
||||
.msg.user .bubble.karaoke .kw.done { color: #ffffff; }
|
||||
.msg.user .bubble.karaoke .kw.now { color: #ffffff; text-shadow: 0 0 9px rgba(255,255,255,.85); }
|
||||
.hint { color: var(--muted); font-size: 13px; text-align: center; margin: auto; }
|
||||
|
||||
/* Barge-in indicator: flashes red whenever the agent is interrupted. */
|
||||
.bargein {
|
||||
display: inline-flex; align-items: center; gap: 7px;
|
||||
font-size: 12px; color: var(--muted);
|
||||
border: 1px solid var(--border); border-radius: 999px;
|
||||
padding: 5px 11px; white-space: nowrap;
|
||||
transition: border-color .2s, box-shadow .2s, color .2s;
|
||||
}
|
||||
.bargein .bi-dot {
|
||||
width: 9px; height: 9px; border-radius: 50%; flex: none;
|
||||
background: var(--err); opacity: .25;
|
||||
transition: opacity .15s, box-shadow .15s, transform .15s;
|
||||
}
|
||||
.bargein b { color: var(--text); font-variant-numeric: tabular-nums; }
|
||||
.bargein.active { border-color: var(--err); color: var(--err); box-shadow: 0 0 12px -2px var(--err); }
|
||||
.bargein.active .bi-dot { opacity: 1; box-shadow: 0 0 9px var(--err); transform: scale(1.35); }
|
||||
|
||||
footer { border-top: 1px solid var(--border); padding: 12px 18px; }
|
||||
.meters { display: flex; align-items: center; gap: 14px; margin-bottom: 10px; }
|
||||
.meter { height: 8px; background: var(--bg); border-radius: 4px; overflow: hidden; flex: 1; }
|
||||
@@ -225,13 +246,82 @@
|
||||
color: var(--muted);
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
/* --- Shared side menu --- */
|
||||
.menu-btn {
|
||||
background: transparent;
|
||||
border: 1px solid var(--border);
|
||||
color: var(--text);
|
||||
border-radius: 8px;
|
||||
padding: 6px 11px;
|
||||
font-size: 16px;
|
||||
line-height: 1;
|
||||
cursor: pointer;
|
||||
}
|
||||
.menu-btn:hover { background: var(--bg); }
|
||||
.sidemenu {
|
||||
position: fixed;
|
||||
top: 0; left: 0;
|
||||
height: 100%;
|
||||
width: 240px;
|
||||
background: var(--panel);
|
||||
border-right: 1px solid var(--border);
|
||||
transform: translateX(-100%);
|
||||
transition: transform .2s ease;
|
||||
z-index: 40;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
padding: 18px 14px;
|
||||
gap: 6px;
|
||||
}
|
||||
.sidemenu.open { transform: translateX(0); }
|
||||
.sidemenu h2 {
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: .7px;
|
||||
color: var(--muted);
|
||||
margin: 0 0 10px;
|
||||
padding: 0 12px;
|
||||
}
|
||||
.sidemenu a {
|
||||
display: block;
|
||||
padding: 10px 12px;
|
||||
border-radius: 8px;
|
||||
color: var(--text);
|
||||
text-decoration: none;
|
||||
font-size: 14px;
|
||||
border: 1px solid transparent;
|
||||
}
|
||||
.sidemenu a:hover { background: var(--bg); }
|
||||
.sidemenu a.active { background: var(--accent-dim); border-color: var(--accent); }
|
||||
.backdrop {
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
background: rgba(0,0,0,.45);
|
||||
opacity: 0;
|
||||
pointer-events: none;
|
||||
transition: opacity .2s;
|
||||
z-index: 30;
|
||||
}
|
||||
.backdrop.open { opacity: 1; pointer-events: auto; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="backdrop" id="menuBackdrop"></div>
|
||||
<nav class="sidemenu" id="sidemenu">
|
||||
<h2>Menu</h2>
|
||||
<a href="/mivocero" class="active">🎙️ Mivocero</a>
|
||||
<a href="/customvoice">🎨 Custom Voice</a>
|
||||
</nav>
|
||||
|
||||
<div class="app">
|
||||
<header>
|
||||
<button class="menu-btn" id="menuBtn" title="Menu">☰</button>
|
||||
<h1>🎙️ Mivocero</h1>
|
||||
<span class="status"><span class="dot" id="connDot"></span><span id="connText">disconnected</span></span>
|
||||
<span class="bargein" id="bargeinInd" title="Agent interrupted by the user (VAD barge-in or the Stop button)">
|
||||
<span class="bi-dot"></span>Barge-in <b id="bargeinCount">0</b>
|
||||
</span>
|
||||
<button id="btnConnect" class="ok">Connect</button>
|
||||
<button id="btnInterrupt" class="danger" disabled>Stop</button>
|
||||
</header>
|
||||
@@ -307,11 +397,24 @@
|
||||
// {"type":"audio_source","state":"…", …} external audio source status
|
||||
// {"type":"cue","kind":"…","name":"…"} filler / start filler / backchannel emitted
|
||||
// {"type":"cues","lang":"…","config":{…}} live cue policy (sent at start)
|
||||
// {"type":"barge_in","source":"…"} agent interrupted (VAD onset)
|
||||
//
|
||||
// The session closes itself after IDLE_TIMEOUT_MS without any traffic in
|
||||
// either direction (see the idle guard below).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// --- Shared side menu (chat / custom voice navigation) ---
|
||||
(() => {
|
||||
const menu = document.getElementById('sidemenu');
|
||||
const backdrop = document.getElementById('menuBackdrop');
|
||||
const setOpen = (open) => {
|
||||
menu.classList.toggle('open', open);
|
||||
backdrop.classList.toggle('open', open);
|
||||
};
|
||||
document.getElementById('menuBtn').addEventListener('click', () => setOpen(true));
|
||||
backdrop.addEventListener('click', () => setOpen(false));
|
||||
})();
|
||||
|
||||
const SAMPLE_RATE_OUT = 44100; // server <-> client (matches the TTS rate)
|
||||
const SAMPLE_RATE_IN = 44100;
|
||||
const FRAME_MS = 20;
|
||||
@@ -347,6 +450,11 @@ const cueEls = {
|
||||
};
|
||||
const cueCounts = { filler: 0, start_filler: 0, backchannel: 0 };
|
||||
const cueTimers = { filler: 0, start_filler: 0, backchannel: 0 };
|
||||
// Barge-in indicator (header pill): flashes red on each interruption.
|
||||
const bargeinInd = $('bargeinInd');
|
||||
const bargeinCountEl = $('bargeinCount');
|
||||
let bargeinTotal = 0;
|
||||
let bargeinTimer = 0;
|
||||
|
||||
let ws = null;
|
||||
let audioCtx = null;
|
||||
@@ -498,6 +606,61 @@ function endAgentMessage() {
|
||||
agentText = '';
|
||||
}
|
||||
|
||||
// --- User-message karaoke --------------------------------------------------
|
||||
// The user's transcript reaches the client only once the utterance has ended
|
||||
// (the STT runs server-side on the whole segment), so there is no per-word
|
||||
// audio timing to follow. The words are instead lit up by a short left-to-right
|
||||
// sweep, giving the same karaoke feel as the agent's reply.
|
||||
const USER_REVEAL_MS_PER_WORD = 90;
|
||||
let userRevealRAF = 0;
|
||||
let userRevealWords = null;
|
||||
|
||||
function addUserMessage(text) {
|
||||
const bubble = addMessage('user', '');
|
||||
bubble.classList.add('karaoke');
|
||||
const words = [];
|
||||
for (const tok of text.split(/(\s+)/)) {
|
||||
if (tok === '') continue;
|
||||
if (/^\s+$/.test(tok)) {
|
||||
bubble.appendChild(document.createTextNode(tok));
|
||||
} else {
|
||||
const span = document.createElement('span');
|
||||
span.className = 'kw';
|
||||
span.textContent = tok;
|
||||
bubble.appendChild(span);
|
||||
words.push(span);
|
||||
}
|
||||
}
|
||||
revealWords(words);
|
||||
return bubble;
|
||||
}
|
||||
|
||||
function revealWords(words) {
|
||||
if (userRevealRAF) { cancelAnimationFrame(userRevealRAF); userRevealRAF = 0; }
|
||||
if (userRevealWords) {
|
||||
for (const w of userRevealWords) { w.classList.add('done'); w.classList.remove('now'); }
|
||||
userRevealWords = null;
|
||||
}
|
||||
if (!words.length) return;
|
||||
const start = performance.now();
|
||||
userRevealWords = words;
|
||||
const tick = (now) => {
|
||||
const p = Math.floor((now - start) / USER_REVEAL_MS_PER_WORD) + 1;
|
||||
for (let i = 0; i < words.length; i++) {
|
||||
words[i].classList.toggle('done', i < p - 1);
|
||||
words[i].classList.toggle('now', i === p - 1);
|
||||
}
|
||||
if (p <= words.length) {
|
||||
userRevealRAF = requestAnimationFrame(tick);
|
||||
} else {
|
||||
for (const w of words) { w.classList.add('done'); w.classList.remove('now'); }
|
||||
userRevealWords = null;
|
||||
userRevealRAF = 0;
|
||||
}
|
||||
};
|
||||
userRevealRAF = requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
// --- Audio worklets --------------------------------------------------------
|
||||
|
||||
const CAPTURE_PROCESSOR = `
|
||||
@@ -680,6 +843,25 @@ function resetCues() {
|
||||
}
|
||||
}
|
||||
|
||||
// Barge-in indicator: bump the counter and flash the pill. Fired by the
|
||||
// server "barge_in" event (VAD onset) and by the local Stop button.
|
||||
function flashBargeIn(source) {
|
||||
bargeinTotal += 1;
|
||||
bargeinCountEl.textContent = bargeinTotal;
|
||||
if (source) bargeinInd.title = 'Last barge-in: ' + source;
|
||||
bargeinInd.classList.add('active');
|
||||
clearTimeout(bargeinTimer);
|
||||
bargeinTimer = setTimeout(() => bargeinInd.classList.remove('active'), 700);
|
||||
}
|
||||
|
||||
function resetBargeIn() {
|
||||
bargeinTotal = 0;
|
||||
clearTimeout(bargeinTimer);
|
||||
bargeinInd.classList.remove('active');
|
||||
bargeinInd.title = 'Agent interrupted by the user (VAD barge-in or the Stop button)';
|
||||
bargeinCountEl.textContent = '0';
|
||||
}
|
||||
|
||||
// --- Cue methods panel -----------------------------------------------------
|
||||
// The method (how each cue decides to fire) plus the live parameters reported
|
||||
// by the server in the "cues" event at session start. `params` turns the
|
||||
@@ -784,7 +966,7 @@ function handleServerText(text) {
|
||||
switch (event.type) {
|
||||
case 'user':
|
||||
endAgentMessage(); // a new turn starts
|
||||
addMessage('user', event.text);
|
||||
addUserMessage(event.text);
|
||||
break;
|
||||
case 'assistant_delta':
|
||||
appendAgentDelta(event.text);
|
||||
@@ -792,6 +974,12 @@ function handleServerText(text) {
|
||||
case 'assistant_end':
|
||||
agentEnded = true; // keep the bubble: karaoke finishes when audio drains
|
||||
break;
|
||||
case 'barge_in':
|
||||
// The server cut the reply (VAD onset): flash the indicator and stop the
|
||||
// karaoke sweep — the audio is gone, so the reply is left as spoken.
|
||||
flashBargeIn(event.source || 'vad');
|
||||
endAgentMessage();
|
||||
break;
|
||||
case 'error':
|
||||
endAgentMessage();
|
||||
addMessage('error', event.text, 'error');
|
||||
@@ -883,6 +1071,7 @@ async function connect() {
|
||||
setConn('on', 'connected');
|
||||
log('connected');
|
||||
resetCues();
|
||||
resetBargeIn();
|
||||
btnConnect.disabled = true;
|
||||
btnInterrupt.disabled = false;
|
||||
textInput.disabled = false;
|
||||
@@ -936,6 +1125,7 @@ btnConnect.onclick = () => (ws ? disconnect() : connect());
|
||||
btnInterrupt.onclick = () => {
|
||||
if (sendJson({ type: 'interrupt' })) {
|
||||
log('→ interrupt');
|
||||
flashBargeIn('client');
|
||||
endAgentMessage();
|
||||
}
|
||||
};
|
||||
@@ -966,7 +1156,7 @@ composer.onsubmit = (e) => {
|
||||
if (!text) return;
|
||||
if (sendJson({ type: 'user_text', text })) {
|
||||
endAgentMessage();
|
||||
addMessage('user', text);
|
||||
addUserMessage(text);
|
||||
textInput.value = '';
|
||||
log('→ user_text');
|
||||
}
|
||||
|
||||
@@ -7,3 +7,7 @@ pub const PLAYGROUND_HTML: &str = include_str!("playground.html");
|
||||
/// The `/mivocero` speech-to-speech chat client, embedded into the binary at
|
||||
/// compile time.
|
||||
pub const MIVOCERO_HTML: &str = include_str!("mivocero.html");
|
||||
|
||||
/// The `/customvoice` voice-cloning builder page, embedded into the binary at
|
||||
/// compile time.
|
||||
pub const CUSTOMVOICE_HTML: &str = include_str!("customvoice.html");
|
||||
|
||||
+43
-4
@@ -235,9 +235,13 @@ pub async fn run_session(socket: WebSocket, state: AppState) {
|
||||
for event in detector.push(&frame) {
|
||||
match event {
|
||||
VadEvent::SpeechOnset { prob } => {
|
||||
// The user started talking: cut the agent off unless
|
||||
// it is already silent.
|
||||
if settings.barge_in && tm_for_vad.is_speaking() {
|
||||
// The user started talking: cut the agent off — but
|
||||
// only once the agent is actually *audible*. While
|
||||
// the reply is still being synthesised or is sitting
|
||||
// in the pre-buffer nothing has reached the user's
|
||||
// ears yet, so a transient (or the tail of the
|
||||
// user's own utterance) must not cancel it.
|
||||
if settings.barge_in && tm_for_vad.is_audible() {
|
||||
info!(
|
||||
prob = format!("{prob:.3}"),
|
||||
"VAD barge-in — interrupting agent"
|
||||
@@ -246,6 +250,13 @@ pub async fn run_session(socket: WebSocket, state: AppState) {
|
||||
"barge_in",
|
||||
json!({ "source": "vad", "prob": prob }),
|
||||
);
|
||||
// Tell the client so it can flash a barge-in
|
||||
// indicator (the VAD onset is server-side, so
|
||||
// the browser otherwise has no way to know).
|
||||
let _ = out_tx_for_mic.send(
|
||||
json!({ "type": "barge_in", "source": "vad", "prob": prob })
|
||||
.to_string(),
|
||||
);
|
||||
tm_for_vad.interrupt();
|
||||
}
|
||||
}
|
||||
@@ -268,7 +279,35 @@ pub async fn run_session(socket: WebSocket, state: AppState) {
|
||||
json!({ "duration_ms": duration_ms, "avg_prob": avg_prob }),
|
||||
);
|
||||
backchannel.on_speech_end();
|
||||
if settings.segment_stt {
|
||||
// Utterance gate (see `VadSettings::min_utterance_ms`
|
||||
// / `min_avg_prob`): drop segments too short or too
|
||||
// soft to be real speech before spending an STT+LLM
|
||||
// turn on them. This is what filters the VAD
|
||||
// false-triggers (breaths, clicks, acoustic echo,
|
||||
// noise) that the STT would otherwise turn into
|
||||
// hallucinated transcripts ("Bye.", "She", empty).
|
||||
let too_short = settings.min_utterance_ms > 0
|
||||
&& duration_ms < settings.min_utterance_ms as u64;
|
||||
let too_soft =
|
||||
settings.min_avg_prob > 0.0 && avg_prob < settings.min_avg_prob;
|
||||
if too_short || too_soft {
|
||||
info!(
|
||||
duration_ms,
|
||||
avg_prob = format!("{avg_prob:.3}"),
|
||||
too_short,
|
||||
too_soft,
|
||||
"utterance dropped (below STT gate)"
|
||||
);
|
||||
audit_for_mic.event(
|
||||
"utterance_dropped",
|
||||
json!({
|
||||
"duration_ms": duration_ms,
|
||||
"avg_prob": avg_prob,
|
||||
"too_short": too_short,
|
||||
"too_soft": too_soft,
|
||||
}),
|
||||
);
|
||||
} else if settings.segment_stt {
|
||||
match &worker_for_mic {
|
||||
Some(worker) => {
|
||||
info!(
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao questo è un test di upload
|
||||
Binary file not shown.
@@ -0,0 +1 @@
|
||||
Ciao, questa è la mia voce personale. Sto leggendo questa frase per creare una voce su misura per me. Parlo in modo naturale e chiaro, così il sistema può imparare il timbro e il ritmo della mia voce.
|
||||
Binary file not shown.
Reference in New Issue
Block a user