roomote_PRE/piperust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

step3

Browse Source
This commit is contained in:
valenti
2026-10-17 22:01:31 +01:00
parent bfed42c3b3
commit 79f927d968
53 changed files with 2158 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
UPDATES.md
+372
View File
@@ -5,6 +5,378 @@ Entries are written in English and ordered newest-first.
---
## 2026-10-17 — Feat: custom voice accepts an uploaded audio file + typed transcript
### Added
- **Upload source on the `/customvoice` page.** A **Source** toggle now offers
two ways to build the reference: **Record now** (the existing microphone flow,
read a predefined phrase for ≥ 10 s) or **Upload a file** — pick any audio
file `ffmpeg` can read (`accept="audio/*"`: wav, mp3, m4a, ogg, flac…) and
**type (or paste) its transcript**. In upload mode the transcript is required
and gates the *Use it* button, since the typed text is what drives the clone.
Both modes share the same *Use it* / *Reset* actions and preview player.
- **`text` query parameter on `POST /api/customvoice`.** The endpoint is now
`POST /api/customvoice?lang=<code>[&text=<transcript>]`. A non-blank `text`
overrides the language's predefined reading phrase for the transcript written
to the sibling `.txt` and passed to `set_reference`; when absent or blank the
predefined phrase is used as before (so a plain recording is unchanged).
Backend: `SetParams` gained `text: Option<String>` and `set_custom_voice`
resolves the transcript from it (`src/customvoice.rs`).
### Notes
- No format constraints are imposed on the upload: the same ffmpeg probe used
for recordings handles whatever container is sent. The 32 MB body cap already
covers uncompressed WAVs.
- README updated (feature bullet, usage comment, *Custom voice builder* flow and
API/route tables, `text` parameter).
---
## 2026-10-17 — Fix: short utterances ("uno", "due") dropped by the STT gate
### Fixed
- **The probability average now counts voiced windows only.** The gate's
`avg_prob` was the mean of the raw VAD probability over **every** window from
speech start to segment close — including the `VAD_MIN_SILENCE_MS` (700 ms)
hangover that ends the utterance. For a long phrase that silence is a small
fraction, but for a short word it is most of the segment: "uno" (~350 ms of
voice + ~700 ms of silence) averaged **0.065–0.199** and was discarded as
`too_soft`, even though the user was plainly speaking. In the same session the
long phrases passed at 0.32–0.60. The average is now taken only over windows
whose probability is ≥ `VAD_NEG_THRESHOLD` (`src/vad/silero.rs`), so the voiced
portion scores its true value and short real words clear the gate while noise
still does not.
### Notes
- `VAD_MIN_AVG_PROB` stays at `0.25`; its meaning is now stricter (it no longer
benefits from the silence dilution), but genuine short words score well above
it. README / `.env` / `.env.example` / `src/vad/mod.rs` docs aligned.
- Tune with `VAD_MIN_AVG_PROB`: raise it if noisy segments become hallucinated
turns, lower it if genuinely soft speech is still dropped.
---
## 2026-10-17 — Fix: too many utterances dropped by the STT gate
### Fixed
- **`VAD_MIN_AVG_PROB` lowered `0.45` → `0.25`.** Log analysis showed **62
drops, all `too_soft`** (none for duration), still occurring after a restart.
Accepted utterances measured `avg_prob` 0.47–0.76 while dropped ones measured
0.08–0.45 — including plainly real speech cut by a hair (2136 ms @ 0.428,
3576 ms @ 0.436). The gate sat almost on top of the per-frame speech threshold
(`VAD_THRESHOLD=0.5`): a real utterance hovers around the threshold on its
onset/release, so its *average* lands just under 0.45 and the segment is
discarded. The `0.25` default leaves a proper margin — genuine soft speech
(0.3–0.45) now passes, while clearly noisy segments (below 0.25) are still
filtered. Update `VAD_MIN_AVG_PROB` in `.env` (default, `.env.example`,
`src/vad/mod.rs` and README aligned).
### Notes
- Tune with `VAD_MIN_AVG_PROB`: lower it further if genuine speech is still
dropped, raise it if noisy segments are turning into hallucinated turns.
`0.0` disables the probability gate.
---
## 2026-10-17 — Fix: custom voice still louder than the default voice
### Fixed
- **Custom voice level pushed below the reference.** Matching the shipped
reference at the *integrated* level was not enough: a measured take lands at
≈`-23.25 LUFS` against `solis.wav`'s ≈`-23.44 LUFS`, i.e. the same loudness,
yet it still sounded louder. The difference is **dynamic range and peaks** —
the user take measures ≈`6.2 LRA` / `-2.2 dBTP` versus `1.6 LRA` / `-7.8 dBTP`
for the mastered reference, so its peaks sit ~5.5 dB hotter and read as
"louder". The normalization target is now `-26 LUFS` with a `-5 dBTP`
ceiling, which drops a take ~3 dB and pulls its peaks down to ≈`-5 dBTP` so a
custom voice sits just under the default instead of on top of it.
- Defaults changed: `CUSTOMVOICE_TARGET_LUFS` `-23` → **`-26`**,
`CUSTOMVOICE_TRUE_PEAK` `-1.5` → **`-5`** (code, `.env`, `.env.example`,
README).
### Notes
- Lower `CUSTOMVOICE_TARGET_LUFS` further for a quieter custom voice; the value
is applied at record time, so a take must be re-recorded (or the endpoint
re-posted) for a change to take effect.
- The custom voice lives only in memory (`TtsEngine::set_reference`): a restart
reverts the reference to the configured `TTS_REFERENCE_AUDIO` voice.
---
## 2026-10-17 — Fix: custom voice louder than the default voice
### Fixed
- **Loudness target aligned with the shipped reference.** The custom-voice
chain normalized to `-18 LUFS` while `voice_profiles/solis.wav` measures
≈`-23.4 LUFS`, so a custom voice came out ~5.4 dB louder than the voice it
replaced. The default is now **`-23`** (`CUSTOMVOICE_TARGET_LUFS`), matching
the reference — swapping voices no longer changes how loud the agent sounds.
- **Mono downmix moved inside the filter chain.** `-ac 1` used to be applied at
the output stage, i.e. *after* loudnorm: a **stereo** upload was measured as
stereo and then folded to mono, landing the written file ~3 dB below the
target. `aformat=channel_layouts=mono` now runs first, so loudnorm measures
the signal that is actually written.
### Notes
- Verified: `solis.wav` (stereo, -23.4 LUFS) and a quiet mono take (-48.6 LUFS)
both render to exactly **-23.00 LUFS**, mono 44.1 kHz. `cargo test` → 25
passed.
---
## 2026-10-17 — Post-processing for the custom voice (level match + cleanup)
### Added
- **Post-processing chain** for a recorded custom voice
(`src/customvoice.rs`), applied before the reference is installed:
`highpass=f=80` (rumble/DC), optional `afftdn` denoise + `deesser`,
leading/trailing silence trim (`silenceremove` + `areverse`) and a
**two-pass EBU R128 `loudnorm` with `linear=true`**.
- **New env knobs**: `CUSTOMVOICE_POSTPROCESS` (default `true`),
`CUSTOMVOICE_DENOISE` (default `true`), `CUSTOMVOICE_TARGET_LUFS`
(default `-18`), `CUSTOMVOICE_TRUE_PEAK` (default `-1.5`).
### Notes
- The loudness stage measures the cleaned signal first, then applies the result
as a **single constant gain** — a per-frame (single-pass) norm is what makes
takes jump in volume, which is the problem this fixes. Verified: a take at
**-48.6 LUFS** and one at **-17.5 LUFS** both render to **≈ -18 LUFS** with
TP at **-1.5 dBTP**.
- Any stage failing (unreadable take, all-silence clip, filter quirk) degrades
to the plain transcode with a warning — a take is never lost.
- Error handling verified: garbage input → `502` with a clean message, unknown
language → `400`. `cargo test` → 25 passed.
---
## 2026-10-17 — Custom voice builder (`/customvoice`) + Mivocero side menu
### Added
- **Custom-voice builder page** at `GET /customvoice`
(`src/web/customvoice.html`). It asks the user for a **language**, shows a
predefined phrase to read (long enough for cloning, ~10 s), records it with
`MediaRecorder` and installs the take as the active voice. The **Stop**
button is disabled until **10 s** have elapsed (timer + progress bar), the
take auto-stops at 20 s, and it can be played back and discarded before use.
- **Backend endpoints** (`src/customvoice.rs`): `GET /api/customvoice/phrases`
(reading phrases per language + the configured default), `POST
/api/customvoice?lang=<code>` (raw recording body), `POST
/api/customvoice/clear`. The body cap of the upload route is raised to 32 MB.
- **Side menu** on `/mivocero` (☰ drawer) linking to the chat and to the
custom-voice page; the same drawer is on `/customvoice`.
### Notes
- Phrases live server-side, so the reference transcript always matches the text
the user read. `POST /api/customvoice` transcodes the upload to mono 16-bit
PCM WAV at `OUTPUT_SAMPLE_RATE` (44.1 kHz) with `ffmpeg`, writes it next to
`TTS_REFERENCE_AUDIO` (the directory the Fish Audio backend can read) with a
sibling `<name>.txt`, and installs it via `TtsEngine::set_reference` — live
for new replies, no restart.
- `POST /api/customvoice/clear` restores the reference configured at startup
(`TTS_REFERENCE_AUDIO`) rather than dropping it entirely.
- Verified: uploading an audio file returns `{"ok":true,…}` and produces a
44.1 kHz/mono `pcm_s16le` WAV with its `.txt`; `clear` logs the restore of
`solis.wav`. Tests: `cargo test` → 25 passed.
---
## 2026-10-17 — Tool calling survives a mis-matched server-side tool parser
### Fixed
- **Tool calls dropped on the streaming path are recovered.** The production
vLLM server runs `--tool-call-parser hermes`, which expects the
`<tool_call>{"name":…,"arguments":{…}}</tool_call>` JSON shape; the Qwen3
model emits the XML shape instead, so the parser throws and the call is
discarded. In streaming the turn then ended with an **empty** reply
(`finish_reason: "stop"`) and no `tool_call` ever fired — the assistant either
stayed silent or narrated a call it never made.
- **Non-streaming fallback** in the turn loop
(`src/orchestrator/turn_manager.rs`): a round that comes back empty while
tools are configured is re-asked once **without streaming** via the new
`LlmClient::chat_reply` (`src/llm.rs`), which returns both the visible text
and the parsed tool calls.
- **Structured calls are parsed too** — `tools::parse_openai_tool_calls`
(`src/tools.rs`) reads the standard `message.tool_calls[]` array, so the
engine keeps working when the server parser *does* match (and returns
`finish_reason: "tool_calls"`). The native `<tool_call>` text path is
unchanged.
### Notes
- End-to-end verified: "Che tempo fa a New York adesso?" now emits
`tool_call {name: get_weather, arguments: {city: "New York"}}` and answers
with live wttr.in data. A tool-free turn is unchanged (no extra request).
- The clean fix remains server-side — `--tool-call-parser qwen3_coder` matches
the Qwen3 XML format. The fallback makes the engine correct regardless, at the
cost of one extra request only on an otherwise-empty round.
- Tests: `cargo test` → 25 passed (2 new in `tools::tests`).
---
## 2026-10-17 — Longer end-of-utterance window (`VAD_MIN_SILENCE_MS` 300 → 700 ms)
### Changed
- **`VAD_MIN_SILENCE_MS` raised** from `300` to `700` ms in `.env` /
`.env.example`, and the fallback default in `src/vad/mod.rs` aligned to
`700`. This is the silence window the VAD waits for before closing an
utterance (see `SileroVad::step`, `silence_run_ms >= min_silence_ms`).
- **README** — updated the config table and the `SpeechEnd` trigger row.
### Notes
- With `300` ms a brief thinking pause (~0,3 s) closed the turn and sent a
partial sentence to the STT/LLM. `700` ms tolerates a natural mid-sentence
pause before the agent answers. The trade-off is a slightly later reply (the
agent now waits ~0,7 s of silence instead of ~0,3 s), which the filler /
start-filler cues already mask.
- The backchannel micro-pause window is unaffected: its `pause_max_ms` (`260`)
is still below the new cap (`min_silence_ms - 32 = 668`).
---
## 2026-10-17 — Mivocero: barge-in indicator + karaoke on the user bubbles
### Added
- **Server `{"type":"barge_in","source":"vad","prob":…}` event** in
`src/ws/session.rs` — emitted when the server-side VAD onset interrupts the
agent. The onset is detected in the session task, so the browser had no other
way to know a barge-in had happened.
- **Barge-in indicator** in `src/web/mivocero.html` — a red **Barge-in** pill in
the header that flashes and bumps a session counter. Driven by the
`barge_in` event (VAD onset) and by the local **Stop** button; the counter
resets on each connect.
### Changed
- **Karaoke on the user bubbles** in `src/web/mivocero.html` — the user's own
messages (STT transcript and typed text) are now rendered word by word like
the assistant's, with colours tuned for the blue bubble. Because the STT
returns the whole transcript only once the utterance has ended, the words are
lit by a short left-to-right sweep (`USER_REVEAL_MS_PER_WORD`, ~90 ms/word)
instead of following the audio playhead.
- **README** — documented the `barge_in` event in both the server→client table
and the audit `out` row, and updated the Mivocero section (karaoke, barge-in
indicator).
---
## 2026-10-17 — VAD barge-in gated on audible output (no more silent cancelled replies)
### Added
- **`AudioMixer::is_audible()`** in `src/audio/mixer.rs` — reports whether the
agent has *audible* output: the reply is playing out (`tts_primed`) or a
latency-mask cue (filler / start filler) is queued. Unlike `is_active()` it
is `false` while the reply is still being synthesised or sits in the TTS
pre-buffer — nothing has reached the user's ears yet.
- **`TurnManager::is_audible()`** in `src/orchestrator/turn_manager.rs` —
forwards the mixer method for the barge-in gate.
### Changed
- **VAD-driven barge-in gate** in `src/ws/session.rs` — a `SpeechOnset` now
interrupts only when `TurnManager::is_audible()` is true, instead of
`is_speaking()` (`is_active()`). `is_speaking()` is retained for the
backchannel gate, where cueing over any in-flight TTS would still be wrong.
### Notes
- Root cause: `is_speaking()` is already `true` as soon as the TTS request is
in flight (`tts_stream_active`), so a VAD onset landing during the
synthesis / pre-buffer window (~1.2–1.6 s) cancelled the reply before a
single sample was played. On the analysed session
(`s-1792251858371-0001`, ~210 s, 21 turns) three replies streamed their text
to the client but produced **zero playback**, and three more were truncated
after 0.1–0.7 s; several of the onsets were false triggers (the utterance
was later dropped by the gate with `avg_prob` 0.28 / 0.21).
- A reply cancelled this way is not orphaned: the next turn's `on_eou` already
cancels the previous in-flight synthesis and restarts the stream, so the
audio follows the newest turn instead of being lost.
---
## 2026-10-17 — Filler fade-out on reply start (no more overlap with the answer)
### Changed
- **Mixer filler fade-out** in `src/audio/mixer.rs` — when the reply's playback
starts (TTS pre-buffer reached) while an end-of-turn filler or a start filler
is still queued, the cue is now **faded out** over `FILLER_FADEOUT_FRAMES`
(4 frames = 80 ms) and then cut, instead of being ducked to `filler_gain_duck`
and left playing underneath the whole answer. The filler is only a latency
mask, so it now hands off cleanly to the reply.
- New `AudioMixer::filler_fade_frames` counter, armed on the priming
transition and reset by `mark_stream_start` / `flush`.
- **`START_FILLER_GO_DELAY` raised** from `1200` to `1800` ms in `.env` /
`.env.example` — the cue now only fires when the first audio chunk is
genuinely late, instead of on the boundary where it used to overlap the
beginning of the reply.
### Notes
- Root cause: with `START_FILLER_GO_DELAY=1200` the first TTS audio arrived
0.75–1.4 s after the cue, while the `sf03` clip is ~2.83 s long — so the cue
played for its first second at full gain and its tail then overlapped the
start of the answer. On the analysed session this happened on 5 of 6 start
fillers (all `sf03`).
---
## 2026-10-17 — Utterance gate: filter spurious VAD turns before STT
### Added
- **Utterance gate** in `src/ws/session.rs` — a VAD segment is now dropped
**before** transcription when it is too short or too "soft", so the STT can no
longer turn a VAD false-trigger (breath, click, acoustic echo, noise) into a
hallucinated text turn that would each otherwise cost a full LLM turn.
Two independent thresholds, both env-driven:
- `VAD_MIN_UTTERANCE_MS` (`VadSettings::min_utterance_ms`, default `300`,
`0` disables) — drop segments shorter than this.
- `VAD_MIN_AVG_PROB` (`VadSettings::min_avg_prob`, default `0.45`, `0.0`
disables) — drop segments whose average speech probability is below this.
A dropped segment is logged and recorded in the audit as `utterance_dropped`
(`duration_ms`, `avg_prob`, `too_short`, `too_soft`).
- **Empty-transcript guard** in `src/orchestrator/turn_manager.rs::on_eou` — a
transcript that decodes to an empty string (or has no tokens at all) no
longer opens an LLM turn.
### Configuration
- New `.env` keys next to the other `VAD_*` options:
```
VAD_MIN_UTTERANCE_MS=300
VAD_MIN_AVG_PROB=0.45
```
### Notes
- The gate runs only when `VAD_SEGMENT_STT=true`. Raise the probability to be
stricter; lower the duration to keep very short backchannels ("sì", "ok").
- Documentation updated: README (VAD configuration table, VAD wiring section,
audit-event table) and project memory.
---
## 2026-10-17 — Cue methods panel in the Mivocero client (live cue policy)
### Added