257 lines
16 KiB
Markdown
257 lines
16 KiB
Markdown
---
|
||
title: HF Realtime Voice
|
||
emoji: 🎙️
|
||
colorFrom: indigo
|
||
colorTo: purple
|
||
sdk: docker
|
||
app_port: 7860
|
||
pinned: false
|
||
short_description: Voice chat over WebSocket against a HF speech-to-speech
|
||
hf_oauth: true
|
||
---
|
||
|
||
# Realtime Voice Demo (WebSocket transport)
|
||
|
||
Browser voice-chat UI for the
|
||
backend. The browser streams mic audio over a WebSocket using the OpenAI
|
||
Realtime **GA** protocol and plays back the assistant's audio as it arrives.
|
||
|
||
---
|
||
|
||
## 🆕 Novedades / Actualizaciones (Azurian Vocero realtime ver.Beta 2.1)
|
||
|
||
> Versión del navegador que debes ver cargada en consola: **`2026-09-04 11:20`**.
|
||
> Si no aparece, haz **borrado de datos del sitio + recarga dura (Ctrl+Shift+R)** o abre una pestaña en modo incógnito.
|
||
|
||
### 🎨 Rediseño visual completo (tema aurora + glassmorphism)
|
||
- **Tema de fondo aurora**: gradiente radial violeta/rosa/cian sobre lienzo oscuro con malla sutil + animación `auroraShift` lenta.
|
||
- **Cromado uniforme glass**: todos los elementos flotantes (tarjeta brand, panel user profile, botones icono, botón enviar chat, panel informes sidebar) usan *glass blur + saturate + borde gradiente brand 1px* (técnica mask `padding-box / border-box` doble linear-gradient).
|
||
- **Tokens de marca 3 colores brand** (violeta `#8b5cf6`, rosa `#ec4899`, cian `#22d3ee`) mezclados con `color-mix(in srgb, brand X%, neutral)` → ningún color hardcodeado deriva; todo el UI sigue estos tres tonos.
|
||
|
||
### 🧭 Estructura (orden vertical de arriba → abajo)
|
||
1. **`header.topbar`** brand "Vocero" → avatar circular iniciales nombre usuario.
|
||
- 🟦 **Avatar user**: círculo 40×40px con gradiente brand, initials mayúsculas bold 800, hover lift/glow + foco teclado. Al hacer clic → diálogo modal perfil usuario (rejilla 2 columnas email/teléfono/username/rol/empresa/userid/país/fecha creación/último login; links `mailto:`/`tel:`; chips mono para códigos; mobile ≤480px apilado 1 col).
|
||
- 🟦 **Botón toggle chat (icono speech bubble) → ELIMINADO PERMANENTEMENTE**. La chat ahora siempre está visible inline; el botón toggle ya no existe en el DOM.
|
||
2. **`main.stage`** (arriba del todo, flexible): orbe/globode voz animado (3 anillos: core, ring, conic spin 3 velocidades + highlight specular + glows breathe `breatheOrb`) + caption `TAP OR SAY "HOLA VOCERO"` micro-bob uppercase gradient.
|
||
- ✅ Antes había una zona vacía sobre el orbe → eliminada, ahora el stage empieza en la parte superior (`justify-content: flex-start`).
|
||
3. **`<section id="chat-panel" class="chat-panel chat-panel-inline">` SIEMPRE VISIBLE, 100% ancho, SOTTO stage, SOPRA footer sticky** (rellena la zona vacía inferior).
|
||
- ✅ Antes: overlay lateral derecho. **Ahora: bloque inline dentro de `#app` como hijo flex-col**.
|
||
- 🟦 Sin marco / sin riquadro (fondo aurora seamless, 0 borde, 0 shadow, 0 backdrop blur).
|
||
- Cabecera `CONVERSACIÓN` h3 uppercase 13px gradiente.
|
||
- `#chat-history` scroll interno. Burbujas de mensaje glass con **código de colores por rol**: usuario (tint 10% voice-user cyan + borde 44% cyan + glow) / assistant (tint 10% voice-assistant violet/pink + borde 42% + glow).
|
||
- Distancias seguras anti-borde: `.hist-msg.user margin-left:6px auto dx` / `.hist-msg.assistant margin-right:22px auto sx` → **nunca las burbujas tocan el borde del viewport** (fix confirmado).
|
||
- Formulario input texto `#chat-form` (arriba del botón enviar ✈️) **actualmente oculto (`hidden`)** para modo voz-puro; elimina ese atributo para reactivarlo. El handler submit que envía `client.sendUserText(text)` al websocket ya está conectado.
|
||
4. **Footer sticky `position: sticky; bottom:0; z:3; backdrop-blur`** "Azurian Vocero realtime ver.Beta 2.1" siempre dentro del viewport, incluso cuando el documento crece y haces scroll (no requiere scroll hacia abajo para verlo). Separador superior línea gradiente brand.
|
||
|
||
### 📜 Sidebar de informes a la IZQUIERDA (antes topbar inline)
|
||
- Desktop **≥ 1024px**: sidebar fija left 320px siempre visible; `#app padding-left 320px` reserva sitio.
|
||
- Tablet/Mobile **< 1024**: offcanvas `translateX(-105%)` slide-in con backdrop blur; toggle mediante botón icono reportes `#reports-sidebar-toggle` en topbar con badge contador.
|
||
- `#reports-hint` siempre visible; `.reports-suggestions` y `.topbar-right` con puerta triple de visibilidad (atributo HTML hidden + CSS rule `[data-debug-only]:not([data-debug-enabled]) !important display:none` + JS `applyDebugOnlyVisibilityGate()` ejecutado 6 veces por lifecycle) → solo aparecen cuando el query param `?debug=true` está presente en la URL.
|
||
|
||
### ⬇️ Auto-scroll chat SIEMPRE hacia la última respuesta del asistente
|
||
Arquitectura anti-clipping definitiva (versión 11:20 fix):
|
||
|
||
1. **CSS sobrante intencional** (`96 → 160px chat-history padding-bottom`, `40→160px chat-panel clamp`, `56→120px #app padding-bottom + safe-area-inset-bottom`) → el último párrafo siempre está por ENCIMA de la línea del footer glass flotante.
|
||
2. **Unidades móviles seguras `svh / svw`** (smallest viewport): evita oscilaciones layout cuando aparece/desaparece la barra URL del móvil.
|
||
3. **JS cancel-and-reschedule** cada vez que `_scrollToBottom()` se llama: si llegan 100 tokens streaming en 1 frame, el último cancela todas las cadenas pendientes y se ejecuta desde cero → **no hay coalescing flag SKIP** (esta era la causa original del scroll insuficiente).
|
||
4. **5 reintentos escalonados `0 / +30 / +70 / +150 / +250ms`**:
|
||
- 0 ms (inmediato): 90% de los casos.
|
||
- 30 ms: tras 1 layout-pass.
|
||
- 70 ms: **corrige el wrapping de la última palabra en una línea nueva** (bug más común).
|
||
- 150 ms: después de reflow Chrome scroll-anchor.
|
||
- 250 ms: después de expandir filas tool / imágenes grandes.
|
||
5. **Snap con `Number.MAX_SAFE_INTEGER`**: el navegador clampa automáticamente al máximo real (`scrollHeight - clientHeight`); no hay race de leer scrollHeight demasiado pronto.
|
||
6. `overflow-anchor: none; scroll-behavior: auto !important;` → sin smooth, sin Chrome scroll-anchor-luchando contra nosotros.
|
||
7. Llamadas scroll explicitas extra en: `onResponseFinished` (fin de respuesta assistant, crítico), `onToolCall`, `onToolResult`.
|
||
|
||
### 🛡️ Protecciones anti-error de refs eliminadas
|
||
- En el refactor de chat eliminamos 4 nodos DOM (`#chat-btn`, `#chat-badge`, `#chat-panel-backdrop`, `#chat-panel-close`).
|
||
- **Antes**: el constructor de `ChatView` usaba `$("#...")` strict → lanzaba `Error("Missing element: #chat-btn")` al arrancar.
|
||
- **Ahora**: esas 4 refs opcionales usan `document.querySelector("#...")` → devuelve `null` cuando no existen, sin throw; todos los listeners guarded `if (el)`. Los 3 IDs obligatorios (`#chat-panel`, `#chat-history`, `#bubble-stack`) siguen con strict para fallar fuerte si alguien los borra por error.
|
||
|
||
### 🔄 Actualizaciones de versión / cache bust
|
||
Para evitar caché HTTP agresiva en navegadores móviles (ESM sin bundler + service worker = nunca refrescan URLs iguales), todas las referencias a recursos estáticos usan query param `?v=YYYYMMDDHHMM` sincronizado en **6 puntos**:
|
||
1. `index.html <link rel="stylesheet" href="style.css?v=...">`
|
||
2. `index.html <script type="module" src="main.js?v=...">`
|
||
3-6. Los **4 imports ESM dentro de main.js** (`ws/s2s-ws-client`, `ui/dom`, `ui/chat`, `ui/account`).
|
||
|
||
Además hay un **script inline en `<head>`** tras el style.css que al `DOMContentLoaded` fuerza a reescribir el href del stylesheet con la última versión si por algún motivo cargó la antigua (anti-cache móvil).
|
||
|
||
### Deploy
|
||
```bash
|
||
cd /home/duxiter/s2s
|
||
docker compose restart s2s_ui
|
||
```
|
||
Limpia siempre datos del sitio en el navegador (Application → Storage → Clear site data) y recarga dura `Ctrl+Shift+R` al finalizar.
|
||
|
||
---
|
||
|
||
## Quick start (local)
|
||
|
||
1. **Start the speech-to-speech backend** in realtime mode (from the repo root;
|
||
```bash
|
||
uv run speech-to-speech \
|
||
--mode realtime \
|
||
--stt parakeet-tdt \
|
||
--llm_backend transformers \
|
||
--tts kokoro \
|
||
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
|
||
--llm_device mps \
|
||
--llm_torch_dtype float16 \
|
||
--enable_live_transcription
|
||
```
|
||
|
||
The realtime server listens on `ws://localhost:8765/v1/realtime` by default
|
||
(`--ws_host` / `--ws_port` to change).
|
||
|
||
2. **Start this app**, pointing it at the backend with `SPEECH_TO_SPEECH_URL`:
|
||
|
||
```bash
|
||
uv pip install -r demo/requirements.txt
|
||
export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
|
||
export SERPER_API_KEY=... # optional; web search is disabled without it
|
||
uv run uvicorn --app-dir demo server:app --reload --port 7860
|
||
```
|
||
|
||
Or with Docker:
|
||
|
||
```bash
|
||
docker build -t s2s-demo demo/
|
||
docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
|
||
```
|
||
|
||
3. Open <http://localhost:7860/>, click the orb, allow the mic, talk.
|
||
|
||
> Browsers require **HTTPS or `localhost`** for `getUserMedia()` (mic + camera).
|
||
> `127.0.0.1` and `localhost` both work; plain `http://192.168.x.y` does NOT.
|
||
|
||
Smoke-test the backend from the shell:
|
||
|
||
```bash
|
||
websocat ws://localhost:8765/v1/realtime
|
||
# -> you should get a session.created event back immediately
|
||
```
|
||
|
||
## How it works
|
||
|
||
1. The browser opens a WebSocket on the configured `/v1/realtime` URL.
|
||
2. Server pushes `session.created` on connect. Client replies with
|
||
`session.update` (OpenAI Realtime **GA** schema: `session.audio.input`,
|
||
`session.audio.output`, `session.output_modalities`).
|
||
3. Client streams mic audio as PCM16 16 kHz mono base64 chunks
|
||
(`input_audio_buffer.append`, one frame every ~40 ms).
|
||
4. Server pushes `response.output_audio.delta` (PCM16 24 kHz mono base64)
|
||
and transcript deltas.
|
||
|
||
The backend exposes one concurrent session per pipeline unit
|
||
(`--num_pipelines` to serve more).
|
||
|
||
## Connecting to a backend
|
||
|
||
Three modes, picked by env (`/api/config` tells the client which one is active):
|
||
|
||
- **`SPEECH_TO_SPEECH_URL` env** — the mode you want for local use, and the
|
||
highest priority. The browser connects **directly** to this realtime
|
||
WebSocket URL; it's shown read-only in Settings. Setting it disables the
|
||
load-balancer logic entirely (no `/api/session` proxy, no queue, no
|
||
metering, no sign-in). Unlike the LB address it is not a secret. Accepts a
|
||
full `ws(s)://host/v1/realtime` URL or a bare host like `localhost:8765`
|
||
(the app adds `/v1/realtime`).
|
||
- **Neither env set** — **Settings → Speech-to-speech server URL**: paste a
|
||
full connect URL or a bare host, and the browser connects to it directly.
|
||
- **`LOAD_BALANCER_URL` env** — multi-compute deployments only: the browser
|
||
POSTs the same-origin `/api/session` proxy, the server forwards to the LB,
|
||
and the browser dials the per-session compute URL the LB hands back. The LB
|
||
address never reaches the browser; the Settings URL field is hidden.
|
||
|
||
| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Metering |
|
||
|:---:|:---:|:---:|---|---|---|
|
||
| ✅ | any | any | direct → pinned URL | visible, locked | off |
|
||
| – | – | any | direct → user URL | editable | off |
|
||
| – | ✅ | ✅ | LB proxy | hidden | **on** |
|
||
| – | ✅ | – | LB proxy | hidden | off |
|
||
|
||
**Settings → Restart** reconnects with the current voice, instructions and URL.
|
||
|
||
## Tools
|
||
|
||
The assistant can call two tools mid-conversation (toggle them from the **Tools**
|
||
button, top-right):
|
||
|
||
- **Web search** — Google results via Serper.dev, proxied server-side so the key
|
||
never reaches the browser. Set `SERPER_API_KEY` as an env var / Space secret.
|
||
Without it, the tool is disabled unless the user pastes their own key in the
|
||
Tools panel.
|
||
- **Camera** — while enabled, a live self-view shows bottom-left; when the model
|
||
calls the tool, the current frame is sent to the vision-language model so it can
|
||
see what you're showing it.
|
||
|
||
## Usage limits (deployed Space only)
|
||
|
||
Conversation time is metered per UTC day by sign-in tier (see `limiter.py` /
|
||
`auth.py`), but **only on the deployed Space** — metering turns on only when BOTH
|
||
`LOAD_BALANCER_URL` and `SPACE_ID` (injected automatically by the HF Space
|
||
runtime) are present. Running locally — even with `LOAD_BALANCER_URL` exported —
|
||
leaves the app unmetered. Tunable via env:
|
||
|
||
| Env | Default | What |
|
||
|-----|---------|------|
|
||
| `LIMIT_ANON_SEC` | `300` | Daily seconds for anonymous visitors (5 min) |
|
||
| `LIMIT_FREE_SEC` | `600` | Daily seconds for signed-in non-PRO users (10 min) |
|
||
| `UNLIMITED_ORGS` | _(adds to defaults)_ | Extra HF org names whose members get **unlimited** usage, like PRO |
|
||
| `USAGE_HASH_SECRET` | _(random)_ | HMAC secret for hashing identity keys + signing the anon cookie |
|
||
|
||
PRO members are always unlimited. Members of `cerebras`, `HuggingFaceM4`,
|
||
`smolagents`, and `pollen-robotics` are unlimited out of the box (shown as
|
||
"Team", not "PRO"); set `UNLIMITED_ORGS=my-team` to add more. Matched
|
||
case-insensitively against the user's organisations from HF OAuth.
|
||
|
||
## Settings (stored in `localStorage`)
|
||
|
||
| Key | What |
|
||
|-----|------|
|
||
| Speech-to-speech server URL | Direct realtime WebSocket URL (hidden/locked when pinned by env) |
|
||
| Voice | Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian) |
|
||
| Instructions | System prompt sent in `session.update` once the WS opens |
|
||
|
||
LocalStorage keys are namespaced `s2s.ws.*` so this app's settings do
|
||
NOT collide with the WebRTC variant.
|
||
|
||
## Files
|
||
|
||
> 🇪🇸 **Notas adicionales (Azurian Vocero)**: `index.html` ahora incluye (1) botón toggle lateral izquierdo reportes `#reports-sidebar-toggle` con badge contador `#reports-sidebar-badge` visible solo <1024px, (2) `<dialog id="user-profile-dialog">` modal perfil usuario que se abre al hacer clic en el avatar circular del encabezado, (3) panel chat `<section id="chat-panel">` permanente dentro de `#app` entre `</main>` y footer sticky (el panel overlay derecho antiguo y el botón toggle chat superior fueron ELIMINADOS completamente).
|
||
|
||
| File | Role |
|
||
|------|------|
|
||
| `index.html` | Single page, orb + settings modal (identical UI to the WebRTC app) |
|
||
| `main.js` | State machine, settings, tools, camera, noise-gate UI wiring + chat-form submit handler (`client.sendUserText`) + user profile dialog open/close + reports sidebar open/close + 3-layer debug gate |
|
||
| `ui/chat.js` | `ChatView`: history panel, ephemeral bubbles, transcript/tool streaming; always-inline 100% width chat panel w/ 5-step retry auto-scroll to last assistant line |
|
||
| `ui/account.js` | `Account`: HF login chip + popover, daily-limit modal |
|
||
| `ui/dom.js` | Shared helpers: strict `$(selector)` that throws on missing, `document.querySelector` fallback, `escHtml`, `truncateError`, `DEBUG` from URL |
|
||
| `auth.py` | HF OAuth + per-request identity (tier, hashed keys) |
|
||
| `limiter.py` | SQLite per-day talk-time budget (chunked server-clock reservation) |
|
||
| `ws/s2s-ws-client.js` | WebSocket handshake + OpenAI Realtime GA protocol |
|
||
| `ws/codec.js` | base64 <-> PCM helpers + transcript extraction (pure) |
|
||
| `ws/orb-visualizer.js` | `OrbVisualiser`: FFT bands -> orb CSS custom properties (violet/pink/cyan brand gradient) |
|
||
| `worklets/mic-capture.js` | AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks |
|
||
| `worklets/audio-playback.js` | AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out |
|
||
| `style.css` | Orb animations, layout, dark aurora gradient theme, glass chat bubbles, sticky footer, reports sidebar desktop left 320px / mobile offcanvas, user profile 2-col grid modal |
|
||
|
||
## Audio pipeline notes
|
||
|
||
- **Input**: `getUserMedia({ echoCancellation, noiseSuppression, autoGainControl })`
|
||
feeds the `mic-capture` worklet at the `AudioContext` rate. The worklet
|
||
resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast
|
||
path, linear interpolation fallback for odd rates) and packs Int16 LE.
|
||
- **Output**: `response.output_audio.delta` decodes to Int16 -> Float32
|
||
and is posted to the `audio-playback` worklet. The worklet maintains a
|
||
per-context ring buffer, linearly interpolates 24 -> 48, and applies
|
||
short 32-frame fades on entry/exit to suppress clicks.
|
||
- **Barge-in**: when the server VAD detects user speech mid-response
|
||
(`input_audio_buffer.speech_started` while `ai-speaking`), the client
|
||
posts `{ kind: "clear" }` to the playback worklet to wipe the queue
|
||
immediately. The server itself cancels the in-flight response.
|
||
|
||
## Credits
|
||
|