vocero-client/s2s-client/README.md

title, emoji, colorFrom, colorTo, sdk, app_port, pinned, short_description, hf_oauth

title	emoji	colorFrom	colorTo	sdk	app_port	pinned	short_description	hf_oauth
HF Realtime Voice	🎙️	indigo	purple	docker	7860	false	Voice chat over WebSocket against a HF speech-to-speech	true

Realtime Voice Demo (WebSocket transport)

Browser voice-chat UI for the backend. The browser streams mic audio over a WebSocket using the OpenAI Realtime GA protocol and plays back the assistant's audio as it arrives.

---

🆕 Novedades / Actualizaciones (Azurian Vocero realtime ver.Beta 2.1)

Versión del navegador que debes ver cargada en consola: 2026-09-04 11:20. Si no aparece, haz borrado de datos del sitio + recarga dura (Ctrl+Shift+R) o abre una pestaña en modo incógnito.

🎨 Rediseño visual completo (tema aurora + glassmorphism)

• Tema de fondo aurora: gradiente radial violeta/rosa/cian sobre lienzo oscuro con malla sutil + animación auroraShift lenta.
• Cromado uniforme glass: todos los elementos flotantes (tarjeta brand, panel user profile, botones icono, botón enviar chat, panel informes sidebar) usan glass blur + saturate + borde gradiente brand 1px (técnica mask padding-box / border-box doble linear-gradient).
• Tokens de marca 3 colores brand (violeta #8b5cf6, rosa #ec4899, cian #22d3ee) mezclados con color-mix(in srgb, brand X%, neutral) → ningún color hardcodeado deriva; todo el UI sigue estos tres tonos.

🧭 Estructura (orden vertical de arriba → abajo)

1. header.topbar brand "Vocero" → avatar circular iniciales nombre usuario.
   • 🟦 Avatar user: círculo 40×40px con gradiente brand, initials mayúsculas bold 800, hover lift/glow + foco teclado. Al hacer clic → diálogo modal perfil usuario (rejilla 2 columnas email/teléfono/username/rol/empresa/userid/país/fecha creación/último login; links mailto:/tel:; chips mono para códigos; mobile ≤480px apilado 1 col).
   • 🟦 Botón toggle chat (icono speech bubble) → ELIMINADO PERMANENTEMENTE. La chat ahora siempre está visible inline; el botón toggle ya no existe en el DOM.
2. main.stage (arriba del todo, flexible): orbe/globode voz animado (3 anillos: core, ring, conic spin 3 velocidades + highlight specular + glows breathe breatheOrb) + caption TAP OR SAY "HOLA VOCERO" micro-bob uppercase gradient.
   • ✅ Antes había una zona vacía sobre el orbe → eliminada, ahora el stage empieza en la parte superior (justify-content: flex-start).
3. <section id="chat-panel" class="chat-panel chat-panel-inline"> SIEMPRE VISIBLE, 100% ancho, SOTTO stage, SOPRA footer sticky (rellena la zona vacía inferior).
   • ✅ Antes: overlay lateral derecho. Ahora: bloque inline dentro de #app como hijo flex-col.
   • 🟦 Sin marco / sin riquadro (fondo aurora seamless, 0 borde, 0 shadow, 0 backdrop blur).
   • Cabecera CONVERSACIÓN h3 uppercase 13px gradiente.
   • #chat-history scroll interno. Burbujas de mensaje glass con código de colores por rol: usuario (tint 10% voice-user cyan + borde 44% cyan + glow) / assistant (tint 10% voice-assistant violet/pink + borde 42% + glow).
   • Distancias seguras anti-borde: .hist-msg.user margin-left:6px auto dx / .hist-msg.assistant margin-right:22px auto sx → nunca las burbujas tocan el borde del viewport (fix confirmado).
   • Formulario input texto #chat-form (arriba del botón enviar ✈️) actualmente oculto (hidden) para modo voz-puro; elimina ese atributo para reactivarlo. El handler submit que envía client.sendUserText(text) al websocket ya está conectado.
4. Footer sticky position: sticky; bottom:0; z:3; backdrop-blur "Azurian Vocero realtime ver.Beta 2.1" siempre dentro del viewport, incluso cuando el documento crece y haces scroll (no requiere scroll hacia abajo para verlo). Separador superior línea gradiente brand.

📜 Sidebar de informes a la IZQUIERDA (antes topbar inline)

• Desktop ≥ 1024px: sidebar fija left 320px siempre visible; #app padding-left 320px reserva sitio.
• Tablet/Mobile < 1024: offcanvas translateX(-105%) slide-in con backdrop blur; toggle mediante botón icono reportes #reports-sidebar-toggle en topbar con badge contador.
• #reports-hint siempre visible; .reports-suggestions y .topbar-right con puerta triple de visibilidad (atributo HTML hidden + CSS rule [data-debug-only]:not([data-debug-enabled]) !important display:none + JS applyDebugOnlyVisibilityGate() ejecutado 6 veces por lifecycle) → solo aparecen cuando el query param ?debug=true está presente en la URL.

⬇️ Auto-scroll chat SIEMPRE hacia la última respuesta del asistente

Arquitectura anti-clipping definitiva (versión 11:20 fix):

1. CSS sobrante intencional (96 → 160px chat-history padding-bottom, 40→160px chat-panel clamp, 56→120px #app padding-bottom + safe-area-inset-bottom) → el último párrafo siempre está por ENCIMA de la línea del footer glass flotante.
2. Unidades móviles seguras svh / svw (smallest viewport): evita oscilaciones layout cuando aparece/desaparece la barra URL del móvil.
3. JS cancel-and-reschedule cada vez que _scrollToBottom() se llama: si llegan 100 tokens streaming en 1 frame, el último cancela todas las cadenas pendientes y se ejecuta desde cero → no hay coalescing flag SKIP (esta era la causa original del scroll insuficiente).
4. 5 reintentos escalonados 0 / +30 / +70 / +150 / +250ms:
   • 0 ms (inmediato): 90% de los casos.
   • 30 ms: tras 1 layout-pass.
   • 70 ms: corrige el wrapping de la última palabra en una línea nueva (bug más común).
   • 150 ms: después de reflow Chrome scroll-anchor.
   • 250 ms: después de expandir filas tool / imágenes grandes.
5. Snap con Number.MAX_SAFE_INTEGER: el navegador clampa automáticamente al máximo real (scrollHeight - clientHeight); no hay race de leer scrollHeight demasiado pronto.
6. overflow-anchor: none; scroll-behavior: auto !important; → sin smooth, sin Chrome scroll-anchor-luchando contra nosotros.
7. Llamadas scroll explicitas extra en: onResponseFinished (fin de respuesta assistant, crítico), onToolCall, onToolResult.

🛡️ Protecciones anti-error de refs eliminadas

• En el refactor de chat eliminamos 4 nodos DOM (#chat-btn, #chat-badge, #chat-panel-backdrop, #chat-panel-close).
• Antes: el constructor de ChatView usaba $("#...") strict → lanzaba Error("Missing element: #chat-btn") al arrancar.
• Ahora: esas 4 refs opcionales usan document.querySelector("#...") → devuelve null cuando no existen, sin throw; todos los listeners guarded if (el). Los 3 IDs obligatorios (#chat-panel, #chat-history, #bubble-stack) siguen con strict para fallar fuerte si alguien los borra por error.

🔄 Actualizaciones de versión / cache bust

Para evitar caché HTTP agresiva en navegadores móviles (ESM sin bundler + service worker = nunca refrescan URLs iguales), todas las referencias a recursos estáticos usan query param ?v=YYYYMMDDHHMM sincronizado en 6 puntos:

1. index.html <link rel="stylesheet" href="style.css?v=...">
2. index.html <script type="module" src="main.js?v=..."> 3-6. Los 4 imports ESM dentro de main.js (ws/s2s-ws-client, ui/dom, ui/chat, ui/account).

Además hay un script inline en <head> tras el style.css que al DOMContentLoaded fuerza a reescribir el href del stylesheet con la última versión si por algún motivo cargó la antigua (anti-cache móvil).

Deploy

cd /home/duxiter/s2s
docker compose restart s2s_ui

Limpia siempre datos del sitio en el navegador (Application → Storage → Clear site data) y recarga dura Ctrl+Shift+R al finalizar.

---

Quick start (local)

1. Start the speech-to-speech backend in realtime mode (from the repo root;

   uv run speech-to-speech \
     --mode realtime \
     --stt parakeet-tdt \
     --llm_backend transformers \
     --tts kokoro \
     --model_name "Qwen/Qwen3-4B-Instruct-2507" \
     --llm_device mps \
     --llm_torch_dtype float16 \
     --enable_live_transcription
   

   The realtime server listens on ws://localhost:8765/v1/realtime by default (--ws_host / --ws_port to change).

2. Start this app, pointing it at the backend with SPEECH_TO_SPEECH_URL:

   uv pip install -r demo/requirements.txt
   export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
   export SERPER_API_KEY=...   # optional; web search is disabled without it
   uv run uvicorn --app-dir demo server:app --reload --port 7860
   

   Or with Docker:

   docker build -t s2s-demo demo/
   docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
   

3. Open http://localhost:7860/, click the orb, allow the mic, talk.

Browsers require HTTPS or localhost for getUserMedia() (mic + camera). 127.0.0.1 and localhost both work; plain http://192.168.x.y does NOT.

Smoke-test the backend from the shell:

websocat ws://localhost:8765/v1/realtime
# -> you should get a session.created event back immediately

How it works

1. The browser opens a WebSocket on the configured /v1/realtime URL.
2. Server pushes session.created on connect. Client replies with session.update (OpenAI Realtime GA schema: session.audio.input, session.audio.output, session.output_modalities).
3. Client streams mic audio as PCM16 16 kHz mono base64 chunks (input_audio_buffer.append, one frame every ~40 ms).
4. Server pushes response.output_audio.delta (PCM16 24 kHz mono base64) and transcript deltas.

The backend exposes one concurrent session per pipeline unit (--num_pipelines to serve more).

Connecting to a backend

Three modes, picked by env (/api/config tells the client which one is active):

• SPEECH_TO_SPEECH_URL env — the mode you want for local use, and the highest priority. The browser connects directly to this realtime WebSocket URL; it's shown read-only in Settings. Setting it disables the load-balancer logic entirely (no /api/session proxy, no queue, no metering, no sign-in). Unlike the LB address it is not a secret. Accepts a full ws(s)://host/v1/realtime URL or a bare host like localhost:8765 (the app adds /v1/realtime).
• Neither env set — Settings → Speech-to-speech server URL: paste a full connect URL or a bare host, and the browser connects to it directly.
• LOAD_BALANCER_URL env — multi-compute deployments only: the browser POSTs the same-origin /api/session proxy, the server forwards to the LB, and the browser dials the per-session compute URL the LB hands back. The LB address never reaches the browser; the Settings URL field is hidden.

SPEECH_TO_SPEECH_URL	LOAD_BALANCER_URL	SPACE_ID	Connection	URL field	Metering
✅	any	any	direct → pinned URL	visible, locked	off
–	–	any	direct → user URL	editable	off
–	✅	✅	LB proxy	hidden	on
–	✅	–	LB proxy	hidden	off

Settings → Restart reconnects with the current voice, instructions and URL.

Tools

The assistant can call two tools mid-conversation (toggle them from the Tools button, top-right):

• Web search — Google results via Serper.dev, proxied server-side so the key never reaches the browser. Set SERPER_API_KEY as an env var / Space secret. Without it, the tool is disabled unless the user pastes their own key in the Tools panel.
• Camera — while enabled, a live self-view shows bottom-left; when the model calls the tool, the current frame is sent to the vision-language model so it can see what you're showing it.

Usage limits (deployed Space only)

Conversation time is metered per UTC day by sign-in tier (see limiter.py / auth.py), but only on the deployed Space — metering turns on only when BOTH LOAD_BALANCER_URL and SPACE_ID (injected automatically by the HF Space runtime) are present. Running locally — even with LOAD_BALANCER_URL exported — leaves the app unmetered. Tunable via env:

Env	Default	What
LIMIT_ANON_SEC	300	Daily seconds for anonymous visitors (5 min)
LIMIT_FREE_SEC	600	Daily seconds for signed-in non-PRO users (10 min)
UNLIMITED_ORGS	(adds to defaults)	Extra HF org names whose members get unlimited usage, like PRO
USAGE_HASH_SECRET	(random)	HMAC secret for hashing identity keys + signing the anon cookie

PRO members are always unlimited. Members of cerebras, HuggingFaceM4, smolagents, and pollen-robotics are unlimited out of the box (shown as "Team", not "PRO"); set UNLIMITED_ORGS=my-team to add more. Matched case-insensitively against the user's organisations from HF OAuth.

Settings (stored in localStorage)

Key	What
Speech-to-speech server URL	Direct realtime WebSocket URL (hidden/locked when pinned by env)
Voice	Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian)
Instructions	System prompt sent in session.update once the WS opens

LocalStorage keys are namespaced s2s.ws.* so this app's settings do NOT collide with the WebRTC variant.

Files

🇪🇸 Notas adicionales (Azurian Vocero): index.html ahora incluye (1) botón toggle lateral izquierdo reportes #reports-sidebar-toggle con badge contador #reports-sidebar-badge visible solo <1024px, (2) <dialog id="user-profile-dialog"> modal perfil usuario que se abre al hacer clic en el avatar circular del encabezado, (3) panel chat <section id="chat-panel"> permanente dentro de #app entre </main> y footer sticky (el panel overlay derecho antiguo y el botón toggle chat superior fueron ELIMINADOS completamente).

File	Role
index.html	Single page, orb + settings modal (identical UI to the WebRTC app)
main.js	State machine, settings, tools, camera, noise-gate UI wiring + chat-form submit handler (client.sendUserText) + user profile dialog open/close + reports sidebar open/close + 3-layer debug gate
ui/chat.js	ChatView: history panel, ephemeral bubbles, transcript/tool streaming; always-inline 100% width chat panel w/ 5-step retry auto-scroll to last assistant line
ui/account.js	Account: HF login chip + popover, daily-limit modal
ui/dom.js	Shared helpers: strict $(selector) that throws on missing, document.querySelector fallback, escHtml, truncateError, DEBUG from URL
auth.py	HF OAuth + per-request identity (tier, hashed keys)
limiter.py	SQLite per-day talk-time budget (chunked server-clock reservation)
ws/s2s-ws-client.js	WebSocket handshake + OpenAI Realtime GA protocol
ws/codec.js	base64 <-> PCM helpers + transcript extraction (pure)
ws/orb-visualizer.js	OrbVisualiser: FFT bands -> orb CSS custom properties (violet/pink/cyan brand gradient)
worklets/mic-capture.js	AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks
worklets/audio-playback.js	AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out
style.css	Orb animations, layout, dark aurora gradient theme, glass chat bubbles, sticky footer, reports sidebar desktop left 320px / mobile offcanvas, user profile 2-col grid modal

Audio pipeline notes

• Input: getUserMedia({ echoCancellation, noiseSuppression, autoGainControl }) feeds the mic-capture worklet at the AudioContext rate. The worklet resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast path, linear interpolation fallback for odd rates) and packs Int16 LE.
• Output: response.output_audio.delta decodes to Int16 -> Float32 and is posted to the audio-playback worklet. The worklet maintains a per-context ring buffer, linearly interpolates 24 -> 48, and applies short 32-frame fades on entry/exit to suppress clicks.
• Barge-in: when the server VAD detects user speech mid-response (input_audio_buffer.speech_started while ai-speaking), the client posts { kind: "clear" } to the playback worklet to wipe the queue immediately. The server itself cancels the in-flight response.

Credits