before language fix
CI / ruff (push) Has been cancelled
CI / mypy (push) Has been cancelled
CI / pytest (push) Has been cancelled
CI / package (push) Has been cancelled
CI / Install smoke (${{ matrix.label }}) (linux, ubuntu-latest) (push) Has been cancelled
CI / Install smoke (${{ matrix.label }}) (macos-arm64, macos-14) (push) Has been cancelled
CI / ruff (push) Has been cancelled
CI / mypy (push) Has been cancelled
CI / pytest (push) Has been cancelled
CI / package (push) Has been cancelled
CI / Install smoke (${{ matrix.label }}) (linux, ubuntu-latest) (push) Has been cancelled
CI / Install smoke (${{ matrix.label }}) (macos-arm64, macos-14) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Vocero — Asistente Digital de Voz
|
||||
|
||||
## ¿Qué es Vocero?
|
||||
|
||||
Vocero es un asistente digital conversacional avanzado que opera completamente a través de voz, implementando una pipeline **Speech-to-Speech (S2S)** en tiempo real. Permite a los usuarios interactuar de forma natural mediante diálogo hablado, sin necesidad de interfaces gráficas ni escritura.
|
||||
|
||||
El nombre "Vocero" evoca la figura del portavoz: un sistema que escucha, comprende, razona y responde con voz clara y coherente, actuando como intermediario entre el usuario y el conocimiento disponible.
|
||||
|
||||
---
|
||||
|
||||
## Arquitectura General
|
||||
|
||||
Vocero se compone de cinco módulos encadenados en una pipeline de baja latencia:
|
||||
|
||||
1. **Detección de Actividad Vocal (VAD)** — Identifica cuándo el usuario empieza y termina de hablar, filtrando el silencio y el ruido de fondo.
|
||||
2. **Reconocimiento de Voz (STT / Speech-to-Text)** — Transcribe el audio del habla a texto utilizando modelos optimizados.
|
||||
3. **Modelo de Lenguaje (LLM)** — Procesa el texto transcrito, razona sobre la consulta y genera una respuesta textual coherente. Incluye un sistema **RAG server-side** que inyecta contexto de la base de conocimiento en cada turno de forma transparente.
|
||||
4. **Síntesis de Voz (TTS / Text-to-Speech)** — Convierte la respuesta textual en audio natural con acento y tono apropiados.
|
||||
5. **Streaming en Tiempo Real** — Todos los módulos se ejecutan en paralelo mediante el protocolo **OpenAI Realtime**, permitiendo respuestas incrementalmente (streaming) en lugar de esperar el final de cada etapa.
|
||||
|
||||
Todo esto está orquestado por un servidor **FastAPI + Uvicorn** con un pool de unidades de pipeline (`PipelineUnit`) que manejan múltiples sesiones concurrentes.
|
||||
|
||||
---
|
||||
|
||||
## Características Principales
|
||||
|
||||
### 🎙️ Conversación por Voz Completamente Natural
|
||||
- Interacción manos libres ideal para entornos industriales, automoción, atención al cliente o asistencia personal.
|
||||
- Latencia baja gracias al procesamiento por fragmentos (chunks) de audio.
|
||||
|
||||
### 📚 Base de Conocimiento (RAG Server-Side)
|
||||
- Vocero cuenta con una **Knowledge Base (KB)** local de documentos en formato Markdown, texto plano y JSONL.
|
||||
- El sistema indexa automáticamente los documentos usando embeddings multilingües (`paraphrase-multilingual-MiniLM-L12-v2`).
|
||||
- Antes de cada respuesta del LLM, el **RAGRetriever** busca los fragmentos (chunks) más relevantes de la KB y los inyecta directamente en el prompt del modelo.
|
||||
- La KB se puede actualizar en caliente vía API HTTP (`/v1/rag/`) sin necesidad de reiniciar el servicio.
|
||||
- La persistencia en disco (archivos NPZ para los embeddings y JSONL para metadatos) evita la reindexación completa en cada arranque.
|
||||
|
||||
### 🌍 Multilingüe
|
||||
- Diseñado principalmente para **español** (`es`) e **italiano** (`it`), con soporte para inglés (`en`) gracias a los modelos de embedding multilingües.
|
||||
- El header del prompt RAG se adapta al idioma configurado.
|
||||
- El TTS utiliza modelos específicos con voces y acentos naturales (ej: acento chileno para el español).
|
||||
|
||||
### ⚙️ Escalable y Concurrente
|
||||
- El parámetro `--num_pipelines` define cuántas sesiones WebSocket simultáneas se pueden atender.
|
||||
- Gestión de procesos en producción mediante **PM2** con logs, reinicio automático y monitorización.
|
||||
- Compatible con arquitecturas **ARM64/NVIDIA Grace Blackwell (GB10)** y GPU CUDA para máxima eficiencia.
|
||||
|
||||
### 🔄 Actualización Dinámica del Conocimiento
|
||||
- La KB se puede recargar en caliente con el endpoint `/v1/rag/reload`.
|
||||
- Los documentos se añaden o eliminan vía API REST sin detener el servicio.
|
||||
- El sistema de **lock reentrante (RLock)** garantiza operaciones thread-safe sobre el índice.
|
||||
|
||||
---
|
||||
|
||||
## Casos de Uso Típicos
|
||||
|
||||
- **Asistente de Información Interna**: Empresas que necesitan un portavoz virtual que conozca manuales, normativas y procedimientos internos.
|
||||
- **Atención al Cliente por Voz**: Call centers automatizados, quioscos interactivos, terminales de atención pública.
|
||||
- **Guía Turístico Audiovisual**: Museos, parques, ciudades que ofrecen información hablada contextualizada.
|
||||
- **Asistente Industrial**: Técnicos que necesitan consultar manuales o procedimientos mientras tienen las manos ocupadas.
|
||||
- **Educación y Formación**: Tutores virtuales que explican conceptos y responden preguntas en voz alta.
|
||||
|
||||
---
|
||||
|
||||
## Ventajas Tecnológicas
|
||||
|
||||
- **RAG "invisible"**: A diferencia de sistemas que muestran fuentes citadas, Vocero integra el contexto directamente en el razonamiento del LLM, ofreciendo respuestas fluidas.
|
||||
- **Sin dependencia de APIs externas en el edge**: Todo el pipeline (STT, LLM, TTS, RAG) se puede ejecutar on-premise o en hardware local.
|
||||
- **Persistencia inteligente**: La primera indexación es lenta, pero las siguientes son casi instantáneas gracias al cache NPZ.
|
||||
- **API de administración completa**: Endpoints `/v1/rag/*` para listar, añadir, eliminar y recargar documentos con simples peticiones HTTP/curl.
|
||||
|
||||
---
|
||||
|
||||
## En Resumen
|
||||
|
||||
**Vocero es mucho más que un chatbot con voz**: es un asistente digital de confianza, que escucha, comprende el contexto de tu organización y responde con voz natural, apoyándose en una base de conocimiento actualizable en tiempo real. Su arquitectura modular y su foco en el procesamiento del habla en tiempo real lo hacen ideal para escenarios donde la conversación humana es el canal principal.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Preguntas Frecuentes - Producto (Ejemplo)
|
||||
|
||||
> Este es un archivo de ejemplo. Sustitúyelo con tus documentos reales.
|
||||
|
||||
## ¿Cómo puedo restablecer mi contraseña?
|
||||
|
||||
Para restablecer tu contraseña ve a la página de inicio de sesión y haz clic en
|
||||
*"Olvidé mi contraseña"*. Recibirás un correo electrónico con un enlace válido
|
||||
durante 15 minutos. Si no recibes el correo, revisa la carpeta de spam o ponte en
|
||||
contacto con soporte en soporte@empresa.es.
|
||||
|
||||
## ¿Cuál es el horario de atención al cliente?
|
||||
|
||||
El horario de atención telefónica y por chat es de **lunes a viernes de 09:00 a 18:00**
|
||||
(hora peninsular española). Los fines de semana solo hay soporte por email con
|
||||
respuesta máxima en 24 horas.
|
||||
|
||||
## ¿Qué métodos de pago aceptáis?
|
||||
|
||||
Aceptamos Visa, Mastercard, American Express, PayPal, transferencia bancaria y
|
||||
Bizum. Para clientes empresariales disponemos de pago a 30 días tras aprobación
|
||||
del crédito.
|
||||
|
||||
## ¿Ofrecéis reembolsos?
|
||||
|
||||
Sí. Todos los planes tienen un periodo de devolución de **14 días naturales**
|
||||
desde la compra, sin necesidad de justificación. Pasado ese plazo, los reembolsos
|
||||
se evalúan caso por caso por el equipo de soporte.
|
||||
|
||||
## ¿En qué idiomas está disponible la plataforma?
|
||||
|
||||
La interfaz web está disponible en español, inglés, francés, italiano y portugués.
|
||||
El soporte por chat se ofrece únicamente en español e inglés de momento.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Políticas Internas - Ejemplo
|
||||
|
||||
> Este es un archivo de ejemplo. Elíminalo o sustitúyelo por tus políticas reales.
|
||||
|
||||
## Política de privacidad de datos de clientes
|
||||
|
||||
Todos los datos de los clientes se almacenan cifrados en repositorios UE (Irlanda
|
||||
y Frankfurt) según el Reglamento General de Protección de Datos (RGPD). Los datos
|
||||
de pago nunca se almacenan en nuestros servidores; son procesados directamente por
|
||||
nuestro proveedor PCI-DSS certificado (Stripe).
|
||||
|
||||
## Retención de registros
|
||||
|
||||
- Datos de usuario activo: mientras dure la relación contractual + 1 mes.
|
||||
- Datos de usuario dado de baja: 6 meses (obligación fiscal).
|
||||
- Registros de auditoría y acceso: 2 años.
|
||||
- Grabaciones de llamadas de soporte: 90 días, salvo controversia judicial.
|
||||
|
||||
## Niveles de servicio (SLA)
|
||||
|
||||
El servicio tiene un **SLA de disponibilidad del 99,9% mensual** para clientes del
|
||||
plan Pro y Enterprise. Si no se cumple, se aplica un crédito proporcional del 10%
|
||||
sobre la cuota mensual. Para incidentes de severidad 1 (servicio caído) el tiempo
|
||||
medio de respuesta es inferior a 1 hora en horario comercial.
|
||||
|
||||
## Seguridad: autenticación y sesiones
|
||||
|
||||
- Todos los usuarios deben usar autenticación en dos pasos (2FA) obligatoria.
|
||||
- Las sesiones web expiran a los 60 minutos de inactividad.
|
||||
- Contraseñas: mínimo 12 caracteres, 1 mayúscula, 1 número, 1 símbolo. Rotación
|
||||
recomendada cada 90 días.
|
||||
Binary file not shown.
@@ -0,0 +1,12 @@
|
||||
.git
|
||||
.gitignore
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
.vscode/
|
||||
.idea/
|
||||
.DS_Store
|
||||
*.log
|
||||
docs/
|
||||
.claude/
|
||||
@@ -0,0 +1,21 @@
|
||||
# macOS metadata
|
||||
.DS_Store
|
||||
|
||||
# Editor scratch
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
|
||||
# Static HTTP server logs (we don't ship them)
|
||||
*.log
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Local-only replica deploy script (never tracked)
|
||||
deploy_replica.py
|
||||
|
||||
.env
|
||||
@@ -0,0 +1,108 @@
|
||||
# Context glossary
|
||||
|
||||
Canonical terms for this space. A glossary, not a spec — it defines what words mean,
|
||||
not how anything is built. Keep design/implementation detail in `DESIGN.md` and the
|
||||
code.
|
||||
|
||||
## Speech-to-speech demo
|
||||
The product: a voice conversation you have with a model by tapping the orb and
|
||||
talking. "The demo" and "the space" refer to this same thing. It runs on Hugging
|
||||
Face's open `speech-to-speech` backend.
|
||||
|
||||
## The pipeline
|
||||
The ordered path a turn travels, from your voice to the orb's reply. Order is
|
||||
meaningful — each stage consumes the previous one's output:
|
||||
|
||||
`you speak → VAD → STT → VLM → TTS → orb replies`
|
||||
|
||||
- **VAD** — voice activity detection. Decides *when* you are speaking, so the system
|
||||
knows a turn has started and ended. Model: silero-vad.
|
||||
- **STT** — speech to text. Transcribes your speech into words. Model:
|
||||
nvidia/parakeet-tdt-1.1b.
|
||||
- **VLM** — the vision-language model that composes the reply. Served via Cerebras.
|
||||
Model: google/gemma-4-31B-it.
|
||||
- **TTS** — text to speech. Speaks the reply back in the chosen voice. Model:
|
||||
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice.
|
||||
|
||||
## Builder
|
||||
A Hugging Face user credited with making the space, shown by HF username. Current
|
||||
builders: tfrere, A-Mahla and andito. Distinct from the *models'* authors (nvidia, google,
|
||||
Qwen, snakers4), who are credited per pipeline stage.
|
||||
|
||||
## Powered by
|
||||
The infrastructure running the pipeline, named in the about panel: Hugging Face
|
||||
Inference Endpoints (hosting) and Cerebras (LLM inference). Distinct from "built by"
|
||||
(the people) and from the model authors (who trained each model).
|
||||
|
||||
## Tool
|
||||
A function the model can call mid-conversation to do something the pipeline
|
||||
can't do on its own (look something up, look through the camera). Tools are
|
||||
declared to the backend in the session config; the model decides when to call
|
||||
one. Distinct from the *pipeline stages* (VAD/STT/VLM/TTS), which always run.
|
||||
|
||||
## Tool executor
|
||||
The client-side component that runs a tool when the model calls it and returns
|
||||
the result to the backend, so the model can speak the answer. It is the missing
|
||||
half of the round-trip: the backend already emits the call, the executor runs it
|
||||
and replies. Distinct from the *tool* itself (the thing being run).
|
||||
|
||||
## Web search tool
|
||||
A tool that looks something up on the web for the model. The model calls it with
|
||||
a query; the tool executor forwards the query to the search proxy and returns the
|
||||
results as the tool result. Activates only when a search key is available.
|
||||
|
||||
## Camera snapshot tool
|
||||
A tool that lets the model look through the user's webcam. While enabled, a live
|
||||
self-view is shown in the page (bottom-left); when the model calls the tool, the
|
||||
executor captures a frame and sends it to the model as an image so the VLM can
|
||||
see it. Distinct from the *preview* (what the user sees) and the *snapshot* (the
|
||||
single frame sent to the model).
|
||||
|
||||
## Search proxy
|
||||
The same-origin server route (`/search`) that holds the search key and calls the
|
||||
external search provider on the client's behalf, so the key never reaches the
|
||||
browser. Lives in the same container as the page. Distinct from the *s2s backend*
|
||||
(the separate load-balanced speech-to-speech service).
|
||||
|
||||
## Tools panel
|
||||
The dialog opened from the "Tools" button in the top-right, holding one switch per
|
||||
tool (and the web-search key status). Turning a switch on/off declares or removes
|
||||
that tool on the live session. Distinct from *Settings* (connection, voice,
|
||||
instructions) and the *About panel* (project info).
|
||||
|
||||
## Identity block
|
||||
The top-left corner of the topbar (replacing the old wordmark): the demo name, a
|
||||
one-line blurb, and the "powered by" / "built by" credits, shown directly rather
|
||||
than hidden behind a click.
|
||||
|
||||
## About panel
|
||||
The popup opened from the (i) icon to the right of the identity block. Holds the
|
||||
general introduction to the speech-to-speech project (with a repo link) and the
|
||||
pipeline. Identity itself now lives in the corner, not here.
|
||||
|
||||
## Queue
|
||||
The line of users waiting for a free conversation slot when every compute is
|
||||
busy. You join the queue instead of being turned away; you leave it by reaching
|
||||
the front or by giving up. Distinct from a *session* (an actual live
|
||||
conversation) — being in the queue is not yet talking, and time spent waiting
|
||||
never counts against your usage limit.
|
||||
|
||||
## Ticket
|
||||
Your held place in the queue. Created when you join, it is what the demo checks
|
||||
to tell you your position and to notice if you have left. A ticket is not a
|
||||
session: it only promises a spot in line, not a compute.
|
||||
|
||||
## Position
|
||||
How many people are ahead of you in the queue, shown while you wait ("You're #3
|
||||
in line"). It only ever counts down. Distinct from an *estimated wait* — the
|
||||
demo shows position, never a time, because wait time is unpredictable.
|
||||
|
||||
## Claim
|
||||
The moment you reach the front and a free slot becomes yours — the queue hands
|
||||
off to a real session and the conversation begins. This is also the point where
|
||||
your usage limit first starts to matter (never while waiting).
|
||||
|
||||
## At capacity
|
||||
The state where the queue itself is full, so new users can't even join the line
|
||||
and are asked to try again shortly. Distinct from simply *busy* (all computes
|
||||
taken but the queue still has room to wait in).
|
||||
@@ -0,0 +1,205 @@
|
||||
# Design language
|
||||
|
||||
The reference for keeping this app visually coherent as it grows. Read it before
|
||||
touching `style.css`, `index.html`, or any DOM-building code in `main.js`. Every
|
||||
rule here is already live in the codebase — this file explains the *why* so changes
|
||||
extend the system instead of drifting from it.
|
||||
|
||||
---
|
||||
|
||||
## The thesis: color belongs to the voice
|
||||
|
||||
This is a voice app. The one thing in the room that should have color is the thing
|
||||
that is talking. So:
|
||||
|
||||
- **The orb** carries saturated color. It glows, and the glow's hue changes with
|
||||
conversational state.
|
||||
- **Everything else is monochrome** — a precise cool-grey dark canvas. Surfaces,
|
||||
borders, buttons, panels, the transcript: all greyscale.
|
||||
- The **only** exceptions are tiny *role echoes* (a one-word mono label, a small
|
||||
icon) that borrow the orb's state hue so the transcript reads in the same color
|
||||
language the orb speaks. They are accents the size of a word, never fills.
|
||||
- **Brand logos keep their own color.** The Hugging Face mark (`#ffd21e`) and the
|
||||
Cerebras mark (`#f15a29`) render in their brand colors in the identity credits and
|
||||
the about panel — a deliberate, owner-approved exception. It applies to those two
|
||||
logos only; do not generalize it to other chrome.
|
||||
|
||||
If you find yourself adding a tinted background, a colored border, or a bright
|
||||
button anywhere outside the orb, stop — that color almost certainly belongs to the
|
||||
orb instead, or shouldn't exist.
|
||||
|
||||
---
|
||||
|
||||
## Color tokens
|
||||
|
||||
All defined in `:root` in `style.css`. Use the variables, never raw hex in rules.
|
||||
|
||||
### Canvas (the monochrome world)
|
||||
| Token | Value | Use |
|
||||
|---|---|---|
|
||||
| `--bg` | `#0a0b10` | Page background (a cool near-black) |
|
||||
| `--bg-elev` | `#13151c` | Raised surfaces: bubbles, panels, icon buttons |
|
||||
| `--bg-elev-2` | `#1b1e29` | Surfaces on surfaces: history bodies, inputs |
|
||||
| `--border` | `rgba(255,255,255,.08)` | Default hairline |
|
||||
| `--border-strong` | `rgba(255,255,255,.16)` | Emphasised hairline |
|
||||
| `--text` | `#f5f6fa` | Primary text; also the *primary button* fill |
|
||||
| `--text-dim` | `rgba(245,246,250,.65)` | Secondary text |
|
||||
| `--text-faint` | `rgba(245,246,250,.42)` | Captions, labels, footer |
|
||||
|
||||
### Voice (the only saturated hues)
|
||||
These are the orb's state colors. They appear on the orb, and as small role echoes
|
||||
in the transcript — nowhere else.
|
||||
|
||||
| Token | Value | Meaning |
|
||||
|---|---|---|
|
||||
| `--accent` / `--speaking` | `#8b7dff` violet | Assistant speaking |
|
||||
| `--accent-2` / `--listening` | `#22d3ee` cyan | You / listening |
|
||||
| `--processing` | `#f59e0b` amber | Thinking / tool call |
|
||||
| `--error` | `#ff6a75` | Error |
|
||||
| `--success` | `#34d399` | Ready / connected |
|
||||
|
||||
### Role echoes (semantic aliases — use these in chat code)
|
||||
| Token | Maps to | Where it shows |
|
||||
|---|---|---|
|
||||
| `--voice-user` | cyan | `YOU` label + user bubble accents |
|
||||
| `--voice-assistant` | violet | `ASSISTANT` label + assistant accents |
|
||||
| `--voice-tool` | amber | `TOOL CALL` label, wrench icon |
|
||||
|
||||
**Why this mapping:** it mirrors the orb exactly — when *you* speak the orb is cyan
|
||||
(`state-listening`), when the *assistant* speaks it's violet (`state-ai-speaking`),
|
||||
when it's working it's amber (`state-processing`). The transcript is a quiet replay
|
||||
of the orb's color story.
|
||||
|
||||
### Orb state → glow (`.circle.state-*` → `--glow`)
|
||||
| State | Glow |
|
||||
|---|---|
|
||||
| `signed-out` | violet `#8b7dff` |
|
||||
| `authenticated` / `ready` | green `#34d399` |
|
||||
| `connecting` / `connected` / `starting` | yellow `#facc15` |
|
||||
| `listening` / `user-speaking` | cyan (`--listening`) |
|
||||
| `processing` | amber (`--processing`) |
|
||||
| `ai-speaking` | violet (`--speaking`) |
|
||||
| `error` | red (`--error`) |
|
||||
|
||||
Adding a new state? Give it a `--glow`, and if it surfaces in the transcript, add a
|
||||
matching `--voice-*` alias rather than a one-off color.
|
||||
|
||||
---
|
||||
|
||||
## Typography
|
||||
|
||||
Two faces, two jobs. Never reach for a third.
|
||||
|
||||
- **Inter** — body and UI. Wordmark, buttons, inputs, panel titles, prose, history
|
||||
message bodies. The workhorse; it should feel neutral and get out of the way.
|
||||
- **Geist Mono** (`--font-mono`) — the **machine voice**. Reserved for text the
|
||||
*system* emits or identifiers it reports, never for human prose.
|
||||
|
||||
### When mono is correct
|
||||
Mono signals "this is the machine talking or naming itself." Use it for:
|
||||
- the orb's status caption (`.circle-caption`)
|
||||
- role eyebrows (`YOU` / `ASSISTANT` / `TOOL CALL`)
|
||||
- tool-call names and argument JSON
|
||||
- the `·WebSocket` transport tag, bitrate readouts, connection identifiers
|
||||
- the empty-state label
|
||||
|
||||
Mono text is set uppercase with `letter-spacing: ~0.1–0.14em` and weight `500`, so it
|
||||
reads as a typed status line, not a headline. Body copy, button labels, and
|
||||
explanatory `small` text stay **Inter** — putting prose in mono breaks the metaphor.
|
||||
|
||||
The font is loaded in `index.html`; the stack falls back to system mono gracefully if
|
||||
the CDN is blocked.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
- **One continuous canvas.** No dividers under the topbar or above the footer, no
|
||||
panel chrome competing with content. The topbar and footer float over the stage.
|
||||
Keep it that way — a new section earns a hairline (`--border`) only if it genuinely
|
||||
needs separating.
|
||||
- **The orb is the hero and the center of gravity.** It sits dead-center on the
|
||||
stage. Controls flank it (mic / stop), captions sit beneath. Don't crowd it.
|
||||
- **Hairlines, not boxes.** Separation comes from `1px` borders at 8–16% white and
|
||||
from spacing, not from heavy fills or shadows. Shadows are soft and low
|
||||
(`0 4px 18px rgba(0,0,0,.32)`), used only to lift floating elements (bubbles,
|
||||
panels, modal).
|
||||
- **Radii:** `--radius-sm: 8px` (buttons, inputs, chips), `--radius-md: 14px`
|
||||
(bubbles, message bodies, modal), `--radius-lg: 22px` (reserved). Pick by element
|
||||
size; don't invent new values.
|
||||
- **Two reading surfaces for the transcript:** ephemeral bubbles top-right (desktop
|
||||
only) that log and fade, and a slide-in history panel for review. On phones the
|
||||
bubble stream is dropped and the panel goes full-screen — the panel is the single
|
||||
source of truth there.
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
- **Buttons.** Default (`.btn`) is a neutral elevated surface. The *primary* button
|
||||
is **near-white on dark** (`--text` fill, `--bg` text) — the highest-contrast thing
|
||||
on the page that *isn't* the orb. There is no colored button; emphasis comes from
|
||||
contrast, not hue.
|
||||
- **Icon buttons** (`.icon-btn`) are `36px`, elevated surface, dim icon that brightens
|
||||
on hover. Side controls (`.side-btn`) are circular, collapse to zero size until the
|
||||
session is live (by width on desktop, by height in the mobile column).
|
||||
- **Chat bubbles & history messages** share one neutral surface. They are
|
||||
distinguished by **side** (you = left, assistant = right) plus the **mono role
|
||||
label** in the role-echo hue — not by tinted fills. Tool entries use the wrench icon
|
||||
+ mono + amber, on the same neutral surface.
|
||||
- **Badge** (new-message dot) is monochrome white — a signal, not a color accent.
|
||||
- **Focus** is visible and neutral: inputs focus to `--text-dim`; the orb uses a
|
||||
`--glow`-colored outline (it's the orb, so color is allowed).
|
||||
|
||||
---
|
||||
|
||||
## Motion
|
||||
|
||||
- **The orb is audio-reactive, not timer-driven.** Mic RMS (`--audio-level`) and the
|
||||
assistant output level (`--ai-audio-level`) drive scale/opacity at display rate, so
|
||||
every syllable moves it. This is the signature animation — keep new motion
|
||||
subordinate to it.
|
||||
- **Quiet by default.** Breathing/glow throbs are slow (1.4–2.4s) and low-contrast.
|
||||
Resist adding scattered micro-animations; an orchestrated moment beats many small
|
||||
ones, and excess motion reads as AI-generated.
|
||||
- **First paint is frozen.** `body.booting` disables all transitions until the first
|
||||
frame commits (stripped after one rAF in `main.js`). Anything new that would
|
||||
otherwise animate-in on load must respect this.
|
||||
- Honor `prefers-reduced-motion` for any motion you add.
|
||||
|
||||
---
|
||||
|
||||
## Writing / copy voice
|
||||
|
||||
- Sentence case, plain verbs, no filler. Tuned and quiet — match the minimal canvas.
|
||||
- Name things by what the user controls, not by the system's internals. A button says
|
||||
exactly what it does, and keeps the same word through the flow.
|
||||
- **Empty states invite action** ("Tap the orb and start talking"), they don't just
|
||||
set a mood.
|
||||
- **Errors state what happened and how to recover**, in the interface's voice — they
|
||||
don't apologize and are never vague.
|
||||
- Mono labels are terse identifiers (`YOU`, `TOOL CALL`); prose stays in Inter.
|
||||
|
||||
---
|
||||
|
||||
## Responsive floor (non-negotiable)
|
||||
|
||||
Every change ships meeting these:
|
||||
- Works down to a `360px`-wide phone. The `@media (max-width: 600px)` block already
|
||||
handles the phone layout — extend it, don't fight it.
|
||||
- Visible keyboard focus on every interactive element.
|
||||
- `prefers-reduced-motion` respected.
|
||||
- Tap targets ≥ `44px`; `touch-action: manipulation` on anything tappable.
|
||||
|
||||
---
|
||||
|
||||
## Before you ship — the mirror check
|
||||
|
||||
1. Is every saturated color either on the orb or a word-sized role echo? If a fill or
|
||||
border is colored, remove that accessory.
|
||||
2. Is mono used only for machine/system text, and Inter for everything human?
|
||||
3. Does any new state have both a `--glow` and (if it appears in chat) a `--voice-*`
|
||||
alias?
|
||||
4. Are separations hairlines + spacing, not boxes and heavy shadows?
|
||||
5. Did you add motion? Is it quieter than the orb and reduced-motion-safe?
|
||||
6. Remove one accessory. The minimal look survives on precision, not addition.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Single container: serves the static front-end AND the /api/search proxy.
|
||||
# HF Spaces (sdk: docker) routes traffic to $PORT, default 7860.
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
EXPOSE 7860
|
||||
|
||||
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "7860"]
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
title: HF Realtime Voice
|
||||
emoji: 🎙️
|
||||
colorFrom: indigo
|
||||
colorTo: purple
|
||||
sdk: docker
|
||||
app_port: 7860
|
||||
pinned: false
|
||||
short_description: Voice chat over WebSocket against a HF speech-to-speech
|
||||
hf_oauth: true
|
||||
---
|
||||
|
||||
# Realtime Voice Demo (WebSocket transport)
|
||||
|
||||
Browser voice-chat UI for the
|
||||
[huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech)
|
||||
backend. The browser streams mic audio over a WebSocket using the OpenAI
|
||||
Realtime **GA** protocol and plays back the assistant's audio as it arrives.
|
||||
|
||||
## Quick start (local)
|
||||
|
||||
1. **Start the speech-to-speech backend** in realtime mode (from the repo root;
|
||||
see the [backend README](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/README.md)
|
||||
for more model combinations):
|
||||
|
||||
```bash
|
||||
uv run speech-to-speech \
|
||||
--mode realtime \
|
||||
--stt parakeet-tdt \
|
||||
--llm_backend transformers \
|
||||
--tts kokoro \
|
||||
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
|
||||
--llm_device mps \
|
||||
--llm_torch_dtype float16 \
|
||||
--enable_live_transcription
|
||||
```
|
||||
|
||||
The realtime server listens on `ws://localhost:8765/v1/realtime` by default
|
||||
(`--ws_host` / `--ws_port` to change).
|
||||
|
||||
2. **Start this app**, pointing it at the backend with `SPEECH_TO_SPEECH_URL`:
|
||||
|
||||
```bash
|
||||
uv pip install -r demo/requirements.txt
|
||||
export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
|
||||
export SERPER_API_KEY=... # optional; web search is disabled without it
|
||||
uv run uvicorn --app-dir demo server:app --reload --port 7860
|
||||
```
|
||||
|
||||
Or with Docker:
|
||||
|
||||
```bash
|
||||
docker build -t s2s-demo demo/
|
||||
docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
|
||||
```
|
||||
|
||||
3. Open <http://localhost:7860/>, click the orb, allow the mic, talk.
|
||||
|
||||
> Browsers require **HTTPS or `localhost`** for `getUserMedia()` (mic + camera).
|
||||
> `127.0.0.1` and `localhost` both work; plain `http://192.168.x.y` does NOT.
|
||||
|
||||
Smoke-test the backend from the shell:
|
||||
|
||||
```bash
|
||||
websocat ws://localhost:8765/v1/realtime
|
||||
# -> you should get a session.created event back immediately
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
1. The browser opens a WebSocket on the configured `/v1/realtime` URL.
|
||||
2. Server pushes `session.created` on connect. Client replies with
|
||||
`session.update` (OpenAI Realtime **GA** schema: `session.audio.input`,
|
||||
`session.audio.output`, `session.output_modalities`).
|
||||
3. Client streams mic audio as PCM16 16 kHz mono base64 chunks
|
||||
(`input_audio_buffer.append`, one frame every ~40 ms).
|
||||
4. Server pushes `response.output_audio.delta` (PCM16 24 kHz mono base64)
|
||||
and transcript deltas.
|
||||
|
||||
The backend exposes one concurrent session per pipeline unit
|
||||
(`--num_pipelines` to serve more).
|
||||
|
||||
## Connecting to a backend
|
||||
|
||||
Three modes, picked by env (`/api/config` tells the client which one is active):
|
||||
|
||||
- **`SPEECH_TO_SPEECH_URL` env** — the mode you want for local use, and the
|
||||
highest priority. The browser connects **directly** to this realtime
|
||||
WebSocket URL; it's shown read-only in Settings. Setting it disables the
|
||||
load-balancer logic entirely (no `/api/session` proxy, no queue, no
|
||||
metering, no sign-in). Unlike the LB address it is not a secret. Accepts a
|
||||
full `ws(s)://host/v1/realtime` URL or a bare host like `localhost:8765`
|
||||
(the app adds `/v1/realtime`).
|
||||
- **Neither env set** — **Settings → Speech-to-speech server URL**: paste a
|
||||
full connect URL or a bare host, and the browser connects to it directly.
|
||||
- **`LOAD_BALANCER_URL` env** — multi-compute deployments only: the browser
|
||||
POSTs the same-origin `/api/session` proxy, the server forwards to the LB,
|
||||
and the browser dials the per-session compute URL the LB hands back. The LB
|
||||
address never reaches the browser; the Settings URL field is hidden.
|
||||
|
||||
| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Metering |
|
||||
|:---:|:---:|:---:|---|---|---|
|
||||
| ✅ | any | any | direct → pinned URL | visible, locked | off |
|
||||
| – | – | any | direct → user URL | editable | off |
|
||||
| – | ✅ | ✅ | LB proxy | hidden | **on** |
|
||||
| – | ✅ | – | LB proxy | hidden | off |
|
||||
|
||||
**Settings → Restart** reconnects with the current voice, instructions and URL.
|
||||
|
||||
## Tools
|
||||
|
||||
The assistant can call two tools mid-conversation (toggle them from the **Tools**
|
||||
button, top-right):
|
||||
|
||||
- **Web search** — Google results via Serper.dev, proxied server-side so the key
|
||||
never reaches the browser. Set `SERPER_API_KEY` as an env var / Space secret.
|
||||
Without it, the tool is disabled unless the user pastes their own key in the
|
||||
Tools panel.
|
||||
- **Camera** — while enabled, a live self-view shows bottom-left; when the model
|
||||
calls the tool, the current frame is sent to the vision-language model so it can
|
||||
see what you're showing it.
|
||||
|
||||
## Usage limits (deployed Space only)
|
||||
|
||||
Conversation time is metered per UTC day by sign-in tier (see `limiter.py` /
|
||||
`auth.py`), but **only on the deployed Space** — metering turns on only when BOTH
|
||||
`LOAD_BALANCER_URL` and `SPACE_ID` (injected automatically by the HF Space
|
||||
runtime) are present. Running locally — even with `LOAD_BALANCER_URL` exported —
|
||||
leaves the app unmetered. Tunable via env:
|
||||
|
||||
| Env | Default | What |
|
||||
|-----|---------|------|
|
||||
| `LIMIT_ANON_SEC` | `300` | Daily seconds for anonymous visitors (5 min) |
|
||||
| `LIMIT_FREE_SEC` | `600` | Daily seconds for signed-in non-PRO users (10 min) |
|
||||
| `UNLIMITED_ORGS` | _(adds to defaults)_ | Extra HF org names whose members get **unlimited** usage, like PRO |
|
||||
| `USAGE_HASH_SECRET` | _(random)_ | HMAC secret for hashing identity keys + signing the anon cookie |
|
||||
|
||||
PRO members are always unlimited. Members of `cerebras`, `HuggingFaceM4`,
|
||||
`smolagents`, and `pollen-robotics` are unlimited out of the box (shown as
|
||||
"Team", not "PRO"); set `UNLIMITED_ORGS=my-team` to add more. Matched
|
||||
case-insensitively against the user's organisations from HF OAuth.
|
||||
|
||||
## Settings (stored in `localStorage`)
|
||||
|
||||
| Key | What |
|
||||
|-----|------|
|
||||
| Speech-to-speech server URL | Direct realtime WebSocket URL (hidden/locked when pinned by env) |
|
||||
| Voice | Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian) |
|
||||
| Instructions | System prompt sent in `session.update` once the WS opens |
|
||||
|
||||
LocalStorage keys are namespaced `s2s.ws.*` so this app's settings do
|
||||
NOT collide with the WebRTC variant.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `index.html` | Single page, orb + settings modal (identical UI to the WebRTC app) |
|
||||
| `main.js` | State machine, settings, tools, camera, noise-gate UI wiring |
|
||||
| `ui/chat.js` | `ChatView`: history panel, ephemeral bubbles, transcript/tool streaming |
|
||||
| `ui/account.js` | `Account`: HF login chip + popover, daily-limit modal |
|
||||
| `ui/dom.js` | Shared helpers: `$`, `escHtml`, `truncateError`, `DEBUG` |
|
||||
| `auth.py` | HF OAuth + per-request identity (tier, hashed keys) |
|
||||
| `limiter.py` | SQLite per-day talk-time budget (chunked server-clock reservation) |
|
||||
| `ws/s2s-ws-client.js` | WebSocket handshake + OpenAI Realtime GA protocol |
|
||||
| `ws/codec.js` | base64 <-> PCM helpers + transcript extraction (pure) |
|
||||
| `ws/orb-visualizer.js` | `OrbVisualiser`: FFT bands -> orb CSS custom properties |
|
||||
| `worklets/mic-capture.js` | AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks |
|
||||
| `worklets/audio-playback.js` | AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out |
|
||||
| `style.css` | Orb animations, layout, dark theme (verbatim from the WebRTC app) |
|
||||
|
||||
## Audio pipeline notes
|
||||
|
||||
- **Input**: `getUserMedia({ echoCancellation, noiseSuppression, autoGainControl })`
|
||||
feeds the `mic-capture` worklet at the `AudioContext` rate. The worklet
|
||||
resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast
|
||||
path, linear interpolation fallback for odd rates) and packs Int16 LE.
|
||||
- **Output**: `response.output_audio.delta` decodes to Int16 -> Float32
|
||||
and is posted to the `audio-playback` worklet. The worklet maintains a
|
||||
per-context ring buffer, linearly interpolates 24 -> 48, and applies
|
||||
short 32-frame fades on entry/exit to suppress clicks.
|
||||
- **Barge-in**: when the server VAD detects user speech mid-response
|
||||
(`input_audio_buffer.speech_started` while `ai-speaking`), the client
|
||||
posts `{ kind: "clear" }` to the playback worklet to wipe the queue
|
||||
immediately. The server itself cancels the in-flight response.
|
||||
|
||||
## Credits
|
||||
|
||||
- Backend: [huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech)
|
||||
- UI verbatim from `amir-tfrere/minimal-conversation-app-s2s-backend` (Pollen Robotics × Hugging Face)
|
||||
@@ -0,0 +1,229 @@
|
||||
"""
|
||||
HF OAuth + per-request identity for the duration limiter.
|
||||
|
||||
Login uses Hugging Face's native Spaces OAuth via `huggingface_hub`
|
||||
(`attach_huggingface_oauth` / `parse_huggingface_oauth`). The OAuth env
|
||||
(`OAUTH_CLIENT_ID`, ...) is injected by the platform when the Space README sets
|
||||
`hf_oauth: true`, so this only activates on a deployed Space — locally and in
|
||||
direct mode there's no OAuth and the limiter treats everyone as anonymous.
|
||||
|
||||
Identity:
|
||||
- signed in -> tier 'pro' | 'free', keyed by hashed HF `sub`
|
||||
- anonymous -> tier 'anon', keyed by BOTH hashed client IP and a hashed
|
||||
signed-cookie id (OR-matched in the limiter)
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
|
||||
import limiter
|
||||
|
||||
logger = logging.getLogger("s2s.auth")
|
||||
|
||||
# huggingface_hub adds these routes when OAuth is attached. Centralised so a
|
||||
# version change is a one-line fix; the paths are handed to the client via
|
||||
# /api/me rather than hardcoded there.
|
||||
OAUTH_LOGIN_PATH = "/oauth/huggingface/login"
|
||||
OAUTH_LOGOUT_PATH = "/oauth/huggingface/logout"
|
||||
|
||||
ANON_COOKIE = "s2s_anon"
|
||||
_COOKIE_MAX_AGE = 60 * 60 * 24 * 30 # 30 days
|
||||
|
||||
|
||||
# Members of these orgs get unlimited usage (like PRO) out of the box. The
|
||||
# UNLIMITED_ORGS env adds to this set; it doesn't replace it.
|
||||
_DEFAULT_UNLIMITED_ORGS = {"cerebras", "huggingfacem4", "smolagents", "pollen-robotics"}
|
||||
|
||||
|
||||
def _unlimited_orgs() -> "set[str]":
|
||||
"""Org usernames whose members get unlimited usage (like PRO).
|
||||
|
||||
Defaults to {cerebras, HuggingFaceM4, smolagents}; the UNLIMITED_ORGS env
|
||||
(comma/space-separated, e.g. `UNLIMITED_ORGS=my-team`) adds more. Matched
|
||||
case-insensitively against the signed-in user's organisations."""
|
||||
raw = os.environ.get("UNLIMITED_ORGS", "")
|
||||
extra = {o.strip().lower() for o in raw.replace(",", " ").split() if o.strip()}
|
||||
return _DEFAULT_UNLIMITED_ORGS | extra
|
||||
|
||||
try:
|
||||
from huggingface_hub import attach_huggingface_oauth, parse_huggingface_oauth
|
||||
_OAUTH_IMPORTABLE = True
|
||||
except Exception as exc: # pragma: no cover - import guard
|
||||
logger.info("huggingface_hub OAuth unavailable (%s); sign-in disabled.", exc)
|
||||
_OAUTH_IMPORTABLE = False
|
||||
|
||||
# Set by attach(): True once OAuth is actually wired (importable + env present).
|
||||
oauth_enabled = False
|
||||
|
||||
|
||||
def attach(app) -> bool:
|
||||
"""Wire HF OAuth onto the app if it's importable and configured. Returns
|
||||
whether sign-in is available."""
|
||||
global oauth_enabled
|
||||
if not _OAUTH_IMPORTABLE or not os.environ.get("OAUTH_CLIENT_ID"):
|
||||
return False
|
||||
try:
|
||||
attach_huggingface_oauth(app)
|
||||
oauth_enabled = True
|
||||
logger.info("HF OAuth attached (sign-in enabled).")
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.warning("Failed to attach HF OAuth: %r", exc)
|
||||
oauth_enabled = False
|
||||
return oauth_enabled
|
||||
|
||||
|
||||
def _field(obj, name, default=None):
|
||||
"""Read a field whether the user-info is an object or a dict."""
|
||||
if obj is None:
|
||||
return default
|
||||
if isinstance(obj, dict):
|
||||
return obj.get(name, default)
|
||||
return getattr(obj, name, default)
|
||||
|
||||
|
||||
# Surface what we detect (orgs, tier) in logs and on /api/me when set. Handy for
|
||||
# verifying org gating on the live Space without guessing.
|
||||
AUTH_DEBUG = bool(os.environ.get("AUTH_DEBUG"))
|
||||
|
||||
# whoami-v2 org lookups are cached for the process lifetime, keyed by token, so
|
||||
# /api/me + /api/session don't each hit the Hub.
|
||||
_orgs_cache: "dict[str, set[str]]" = {}
|
||||
|
||||
|
||||
def current_oauth(request):
|
||||
"""The parsed HF OAuth info (user_info + access_token), or None."""
|
||||
if not oauth_enabled:
|
||||
return None
|
||||
try:
|
||||
return parse_huggingface_oauth(request)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def current_user(request):
|
||||
"""The signed-in HF user-info, or None."""
|
||||
return _field(current_oauth(request), "user_info")
|
||||
|
||||
|
||||
def _user_org_names(user) -> "set[str]":
|
||||
"""The user's organisations from the OAuth userinfo, by username/name/id."""
|
||||
names = set()
|
||||
for org in _field(user, "orgs", []) or []:
|
||||
for key in ("preferred_username", "name", "sub"):
|
||||
val = _field(org, key)
|
||||
if val:
|
||||
names.add(str(val).lower())
|
||||
return names
|
||||
|
||||
|
||||
def _orgs_via_token(token: str) -> "set[str]":
|
||||
"""Fallback org lookup via the Hub `whoami-v2` API, using the user's OAuth
|
||||
access token. Covers the case where the userinfo claim omits `orgs`."""
|
||||
if not token:
|
||||
return set()
|
||||
if token in _orgs_cache:
|
||||
return _orgs_cache[token]
|
||||
names: "set[str]" = set()
|
||||
try:
|
||||
import httpx
|
||||
|
||||
resp = httpx.get(
|
||||
"https://huggingface.co/api/whoami-v2",
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
timeout=5.0,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
for org in resp.json().get("orgs", []) or []:
|
||||
for key in ("name", "fullname"):
|
||||
val = org.get(key)
|
||||
if val:
|
||||
names.add(str(val).lower())
|
||||
except Exception as exc: # pragma: no cover - network/permission dependent
|
||||
logger.info("whoami-v2 org lookup failed: %r", exc)
|
||||
_orgs_cache[token] = names
|
||||
return names
|
||||
|
||||
|
||||
def _org_names(user, token=None, allow=None) -> "set[str]":
|
||||
"""The user's org usernames from the OAuth userinfo claim. If that doesn't
|
||||
already satisfy `allow`, fall back to the Hub `whoami-v2` API (the claim is
|
||||
often empty or partial), so membership is resolved either way."""
|
||||
names = _user_org_names(user)
|
||||
if token and (allow is None or not (allow & names)):
|
||||
names = names | _orgs_via_token(token)
|
||||
return names
|
||||
|
||||
|
||||
def resolve_tier(user, token=None) -> str:
|
||||
"""Tier for a signed-in user: 'pro' (paying), 'org' (allow-listed org
|
||||
member, unlimited), or 'free'. PRO wins over org if both apply."""
|
||||
if bool(_field(user, "is_pro", False)):
|
||||
return "pro"
|
||||
allow = _unlimited_orgs()
|
||||
names = _org_names(user, token, allow)
|
||||
tier = "org" if (allow & names) else "free"
|
||||
if AUTH_DEBUG:
|
||||
logger.info("tier=%s orgs=%s allow=%s", tier, sorted(names), sorted(allow))
|
||||
return tier
|
||||
|
||||
|
||||
def user_view(request) -> dict:
|
||||
"""Public profile for /api/me."""
|
||||
info = current_oauth(request)
|
||||
user = _field(info, "user_info")
|
||||
if not user:
|
||||
return {"loggedIn": False, "tier": "anon"}
|
||||
token = _field(info, "access_token")
|
||||
out = {
|
||||
"loggedIn": True,
|
||||
"username": _field(user, "preferred_username") or _field(user, "name") or "you",
|
||||
"avatar": _field(user, "picture"),
|
||||
"tier": resolve_tier(user, token),
|
||||
}
|
||||
if AUTH_DEBUG:
|
||||
out["orgs"] = sorted(_org_names(user, token))
|
||||
return out
|
||||
|
||||
|
||||
def _client_ip(request) -> str:
|
||||
"""Real client IP. On HF the app sits behind a proxy, so the user's address
|
||||
is the first hop in X-Forwarded-For, not request.client.host."""
|
||||
xff = request.headers.get("x-forwarded-for", "")
|
||||
if xff:
|
||||
return xff.split(",")[0].strip()
|
||||
return request.client.host if request.client else "unknown"
|
||||
|
||||
|
||||
def resolve_identity(request):
|
||||
"""Resolve (tier, keys, set_cookie) for this request.
|
||||
|
||||
`keys` are the limiter usage_daily keys to debit (one for signed-in, two for
|
||||
anonymous). `set_cookie` is a signed value to Set-Cookie when we minted a new
|
||||
anonymous id, else None.
|
||||
"""
|
||||
info = current_oauth(request)
|
||||
user = _field(info, "user_info")
|
||||
if user:
|
||||
sub = _field(user, "sub") or _field(user, "preferred_username")
|
||||
token = _field(info, "access_token")
|
||||
return resolve_tier(user, token), [limiter.hash_key(f"sub:{sub}")], None
|
||||
|
||||
# Anonymous: key by IP and a signed cookie id, minting the cookie if absent.
|
||||
ip = _client_ip(request)
|
||||
cookie_id = limiter.verify_cookie(request.cookies.get(ANON_COOKIE, ""))
|
||||
set_cookie = None
|
||||
if not cookie_id:
|
||||
cookie_id = secrets.token_urlsafe(18)
|
||||
set_cookie = limiter.sign_cookie(cookie_id)
|
||||
keys = [limiter.hash_key(f"ip:{ip}"), limiter.hash_key(f"cookie:{cookie_id}")]
|
||||
return "anon", keys, set_cookie
|
||||
|
||||
|
||||
def set_anon_cookie(response, signed: str) -> None:
|
||||
# The Space runs inside an iframe on huggingface.co, so the cookie lives in a
|
||||
# cross-site context — it must be SameSite=None; Secure or the browser drops it.
|
||||
response.set_cookie(
|
||||
ANON_COOKIE, signed,
|
||||
max_age=_COOKIE_MAX_AGE, httponly=True, samesite="none", secure=True,
|
||||
)
|
||||
@@ -0,0 +1,29 @@
|
||||
# Convert the static Space into a Docker app with a search proxy
|
||||
|
||||
To give the model a web search tool, the executor (which runs in the browser of a
|
||||
public Space) needs a search key. A `sdk: static` Space serves files as-is with no
|
||||
runtime process, so it cannot hold a secret the browser uses without exposing it.
|
||||
We convert the Space from `sdk: static` to `sdk: docker`: a single container runs a
|
||||
small server (FastAPI + uvicorn) that both serves the existing front-end *unchanged*
|
||||
and exposes a same-origin `/search` proxy holding `SERPER_API_KEY` server-side. The
|
||||
whole app lives in that one container; the s2s speech-to-speech backend stays the
|
||||
separate load-balanced service it already is.
|
||||
|
||||
## Considered options
|
||||
|
||||
- **Stay static, user-supplied key only** — no owner key; search works only if each
|
||||
user pastes their own. Rejected as the default because it leaves the deployed demo
|
||||
with no working search.
|
||||
- **Separate proxy service** — same secrecy, but splits the app across two deploys.
|
||||
Rejected: the app must live in one container.
|
||||
- **Key baked into client JS at build time** — would be readable in the served
|
||||
bundle. Rejected: defeats the point.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Deployment is no longer static: there is a Dockerfile and a server process; the
|
||||
README front-matter changes from `sdk: static` to `sdk: docker`.
|
||||
- The client calls `/search` same-origin; the server reads the key from env. A user
|
||||
may still supply their own key as a fallback, sent per-request to the proxy.
|
||||
- The front-end, audio pipeline, and s2s handshake are untouched — only the hosting
|
||||
shape and the new route are added.
|
||||
@@ -0,0 +1,373 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
|
||||
<title>Minimal Conversation · S2S backend (WebSocket)</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link
|
||||
href="https://fonts.googleapis.com/css2?family=Geist+Mono:wght@400;500;600&family=Inter:wght@400;500;600;700;800&display=swap"
|
||||
rel="stylesheet"
|
||||
/>
|
||||
<link rel="stylesheet" href="style.css" />
|
||||
</head>
|
||||
<!--
|
||||
`booting` disables transitions/animations until the first paint commits.
|
||||
Without this the orb visibly fades/scales in on the very first frame
|
||||
because `state-idle` and the generic `.ind` defaults differ. The
|
||||
bootstrap script at the end of main.js strips the class after one rAF.
|
||||
-->
|
||||
<body class="booting">
|
||||
<div id="app">
|
||||
<header class="topbar">
|
||||
<div class="brand">
|
||||
<div class="ident">
|
||||
<div class="ident-head">
|
||||
<a class="ident-title" href="https://github.com/huggingface/speech-to-speech" target="_blank" rel="noopener">Hugging Face Realtime</a>
|
||||
<button id="about-btn" class="about-btn" title="About this space" aria-label="About this space">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" y1="16" x2="12" y2="12"/><line x1="12" y1="8" x2="12.01" y2="8"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
<p class="ident-blurb">An open, real-time voice chat built on Hugging Face's speech-to-speech stack.</p>
|
||||
<div class="ident-meta">
|
||||
<span class="ident-row">
|
||||
<span class="ident-label">Powered by</span>
|
||||
<a href="https://huggingface.co/inference-endpoints" target="_blank" rel="noopener">Inference Endpoints</a>
|
||||
<span class="sep" aria-hidden="true">·</span>
|
||||
<a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a>
|
||||
</span>
|
||||
<span class="ident-row">
|
||||
<span class="ident-label">Built by</span>
|
||||
<a class="hf-credit" href="https://huggingface.co" target="_blank" rel="noopener"><svg class="hf-mark" viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M12.025 1.13c-5.77 0-10.449 4.647-10.449 10.378 0 1.112.178 2.181.503 3.185.064-.222.203-.444.416-.577a.96.96 0 0 1 .524-.15c.293 0 .584.124.84.284.278.173.48.408.71.694.226.282.458.611.684.951v-.014c.017-.324.106-.622.264-.874s.403-.487.762-.543c.3-.047.596.06.787.203s.31.313.4.467c.15.257.212.468.233.542.01.026.653 1.552 1.657 2.54.616.605 1.01 1.223 1.082 1.912.055.537-.096 1.059-.38 1.572.637.121 1.294.187 1.967.187.657 0 1.298-.063 1.921-.178-.287-.517-.44-1.041-.384-1.581.07-.69.465-1.307 1.081-1.913 1.004-.987 1.647-2.513 1.657-2.539.021-.074.083-.285.233-.542.09-.154.208-.323.4-.467a1.08 1.08 0 0 1 .787-.203c.359.056.604.29.762.543s.247.55.265.874v.015c.225-.34.457-.67.683-.952.23-.286.432-.52.71-.694.257-.16.547-.284.84-.285a.97.97 0 0 1 .524.151c.228.143.373.388.43.625l.006.04a10.3 10.3 0 0 0 .534-3.273c0-5.731-4.678-10.378-10.449-10.378M8.327 6.583a1.5 1.5 0 0 1 .713.174 1.487 1.487 0 0 1 .617 2.013c-.183.343-.762-.214-1.102-.094-.38.134-.532.914-.917.71a1.487 1.487 0 0 1 .69-2.803m7.486 0a1.487 1.487 0 0 1 .689 2.803c-.385.204-.536-.576-.916-.71-.34-.12-.92.437-1.103.094a1.487 1.487 0 0 1 .617-2.013 1.5 1.5 0 0 1 .713-.174m-10.68 1.55a.96.96 0 1 1 0 1.921.96.96 0 0 1 0-1.92m13.838 0a.96.96 0 1 1 0 1.92.96.96 0 0 1 0-1.92M8.489 11.458c.588.01 1.965 1.157 3.572 1.164 1.607-.007 2.984-1.155 3.572-1.164.196-.003.305.12.305.454 0 .886-.424 2.328-1.563 3.202-.22-.756-1.396-1.366-1.63-1.32q-.011.001-.02.006l-.044.026-.01.008-.03.024q-.018.017-.035.036l-.032.04a1 1 0 0 0-.058.09l-.014.025q-.049.088-.11.19a1 1 0 0 1-.083.116 1.2 1.2 0 0 1-.173.18q-.035.029-.075.058a1.3 1.3 0 0 1-.251-.243 1 1 0 0 1-.076-.107c-.124-.193-.177-.363-.337-.444-.034-.016-.104-.008-.2.022q-.094.03-.216.087-.06.028-.125.063l-.13.074q-.067.04-.136.086a3 3 0 0 0-.135.096 3 3 0 0 0-.26.219 2 2 0 0 0-.12.121 2 2 0 0 0-.106.128l-.002.002a2 2 0 0 0-.09.132l-.001.001a1.2 1.2 0 0 0-.105.212q-.013.036-.024.073c-1.139-.875-1.563-2.317-1.563-3.203 0-.334.109-.457.305-.454m.836 10.354c.824-1.19.766-2.082-.365-3.194-1.13-1.112-1.789-2.738-1.789-2.738s-.246-.945-.806-.858-.97 1.499.202 2.362c1.173.864-.233 1.45-.685.64-.45-.812-1.683-2.896-2.322-3.295s-1.089-.175-.938.647 2.822 2.813 2.562 3.244-1.176-.506-1.176-.506-2.866-2.567-3.49-1.898.473 1.23 2.037 2.16c1.564.932 1.686 1.178 1.464 1.53s-3.675-2.511-4-1.297c-.323 1.214 3.524 1.567 3.287 2.405-.238.839-2.71-1.587-3.216-.642-.506.946 3.49 2.056 3.522 2.064 1.29.33 4.568 1.028 5.713-.624m5.349 0c-.824-1.19-.766-2.082.365-3.194 1.13-1.112 1.789-2.738 1.789-2.738s.246-.945.806-.858.97 1.499-.202 2.362c-1.173.864.233 1.45.685.64.451-.812 1.683-2.896 2.322-3.295s1.089-.175.938.647-2.822 2.813-2.562 3.244 1.176-.506 1.176-.506 2.866-2.567 3.49-1.898-.473 1.23-2.037 2.16c-1.564.932-1.686 1.178-1.464 1.53s3.675-2.511 4-1.297c.323 1.214-3.524 1.567-3.287 2.405.238.839 2.71-1.587 3.216-.642.506.946-3.49 2.056-3.522 2.064-1.29.33-4.568 1.028-5.713-.624"/></svg>Hugging Face</a>
|
||||
<span class="sep" aria-hidden="true">·</span>
|
||||
<a class="handle" href="https://huggingface.co/tfrere" target="_blank" rel="noopener">tfrere</a>
|
||||
<span class="sep" aria-hidden="true">·</span>
|
||||
<a class="handle" href="https://huggingface.co/A-Mahla" target="_blank" rel="noopener">A-Mahla</a>
|
||||
<span class="sep" aria-hidden="true">·</span>
|
||||
<a class="handle" href="https://huggingface.co/andito" target="_blank" rel="noopener">andito</a>
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="topbar-right">
|
||||
<!-- HF login chip / sign-in pill. Populated by ui/account.js; only
|
||||
shown when the deploy runs behind a load balancer with OAuth. -->
|
||||
<div id="account" class="account" hidden></div>
|
||||
<button id="about-btn-m" class="icon-btn about-btn-mobile" title="About this space" aria-label="About this space">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" y1="16" x2="12" y2="12"/><line x1="12" y1="8" x2="12.01" y2="8"/></svg>
|
||||
</button>
|
||||
<button id="tools-btn" class="icon-btn" title="Tools" aria-label="Tools">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/></svg>
|
||||
</button>
|
||||
<button id="chat-btn" class="icon-btn" title="Conversation history" aria-label="Conversation history">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15a2 2 0 0 1-2 2H7l-4 4V5a2 2 0 0 1 2-2h14a2 2 0 0 1 2 2z"/></svg>
|
||||
<span id="chat-badge" class="chat-badge" aria-hidden="true"></span>
|
||||
</button>
|
||||
<button id="settings-btn" class="icon-btn" title="Settings" aria-label="Settings">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 1 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 1 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 1 1-2.83-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 1 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 1 1 2.83-2.83l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 1 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 1 1 2.83 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9c0 .66.39 1.25 1 1.51H21a2 2 0 1 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main class="stage">
|
||||
<div class="orb-wrap">
|
||||
<div id="mic-gate" class="mic-gate">
|
||||
<svg id="mic-gate-arc" class="mic-gate-arc" viewBox="0 0 100 100" aria-hidden="true">
|
||||
<path id="mga-track" class="mga-track" fill="none" />
|
||||
<path id="mga-fill" class="mga-fill" fill="none" />
|
||||
<path id="mga-hit" class="mga-hit" fill="none" />
|
||||
<circle id="mga-handle" class="mga-handle" r="3" />
|
||||
</svg>
|
||||
<button id="mic-btn" class="side-btn" type="button" aria-label="Mute" title="Mute" aria-hidden="true">
|
||||
<svg class="mic-on" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="2" width="6" height="12" rx="3"/><path d="M5 10a7 7 0 0 0 14 0"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
|
||||
<svg class="mic-off" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><line x1="2" y1="2" x2="22" y2="22"/><path d="M9 5a3 3 0 0 1 6 0v4"/><path d="M9 10v1a3 3 0 0 0 5.1 2.1"/><path d="M19 10a7 7 0 0 1-1.24 3.97"/><path d="M5 10a7 7 0 0 0 11 5.67"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<button
|
||||
id="main-circle"
|
||||
class="circle state-idle"
|
||||
type="button"
|
||||
aria-label="Start voice conversation"
|
||||
>
|
||||
<span class="circle-glow" aria-hidden="true"></span>
|
||||
<span class="circle-ring" aria-hidden="true"></span>
|
||||
<span class="circle-ring-outer" aria-hidden="true"></span>
|
||||
<span class="circle-core">
|
||||
<span class="circle-indicator" aria-hidden="true">
|
||||
<svg class="ind ind-mic" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
|
||||
<rect x="9" y="2" width="6" height="12" rx="3" fill="currentColor" stroke="none"/>
|
||||
<path d="M5 10a7 7 0 0 0 14 0"/>
|
||||
<line x1="12" y1="19" x2="12" y2="22"/>
|
||||
<line x1="8" y1="22" x2="16" y2="22"/>
|
||||
</svg>
|
||||
<svg class="ind ind-error" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="9"/><line x1="12" y1="8" x2="12" y2="13"/><line x1="12" y1="16" x2="12" y2="16"/></svg>
|
||||
<span class="ind ind-spinner"></span>
|
||||
<span class="ind ind-thinking">
|
||||
<span class="dot"></span>
|
||||
<span class="dot"></span>
|
||||
<span class="dot"></span>
|
||||
</span>
|
||||
<span class="ind ind-bars">
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
</span>
|
||||
<svg class="ind ind-voice" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
|
||||
<path d="M3 10v4a1 1 0 0 0 1 1h3l5 4V5L7 9H4a1 1 0 0 0-1 1z" fill="currentColor" stroke="none"/>
|
||||
<path class="wave wave-1" d="M16 8a5 5 0 0 1 0 8"/>
|
||||
<path class="wave wave-2" d="M19 5a9 9 0 0 1 0 14"/>
|
||||
</svg>
|
||||
</span>
|
||||
</span>
|
||||
</button>
|
||||
|
||||
<button id="stop-btn" class="side-btn" type="button" aria-label="End" title="End" aria-hidden="true">
|
||||
<svg viewBox="0 0 24 24" fill="currentColor"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<p id="circle-caption" class="circle-caption" role="status">Tap to start</p>
|
||||
<p id="circle-subcaption" class="circle-subcaption" hidden></p>
|
||||
|
||||
<div id="queue-actions" class="queue-actions" hidden>
|
||||
<button id="join-queue-btn" class="join-queue-btn" type="button" hidden>
|
||||
Join now
|
||||
</button>
|
||||
<button id="leave-queue-btn" class="leave-queue-btn" type="button" hidden>
|
||||
Leave queue
|
||||
</button>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="footer">
|
||||
<span>
|
||||
Powered by
|
||||
<a href="https://github.com/huggingface/speech-to-speech" target="_blank" rel="noopener">huggingface/speech-to-speech</a>
|
||||
</span>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
<!-- Ephemeral chat bubbles anchored to the top-right -->
|
||||
<div id="bubble-stack" class="bubble-stack" aria-live="polite" aria-atomic="false"></div>
|
||||
|
||||
<!-- Webcam self-view, shown bottom-left while the camera tool is enabled -->
|
||||
<div id="cam-pip" class="cam-pip" aria-hidden="true">
|
||||
<video id="cam-video" class="cam-video" autoplay playsinline muted></video>
|
||||
<span class="cam-flash" aria-hidden="true"></span>
|
||||
<span class="cam-label">camera</span>
|
||||
</div>
|
||||
|
||||
<!-- Conversation history panel (overlay) -->
|
||||
<div id="chat-panel" class="chat-panel">
|
||||
<div id="chat-panel-backdrop" class="chat-panel-backdrop"></div>
|
||||
<div class="chat-panel-inner">
|
||||
<header class="chat-panel-header">
|
||||
<h3>Conversation</h3>
|
||||
<button id="chat-panel-close" class="icon-btn" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
<div id="chat-history" class="chat-history"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<dialog id="about-modal" class="modal about-modal">
|
||||
<div class="modal-content">
|
||||
<header class="modal-header">
|
||||
<h2>About</h2>
|
||||
<button id="about-close" class="icon-btn" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
|
||||
<!-- General intro to the speech-to-speech project -->
|
||||
<div class="about-intro">
|
||||
<p>Speech-to-speech is Hugging Face's open framework for real-time voice agents. Rather than one end-to-end model, it chains four open models from the Hub (speech detection, transcription, a vision-language model, and synthesis), so any stage can be swapped or run locally. This demo wires that pipeline to hosted inference.</p>
|
||||
<a class="about-repo" href="https://github.com/huggingface/speech-to-speech" target="_blank" rel="noopener">View the project on GitHub<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</div>
|
||||
|
||||
<!-- The backend, shown as the path a turn actually travels -->
|
||||
<div class="about-pipeline">
|
||||
<p class="pipeline-title">The pipeline</p>
|
||||
<ol class="pipeline">
|
||||
<li class="pipe-endpoint">You speak</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">VAD</span>
|
||||
<span class="pipe-job">detects speech</span>
|
||||
<a class="pipe-model" href="https://github.com/snakers4/silero-vad" target="_blank" rel="noopener">silero-vad<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">STT</span>
|
||||
<span class="pipe-job">transcribes it</span>
|
||||
<a class="pipe-model" href="https://huggingface.co/nvidia/parakeet-tdt-1.1b" target="_blank" rel="noopener">nvidia/parakeet-tdt-1.1b<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">VLM</span>
|
||||
<span class="pipe-job">analyses and composes the reply <span class="pipe-note">· via <a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a></span></span>
|
||||
<a class="pipe-model" href="https://huggingface.co/google/gemma-4-31B-it" target="_blank" rel="noopener">google/gemma-4-31B-it<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">TTS</span>
|
||||
<span class="pipe-job">speaks it back</span>
|
||||
<a class="pipe-model" href="https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" target="_blank" rel="noopener">Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-endpoint">The orb replies</li>
|
||||
</ol>
|
||||
</div>
|
||||
</div>
|
||||
</dialog>
|
||||
|
||||
<dialog id="settings-modal" class="modal">
|
||||
<form method="dialog" class="modal-content">
|
||||
<header class="modal-header">
|
||||
<h2>Settings</h2>
|
||||
<button class="icon-btn" value="close" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
|
||||
<div class="tab-panels">
|
||||
<section class="tab-panel active" role="tabpanel">
|
||||
<label class="field" id="conn-field">
|
||||
<span id="conn-label">Speech-to-speech server URL</span>
|
||||
<input id="lb-url" type="text" autocomplete="off" spellcheck="false" placeholder="http://localhost:port" />
|
||||
<small id="conn-hint">
|
||||
URL of your speech-to-speech server, e.g.
|
||||
<code>http://localhost:8080</code> (the app adds <code>/v1/realtime</code>).
|
||||
</small>
|
||||
</label>
|
||||
|
||||
<div class="field-row">
|
||||
<label class="field">
|
||||
<span>Voice</span>
|
||||
<select id="voice">
|
||||
<option value="Aiden" selected>Aiden</option>
|
||||
<option value="Ryan">Ryan</option>
|
||||
<option value="Dylan">Dylan</option>
|
||||
<option value="Eric">Eric</option>
|
||||
<option value="Ono_Anna">Ono_Anna</option>
|
||||
<option value="Serena">Serena</option>
|
||||
<option value="Sohee">Sohee</option>
|
||||
<option value="Uncle_Fu">Uncle_Fu</option>
|
||||
<option value="Vivian">Vivian</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<span class="field-head">
|
||||
Noise gate
|
||||
<span id="gate-value" class="field-value">Off</span>
|
||||
</span>
|
||||
<div class="gate">
|
||||
<div class="gate-track" aria-hidden="true">
|
||||
<div id="gate-meter-fill" class="gate-meter-fill"></div>
|
||||
<input
|
||||
id="noise-gate"
|
||||
type="range"
|
||||
min="-66"
|
||||
max="-3"
|
||||
step="1"
|
||||
value="-50"
|
||||
aria-label="Noise gate threshold"
|
||||
/>
|
||||
</div>
|
||||
<div class="gate-ends">
|
||||
<span>Off</span>
|
||||
<span>−3 dB</span>
|
||||
</div>
|
||||
</div>
|
||||
<small>
|
||||
Mutes the mic below the handle so room noise isn't sent. Slide fully
|
||||
left to turn it off; the bar shows your live input while in a call.
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<label class="field">
|
||||
<span>Instructions</span>
|
||||
<textarea id="instructions" rows="5" placeholder="You are a friendly voice assistant..."></textarea>
|
||||
</label>
|
||||
|
||||
<div class="field">
|
||||
<button id="restart-conversation" type="button" class="btn primary wide" disabled>
|
||||
Restart conversation with these settings
|
||||
</button>
|
||||
<small id="restart-hint">Connect first, then come back here to apply live changes.</small>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<footer class="modal-footer">
|
||||
<button id="settings-save" type="submit" class="btn primary" value="save">Save</button>
|
||||
</footer>
|
||||
</form>
|
||||
</dialog>
|
||||
|
||||
<dialog id="tools-modal" class="modal">
|
||||
<div class="modal-content">
|
||||
<header class="modal-header">
|
||||
<h2>Tools</h2>
|
||||
<button id="tools-close" class="icon-btn" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
|
||||
<p class="tools-intro">Let the assistant act during the conversation. Changes apply live.</p>
|
||||
|
||||
<div class="tool-list">
|
||||
<div class="tool-row" id="tool-web-row">
|
||||
<div class="tool-info">
|
||||
<span class="tool-name">Web search</span>
|
||||
<span class="tool-desc">Look things up on Google, via Serper.</span>
|
||||
</div>
|
||||
<label class="switch">
|
||||
<input id="tool-web" type="checkbox" />
|
||||
<span class="switch-track" aria-hidden="true"></span>
|
||||
</label>
|
||||
</div>
|
||||
<label class="field tools-key">
|
||||
<span>Search API key</span>
|
||||
<input id="search-key" type="password" autocomplete="off" spellcheck="false" />
|
||||
<small id="tool-web-hint"></small>
|
||||
</label>
|
||||
|
||||
<div class="tool-row tool-row-sep" id="tool-cam-row">
|
||||
<div class="tool-info">
|
||||
<span class="tool-name">Camera</span>
|
||||
<span class="tool-desc">Let the assistant see through your webcam.</span>
|
||||
</div>
|
||||
<label class="switch">
|
||||
<input id="tool-cam" type="checkbox" />
|
||||
<span class="switch-track" aria-hidden="true"></span>
|
||||
</label>
|
||||
</div>
|
||||
<small id="tool-cam-hint" class="tool-hint">A live preview shows bottom-left while the camera is on.</small>
|
||||
</div>
|
||||
</div>
|
||||
</dialog>
|
||||
|
||||
<!-- Shown when the per-day conversation budget is spent (mid-call or at
|
||||
start). Title / message / CTA / note are filled per tier by
|
||||
ui/account.js. The hero is the HF mark — a friendly smiling face. -->
|
||||
<dialog id="limit-modal" class="modal limit-modal">
|
||||
<div class="modal-content limit-card">
|
||||
<button id="limit-close" class="icon-btn limit-close" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
<div class="limit-badge" aria-hidden="true">
|
||||
<svg class="hf-logo" viewBox="0 0 95 88" fill="none" aria-hidden="true"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg>
|
||||
</div>
|
||||
<h2 id="limit-title" class="limit-title">That's a wrap for now</h2>
|
||||
<p id="limit-msg" class="limit-msg"></p>
|
||||
<a id="limit-cta" class="btn primary wide limit-cta" href="#"></a>
|
||||
<p id="limit-note" class="limit-note"></p>
|
||||
</div>
|
||||
</dialog>
|
||||
|
||||
<script type="module" src="main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,276 @@
|
||||
"""
|
||||
Per-day talk-time budget for the speech-to-speech demo.
|
||||
|
||||
Our server isn't in the audio path (the browser dials the compute WebSocket
|
||||
directly), so it can't cut a live stream. What it *can* do is meter time with a
|
||||
server-clock, chunked reservation:
|
||||
|
||||
- At grant we reserve the first chunk (CHUNK_SEC) and debit it from the day's
|
||||
budget. A parallel grant therefore sees the budget already spent.
|
||||
- The client heartbeats; each heartbeat extends the reservation one chunk at a
|
||||
time until the daily budget runs out, then we report `expired` so the client
|
||||
tears down.
|
||||
- On a clean end (sendBeacon) we reconcile to the real elapsed time and refund
|
||||
the unused chunk. A crash (no end, no heartbeats) is reaped by a sweep and
|
||||
forfeits at most one chunk.
|
||||
|
||||
All time is the server's clock. Budgets are per UTC day; a new day is simply a
|
||||
new row (no explicit reset). Logged-in users are keyed by a hashed HF `sub`;
|
||||
anonymous users by BOTH a hashed IP and a hashed signed-cookie id, OR-matched
|
||||
(spent = max of the two) so clearing one identifier doesn't reset the budget.
|
||||
|
||||
Storage is SQLite at $USAGE_DB_PATH, else /data (persistent Spaces storage),
|
||||
else a /tmp fallback. On /tmp the budget is only per-uptime — flagged in logs.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger("s2s.limiter")
|
||||
|
||||
# ── Tunables (env-overridable) ───────────────────────────────────────────────
|
||||
ANON_SEC = int(os.environ.get("LIMIT_ANON_SEC", "300")) # 5 min/day, not signed in
|
||||
FREE_SEC = int(os.environ.get("LIMIT_FREE_SEC", "600")) # 10 min/day, signed in, no PRO
|
||||
CHUNK_SEC = int(os.environ.get("RESERVE_CHUNK_SEC", "10")) # reservation granularity
|
||||
HEARTBEAT_SEC = int(os.environ.get("HEARTBEAT_SEC", "5")) # advertised client cadence
|
||||
REAP_AFTER_SEC = int(os.environ.get("SESSION_REAP_SEC", "15")) # silence before sweep
|
||||
|
||||
# Stable across restarts or the hashed keys (and signed cookies) rotate and the
|
||||
# budget effectively resets. Set it as a Space secret. Falls back to a per-boot
|
||||
# random value (keys then only hold within one uptime).
|
||||
_HASH_SECRET = (os.environ.get("USAGE_HASH_SECRET", "").strip() or os.urandom(32).hex()).encode()
|
||||
|
||||
_lock = threading.Lock()
|
||||
_db_path: "Path | None" = None
|
||||
|
||||
|
||||
def budget_for(tier: str) -> "int | None":
|
||||
"""Daily second-budget for a tier, or None for unlimited.
|
||||
|
||||
Unlimited tiers: 'pro' (paying PRO members) and 'org' (members of an
|
||||
allow-listed organisation, see UNLIMITED_ORGS in auth.py)."""
|
||||
if tier in ("pro", "org"):
|
||||
return None
|
||||
if tier == "free":
|
||||
return FREE_SEC
|
||||
return ANON_SEC
|
||||
|
||||
|
||||
def hash_key(raw: str) -> str:
|
||||
"""HMAC a raw identifier (sub / ip / cookie id) into an opaque storage key."""
|
||||
digest = hmac.new(_HASH_SECRET, raw.encode("utf-8"), hashlib.sha256).hexdigest()
|
||||
return f"k_{digest}"
|
||||
|
||||
|
||||
def sign_cookie(value: str) -> str:
|
||||
"""`<id>.<sig>` so a forged anon-cookie id is rejected on read."""
|
||||
sig = hmac.new(_HASH_SECRET, value.encode("utf-8"), hashlib.sha256).hexdigest()[:32]
|
||||
return f"{value}.{sig}"
|
||||
|
||||
|
||||
def verify_cookie(signed: str) -> "str | None":
|
||||
"""Return the id if the signature checks out, else None."""
|
||||
if not signed or "." not in signed:
|
||||
return None
|
||||
value, _, sig = signed.rpartition(".")
|
||||
want = hmac.new(_HASH_SECRET, value.encode("utf-8"), hashlib.sha256).hexdigest()[:32]
|
||||
return value if hmac.compare_digest(sig, want) else None
|
||||
|
||||
|
||||
def _today() -> str:
|
||||
return datetime.now(timezone.utc).date().isoformat()
|
||||
|
||||
|
||||
def _resolve_db_path() -> Path:
|
||||
explicit = os.environ.get("USAGE_DB_PATH", "").strip()
|
||||
if explicit:
|
||||
return Path(explicit)
|
||||
data = Path("/data")
|
||||
if data.is_dir() and os.access(data, os.W_OK):
|
||||
return data / "s2s-usage.sqlite3"
|
||||
logger.warning("No persistent /data — usage budget falls back to /tmp (per-uptime only).")
|
||||
return Path(tempfile.gettempdir()) / "s2s-usage.sqlite3"
|
||||
|
||||
|
||||
def _connect() -> sqlite3.Connection:
|
||||
con = sqlite3.connect(_db_path, timeout=5.0)
|
||||
con.execute("PRAGMA journal_mode=WAL")
|
||||
con.execute("PRAGMA busy_timeout=5000")
|
||||
return con
|
||||
|
||||
|
||||
def init() -> None:
|
||||
"""Create the schema. Call once at startup."""
|
||||
global _db_path
|
||||
_db_path = _resolve_db_path()
|
||||
with _lock, _connect() as con:
|
||||
con.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS usage_daily (
|
||||
user_key TEXT NOT NULL,
|
||||
day TEXT NOT NULL,
|
||||
spent_sec INTEGER NOT NULL DEFAULT 0,
|
||||
updated_at INTEGER NOT NULL,
|
||||
PRIMARY KEY (user_key, day)
|
||||
)"""
|
||||
)
|
||||
con.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS sessions (
|
||||
session_id TEXT PRIMARY KEY,
|
||||
keys TEXT NOT NULL, -- comma-joined usage_daily keys to debit
|
||||
day TEXT NOT NULL,
|
||||
tier TEXT NOT NULL,
|
||||
grant_ts REAL NOT NULL,
|
||||
last_seen_ts REAL NOT NULL,
|
||||
reserved_sec INTEGER NOT NULL,
|
||||
ended INTEGER NOT NULL DEFAULT 0
|
||||
)"""
|
||||
)
|
||||
logger.info("Usage limiter ready at %s (anon=%ss free=%ss chunk=%ss)", _db_path, ANON_SEC, FREE_SEC, CHUNK_SEC)
|
||||
|
||||
|
||||
# ── Internal helpers (call under _lock) ───────────────────────────────────────
|
||||
|
||||
def _spent(con, key: str, day: str) -> int:
|
||||
row = con.execute(
|
||||
"SELECT spent_sec FROM usage_daily WHERE user_key=? AND day=?", (key, day)
|
||||
).fetchone()
|
||||
return int(row[0]) if row else 0
|
||||
|
||||
|
||||
def _spent_max(con, keys, day: str) -> int:
|
||||
"""OR-match: the most-spent identifier governs."""
|
||||
return max((_spent(con, k, day) for k in keys), default=0)
|
||||
|
||||
|
||||
def _add(con, keys, day: str, delta: int) -> None:
|
||||
now = int(time.time())
|
||||
for k in keys:
|
||||
cur = _spent(con, k, day)
|
||||
nxt = max(0, cur + delta)
|
||||
con.execute(
|
||||
"""INSERT INTO usage_daily (user_key, day, spent_sec, updated_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(user_key, day) DO UPDATE SET
|
||||
spent_sec = excluded.spent_sec, updated_at = excluded.updated_at""",
|
||||
(k, day, nxt, now),
|
||||
)
|
||||
|
||||
|
||||
# ── Public API ────────────────────────────────────────────────────────────────
|
||||
|
||||
def remaining(keys, tier: str) -> "int | None":
|
||||
"""Seconds left today for these keys (None = unlimited). No mutation."""
|
||||
budget = budget_for(tier)
|
||||
if budget is None:
|
||||
return None
|
||||
with _lock, _connect() as con:
|
||||
return max(0, budget - _spent_max(con, keys, _today()))
|
||||
|
||||
|
||||
def begin(session_id: str, keys, tier: str) -> int:
|
||||
"""Reserve the first chunk for a new session and record it. Returns the
|
||||
chunk reserved (0 if the budget is already exhausted — the first heartbeat
|
||||
will then expire it). PRO (unlimited) is never tracked; don't call it here."""
|
||||
day = _today()
|
||||
budget = budget_for(tier)
|
||||
now = time.time()
|
||||
with _lock, _connect() as con:
|
||||
avail = budget - _spent_max(con, keys, day) if budget is not None else CHUNK_SEC
|
||||
chunk = max(0, min(CHUNK_SEC, avail))
|
||||
if chunk:
|
||||
_add(con, keys, day, chunk)
|
||||
con.execute(
|
||||
"""INSERT OR REPLACE INTO sessions
|
||||
(session_id, keys, day, tier, grant_ts, last_seen_ts, reserved_sec, ended)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, 0)""",
|
||||
(session_id, ",".join(keys), day, tier, now, now, chunk),
|
||||
)
|
||||
return chunk
|
||||
|
||||
|
||||
def heartbeat(session_id: str) -> bool:
|
||||
"""Keep a session alive: extend the reservation toward `elapsed + 1 chunk`,
|
||||
debiting the budget chunk by chunk. Returns True while alive, False once the
|
||||
budget is spent (caller should tear down) or the session is unknown/ended."""
|
||||
now = time.time()
|
||||
with _lock, _connect() as con:
|
||||
row = con.execute(
|
||||
"SELECT keys, day, tier, grant_ts, reserved_sec, ended FROM sessions WHERE session_id=?",
|
||||
(session_id,),
|
||||
).fetchone()
|
||||
if not row or row[5]:
|
||||
return False
|
||||
keys = row[0].split(",")
|
||||
day, tier, grant_ts, reserved = row[1], row[2], row[3], int(row[4])
|
||||
budget = budget_for(tier)
|
||||
elapsed = now - grant_ts
|
||||
|
||||
# Grow the reservation one chunk at a time until it covers elapsed + a
|
||||
# one-chunk lookahead, or the budget runs dry.
|
||||
while reserved < elapsed + CHUNK_SEC:
|
||||
if budget is not None and _spent_max(con, keys, day) >= budget:
|
||||
break
|
||||
_add(con, keys, day, CHUNK_SEC)
|
||||
reserved += CHUNK_SEC
|
||||
|
||||
alive = reserved > elapsed # could we cover the time already elapsed?
|
||||
con.execute(
|
||||
"UPDATE sessions SET last_seen_ts=?, reserved_sec=?, ended=? WHERE session_id=?",
|
||||
(now, reserved, 0 if alive else 1, session_id),
|
||||
)
|
||||
if not alive:
|
||||
_reconcile(con, keys, day, grant_ts, reserved, end_ts=now)
|
||||
return alive
|
||||
|
||||
|
||||
def end(session_id: str) -> None:
|
||||
"""Clean teardown: reconcile to actual elapsed time and refund the unused
|
||||
reservation. Idempotent."""
|
||||
now = time.time()
|
||||
with _lock, _connect() as con:
|
||||
row = con.execute(
|
||||
"SELECT keys, day, grant_ts, reserved_sec, ended FROM sessions WHERE session_id=?",
|
||||
(session_id,),
|
||||
).fetchone()
|
||||
if not row or row[4]:
|
||||
return
|
||||
keys, day, grant_ts, reserved = row[0].split(","), row[1], row[2], int(row[3])
|
||||
_reconcile(con, keys, day, grant_ts, reserved, end_ts=now)
|
||||
con.execute("UPDATE sessions SET ended=1, last_seen_ts=? WHERE session_id=?", (now, session_id))
|
||||
|
||||
|
||||
def sweep() -> None:
|
||||
"""Reap sessions that went silent (crash / closed without a beacon): bill
|
||||
their elapsed time, refund the rest, mark ended. Forfeits ≤ one chunk."""
|
||||
now = time.time()
|
||||
cutoff = now - REAP_AFTER_SEC
|
||||
with _lock, _connect() as con:
|
||||
stale = con.execute(
|
||||
"SELECT session_id, keys, day, grant_ts, last_seen_ts, reserved_sec FROM sessions "
|
||||
"WHERE ended=0 AND last_seen_ts < ?",
|
||||
(cutoff,),
|
||||
).fetchall()
|
||||
for session_id, keys_s, day, grant_ts, last_seen, reserved in stale:
|
||||
_reconcile(con, keys_s.split(","), day, grant_ts, int(reserved), end_ts=last_seen)
|
||||
con.execute("UPDATE sessions SET ended=1 WHERE session_id=?", (session_id,))
|
||||
if stale:
|
||||
logger.debug("swept %d stale session(s)", len(stale))
|
||||
|
||||
|
||||
def _reconcile(con, keys, day: str, grant_ts: float, reserved: int, end_ts: float) -> None:
|
||||
"""Refund reserved-but-unused time. Bill elapsed rounded up to a chunk,
|
||||
capped at what was reserved."""
|
||||
elapsed = max(0.0, end_ts - grant_ts)
|
||||
billed = min(reserved, int(math.ceil(elapsed / CHUNK_SEC) * CHUNK_SEC))
|
||||
refund = reserved - billed
|
||||
if refund > 0:
|
||||
_add(con, keys, day, -refund)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
fastapi==0.115.6
|
||||
uvicorn[standard]==0.34.0
|
||||
httpx==0.28.1
|
||||
# [oauth] extra pulls authlib + itsdangerous for attach_huggingface_oauth.
|
||||
huggingface_hub[oauth]>=0.30,<1.0
|
||||
@@ -0,0 +1,447 @@
|
||||
"""
|
||||
Tiny server for the speech-to-speech demo.
|
||||
|
||||
The demo used to ship as a `sdk: static` Space, but the web-search tool needs a
|
||||
search key the browser must NOT see. A static Space has no runtime process, so it
|
||||
can't hold a secret the front-end uses. This server fixes that: it serves the
|
||||
unchanged front-end AND exposes a same-origin `/api/search` proxy that holds the
|
||||
Serper key server-side (see docs/adr/0001).
|
||||
|
||||
Everything lives in one container; the speech-to-speech backend stays a separate,
|
||||
load-balanced service the browser talks to over WebSocket as before. The load
|
||||
balancer's address is a secret too (like the Serper key): the browser never sees
|
||||
it. `/api/session` proxies the session handshake server-side so only the
|
||||
per-session compute URL the LB hands back (which the browser must dial) is exposed.
|
||||
|
||||
On the deployed Space the server also meters conversation time by HF login tier
|
||||
(anonymous / signed-in / PRO) — see `limiter.py` and `auth.py`. That whole feature
|
||||
is off unless BOTH `LOAD_BALANCER_URL` and `SPACE_ID` are set, so it runs only on
|
||||
the live Space, never locally (even with the LB exported for testing).
|
||||
|
||||
`SPEECH_TO_SPEECH_URL` overrides everything: when set, the LB logic above is
|
||||
disabled entirely (no session proxy, no queue, no metering, no sign-in) and the
|
||||
browser connects directly to that URL, shown read-only in Settings.
|
||||
|
||||
Endpoints:
|
||||
GET /api/config -> { search, lb, allowDirect, s2sUrl, auth }
|
||||
GET /api/me -> login + tier + remaining budget (LB mode only)
|
||||
POST /api/search -> { results, answer } Google via Serper.dev
|
||||
POST /api/session -> proxies <LB>/session: a grant, or a queue ticket
|
||||
GET /api/queue/{id} -> proxies <LB>/queue/{id}: position, or a grant on claim
|
||||
DELETE /api/queue/{id} -> leave the queue (explicit "Leave queue" button)
|
||||
POST /api/queue/end -> leave the queue (sendBeacon on teardown)
|
||||
POST /api/session/heartbeat-> extend the reservation; { expired }
|
||||
POST /api/session/end -> reconcile + refund (sendBeacon on teardown)
|
||||
/* -> static files (index.html, main.js, ...)
|
||||
|
||||
When every compute slot is busy the load balancer hands back a queue ticket
|
||||
instead of a grant; the browser polls /api/queue/{id} until it reaches the front
|
||||
and a slot frees. Waiting reserves nothing — the daily budget is only reserved at
|
||||
the moment a slot is actually claimed (a grant), never while queued.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
|
||||
import auth
|
||||
import limiter
|
||||
|
||||
logger = logging.getLogger("s2s.search")
|
||||
|
||||
SERPER_KEY = os.environ.get("SERPER_API_KEY", "").strip()
|
||||
# Speech-to-speech load balancer URL. When set, the browser POSTs /api/session
|
||||
# (which proxies <lb>/session here, server-side) and connects to the URL the LB
|
||||
# returns (the original flow). The LB address itself is never sent to the browser.
|
||||
# When empty, the user may instead set a direct s2s server URL in Settings and the
|
||||
# browser connects to it straight (no load balancer).
|
||||
LOAD_BALANCER_URL = os.environ.get("LOAD_BALANCER_URL", "").strip()
|
||||
# Direct s2s server URL pinned by the deploy. Takes priority over the load
|
||||
# balancer: when set, ALL LB logic is disabled (no /api/session proxy, no queue,
|
||||
# no limiter, no sign-in) and the browser connects to this URL directly. Unlike
|
||||
# the LB address it is NOT a secret — /api/config sends it to the client, which
|
||||
# shows it read-only in Settings.
|
||||
SPEECH_TO_SPEECH_URL = os.environ.get("SPEECH_TO_SPEECH_URL", "").strip()
|
||||
if SPEECH_TO_SPEECH_URL:
|
||||
LOAD_BALANCER_URL = ""
|
||||
# HF injects SPACE_ID ("owner/space") into every Space runtime; it's absent
|
||||
# locally and on a plain `docker run`. We meter conversation time ONLY on the
|
||||
# deployed Space — i.e. when BOTH the LB is configured AND we're on a Space.
|
||||
# Off-Space (local dev, even with the LB exported) the app still proxies the LB,
|
||||
# but nothing is metered: no budget, no reservations, no sign-in gating.
|
||||
SPACE_ID = os.environ.get("SPACE_ID", "").strip()
|
||||
LIMITER_ENABLED = bool(LOAD_BALANCER_URL) and bool(SPACE_ID)
|
||||
SERPER_URL = "https://google.serper.dev/search"
|
||||
# Cap results so the tool output stays small enough to feed back to the model.
|
||||
MAX_RESULTS = 5
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
app = FastAPI(title="s2s-demo")
|
||||
|
||||
# Wire HF OAuth before the app serves (no-op unless the OAuth env is present).
|
||||
# Sign-in only matters when we're metering (prod Space), so gate it on that.
|
||||
AUTH_ENABLED = LIMITER_ENABLED and auth.attach(app)
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def _startup():
|
||||
"""Stand up the usage DB and a periodic sweeper — metered (prod Space) only."""
|
||||
if not LIMITER_ENABLED:
|
||||
return
|
||||
limiter.init()
|
||||
asyncio.create_task(_sweeper())
|
||||
|
||||
|
||||
async def _sweeper():
|
||||
while True:
|
||||
await asyncio.sleep(limiter.REAP_AFTER_SEC)
|
||||
try:
|
||||
await asyncio.to_thread(limiter.sweep)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.warning("usage sweep failed: %r", exc)
|
||||
|
||||
|
||||
class SearchRequest(BaseModel):
|
||||
query: str
|
||||
# Optional user-supplied key (fallback when the deploy has no server key).
|
||||
# Used for this request only; never stored.
|
||||
key: str | None = None
|
||||
|
||||
|
||||
@app.get("/api/config")
|
||||
def config():
|
||||
"""Client bootstrap: whether web search is available, whether the deploy runs
|
||||
behind a load balancer (so the browser uses the /api/session proxy + limiter),
|
||||
whether HF sign-in is available, and whether the user may instead set a direct
|
||||
s2s server URL. The LB address itself is intentionally NOT included."""
|
||||
return {
|
||||
"search": bool(SERPER_KEY),
|
||||
"lb": bool(LOAD_BALANCER_URL),
|
||||
"allowDirect": not LOAD_BALANCER_URL,
|
||||
# Deploy-pinned direct s2s URL (empty when unset). Not a secret: the
|
||||
# browser dials it itself, and Settings shows it locked.
|
||||
"s2sUrl": SPEECH_TO_SPEECH_URL,
|
||||
"auth": AUTH_ENABLED,
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/me")
|
||||
async def me(request: Request):
|
||||
"""Login state, tier, and remaining daily budget. Only meaningful in LB mode;
|
||||
sets the anonymous tracking cookie when first seen."""
|
||||
if not LIMITER_ENABLED:
|
||||
return {"enabled": False}
|
||||
view = auth.user_view(request)
|
||||
tier, keys, set_cookie = auth.resolve_identity(request)
|
||||
unlimited = limiter.budget_for(tier) is None
|
||||
rem = None if unlimited else await asyncio.to_thread(limiter.remaining, keys, tier)
|
||||
out = {
|
||||
"enabled": True,
|
||||
"auth": AUTH_ENABLED,
|
||||
**view,
|
||||
"remainingSec": rem,
|
||||
"limitSec": limiter.budget_for(tier),
|
||||
"loginUrl": auth.OAUTH_LOGIN_PATH if AUTH_ENABLED else None,
|
||||
"logoutUrl": auth.OAUTH_LOGOUT_PATH if AUTH_ENABLED else None,
|
||||
}
|
||||
resp = JSONResponse(out)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
|
||||
@app.post("/api/search")
|
||||
async def search(req: SearchRequest):
|
||||
"""Proxy a Google search via Serper.dev. The key stays on the server unless
|
||||
the user brought their own (then theirs is used for this request only)."""
|
||||
query = (req.query or "").strip()
|
||||
if not query:
|
||||
raise HTTPException(status_code=400, detail="Empty query.")
|
||||
|
||||
key = (req.key or "").strip() or SERPER_KEY
|
||||
if not key:
|
||||
# No server key and the user didn't supply one — search is unavailable.
|
||||
raise HTTPException(status_code=503, detail="Search is not configured.")
|
||||
|
||||
headers = {"X-API-KEY": key, "Content-Type": "application/json"}
|
||||
payload = {"q": query, "num": MAX_RESULTS}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=12.0) as http:
|
||||
resp = await http.post(SERPER_URL, headers=headers, json=payload)
|
||||
except httpx.RequestError as exc:
|
||||
logger.warning("Serper unreachable: %r", exc)
|
||||
raise HTTPException(status_code=502, detail="Search provider unreachable.")
|
||||
|
||||
if resp.status_code != 200:
|
||||
# Serper's error body carries the real reason (e.g. "Not enough
|
||||
# credits") and contains no key, so it's safe to log and relay.
|
||||
body = resp.text[:300]
|
||||
logger.warning("Serper error %s: %s", resp.status_code, body)
|
||||
msg = None
|
||||
try:
|
||||
msg = resp.json().get("message")
|
||||
except Exception:
|
||||
pass
|
||||
detail = f"Search provider error ({resp.status_code})"
|
||||
if msg:
|
||||
detail += f": {msg}"
|
||||
raise HTTPException(status_code=502, detail=detail)
|
||||
|
||||
data = resp.json()
|
||||
results = []
|
||||
for item in (data.get("organic") or [])[:MAX_RESULTS]:
|
||||
results.append(
|
||||
{
|
||||
"title": item.get("title", ""),
|
||||
"snippet": item.get("snippet", ""),
|
||||
"url": item.get("link", ""),
|
||||
}
|
||||
)
|
||||
|
||||
# A direct answer when Google has one — saves the model a hop.
|
||||
box = data.get("answerBox") or {}
|
||||
answer = box.get("answer") or box.get("snippet") or None
|
||||
if not answer:
|
||||
kg = data.get("knowledgeGraph") or {}
|
||||
answer = kg.get("description") or None
|
||||
|
||||
return JSONResponse({"query": query, "answer": answer, "results": results})
|
||||
|
||||
|
||||
@app.post("/api/session")
|
||||
async def session(request: Request):
|
||||
"""Proxy the session handshake to the load balancer, keeping its URL secret,
|
||||
and meter conversation time by tier.
|
||||
|
||||
The browser POSTs here (same-origin); we resolve the caller's tier, refuse if
|
||||
today's budget is already spent (402), otherwise POST <LOAD_BALANCER_URL>/session
|
||||
and relay the JSON back. The LB body carries a per-session `connect_url`
|
||||
(compute host + short-lived token) the browser must dial directly — that one
|
||||
URL is unavoidably exposed, but the stable load-balancer address is not. On a
|
||||
successful grant we reserve the first time chunk against the day's budget."""
|
||||
if not LOAD_BALANCER_URL:
|
||||
# No LB configured — this deploy is direct-mode only; the browser should
|
||||
# never call this. 404 so it's indistinguishable from a missing route.
|
||||
raise HTTPException(status_code=404, detail="Not found.")
|
||||
|
||||
tier, keys, set_cookie = auth.resolve_identity(request)
|
||||
# Metering runs only on the deployed Space; off-Space the LB still proxies but
|
||||
# nothing is tracked. Within metering, unlimited tiers (pro, org) aren't either.
|
||||
tracked = LIMITER_ENABLED and limiter.budget_for(tier) is not None
|
||||
|
||||
# Refuse before troubling the LB if the day's budget is already gone. Done
|
||||
# here (at enqueue) so we never put a user who can't talk into the queue.
|
||||
if tracked:
|
||||
rem = await asyncio.to_thread(limiter.remaining, keys, tier)
|
||||
if rem is not None and rem <= 0:
|
||||
resp = JSONResponse(
|
||||
{"tier": tier, "reason": "limit", "remainingSec": 0}, status_code=402
|
||||
)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
url = f"{LOAD_BALANCER_URL.rstrip('/')}/session"
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=15.0) as http:
|
||||
lb = await http.post(url, headers={"Content-Type": "application/json"}, content="{}")
|
||||
except httpx.RequestError as exc:
|
||||
logger.warning("Load balancer unreachable: %r", exc)
|
||||
raise HTTPException(status_code=502, detail="Speech service unreachable.")
|
||||
|
||||
# The queue is full: the LB replies 503 {state:"at_capacity"}. Relay it as-is
|
||||
# so the client shows a soft "try again shortly", not a hard error.
|
||||
if lb.status_code == 503:
|
||||
body = _safe_json(lb)
|
||||
if body.get("state") == "at_capacity":
|
||||
resp = JSONResponse({"state": "at_capacity"}, status_code=503)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
if lb.status_code != 200:
|
||||
# The LB's error body may name the reason (e.g. capacity); it carries no
|
||||
# secret, so relay a trimmed copy.
|
||||
logger.warning("Session handshake failed %s: %s", lb.status_code, lb.text[:300])
|
||||
raise HTTPException(status_code=502, detail=f"Session handshake failed ({lb.status_code}).")
|
||||
|
||||
data = lb.json()
|
||||
|
||||
# Busy pool: the LB queued us. Relay the ticket untouched — crucially with NO
|
||||
# reservation, so waiting in line never costs the day's budget.
|
||||
if data.get("state") == "queued":
|
||||
data["tier"] = tier
|
||||
resp = JSONResponse(data)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
# A slot was free: reserve the first chunk now and return the grant.
|
||||
return await _finalize_grant(data, keys, tier, tracked, set_cookie)
|
||||
|
||||
|
||||
@app.get("/api/queue/{queue_id}")
|
||||
async def queue_status(queue_id: str, request: Request):
|
||||
"""Poll a waiting ticket: relay the position, or — when the head of the line
|
||||
claims a freed slot — reserve the budget now and return the grant. Re-checks the
|
||||
daily budget at claim, since a multi-minute wait could have spent it elsewhere."""
|
||||
if not LOAD_BALANCER_URL:
|
||||
raise HTTPException(status_code=404, detail="Not found.")
|
||||
|
||||
tier, keys, set_cookie = auth.resolve_identity(request)
|
||||
tracked = LIMITER_ENABLED and limiter.budget_for(tier) is not None
|
||||
|
||||
url = f"{LOAD_BALANCER_URL.rstrip('/')}/queue/{queue_id}"
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=15.0) as http:
|
||||
lb = await http.get(url)
|
||||
except httpx.RequestError as exc:
|
||||
logger.warning("Load balancer unreachable: %r", exc)
|
||||
raise HTTPException(status_code=502, detail="Speech service unreachable.")
|
||||
|
||||
if lb.status_code == 404:
|
||||
# Ticket unknown/expired (reaped after we stopped polling). Tell the client
|
||||
# to start over rather than spin.
|
||||
resp = JSONResponse({"state": "expired"}, status_code=404)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
if lb.status_code != 200:
|
||||
logger.warning("Queue poll failed %s: %s", lb.status_code, lb.text[:300])
|
||||
raise HTTPException(status_code=502, detail=f"Queue poll failed ({lb.status_code}).")
|
||||
|
||||
data = lb.json()
|
||||
|
||||
if data.get("state") == "queued":
|
||||
data["tier"] = tier
|
||||
resp = JSONResponse(data)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
# Claimed a slot. Re-check the budget: it may have been spent in another tab
|
||||
# during the wait. If so, refuse — the just-claimed slot is now a pending
|
||||
# session on the LB and its pending-timeout reaper reclaims it shortly.
|
||||
if tracked:
|
||||
rem = await asyncio.to_thread(limiter.remaining, keys, tier)
|
||||
if rem is not None and rem <= 0:
|
||||
resp = JSONResponse(
|
||||
{"tier": tier, "reason": "limit", "remainingSec": 0}, status_code=402
|
||||
)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
return await _finalize_grant(data, keys, tier, tracked, set_cookie)
|
||||
|
||||
|
||||
@app.delete("/api/queue/{queue_id}")
|
||||
async def queue_leave(queue_id: str):
|
||||
"""Leave the queue from the explicit 'Leave queue' button (a real fetch)."""
|
||||
if not LOAD_BALANCER_URL:
|
||||
raise HTTPException(status_code=404, detail="Not found.")
|
||||
await _lb_leave(queue_id)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@app.post("/api/queue/end")
|
||||
async def queue_end(request: Request):
|
||||
"""Leave the queue on teardown/tab-close (navigator.sendBeacon, which can only
|
||||
POST). Body: { queueId }. Best-effort; the LB reaps the ticket on TTL anyway."""
|
||||
if not LOAD_BALANCER_URL:
|
||||
raise HTTPException(status_code=404, detail="Not found.")
|
||||
qid = await _queue_id(request)
|
||||
if qid:
|
||||
await _lb_leave(qid)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
async def _finalize_grant(data, keys, tier, tracked, set_cookie):
|
||||
"""Shared grant tail (fast path or queue claim): reserve the first chunk, attach
|
||||
the metering fields the client needs, and set the anon cookie."""
|
||||
remaining = None
|
||||
if tracked and data.get("session_id"):
|
||||
await asyncio.to_thread(limiter.begin, data["session_id"], keys, tier)
|
||||
remaining = await asyncio.to_thread(limiter.remaining, keys, tier)
|
||||
|
||||
data.update({
|
||||
"tier": tier,
|
||||
"limited": tracked,
|
||||
"remainingSec": remaining,
|
||||
"heartbeatSec": limiter.HEARTBEAT_SEC,
|
||||
})
|
||||
resp = JSONResponse(data)
|
||||
if set_cookie:
|
||||
auth.set_anon_cookie(resp, set_cookie)
|
||||
return resp
|
||||
|
||||
|
||||
async def _lb_leave(queue_id: str) -> None:
|
||||
"""Best-effort: tell the LB to drop a waiting ticket."""
|
||||
url = f"{LOAD_BALANCER_URL.rstrip('/')}/queue/{queue_id}"
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as http:
|
||||
await http.delete(url)
|
||||
except httpx.RequestError as exc:
|
||||
logger.warning("Queue leave failed: %r", exc)
|
||||
|
||||
|
||||
def _safe_json(response) -> dict:
|
||||
try:
|
||||
body = response.json()
|
||||
except Exception:
|
||||
return {}
|
||||
return body if isinstance(body, dict) else {}
|
||||
|
||||
|
||||
async def _queue_id(request: Request) -> str:
|
||||
"""Pull `queueId` from a JSON body, tolerating sendBeacon's blob posts."""
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception:
|
||||
return ""
|
||||
return (data or {}).get("queueId", "") if isinstance(data, dict) else ""
|
||||
|
||||
|
||||
async def _session_id(request: Request) -> str:
|
||||
"""Pull `sessionId` from a JSON body, tolerating sendBeacon's blob posts."""
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception:
|
||||
return ""
|
||||
return (data or {}).get("sessionId", "") if isinstance(data, dict) else ""
|
||||
|
||||
|
||||
@app.post("/api/session/heartbeat")
|
||||
async def session_heartbeat(request: Request):
|
||||
"""Extend the live reservation one chunk at a time. `expired` once the day's
|
||||
budget is spent — the client then tears down."""
|
||||
if not LIMITER_ENABLED:
|
||||
raise HTTPException(status_code=404, detail="Not found.")
|
||||
sid = await _session_id(request)
|
||||
alive = bool(sid) and await asyncio.to_thread(limiter.heartbeat, sid)
|
||||
return {"expired": not alive}
|
||||
|
||||
|
||||
@app.post("/api/session/end")
|
||||
async def session_end(request: Request):
|
||||
"""Clean teardown: reconcile to real elapsed time and refund the unused
|
||||
chunk. Sent via navigator.sendBeacon, so it must succeed without a response."""
|
||||
if not LIMITER_ENABLED:
|
||||
raise HTTPException(status_code=404, detail="Not found.")
|
||||
sid = await _session_id(request)
|
||||
if sid:
|
||||
await asyncio.to_thread(limiter.end, sid)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
# Static front-end. Registered last so the /api routes win. `html=True` serves
|
||||
# index.html at "/". The repo is public anyway, so serving the dir is fine.
|
||||
app.mount("/", StaticFiles(directory=HERE, html=True), name="static")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,189 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* Account — the HF login chip and the daily-limit modal.
|
||||
*
|
||||
* Reads `/api/me` to learn the current tier (anonymous / signed-in / PRO) and
|
||||
* remaining daily talk-time, renders a sign-in pill or a signed-in chip with a
|
||||
* small popover (tier, remaining, sign out, upgrade), and shows the limit modal
|
||||
* when a conversation is refused or cut. The time metering itself lives in
|
||||
* main.js (heartbeat loop) + the server; this module is just the surface.
|
||||
*
|
||||
* Inert unless the deploy is in LB mode (`/api/me` → `{enabled:true}`).
|
||||
*/
|
||||
|
||||
import { $, escHtml } from "./dom.js";
|
||||
|
||||
const PRO_URL = "https://huggingface.co/subscribe/pro";
|
||||
|
||||
// Official multi-color Hugging Face logo, used in the badge + sign-in CTA.
|
||||
const HF_MARK = `<svg class="hf-logo" viewBox="0 0 95 88" fill="none" aria-hidden="true"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg>`;
|
||||
|
||||
/** @param {number} sec @returns {string} "m:ss" */
|
||||
function fmt(sec) {
|
||||
const s = Math.max(0, Math.round(sec));
|
||||
return `${Math.floor(s / 60)}:${String(s % 60).padStart(2, "0")}`;
|
||||
}
|
||||
|
||||
export class Account {
|
||||
constructor() {
|
||||
/** @type {HTMLElement} */
|
||||
this._root = $("#account");
|
||||
/** @type {HTMLDialogElement} */
|
||||
this._modal = $("#limit-modal");
|
||||
this._modalTitle = $("#limit-title");
|
||||
this._modalMsg = $("#limit-msg");
|
||||
this._modalNote = $("#limit-note");
|
||||
/** @type {HTMLAnchorElement} */
|
||||
this._modalCta = /** @type {any} */ ($("#limit-cta"));
|
||||
|
||||
/** @type {{enabled:boolean, auth?:boolean, loggedIn?:boolean, username?:string, avatar?:string, tier?:string, remainingSec?:number|null, limitSec?:number|null, loginUrl?:string|null, logoutUrl?:string|null}} */
|
||||
this._me = { enabled: false };
|
||||
this._popoverOpen = false;
|
||||
|
||||
$("#limit-close").addEventListener("click", () => this._modal.close());
|
||||
this._modal.addEventListener("click", (e) => {
|
||||
if (e.target === this._modal) this._modal.close();
|
||||
});
|
||||
// Close the popover on an outside click.
|
||||
document.addEventListener("click", (e) => {
|
||||
if (this._popoverOpen && !this._root.contains(/** @type {Node} */ (e.target))) {
|
||||
this._closePopover();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
get tier() {
|
||||
return this._me.tier || "anon";
|
||||
}
|
||||
|
||||
/** Fetch `/api/me` and (re)render the chip. Safe to call repeatedly (load,
|
||||
* after the OAuth redirect, after a conversation ends). */
|
||||
async refresh() {
|
||||
try {
|
||||
const res = await fetch("api/me");
|
||||
this._me = res.ok ? await res.json() : { enabled: false };
|
||||
} catch {
|
||||
this._me = { enabled: false };
|
||||
}
|
||||
this._render();
|
||||
}
|
||||
|
||||
_render() {
|
||||
const me = this._me;
|
||||
if (!me.enabled) {
|
||||
this._root.hidden = true;
|
||||
this._root.innerHTML = "";
|
||||
return;
|
||||
}
|
||||
this._root.hidden = false;
|
||||
|
||||
if (!me.loggedIn) {
|
||||
// Signed-out: a sign-in pill (only when OAuth is actually available).
|
||||
if (me.auth && me.loginUrl) {
|
||||
this._root.innerHTML = `<a class="signin-pill" href="${escHtml(me.loginUrl)}" title="Sign in for more time"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M15 3h4a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2h-4"/><polyline points="10 17 15 12 10 7"/><line x1="15" y1="12" x2="3" y2="12"/></svg><span>Sign in</span></a>`;
|
||||
} else {
|
||||
this._root.innerHTML = "";
|
||||
this._root.hidden = true;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Signed-in: avatar + handle chip that toggles a popover.
|
||||
const isPro = me.tier === "pro";
|
||||
// Org members get unlimited usage too, but aren't PRO — don't brand them so.
|
||||
const isUnlimited = isPro || me.tier === "org";
|
||||
const avatar = me.avatar
|
||||
? `<img class="account-avatar" src="${escHtml(me.avatar)}" alt="" />`
|
||||
: `<span class="account-avatar account-avatar-fallback">${escHtml((me.username || "?")[0].toUpperCase())}</span>`;
|
||||
const remaining =
|
||||
isUnlimited || me.remainingSec == null
|
||||
? "Unlimited"
|
||||
: `${fmt(me.remainingSec)} left today`;
|
||||
const tierLabel = isPro ? "PRO" : isUnlimited ? "Team" : "Free";
|
||||
|
||||
this._root.innerHTML = `
|
||||
<button id="account-chip" class="account-chip" aria-haspopup="true" aria-expanded="false">
|
||||
${avatar}
|
||||
<span class="account-handle">${escHtml(me.username || "you")}</span>
|
||||
${isPro
|
||||
? '<span class="account-pro">PRO</span>'
|
||||
: me.tier === "org"
|
||||
? '<span class="account-pro account-team">TEAM</span>'
|
||||
: ""}
|
||||
</button>
|
||||
<div id="account-pop" class="account-pop" hidden>
|
||||
<div class="account-pop-row account-pop-name">${escHtml(me.username || "you")}</div>
|
||||
<div class="account-pop-row account-pop-meta">
|
||||
<span class="account-tier">${tierLabel}</span>
|
||||
<span class="account-remaining">${escHtml(remaining)}</span>
|
||||
</div>
|
||||
${isUnlimited ? "" : `<a class="account-pop-link" href="${PRO_URL}" target="_blank" rel="noopener">Upgrade to PRO</a>`}
|
||||
<a class="account-pop-link account-signout" href="${escHtml(me.logoutUrl || "#")}">Sign out</a>
|
||||
</div>`;
|
||||
|
||||
const chip = $("#account-chip");
|
||||
chip.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
this._popoverOpen ? this._closePopover() : this._openPopover();
|
||||
});
|
||||
}
|
||||
|
||||
_openPopover() {
|
||||
const pop = document.getElementById("account-pop");
|
||||
const chip = document.getElementById("account-chip");
|
||||
if (!pop || !chip) return;
|
||||
pop.hidden = false;
|
||||
chip.setAttribute("aria-expanded", "true");
|
||||
this._popoverOpen = true;
|
||||
}
|
||||
|
||||
_closePopover() {
|
||||
const pop = document.getElementById("account-pop");
|
||||
const chip = document.getElementById("account-chip");
|
||||
if (pop) pop.hidden = true;
|
||||
if (chip) chip.setAttribute("aria-expanded", "false");
|
||||
this._popoverOpen = false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Show the limit modal for a tier — used both when a conversation is refused
|
||||
* at start (402) and when a live one is cut (heartbeat `expired`).
|
||||
* @param {string} [tier]
|
||||
*/
|
||||
showLimit(tier = this.tier) {
|
||||
const canSignIn = this._me.auth && this._me.loginUrl;
|
||||
this._modalTitle.textContent = "Thanks for chatting!";
|
||||
if (tier === "anon") {
|
||||
this._modalMsg.textContent =
|
||||
"Guest conversations run for 5 minutes. Sign in with Hugging Face to get 10 minutes a day for free, and PRO members chat with no limit at all.";
|
||||
this._modalNote.textContent = "Your free minutes refresh tomorrow.";
|
||||
if (canSignIn) {
|
||||
this._modalCta.innerHTML = `${HF_MARK}<span>Sign in with Hugging Face</span>`;
|
||||
this._modalCta.href = /** @type {string} */ (this._me.loginUrl);
|
||||
this._modalCta.hidden = false;
|
||||
} else {
|
||||
this._modalCta.hidden = true;
|
||||
}
|
||||
} else {
|
||||
// Signed-in, non-PRO.
|
||||
this._modalMsg.textContent =
|
||||
"You've enjoyed your 10 minutes for today. Go PRO for unlimited conversations and to support open source AI.";
|
||||
this._modalNote.textContent = "Or come back tomorrow. Your minutes reset daily.";
|
||||
this._modalCta.innerHTML = "<span>Upgrade to PRO</span>";
|
||||
this._modalCta.href = PRO_URL;
|
||||
this._modalCta.hidden = false;
|
||||
}
|
||||
if (!this._modal.open) this._modal.showModal();
|
||||
}
|
||||
|
||||
/** Show a warm "we're at capacity" message when even the waiting line is full.
|
||||
* Reuses the limit modal shell; no call-to-action, just reassurance. */
|
||||
showBusy() {
|
||||
this._modalTitle.textContent = "Hugged to the limit 🤗";
|
||||
this._modalMsg.textContent =
|
||||
"Every slot and the whole line are full right now. Too much love! Grab a coffee and pop back in a minute.";
|
||||
this._modalNote.textContent = "A spot usually opens up within a few minutes.";
|
||||
this._modalCta.hidden = true;
|
||||
if (!this._modal.open) this._modal.showModal();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,425 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* ChatView — owns the whole conversation surface: the slide-in history panel,
|
||||
* the ephemeral on-orb bubbles, and all the transcript/tool/streaming
|
||||
* bookkeeping. main.js wires the realtime client's events straight to the
|
||||
* `on*` methods here and otherwise doesn't touch chat state.
|
||||
*
|
||||
* Two parallel surfaces share one shape (see `_buildMessageEl`):
|
||||
* - ephemeral bubbles (`.bubble` / `.bubble-*`) fade on a timer
|
||||
* - persistent history (`.hist-msg` / `.hist-*`) the durable panel log
|
||||
*
|
||||
* Keying:
|
||||
* - user transcripts by the server's `item_id` — a speculative continuation
|
||||
* REUSES it, so both segments land in one row/bubble; deltas are CUMULATIVE
|
||||
* (each carries the full sentence so far), so we replace text wholesale.
|
||||
* - assistant transcripts by `response_id`, so a cancelled speculative reply
|
||||
* can be marked interrupted without erasing what was already shown.
|
||||
*/
|
||||
|
||||
import { $, escHtml, DEBUG } from "./dom.js";
|
||||
|
||||
const WRENCH_PATH = `<path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/>`;
|
||||
const CHAT_BUBBLE_SVG = `<svg width="28" height="28" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15a2 2 0 0 1-2 2H7l-4 4V5a2 2 0 0 1 2-2h14a2 2 0 0 1 2 2z"/></svg>`;
|
||||
const EMPTY_STATE_HTML = `<div id="chat-empty" class="chat-empty">${CHAT_BUBBLE_SVG}<span class="chat-empty-title">No messages yet</span><span class="chat-empty-hint">Tap the orb and start talking</span></div>`;
|
||||
|
||||
export class ChatView {
|
||||
constructor() {
|
||||
/** @type {HTMLButtonElement} */
|
||||
this._chatBtn = $("#chat-btn");
|
||||
/** @type {HTMLSpanElement} */
|
||||
this._chatBadge = $("#chat-badge");
|
||||
/** @type {HTMLDivElement} */
|
||||
this._chatPanel = $("#chat-panel");
|
||||
/** @type {HTMLDivElement} */
|
||||
this._chatPanelBackdrop = $("#chat-panel-backdrop");
|
||||
/** @type {HTMLButtonElement} */
|
||||
this._chatPanelClose = $("#chat-panel-close");
|
||||
/** @type {HTMLDivElement} */
|
||||
this._chatHistory = $("#chat-history");
|
||||
/** @type {HTMLDivElement} */
|
||||
this._bubbleStack = $("#bubble-stack");
|
||||
|
||||
this._panelOpen = false;
|
||||
this._scrollQueued = false;
|
||||
|
||||
// ── User transcript state (keyed by item_id) ───────────────────────────
|
||||
/** @type {Map<string, HTMLElement>} */
|
||||
this._userHistByItem = new Map();
|
||||
/** @type {HTMLElement | null} */
|
||||
this._activeUserBubble = null;
|
||||
this._activeUserItemId = "";
|
||||
// Monotonic counter for synthesizing unique keys when the server omits an
|
||||
// item_id / response_id, so id-less messages never collapse onto each other.
|
||||
this._anonSeq = 0;
|
||||
|
||||
// ── Assistant transcript state (keyed by response_id) ──────────────────
|
||||
/** @type {Map<string, { bubble: HTMLElement, hist: HTMLElement }>} */
|
||||
this._asstByResp = new Map();
|
||||
|
||||
// ── Ephemeral bubble auto-dismiss ──────────────────────────────────────
|
||||
// Per-element expiry (epoch ms). A bubble fades once its expiry passes —
|
||||
// but only in stack order (see _reapBubbles). Refreshing the expiry keeps a
|
||||
// bubble alive while it updates (e.g. the live user utterance).
|
||||
/** @type {WeakMap<HTMLElement, number>} */
|
||||
this._bubbleExpiry = new WeakMap();
|
||||
// Single pending reaper handle: one timer for the whole stack (not one per
|
||||
// bubble) so dismissal is strictly oldest-first regardless of per-bubble delays.
|
||||
this._reaperHandle = 0;
|
||||
|
||||
this._chatBtn.addEventListener("click", () => (this._panelOpen ? this._closePanel() : this._openPanel()));
|
||||
this._chatPanelClose.addEventListener("click", () => this._closePanel());
|
||||
this._chatPanelBackdrop.addEventListener("click", () => this._closePanel());
|
||||
document.addEventListener("keydown", (e) => {
|
||||
if (e.key === "Escape" && this._panelOpen) this._closePanel();
|
||||
});
|
||||
}
|
||||
|
||||
// ── Panel ───────────────────────────────────────────────────────────────
|
||||
|
||||
_openPanel() {
|
||||
this._panelOpen = true;
|
||||
this._chatPanel.classList.add("open");
|
||||
this._chatBadge.classList.remove("visible");
|
||||
this._scrollToBottom();
|
||||
}
|
||||
|
||||
_closePanel() {
|
||||
this._panelOpen = false;
|
||||
this._chatPanel.classList.remove("open");
|
||||
}
|
||||
|
||||
// Coalesce scroll-to-bottom: a burst of cumulative transcript deltas would
|
||||
// otherwise queue one rAF per delta, all writing the same scrollTop.
|
||||
_scrollToBottom() {
|
||||
if (!this._panelOpen || this._scrollQueued) return;
|
||||
this._scrollQueued = true;
|
||||
requestAnimationFrame(() => {
|
||||
this._scrollQueued = false;
|
||||
this._chatHistory.scrollTop = this._chatHistory.scrollHeight;
|
||||
});
|
||||
}
|
||||
|
||||
_markUnread() {
|
||||
if (this._panelOpen) {
|
||||
this._scrollToBottom();
|
||||
return;
|
||||
}
|
||||
this._chatBadge.classList.add("visible");
|
||||
}
|
||||
|
||||
// ── Shared rendering ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Build a role-labelled message element. Ephemeral bubbles and persistent
|
||||
* history rows share the same shape and differ only in their class prefix
|
||||
* (`bubble`/`bubble-*` vs `hist-msg`/`hist-*`).
|
||||
* @param {{ container: string, prefix: string, role: "user"|"assistant", text: string, partial?: boolean }} o
|
||||
* @returns {HTMLElement}
|
||||
*/
|
||||
_buildMessageEl({ container, prefix, role, text, partial = false }) {
|
||||
const el = document.createElement("div");
|
||||
el.className = `${container} ${role}`;
|
||||
const label = role === "user" ? "You" : "Assistant";
|
||||
el.innerHTML = `<div class="${prefix}-role">${label}</div><div class="${prefix}-body${partial ? " partial" : ""}">${escHtml(text)}</div>`;
|
||||
return el;
|
||||
}
|
||||
|
||||
// ── Ephemeral bubbles ───────────────────────────────────────────────────
|
||||
|
||||
/** @param {"user"|"assistant"|"tool"} role @param {string} text @returns {HTMLElement} */
|
||||
_spawnBubble(role, text) {
|
||||
let el;
|
||||
if (role === "tool") {
|
||||
el = document.createElement("div");
|
||||
el.className = "bubble tool";
|
||||
el.innerHTML = `<svg class="bubble-tool-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">${WRENCH_PATH}</svg><span class="bubble-tool-text">${escHtml(text)}</span>`;
|
||||
} else {
|
||||
el = this._buildMessageEl({ container: "bubble", prefix: "bubble", role, text });
|
||||
}
|
||||
this._bubbleStack.appendChild(el);
|
||||
// Cap the stack at 3, but never evict the bubble the caller is still
|
||||
// actively updating (the live user bubble) — drop the next-oldest instead.
|
||||
const visible = /** @type {HTMLElement[]} */ ([...this._bubbleStack.querySelectorAll(".bubble:not(.out)")]);
|
||||
if (visible.length > 3) {
|
||||
this._dismissBubble(visible.find((b) => b !== this._activeUserBubble) ?? visible[0]);
|
||||
}
|
||||
requestAnimationFrame(() => el.classList.add("in"));
|
||||
return el;
|
||||
}
|
||||
|
||||
/** @param {HTMLElement} el @param {string} text */
|
||||
_updateBubbleText(el, text) {
|
||||
const t = el.querySelector(".bubble-body");
|
||||
if (t) t.textContent = text;
|
||||
}
|
||||
|
||||
/** @param {HTMLElement} el */
|
||||
_dismissBubble(el) {
|
||||
if (!el || el.classList.contains("out")) return; // idempotent
|
||||
this._bubbleExpiry.delete(el);
|
||||
el.classList.remove("in");
|
||||
el.classList.add("out");
|
||||
const remove = () => el.remove();
|
||||
el.addEventListener("transitionend", remove, { once: true });
|
||||
// Fallback: transitionend never fires if the bubble's visual state didn't
|
||||
// change (dismissed pre-paint) or the tab is backgrounded. The transition
|
||||
// is 0.3s, so force removal a little after.
|
||||
setTimeout(remove, 400);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fade bubbles whose expiry has passed — strictly oldest-first. We walk the
|
||||
* stack top (oldest) to bottom (newest) and stop at the first bubble still
|
||||
* alive: nothing newer may leave while an older bubble is still on screen. A
|
||||
* bubble that keeps updating pushes its own expiry forward, so it (and
|
||||
* everything behind it) stays put until it finally goes quiet.
|
||||
*/
|
||||
_reapBubbles() {
|
||||
this._reaperHandle = 0;
|
||||
const now = Date.now();
|
||||
const visible = /** @type {HTMLElement[]} */ ([...this._bubbleStack.querySelectorAll(".bubble:not(.out)")]);
|
||||
let nextWake = Infinity;
|
||||
for (const el of visible) {
|
||||
const exp = this._bubbleExpiry.get(el) ?? now; // no expiry recorded → treat as due
|
||||
if (exp <= now) {
|
||||
this._dismissBubble(el);
|
||||
} else {
|
||||
// Oldest survivor isn't due yet; stop so nothing newer leaves before it.
|
||||
nextWake = exp;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (nextWake !== Infinity) {
|
||||
this._reaperHandle = setTimeout(() => this._reapBubbles(), Math.max(50, nextWake - Date.now()));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* (Re)arm a bubble's auto-dismiss by pushing its expiry out by `delay`.
|
||||
* Calling it again resets the countdown — so a bubble that keeps updating
|
||||
* stays on screen and only fades once it goes quiet. Removal is ordered by the
|
||||
* shared reaper, so the oldest bubble always disappears first.
|
||||
* @param {HTMLElement} el @param {number} [delay]
|
||||
*/
|
||||
_bumpDismiss(el, delay = 4000) {
|
||||
this._bubbleExpiry.set(el, Date.now() + delay);
|
||||
if (!this._reaperHandle) this._reaperHandle = setTimeout(() => this._reapBubbles(), delay);
|
||||
}
|
||||
|
||||
// ── History ───────────────────────────────────────────────────────────────
|
||||
|
||||
/** Render the empty-state placeholder into the history panel. */
|
||||
renderEmptyState() {
|
||||
this._chatHistory.innerHTML = EMPTY_STATE_HTML;
|
||||
}
|
||||
|
||||
/** Reset the panel to the empty state and clear the unread badge. */
|
||||
clear() {
|
||||
this.renderEmptyState();
|
||||
this._chatBadge.classList.remove("visible");
|
||||
}
|
||||
|
||||
/** @param {"user"|"assistant"} role @param {string} text @param {boolean} partial @returns {HTMLElement} */
|
||||
_appendHistMsg(role, text, partial) {
|
||||
const empty = this._chatHistory.querySelector(".chat-empty");
|
||||
if (empty) empty.remove();
|
||||
const el = this._buildMessageEl({ container: "hist-msg", prefix: "hist", role, text, partial });
|
||||
this._chatHistory.appendChild(el);
|
||||
this._scrollToBottom();
|
||||
return el;
|
||||
}
|
||||
|
||||
/** @param {HTMLElement | null} el @param {string} text @param {boolean} partial */
|
||||
_updateHistMsg(el, text, partial) {
|
||||
if (!el) return;
|
||||
const body = /** @type {HTMLElement | null} */ (el.querySelector(".hist-body"));
|
||||
if (!body) return;
|
||||
body.textContent = text;
|
||||
body.classList.toggle("partial", partial);
|
||||
this._scrollToBottom();
|
||||
}
|
||||
|
||||
/**
|
||||
* Append a tool-call row to the conversation. We only add it once the tool
|
||||
* has run, so the expandable toggle carries BOTH the call input and its result.
|
||||
* @param {string} name @param {string} argsJson @param {string} output
|
||||
*/
|
||||
_appendHistTool(name, argsJson, output) {
|
||||
const empty = this._chatHistory.querySelector(".chat-empty");
|
||||
if (empty) empty.remove();
|
||||
let pretty = argsJson;
|
||||
try { pretty = JSON.stringify(JSON.parse(argsJson), null, 2); } catch {}
|
||||
const el = document.createElement("div");
|
||||
el.className = "hist-msg tool";
|
||||
el.innerHTML = `
|
||||
<div class="hist-role">Tool call</div>
|
||||
<button class="hist-tool-header" aria-expanded="false">
|
||||
<svg class="hist-tool-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">${WRENCH_PATH}</svg>
|
||||
<span class="hist-tool-name">${escHtml(name)}</span>
|
||||
<svg class="hist-tool-chevron" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="6 9 12 15 18 9"/></svg>
|
||||
</button>
|
||||
<div class="hist-tool-body">
|
||||
<div class="hist-tool-label">Input</div>
|
||||
<div class="hist-tool-block">${escHtml(pretty)}</div>
|
||||
<div class="hist-tool-label">Output</div>
|
||||
<div class="hist-tool-block hist-tool-output">${escHtml(output || "(no output)")}</div>
|
||||
</div>
|
||||
`;
|
||||
const header = /** @type {HTMLButtonElement} */ (el.querySelector(".hist-tool-header"));
|
||||
const body = /** @type {HTMLDivElement} */ (el.querySelector(".hist-tool-body"));
|
||||
header.addEventListener("click", () => {
|
||||
const expanded = header.getAttribute("aria-expanded") === "true";
|
||||
header.setAttribute("aria-expanded", String(!expanded));
|
||||
body.classList.toggle("open", !expanded);
|
||||
});
|
||||
this._chatHistory.appendChild(el);
|
||||
this._scrollToBottom();
|
||||
}
|
||||
|
||||
/** Tag an assistant history row as interrupted (user barged in mid-reply).
|
||||
* @param {HTMLElement | null} hist */
|
||||
_markHistInterrupted(hist) {
|
||||
if (!hist || hist.querySelector(".hist-note")) return;
|
||||
hist.classList.add("interrupted");
|
||||
const note = document.createElement("div");
|
||||
note.className = "hist-note";
|
||||
note.textContent = "Interrupted";
|
||||
hist.appendChild(note);
|
||||
}
|
||||
|
||||
/** Render a captured webcam frame in the transcript (the camera tool result).
|
||||
* @param {string} dataUrl */
|
||||
_appendHistImage(dataUrl) {
|
||||
const empty = this._chatHistory.querySelector(".chat-empty");
|
||||
if (empty) empty.remove();
|
||||
const el = document.createElement("div");
|
||||
el.className = "hist-msg tool";
|
||||
el.innerHTML = `<div class="hist-role">Snapshot</div><img class="hist-image" alt="Webcam snapshot sent to the model" />`;
|
||||
const img = /** @type {HTMLImageElement} */ (el.querySelector("img"));
|
||||
img.src = dataUrl;
|
||||
this._chatHistory.appendChild(el);
|
||||
this._scrollToBottom();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset all streaming bookkeeping for session start / teardown. Pass
|
||||
* `dismiss` to also fade any bubbles still on screen.
|
||||
* @param {{ dismiss?: boolean }} [opts]
|
||||
*/
|
||||
reset(opts) {
|
||||
if (opts?.dismiss) {
|
||||
if (this._activeUserBubble) this._dismissBubble(this._activeUserBubble);
|
||||
for (const { bubble } of this._asstByResp.values()) this._dismissBubble(bubble);
|
||||
}
|
||||
this._userHistByItem.clear();
|
||||
this._activeUserBubble = null;
|
||||
this._activeUserItemId = "";
|
||||
this._asstByResp.clear();
|
||||
}
|
||||
|
||||
// ── Client event handlers ─────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* A streamed transcript delta (user or assistant).
|
||||
* @param {{ role: "user" | "assistant"; text: string; partial: boolean; itemId?: string; responseId?: string }} d
|
||||
*/
|
||||
onTranscript(d) {
|
||||
if (DEBUG) console.debug(`[ui] transcript role=${d.role} partial=${d.partial} item=${d.itemId} resp=${d.responseId} text=${JSON.stringify(d.text)}`);
|
||||
|
||||
if (d.role === "user") {
|
||||
// Group by item_id: a speculative continuation reuses the same id, so it
|
||||
// updates the same row/bubble. A missing id falls back to the active item
|
||||
// (same utterance) or a fresh unique key, never a shared sentinel that
|
||||
// would collapse distinct turns into one row.
|
||||
const id = d.itemId || this._activeUserItemId || `_u${++this._anonSeq}`;
|
||||
const text = d.text;
|
||||
|
||||
let hist = this._userHistByItem.get(id);
|
||||
if (!hist) {
|
||||
hist = this._appendHistMsg("user", text, d.partial);
|
||||
this._userHistByItem.set(id, hist);
|
||||
} else {
|
||||
this._updateHistMsg(hist, text, d.partial);
|
||||
}
|
||||
|
||||
// One ephemeral bubble per active item. Purely timer-based: the timer is
|
||||
// refreshed on every delta, so it stays while the user keeps talking and
|
||||
// fades a few seconds after they stop — no dependency on a response ever
|
||||
// arriving, so it can never get stuck.
|
||||
if (this._activeUserItemId !== id || !this._activeUserBubble) {
|
||||
this._activeUserBubble = this._spawnBubble("user", text);
|
||||
this._activeUserItemId = id;
|
||||
} else {
|
||||
this._updateBubbleText(this._activeUserBubble, text);
|
||||
}
|
||||
this._bumpDismiss(this._activeUserBubble, 6000);
|
||||
this._markUnread();
|
||||
} else if (d.role === "assistant") {
|
||||
// Assistant transcript arrives once, as the full text, keyed by
|
||||
// response_id so a cancelled speculative response can be removed later. A
|
||||
// missing id gets a unique key so two id-less replies never collide.
|
||||
const rid = d.responseId || `_a${++this._anonSeq}`;
|
||||
const entry = this._asstByResp.get(rid);
|
||||
if (!entry) {
|
||||
const bubble = this._spawnBubble("assistant", d.text);
|
||||
this._asstByResp.set(rid, { bubble, hist: this._appendHistMsg("assistant", d.text, false) });
|
||||
this._bumpDismiss(bubble);
|
||||
} else {
|
||||
this._updateBubbleText(entry.bubble, d.text);
|
||||
this._updateHistMsg(entry.hist, d.text, false);
|
||||
this._bumpDismiss(entry.bubble);
|
||||
}
|
||||
this._markUnread();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A response closed (completed or cancelled).
|
||||
* @param {{ responseId: string; status: string; audible?: boolean; transcript?: string }} detail
|
||||
*/
|
||||
onResponseFinished(detail) {
|
||||
const { responseId, status, audible, transcript } = detail;
|
||||
if (DEBUG) console.debug(`[ui] response-finished resp=${responseId} status=${status} audible=${audible} known=${this._asstByResp.has(responseId)}`);
|
||||
// Without an id we can't target a specific response; the bubble will
|
||||
// auto-dismiss on its own timer regardless.
|
||||
if (!responseId) return;
|
||||
const entry = this._asstByResp.get(responseId);
|
||||
|
||||
if (status === "cancelled") {
|
||||
// Keep every transcript that was received — mark it interrupted rather
|
||||
// than erasing it. If the `*.transcript.done` never fired, build the row
|
||||
// from the text carried in response.done.
|
||||
let hist = entry?.hist ?? null;
|
||||
if (!hist && transcript) {
|
||||
hist = this._appendHistMsg("assistant", transcript, false);
|
||||
} else if (hist && transcript) {
|
||||
this._updateHistMsg(hist, transcript, false);
|
||||
}
|
||||
if (hist) this._markHistInterrupted(hist);
|
||||
this._asstByResp.delete(responseId);
|
||||
return;
|
||||
}
|
||||
|
||||
// Any other terminal close (completed / failed / incomplete / …): just
|
||||
// release the map entry. The bubble already auto-dismisses on its timer and
|
||||
// the history row persists as the conversation log. Crucially we do NOT
|
||||
// touch user state here — that lifecycle is fully independent.
|
||||
this._asstByResp.delete(responseId);
|
||||
}
|
||||
|
||||
/** The model called a tool — show an ephemeral "running" bubble.
|
||||
* @param {string} name */
|
||||
onToolCall(name) {
|
||||
this._bumpDismiss(this._spawnBubble("tool", name));
|
||||
this._markUnread();
|
||||
}
|
||||
|
||||
/** The tool finished — append its call+result row (and any captured image).
|
||||
* @param {string} name @param {string} argsJson @param {string} output @param {string} [image] */
|
||||
onToolResult(name, argsJson, output, image) {
|
||||
this._appendHistTool(name, argsJson, output);
|
||||
if (image) this._appendHistImage(image); // show the captured frame below the call
|
||||
this._markUnread();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
// @ts-check
|
||||
/** Small shared helpers used across the UI modules: a strict query selector,
|
||||
* HTML escaping for text we drop into innerHTML, error-string trimming, and
|
||||
* the opt-in debug flag. */
|
||||
|
||||
/** Opt-in tracing: `localStorage.setItem("s2s.debug", "1")` then reload. */
|
||||
export const DEBUG = (() => {
|
||||
try {
|
||||
return localStorage.getItem("s2s.debug") === "1";
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})();
|
||||
|
||||
/**
|
||||
* Query a single element, throwing if it's missing (so a broken selector fails
|
||||
* loudly at startup rather than as a later null-deref).
|
||||
* @template {HTMLElement} T
|
||||
* @param {string} selector
|
||||
* @returns {T}
|
||||
*/
|
||||
export function $(selector) {
|
||||
const el = document.querySelector(selector);
|
||||
if (!el) throw new Error(`Missing element: ${selector}`);
|
||||
return /** @type {T} */ (el);
|
||||
}
|
||||
|
||||
/** @param {string} s @returns {string} */
|
||||
export function escHtml(s) {
|
||||
return s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
|
||||
}
|
||||
|
||||
/** Trim a long error message to fit the orb caption. @param {string} text */
|
||||
export function truncateError(text) {
|
||||
if (text.length <= 90) return text;
|
||||
return text.slice(0, 87) + "…";
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* AudioWorkletProcessor that plays back Float32 mono samples received from
|
||||
* the main thread, upsampling whatever incoming rate the server uses
|
||||
* (typically 24 kHz PCM16) to the AudioContext rate (typically 48 kHz).
|
||||
*
|
||||
* Lifecycle / messaging:
|
||||
*
|
||||
* main -> worklet:
|
||||
* { kind: "config", inputRate: 24000 } one-shot at startup
|
||||
* { kind: "audio", samples: Float32Array } (transferable) per chunk
|
||||
* { kind: "clear" } wipe queue (barge-in)
|
||||
*
|
||||
* worklet -> main:
|
||||
* { kind: "stats", queuedMs, played } every ~250 ms
|
||||
* { kind: "underrun" } every time the queue
|
||||
* runs dry mid-playback
|
||||
*
|
||||
* Underrun strategy: output silence. We do NOT hold the last sample (that
|
||||
* tends to produce audible clicks/buzzes when long gaps appear between
|
||||
* TTS chunks). A short ramp-out + ramp-in at boundaries would be nicer but
|
||||
* the server's 30 ms cadence makes underruns visible only at end of turn.
|
||||
*/
|
||||
|
||||
const STATS_INTERVAL_FRAMES = 12000;
|
||||
const FADE_FRAMES = 32;
|
||||
|
||||
class AudioPlaybackProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this._inputRate = 24000;
|
||||
this._stepRatio = this._inputRate / sampleRate;
|
||||
this._queue = [];
|
||||
this._readIdx = 0;
|
||||
this._fracPos = 0;
|
||||
this._playing = false;
|
||||
this._framesSinceStats = 0;
|
||||
this._totalPlayed = 0;
|
||||
this._fadeIn = 0;
|
||||
this._fadeOut = 0;
|
||||
this._lastSample = 0;
|
||||
|
||||
this.port.onmessage = (e) => {
|
||||
const data = e.data;
|
||||
if (!data || typeof data !== "object") return;
|
||||
switch (data.kind) {
|
||||
case "config":
|
||||
if (typeof data.inputRate === "number" && data.inputRate > 0) {
|
||||
this._inputRate = data.inputRate;
|
||||
this._stepRatio = this._inputRate / sampleRate;
|
||||
}
|
||||
break;
|
||||
case "audio":
|
||||
if (data.samples instanceof Float32Array && data.samples.length > 0) {
|
||||
this._queue.push(data.samples);
|
||||
if (!this._playing) {
|
||||
this._playing = true;
|
||||
this._fadeIn = FADE_FRAMES;
|
||||
this._fadeOut = 0;
|
||||
}
|
||||
}
|
||||
break;
|
||||
case "clear":
|
||||
this._queue.length = 0;
|
||||
this._readIdx = 0;
|
||||
this._fracPos = 0;
|
||||
this._fadeOut = FADE_FRAMES;
|
||||
break;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
_queuedSamples() {
|
||||
let total = -this._readIdx;
|
||||
for (const buf of this._queue) total += buf.length;
|
||||
return Math.max(0, total);
|
||||
}
|
||||
|
||||
/** Linear-interp read at the current fractional position. */
|
||||
_readInterpolated() {
|
||||
if (this._queue.length === 0) return null;
|
||||
const head = this._queue[0];
|
||||
const idx = this._readIdx;
|
||||
const frac = this._fracPos;
|
||||
|
||||
let a = head[idx];
|
||||
let b;
|
||||
if (idx + 1 < head.length) {
|
||||
b = head[idx + 1];
|
||||
} else if (this._queue.length > 1) {
|
||||
b = this._queue[1][0];
|
||||
} else {
|
||||
b = a;
|
||||
}
|
||||
return a + (b - a) * frac;
|
||||
}
|
||||
|
||||
/** Advance the read position by `stepRatio`; pop consumed buffers. */
|
||||
_advance() {
|
||||
this._fracPos += this._stepRatio;
|
||||
while (this._fracPos >= 1) {
|
||||
this._fracPos -= 1;
|
||||
this._readIdx += 1;
|
||||
}
|
||||
while (this._queue.length > 0 && this._readIdx >= this._queue[0].length) {
|
||||
this._readIdx -= this._queue[0].length;
|
||||
this._queue.shift();
|
||||
}
|
||||
}
|
||||
|
||||
process(_, outputs) {
|
||||
const channels = outputs[0];
|
||||
if (!channels || channels.length === 0) return true;
|
||||
const out = channels[0];
|
||||
const stereo = channels.length > 1 ? channels[1] : null;
|
||||
|
||||
for (let i = 0; i < out.length; i++) {
|
||||
let sample = 0;
|
||||
|
||||
if (this._playing) {
|
||||
const v = this._readInterpolated();
|
||||
if (v === null) {
|
||||
// Underrun: try to ramp out cleanly to avoid clicks.
|
||||
sample = this._lastSample * Math.max(0, 1 - 1 / FADE_FRAMES);
|
||||
this._lastSample = sample;
|
||||
if (Math.abs(sample) < 1e-4) {
|
||||
this._playing = false;
|
||||
this._lastSample = 0;
|
||||
this.port.postMessage({ kind: "underrun" });
|
||||
}
|
||||
} else {
|
||||
sample = v;
|
||||
this._lastSample = v;
|
||||
this._advance();
|
||||
}
|
||||
|
||||
if (this._fadeIn > 0) {
|
||||
const gain = 1 - this._fadeIn / FADE_FRAMES;
|
||||
sample *= gain;
|
||||
this._fadeIn -= 1;
|
||||
}
|
||||
if (this._fadeOut > 0) {
|
||||
const gain = this._fadeOut / FADE_FRAMES;
|
||||
sample *= gain;
|
||||
this._fadeOut -= 1;
|
||||
if (this._fadeOut === 0) {
|
||||
this._playing = false;
|
||||
this._lastSample = 0;
|
||||
}
|
||||
}
|
||||
|
||||
this._totalPlayed += 1;
|
||||
}
|
||||
|
||||
out[i] = sample;
|
||||
if (stereo) stereo[i] = sample;
|
||||
}
|
||||
|
||||
this._framesSinceStats += out.length;
|
||||
if (this._framesSinceStats >= STATS_INTERVAL_FRAMES) {
|
||||
this._framesSinceStats = 0;
|
||||
const queuedSamples = this._queuedSamples();
|
||||
const queuedMs = (queuedSamples / this._inputRate) * 1000;
|
||||
this.port.postMessage({ kind: "stats", queuedMs, played: this._totalPlayed });
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor("audio-playback", AudioPlaybackProcessor);
|
||||
@@ -0,0 +1,159 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* AudioWorkletProcessor that resamples the AudioContext rate (typically 48 kHz)
|
||||
* down to 16 kHz, packs the result as little-endian Int16 PCM, and posts it
|
||||
* back to the main thread in fixed-size chunks.
|
||||
*
|
||||
* The Hugging Face speech-to-speech WebSocket route expects the
|
||||
* `input_audio_buffer.append` payload at 16 kHz PCM16 mono.
|
||||
*
|
||||
* Design notes:
|
||||
* - 48 -> 16 is an exact 3:1 ratio so we use a 3-tap boxcar average as a
|
||||
* cheap low-pass before decimating. Good enough for voice STT; we lose
|
||||
* a tiny bit of >8 kHz content which the pipeline discards anyway.
|
||||
* - Output frames are emitted at the cadence dictated by `chunkMs`
|
||||
* (default 40 ms = 640 samples = 1280 bytes). The OpenAI Realtime
|
||||
* server batches incoming audio so the cadence is flexible; 20-100 ms
|
||||
* is the sweet spot.
|
||||
* - Float -> Int16 saturates to [-1, 1] before scaling.
|
||||
* - Optional noise gate: per-chunk RMS decides open/closed against a
|
||||
* threshold; the gain ramps (fast attack, hold, slow release) so word
|
||||
* onsets aren't clipped and quiet tails don't click. The gate only
|
||||
* affects the audio we SEND; the main-thread visualiser taps the raw
|
||||
* mic separately. We post the chunk RMS up every frame so the Settings
|
||||
* mic meter can show the live level against the threshold.
|
||||
*/
|
||||
|
||||
const TARGET_RATE = 16000;
|
||||
const DEFAULT_CHUNK_MS = 40;
|
||||
// Gate envelope timing (fixed; only the threshold is user-tunable).
|
||||
const GATE_ATTACK_MS = 5; // open almost instantly so word onsets survive
|
||||
const GATE_HOLD_MS = 250; // stay open this long after the level drops back under
|
||||
const GATE_RELEASE_MS = 80; // then fade closed over this long (no click)
|
||||
|
||||
class MicCaptureProcessor extends AudioWorkletProcessor {
|
||||
constructor(options) {
|
||||
super();
|
||||
const chunkMs = options?.processorOptions?.chunkMs ?? DEFAULT_CHUNK_MS;
|
||||
this._inputRate = sampleRate;
|
||||
this._ratio = this._inputRate / TARGET_RATE;
|
||||
this._chunkSamples16k = Math.round((TARGET_RATE * chunkMs) / 1000);
|
||||
this._scratch = new Float32Array(0);
|
||||
this._decimated = new Float32Array(this._chunkSamples16k);
|
||||
this._enabled = true;
|
||||
|
||||
// Noise gate state. Disabled by default (pure passthrough).
|
||||
this._gateEnabled = false;
|
||||
this._thresholdLin = 0; // linear amplitude; signal RMS must exceed this to open
|
||||
this._gateGain = 1; // smoothed gain currently applied
|
||||
this._holdRemaining = 0; // samples left before the gate may start closing
|
||||
this._attackCoef = Math.exp(-1 / ((GATE_ATTACK_MS / 1000) * TARGET_RATE));
|
||||
this._releaseCoef = Math.exp(-1 / ((GATE_RELEASE_MS / 1000) * TARGET_RATE));
|
||||
this._holdSamples = Math.round((GATE_HOLD_MS / 1000) * TARGET_RATE);
|
||||
|
||||
this.port.onmessage = (e) => {
|
||||
const data = e.data;
|
||||
if (data?.kind === "enable") this._enabled = !!data.value;
|
||||
else if (data?.kind === "gate") {
|
||||
this._gateEnabled = !!data.enabled;
|
||||
// dB -> linear amplitude. When off, threshold 0 keeps the gate open.
|
||||
this._thresholdLin = data.enabled ? Math.pow(10, data.thresholdDb / 20) : 0;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Append `incoming` to the internal scratch buffer, then emit as many
|
||||
* full output chunks as we have material for.
|
||||
* @param {Float32Array} incoming
|
||||
*/
|
||||
_ingest(incoming) {
|
||||
if (incoming.length === 0) return;
|
||||
const next = new Float32Array(this._scratch.length + incoming.length);
|
||||
next.set(this._scratch, 0);
|
||||
next.set(incoming, this._scratch.length);
|
||||
this._scratch = next;
|
||||
this._maybeEmit();
|
||||
}
|
||||
|
||||
_maybeEmit() {
|
||||
const r = this._ratio;
|
||||
const n = this._chunkSamples16k;
|
||||
const needIn = Math.ceil(n * r);
|
||||
const dec = this._decimated;
|
||||
while (this._scratch.length >= needIn) {
|
||||
// 1. Decimate to 16 kHz floats and accumulate energy for the gate/meter.
|
||||
let sumSq = 0;
|
||||
if (Math.abs(r - 3) < 1e-6) {
|
||||
// 48 kHz -> 16 kHz fast path with boxcar lowpass.
|
||||
for (let i = 0; i < n; i++) {
|
||||
const idx = i * 3;
|
||||
const s = (this._scratch[idx] + this._scratch[idx + 1] + this._scratch[idx + 2]) / 3;
|
||||
dec[i] = s;
|
||||
sumSq += s * s;
|
||||
}
|
||||
} else {
|
||||
// Generic path: linear interpolation. Slower but works at any rate
|
||||
// (e.g. some Windows boxes report sampleRate=44100).
|
||||
for (let i = 0; i < n; i++) {
|
||||
const srcPos = i * r;
|
||||
const idx = Math.floor(srcPos);
|
||||
const frac = srcPos - idx;
|
||||
const a = this._scratch[idx];
|
||||
const b = this._scratch[idx + 1] ?? a;
|
||||
const s = a + (b - a) * frac;
|
||||
dec[i] = s;
|
||||
sumSq += s * s;
|
||||
}
|
||||
}
|
||||
const rms = Math.sqrt(sumSq / n);
|
||||
|
||||
// 2. Decide the gate target for this chunk, then ramp sample-by-sample.
|
||||
let target = 1;
|
||||
if (this._gateEnabled) {
|
||||
if (rms >= this._thresholdLin) {
|
||||
this._holdRemaining = this._holdSamples; // re-arm the hold
|
||||
} else if (this._holdRemaining > 0) {
|
||||
this._holdRemaining -= n; // coasting through the hold window
|
||||
} else {
|
||||
target = 0;
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Apply the (smoothed) gain and pack to Int16.
|
||||
const out = new Int16Array(n);
|
||||
let gain = this._gateGain;
|
||||
for (let i = 0; i < n; i++) {
|
||||
const coef = target > gain ? this._attackCoef : this._releaseCoef;
|
||||
gain = target + (gain - target) * coef;
|
||||
const s = dec[i] * gain;
|
||||
const clamped = s < -1 ? -1 : s > 1 ? 1 : s;
|
||||
out[i] = clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff;
|
||||
}
|
||||
this._gateGain = gain;
|
||||
|
||||
// Shift the scratch buffer to keep only the trailing unused samples.
|
||||
const consumed = Math.floor(n * r);
|
||||
this._scratch = this._scratch.slice(consumed);
|
||||
|
||||
// Live input level for the Settings meter (raw RMS, pre-gate).
|
||||
this.port.postMessage({ kind: "level", rms });
|
||||
|
||||
if (this._enabled) {
|
||||
this.port.postMessage(out.buffer, [out.buffer]);
|
||||
}
|
||||
// When disabled (mic muted) we silently consume input so the worklet
|
||||
// stays alive and the buffer never grows unbounded.
|
||||
}
|
||||
}
|
||||
|
||||
process(inputs) {
|
||||
const input = inputs[0];
|
||||
if (!input || input.length === 0 || !input[0]) return true;
|
||||
const mono = input[0];
|
||||
if (mono.length > 0) this._ingest(mono);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor("mic-capture", MicCaptureProcessor);
|
||||
@@ -0,0 +1,57 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* Pure, stateless helpers for the WebSocket realtime client: base64 <-> PCM
|
||||
* conversion for the audio frames on the wire, transcript extraction from a
|
||||
* `response.done` payload, and a tiny URL helper. Kept separate from the client
|
||||
* so the protocol/state logic stays readable.
|
||||
*/
|
||||
|
||||
/** @param {string} url */
|
||||
export function trimTrailingSlash(url) {
|
||||
return url.endsWith("/") ? url.slice(0, -1) : url;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull the assistant transcript out of a `response.done` payload. The text
|
||||
* lives in `response.output[].content[].transcript` (audio) or `.text`. Used as
|
||||
* the source of truth for interrupted replies, where the dedicated
|
||||
* `*.transcript.done` event may never arrive.
|
||||
* @param {any} response
|
||||
* @returns {string}
|
||||
*/
|
||||
export function extractResponseTranscript(response) {
|
||||
const output = response?.output;
|
||||
if (!Array.isArray(output)) return "";
|
||||
/** @type {string[]} */
|
||||
const parts = [];
|
||||
for (const item of output) {
|
||||
for (const part of item?.content ?? []) {
|
||||
const text = part?.transcript ?? part?.text;
|
||||
if (typeof text === "string" && text.trim()) parts.push(text.trim());
|
||||
}
|
||||
}
|
||||
return parts.join(" ").trim();
|
||||
}
|
||||
|
||||
/** @param {ArrayBuffer} buf */
|
||||
export function base64FromArrayBuffer(buf) {
|
||||
const bytes = new Uint8Array(buf);
|
||||
// Chunked encoding so we don't blow up the call stack on long buffers.
|
||||
let binary = "";
|
||||
const chunk = 0x8000;
|
||||
for (let i = 0; i < bytes.length; i += chunk) {
|
||||
binary += String.fromCharCode.apply(null, /** @type {number[]} */ (
|
||||
/** @type {unknown} */ (bytes.subarray(i, i + chunk))
|
||||
));
|
||||
}
|
||||
return btoa(binary);
|
||||
}
|
||||
|
||||
/** @param {string} b64 */
|
||||
export function base64ToBytes(b64) {
|
||||
const binary = atob(b64);
|
||||
const len = binary.length;
|
||||
const out = new Uint8Array(len);
|
||||
for (let i = 0; i < len; i++) out[i] = binary.charCodeAt(i);
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* Orb spectrum visualiser. Each animation frame it reads two AnalyserNodes (the
|
||||
* mic input and the TTS output) and maps the low-frequency speech energy onto
|
||||
* the orb's CSS custom properties:
|
||||
* - `--bar0`..`--bar4` the 5-band level meter
|
||||
* - `--ai-audio-level` the global "Reachy talks" glow / scale pulse
|
||||
*
|
||||
* The bottom of the FFT is where speech energy lives, so the band edges stay
|
||||
* low — that keeps the bars dancing on voice rather than on noise. While the AI
|
||||
* is speaking we source the bars from the OUTPUT analyser so the orb pulses with
|
||||
* Reachy's voice instead of sitting dead while the user is silent.
|
||||
*/
|
||||
|
||||
// Exported so the client can size its AnalyserNodes to match our buffer.
|
||||
export const VIS_FFT_SIZE = 256;
|
||||
const VIS_BAND_COUNT = 5;
|
||||
const VIS_BAND_EDGES = [2, 5, 9, 16, 28, 52];
|
||||
const VIS_ATTACK = 0.6; // weight for new sample on upswing (snappy)
|
||||
const VIS_RELEASE = 0.18; // weight for new sample on decay (gentle fade)
|
||||
|
||||
export class OrbVisualiser {
|
||||
/**
|
||||
* @param {AnalyserNode} micAnalyser
|
||||
* @param {AnalyserNode} outAnalyser
|
||||
* @param {() => boolean} isAiSpeaking Source the bars from the AI output when
|
||||
* true, otherwise from the mic.
|
||||
*/
|
||||
constructor(micAnalyser, outAnalyser, isAiSpeaking) {
|
||||
this._mic = micAnalyser;
|
||||
this._out = outAnalyser;
|
||||
this._isAiSpeaking = isAiSpeaking;
|
||||
this._buf = new Uint8Array(micAnalyser.frequencyBinCount);
|
||||
this._bands = new Float32Array(VIS_BAND_COUNT);
|
||||
this._aiLevel = 0;
|
||||
/** @type {number | null} */
|
||||
this._frame = null;
|
||||
}
|
||||
|
||||
/** Begin the rAF loop (idempotent). */
|
||||
start() {
|
||||
if (this._frame !== null) return;
|
||||
const root = document.documentElement;
|
||||
const tick = () => {
|
||||
this._frame = requestAnimationFrame(tick);
|
||||
this._update(root);
|
||||
};
|
||||
this._frame = requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
/** Stop the loop and clear the CSS vars so the orb returns to rest. */
|
||||
stop() {
|
||||
if (this._frame !== null) {
|
||||
cancelAnimationFrame(this._frame);
|
||||
this._frame = null;
|
||||
}
|
||||
const root = document.documentElement;
|
||||
for (let i = 0; i < VIS_BAND_COUNT; i++) root.style.removeProperty(`--bar${i}`);
|
||||
root.style.removeProperty("--ai-audio-level");
|
||||
}
|
||||
|
||||
/** @param {HTMLElement} root */
|
||||
_update(root) {
|
||||
// Mic bars: split FFT into 5 log-ish bands, smooth, write CSS vars.
|
||||
const source = this._isAiSpeaking() ? this._out : this._mic;
|
||||
source.getByteFrequencyData(this._buf);
|
||||
|
||||
for (let b = 0; b < VIS_BAND_COUNT; b++) {
|
||||
const lo = VIS_BAND_EDGES[b];
|
||||
const hi = VIS_BAND_EDGES[b + 1];
|
||||
let sum = 0;
|
||||
let n = 0;
|
||||
for (let i = lo; i < hi && i < this._buf.length; i++) {
|
||||
sum += this._buf[i];
|
||||
n += 1;
|
||||
}
|
||||
const target = n > 0 ? sum / (n * 255) : 0;
|
||||
const prev = this._bands[b];
|
||||
const k = target > prev ? VIS_ATTACK : VIS_RELEASE;
|
||||
const next = prev + (target - prev) * k;
|
||||
this._bands[b] = next;
|
||||
root.style.setProperty(`--bar${b}`, next.toFixed(3));
|
||||
}
|
||||
|
||||
// Global AI audio level: peak of the output analyser, used by the CSS to
|
||||
// make the orb's glow / scale react to Reachy's voice.
|
||||
this._out.getByteFrequencyData(this._buf);
|
||||
let peak = 0;
|
||||
const limit = Math.min(this._buf.length, VIS_BAND_EDGES[VIS_BAND_COUNT]);
|
||||
for (let i = 0; i < limit; i++) {
|
||||
if (this._buf[i] > peak) peak = this._buf[i];
|
||||
}
|
||||
const aiTarget = peak / 255;
|
||||
const k = aiTarget > this._aiLevel ? VIS_ATTACK : VIS_RELEASE;
|
||||
this._aiLevel = this._aiLevel + (aiTarget - this._aiLevel) * k;
|
||||
root.style.setProperty("--ai-audio-level", this._aiLevel.toFixed(3));
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user