docs: consolidate English documentation under root docs
This commit is contained in:
@@ -1,188 +0,0 @@
|
||||
# Demo S2S — schema di funzionamento
|
||||
|
||||
> Documento orientativo del modulo `demo/` (UI + proxy S2S + reasoning). Non sostituisce
|
||||
> il codice: serve a trovare il proprio posto quando si apre il progetto.
|
||||
|
||||
---
|
||||
|
||||
## 1. Cosa c'è in `demo/`
|
||||
|
||||
Il modulo fa **tre cose contemporaneamente**:
|
||||
|
||||
| Ruolo | Cosa fa | Entry point |
|
||||
|---|---|---|
|
||||
| **HTTP server statico + API** | Serve l'UI, offre endpoint REST, gira un WebSocket proxy | [server.py](file:///home/duxiter/s2s/demo/server.py) |
|
||||
| **Voice UI (browser)** | Orba, mic, transcriptioni, audio playback, pannelli | [index.html](file:///home/duxiter/s2s/demo/index.html) + [main.js](file:///home/duxiter/s2s/demo/main.js) |
|
||||
| **Reasoning hook** | Inoltra transcript utente a un motore di ragionamento esterno (NODE_RED o OpenAI) | [server.py](file:///home/duxiter/s2s/demo/server.py#L658-L834) `/api/reasoning` |
|
||||
|
||||
La configurazione (URL backend S2S, tipo di reasoning, chiavi API) viene presa dal file `.env` nella root del progetto.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architettura in un diagramma
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ BROWSER (demo UI) │
|
||||
│ [index.html] → [main.js] (ESM type=module) │
|
||||
│ │ ▲ │
|
||||
│ │ audio + WS events │ transcript chunks │
|
||||
│ ▼ │ │
|
||||
│ ┌──────────────────────────────────────────┐ │
|
||||
│ │ s2s-ws-client.js → WebSocket /v1/realtime │ │
|
||||
│ │ + codec.js + orb-visualizer.js │ │
|
||||
│ └──────────────────────────────────────────┘ │
|
||||
└─────────────────────────┬────────────────────────┘
|
||||
│ WebSocket /v1/realtime
|
||||
▼
|
||||
┌───────────────────────────────────────────────┐
|
||||
│ server.py (FastAPI) │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────┐ │
|
||||
│ │ realtime_proxy ← TUNNEL → upstream │ │
|
||||
│ │ (WS relay puro, copia byte-per-byte)│ │
|
||||
│ │ _client_to_upstream ─┐ │ │
|
||||
│ │ _upstream_to_client ─┤ session_ctx │ │
|
||||
│ │ session_ctx{user,id} │ shared dict │ │
|
||||
│ └───────────────────────┴──────────────┘ │
|
||||
│ │
|
||||
│ endpoint REST: │
|
||||
│ GET / → index.html │
|
||||
│ GET /api/config → env flags │
|
||||
│ POST /api/reasoning → NODE_RED / │
|
||||
│ OPENAI │
|
||||
│ POST /api/auth/* → Aquachile │
|
||||
│ │
|
||||
└──────┬──────────────────────────────┬─────────┘
|
||||
│ POST /api/reasoning │ fire-and-forget
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌───────────────────────────────┐
|
||||
│ Node-RED (port 1881)│ │ Node-RED (port 1881) │
|
||||
│ POST /api/reasoning │ │ POST /api/log/system-response │
|
||||
│ log user transcript │ │ log assistant TTS + pairing │
|
||||
└──────────────────────┘ └───────────────────────────────┘
|
||||
^
|
||||
│ forward 4 eventi realtime:
|
||||
│ audio_transcript.done
|
||||
│ content_part.done
|
||||
│ output_item.done role=assistant
|
||||
│ conversation.item.created role=assistant
|
||||
```
|
||||
|
||||
> ### 📝 Nota importante sul flusso AUDIO con l'attuale `.env`
|
||||
>
|
||||
> Con `.env` attuale che ha `S2S_PROXY_TARGET=ws://ai-server:12345/v1/realtime`:
|
||||
>
|
||||
> ✅ **Sì, TUTTO l'AUDIO utente (PCM16 16kHz mono, eventi `input_audio_buffer.append`) arriva a `ws://ai-server:12345/v1/realtime`** — ma **non direttamente**: passa SEMPRE per il **tunnel 1:1 WebSocket di `server.py`** (`_client_to_upstream`), che copia byte-per-byte senza modifiche.
|
||||
>
|
||||
> ✅ Allo stesso modo l'AUDIO risposta (TTS, `response.output_audio.delta`) ritorna da `ai-server:12345` → tunnel `_upstream_to_client` → browser → playback nativo.
|
||||
>
|
||||
> ❌ **Cosa invece NON va a `ai-server:12345`** (canale HTTP separato parallelo, non tocca l'audio):
|
||||
> - Il **transcript STT** viene mandato via HTTP a `http://localhost:1881/api/reasoning` (Node-RED, hook LOG_ONLY oggi)
|
||||
> - Il **testo risposta TTS** viene mandato via HTTP a `http://localhost:1881/api/log/system-response` (Node-RED, forward fire-and-forget)
|
||||
> - Chiamate **auth / profilo Aquachile** verso `aquachile.azurianlab.com`
|
||||
>
|
||||
> Questi 3 flussi REST sono **aggiuntivi** e **non alterano minimamente il tunnel audio**: browser ↔ server.py ↔ `ai-server:12345` rimane il percorso esclusivo per voce e risposta.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cose importanti di [server.py](file:///home/duxiter/s2s/demo/server.py)
|
||||
|
||||
### File → ruoli
|
||||
- **`/`** → StaticFast `StaticFiles` su `demo/` (index.html + assets)
|
||||
- **`@app.middleware http` no_cache_static_assets** → `Cache-Control: no-store` per `.js/.html/.css/.mjs/.wasm` per sconfiggere cache browser durante sviluppo
|
||||
- **`@app.get /api/config`** → manda al browser variabili env (`S2S_REASONING_TYPE`, `S2S_PROXY_TARGET`, auth, features). Il client decida da sé cosa mostrare in UI.
|
||||
- **`@app.post /api/reasoning` ([L658-L834](file:///home/duxiter/s2s/demo/server.py#L658-L834))** → cuore del reasoning. Due modalità, scelta da env `S2S_REASONING_TYPE`:
|
||||
- **Modalità A — NODE_RED**: inoltra `POST {NODE_RED_URL}/api/reasoning` timeout 30s. Accetta `log_only=true` + `answer=""` (Fase 1).
|
||||
- **Modalità B — OPENAI**: chiamata diretta `POST $OPENAI_BASE_URL/chat/completions` (Fasi successive).
|
||||
- **`@app.websocket /v1/realtime` ([L874-L980](file:///home/duxiter/s2s/demo/server.py#L874-L980))** → **tunnel S2S nativo**. Non traduce messaggi: copia + registra + forwarda. 2 task paralleli:
|
||||
- `_client_to_upstream(ws, upstream, client_ip, session_ctx)`: browser → AI server
|
||||
- `_upstream_to_client(ws, upstream, client_ip, session_ctx)`: AI server → browser
|
||||
- `session_ctx = {last_user_transcript, last_item_id}` **dizionario mutabile condiviso** per accoppiare input↔output
|
||||
- **`async def _forward_agent_text_to_nodered` ([L837-L873](file:///home/duxiter/s2s/demo/server.py#L837-L873))** → helper fire-and-forget. Tutti errori → `logger.debug` (non spezza il tunnel WS).
|
||||
- **`_upstream_to_client` hook eventi ([L1305-L1500](file:///home/duxiter/s2s/demo/server.py#L1305-L1500))**: per ogni evento output agente avvia `asyncio.create_task(_forward_agent_text_to_nodered(...))` → non blocca lo streaming audio.
|
||||
- **Sidecar Python locali**: [auth.py](file:///home/duxiter/s2s/demo/auth.py) (Aquachile profile+user+phone), [limiter.py](file:///home/duxiter/s2s/demo/limiter.py) (rate limiter).
|
||||
|
||||
### Ordine di chiamata in un turno vocale
|
||||
1. Browser clicca orb → `ws /v1/realtime` → `realtime_proxy()` accetta
|
||||
2. Browser manda `session.update` → `input_audio_buffer.append` in loop
|
||||
3. Upstream genera `conversation.item.input_audio_transcription.completed` → server cattura → **`session_ctx["last_user_transcript"] = transcript`**
|
||||
4. `_upstream_to_client` vede transcript STT completo → server invoca in parallelo **`POST /api/reasoning`** con quel transcript
|
||||
5. Upstream manda `response.output_audio.delta` audio chunks → tunnelati diretti al browser → playback nativo
|
||||
6. Upstream manda `response.output_audio_transcript.done` → server cattura → `create_task(_forward_agent_text_to_nodered({tts, source_event, user_transcript, …}))` → Node-RED SYSTEM_RESPONSE log
|
||||
7. Audio finisce → turno chiuso
|
||||
|
||||
---
|
||||
|
||||
## 4. Cose importanti del lato client ([main.js](file:///home/duxiter/s2s/demo/main.js))
|
||||
|
||||
### Moduli ESM importati (tutti con `?v=VERSIONE` per cache-bust)
|
||||
```
|
||||
main.js
|
||||
├─ dom.js → helper DOM (create, qs, bubble, ecc.)
|
||||
├─ chat.js → pannello messaggi + appendAssistantText
|
||||
├─ account.js → pannello auth + profilo utente
|
||||
└─ ws/s2s-ws-client.js
|
||||
├─ codec.js → codec pcm + base64
|
||||
└─ orb-visualizer.js → animazione orba su eventi volume/state
|
||||
```
|
||||
|
||||
### Funzioni chiave
|
||||
| Funzione | Ruolo |
|
||||
|---|---|
|
||||
| `fetchOpenAiReasoningAnswer(transcript)` | `POST /api/reasoning`, ritorna `{answer, log_only, raw}` |
|
||||
| `maybeRunOpenAiReasoning(transcript, itemId)` | guardia "ho già processato questo turn?" (`processedReasoningTurns`). Se `result.log_only === true` → **early return, zero side-effect**. Altrimenti branch standard (Fasi successive). |
|
||||
| `cancelActiveResponse()` | Fasi successive: ferma risposta corrente prima di inject custom |
|
||||
| `appendAssistantText(answer)` / `sendUserText(renderMarker)+requestResponse` | Fasi successive: sostituisce testo risposta backend nativo |
|
||||
| `speechSynthesis.speak(utterance)` | Fasi successive: TTS *browser* per answer custom, invece del backend nativo |
|
||||
|
||||
### Regole per non rompere Fase 1
|
||||
- Se `S2S_REASONING_TYPE=NODE_RED` **e** Node-RED torna `log_only=true` → il client non tocca nulla. Zero `cancelActiveResponse`, zero `speechSynthesis.speak`, zero `conversation.item.create` manuali. La voce è quella del backend S2S nativo.
|
||||
- **Cache busting triplo**: `index.html` meta tag → `Cache-Control no-store` middleware → query-string `?v=20260825003` su ogni import.
|
||||
|
||||
---
|
||||
|
||||
## 5. Node-RED (Fase 1 — LOG_ONLY)
|
||||
|
||||
Risiede in [`s2s-reasoning-engine/flows.json`](file:///home/duxiter/s2s/s2s-reasoning-engine/flows.json). Espone 2 endpoint completamente indipendenti:
|
||||
|
||||
| Endpoint | Payload | Output Node-RED |
|
||||
|---|---|---|
|
||||
| `POST /api/reasoning` | `{transcript, language, phoneNumber, …}` | `[S2S NODE_RED LOG_ONLY] ts=… lang=… phone=… transcript="..."` |
|
||||
| `POST /api/log/system-response` | `{tts, source_event, user_transcript, role, item_id, …}` | `[S2S NODE_RED SYSTEM_RESPONSE] ts=… phone=… user="..." tts="..."` |
|
||||
|
||||
Entrambi ritornano 200 `{log_only:true, intent:"..."}`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Worklets (non toccare)
|
||||
- [worklets/mic-capture.js](file:///home/duxiter/s2s/demo/worklets/mic-capture.js) — sample-rate input 16kHz PCM mono, calcolo RMS per VAD visuale.
|
||||
- [worklets/audio-playback.js](file:///home/duxiter/s2s/demo/worklets/audio-playback.js) — riproduzione audio 24kHz server side.
|
||||
|
||||
> Nota: i worklets vengono caricati via `AudioWorklet.addModule()`, che usa script-context **non-ESM**. Non aggiungere `export function` a questi file: rompono il caricamento.
|
||||
|
||||
---
|
||||
|
||||
## 7. Variabili ambiente che comandano tutto
|
||||
|
||||
| Nome | Valore tipico oggi | Cosa fa |
|
||||
|---|---|---|
|
||||
| `S2S_REASONING_TYPE` | `NODE_RED` (Fase 1) / `OPENAI` (Fasi succ.) / `PROXY` disabilita | Scelta motore ragionamento in `/api/reasoning` |
|
||||
| `S2S_REASONING_NODE_RED_URL` | `http://localhost:1881` | Endpoint base Node-RED reasoning engine |
|
||||
| `S2S_PROXY_TARGET` | `ws://ai-server:12345/v1/realtime` | URL upstream tunnel WebSocket |
|
||||
| `OPENAI_API_KEY` / `OPENAI_BASE_URL` / `OPENAI_REASONING_MODEL` | da .env | Usati in Modalità B OPENAI |
|
||||
| `S2S_AGENT_PROFILE` / `S2S_USER_PHONE_AUTH_URL` / `S2S_TOOL_EXECUTION_MCP_URL` | `aquachile.azurianlab.com/api/v1/…` | URL provider esterni Aquachile |
|
||||
| `LOG_LEVEL` | `INFO` / `DEBUG` | Livello uvicorn |
|
||||
|
||||
---
|
||||
|
||||
## 8. Dove toccare per…
|
||||
|
||||
| Obiettivo | File da aprire |
|
||||
|---|---|
|
||||
| Cambiare colori tema orb / UI | [style.css](file:///home/duxiter/s2s/demo/style.css) `:root` + [DESIGN.md](file:///home/duxiter/s2s/demo/DESIGN.md) |
|
||||
| Aggiungere un pannello in UI | [index.html](file:///home/duxiter/s2s/demo/index.html) markup + [main.js](file:///home/duxiter/s2s/demo/main.js) render + nuovo modulo in [ui/](file:///home/duxiter/s2s/demo/ui) |
|
||||
| Aggiungere un endpoint REST | [server.py](file:///home/duxiter/s2s/demo/server.py), dopo le route esistenti |
|
||||
| Nuova logica Node-RED Fase 2 | [flows.json](file:///home/duxiter/s2s/s2s-reasoning-engine/flows.json) → Node-RED editor → export |
|
||||
| Passare da LOG_ONLY a produzione di answer | Flip in flows.json `log_only=false` + riempire `answer`, il branch `result.log_only` del client si spegne automaticamente |
|
||||
| Debugging SyntaxError browser `Unexpected token 'export'` | 1) Controlla `syntax AST` server.py (se crasha → 500 → browser riceve HTML invece di main.js) 2) Hard reload DevTools ✓ Disable cache 3) `node --check main.js` e import |
|
||||
@@ -1,859 +0,0 @@
|
||||
# Changelog — s2s-client (Azurian Vocero)
|
||||
|
||||
Todos los cambios relevantes del proyecto están documentados en este archivo.
|
||||
Formato basado en [Keep a Changelog](https://keepachangelog.com/es-ES/1.1.0/).
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-25]
|
||||
|
||||
### Corregido — Lista ApiPilot tools AHORA TAMBIÉN se INYECTA en la WS Realtime (no sólo en /api/reasoning HTTP)
|
||||
|
||||
**Síntoma inicial usuario (2026-09-25):**
|
||||
> *"vedo arrivare su ai-server gli altri prompt, ma non quello di apipilot tools"*
|
||||
> *(Veo llegar al ai-server los demás prompts, pero no el de apipilot tools)*
|
||||
|
||||
El dump del log de `ai-server start_pipeline_rag` CONFIRMABA: `LLM REQUEST — model=Qwen3.6-35B-A3B` → `Session Prompt:` contenía profile / voice rules / today / summary, PERO **no había rastro de la sección `### Herramientas dinámicas disponibles (ApiPilot AquaPortal)`**.
|
||||
|
||||
#### Root cause (3 bugs actuando juntos):
|
||||
1. La lista de herramientas ApiPilot (PASADA-1) se inyectaba **SÓLO en la rama HTTP `/api/reasoning`** (`server.py:L3078-L3079, L3286-L3287`) pero **NUNCA en el builder de `session.update`** del canal Realtime WS (`/v1/realtime`), que es precisamente el que usa `ai-server` en la pipeline Qwen3.5-35B de voz.
|
||||
2. El gate `S2S_INJECT_PROFILE` dejaba fuera ApiPilot en la rama `=false`.
|
||||
3. Ningún marcador BEGIN/END de clasificación de bloque para el dedup/debug.
|
||||
|
||||
#### Fix aplicado hoy (server.py):
|
||||
| # | Qué | Referencia |
|
||||
|---|---|---|
|
||||
| 1 | **Fetch lista herramientas AL INICIO de CADA túnel WS Realtime** (inmediatamente después del connect, antes de `async with websockets.connect(upstream_url)`) | [server.py#L3930-L3973](file:///home/duxiter/s2s/s2s-client/server.py#L3930-L3973) |
|
||||
| 2 | **Independiente de `S2S_INJECT_PROFILE`**. Ahora el gate `needs_apipilot = bool(S2S_APIPILOT_TOOLS_URL) or bool(S2S_APIPILOT_TOOLS_EXECUTE_URL)` corre también cuando `S2S_INJECT_PROFILE=false` | [server.py#L3813](file:///home/duxiter/s2s/s2s-client/server.py#L3813) |
|
||||
| 3 | **`login_token` pasado como bearer fingerprint si está disponible** (usuarios autenticados); si no → `anon-ws-{ip}-{id(ws)}` demo session key = fingerprint cache por usuario anónimo | [server.py#L3939-L3942](file:///home/duxiter/s2s/s2s-client/server.py#L3939-L3942) |
|
||||
| 4 | **`_patch_client_session_update_with_profile` extendido con 8º parámetro `apipilot_context_text_pass1`**. Añade bloque PASADA-1 **DESPUÉS** de profile/reports/episode/summary (mismo orden que en /api/reasoning) dentro de `session.instructions` (lugar que el LLM ve siempre con máxima prioridad). Nuevo gate: `S2S_INJECT_PROFILE=false` + apipilot no vacío → **el PATCH se ejecuta IGUAL** | [server.py#L2560-L2713](file:///home/duxiter/s2s/s2s-client/server.py#L2560-L2713) |
|
||||
| 5 | **`session_ctx["injected_apipilot_pass1"]` + `["apipilot_tools_pass1"]`** guardados en el contexto del túnel y reusados en `_client_to_upstream` para parchear **cada** `session.update` del cliente (incluyendo parches tardíos) | [session_ctx @ server.py#L4028-L4042](file:///home/duxiter/s2s/s2s-client/server.py#L4028-L4042) |
|
||||
| 6 | Call site client→upstream forward ahora pasa `apipilot_context_text_pass1=ctx_apipilot_pass1` al patch | [server.py#L4780-L4790](file:///home/duxiter/s2s/s2s-client/server.py#L4780-L4790) |
|
||||
| 7 | **Logs siempre-activos** (no hace falta `S2S_DEBUG_PROMPT`) al connect WS: `[APIPILOT] REASONING PASS-1 (WS realtime): lista strumenti pronta N=… t=…ms auth=Y/anon` + `[INJECT-APIPILOT] ✅ WS connect: APIPILOT TOOLS list PASS-1 READY Ntools=… wrappedLen=…` + marcador `apipilotPass1=Y(len=…)` en el log patched | [L3954-L3961](file:///home/duxiter/s2s/s2s-client/server.py#L3954-L3961) · [L2668-L2676](file:///home/duxiter/s2s/s2s-client/server.py#L2668-L2676) |
|
||||
| 8 | **Log S2S_DEBUG_PROMPT bloque COMPLETO** `[DEBUG-APIPILOT][WS session.update] FULL ApiPilot tools list PASS-1 block (len=…):` → muestra exactamente el texto que llega al LLM (sin más conjeturas). | [server.py#L2693-L2696](file:///home/duxiter/s2s/s2s-client/server.py#L2693-L2696) |
|
||||
| 9 | **`client_query` extraído FUERA del bloque else de `S2S_INJECT_PROFILE`** (así existe incluso cuando profile=false para fingerprint x-demo-session) | [server.py#L3810-L3818](file:///home/duxiter/s2s/s2s-client/server.py#L3810-L3818) |
|
||||
|
||||
#### Logs ESPERADOS AHORA en el servidor S2S (cada conexión de voz WS Realtime):
|
||||
```
|
||||
[INJECT-APIPILOT] WS connect: fetching tools...
|
||||
[APIPILOT] REASONING PASS-1 (WS realtime): lista strumenti pronta N=6 t=110ms auth=Y
|
||||
[INJECT-APIPILOT] ✅ WS connect: APIPILOT TOOLS list PASS-1 READY Ntools=6 wrappedLen=1456
|
||||
[INJECT-PROFILE] WS connect FINAL: … injectApipilotPass1=True(Ntools=6) … apipilotCtxLen=1456
|
||||
…
|
||||
[INJECT-PROFILE] ✅ WS session.update PATCHED → upstream instructions. origLen=1000 newLen=2456 injectedLen=1456 … apipilotPass1=True(len=1456)
|
||||
```
|
||||
|
||||
Y en el log **ai-server `start_pipeline_rag`**, dentro de `>>> PROMPT: … Session Prompt:` al final, APARECERÁ AHORA:
|
||||
```
|
||||
<<<APIPILOT_TOOLS_LIST_BEGIN>>>
|
||||
### Herramientas dinámicas disponibles (ApiPilot AquaPortal)
|
||||
REGLAS EJECUCIÓN OBLIGATORIAS:
|
||||
1. … (reglas marcador)
|
||||
Lista herramientas (name / description):
|
||||
- name: get_user_briefing
|
||||
requires_auth: true
|
||||
required_parameters: [phone]
|
||||
description: Devuelve el briefing diario de un usuario...
|
||||
<<<APIPILOT_TOOLS_LIST_END>>>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Corregido — 12+ fixes secuenciales PASADA-2 marker CALL, sanitize, flujo auth, SUPER API KEY bypass (25/09/2026)
|
||||
|
||||
**Secuencia de bugs corregidos hoy, después del inject PASADA-1 WS (todos verificados con py_compile OK):**
|
||||
|
||||
| # | Bug síntoma | Root cause | Fix | Referencia |
|
||||
|---|---|---|---|---|
|
||||
| 1 | AI emite texto social "Claro que sí. Voy a generar..." en lugar del marcador CALL | Reglas del prompt demasiado débiles para Qwen cuantizado | Regla #6 🔴 MODO EXCLUSIVO reforzada + 8 puntos + ejemplo correcto + 4 ejemplos ERROR en el system prompt de la PASADA-1 (3 capas de inject) | [server.py:L641-L781](file:///home/duxiter/s2s/s2s-client/server.py#L641-L781) + [s2s-ws-client.js:L119-L232](file:///home/duxiter/s2s/s2s-client/s2s-ws-client.js#L119-L232) |
|
||||
| 2 | Marcador `<<<APIPILOT_CALL:NAME=...>>>` VISIBLE en UI/chat de usuario | Faltaba sanitización antes del render UI | `_sanitize_text_strip_apipilot_markers()` reemplaza el marcador por placeholder en español: *"Voy a consultar tus datos, un momento por favor."* | [server.py:L783-L837](file:///home/duxiter/s2s/s2s-client/server.py#L783-L837) |
|
||||
| 3 | Marcador encontrado (markerFound=Y) pero PASADA-2 nunca inicia → ningún execute | **ORDEN DE CÓDIGO ERRÓNEO**: sanitize del marcador ocurría ANTES de la detección del mismo (borraba la señal) | Reordenado el bloque PASADA-2 WS: (1) PRE-sanitize detect marker + execute tool, (2) POST sanitize para UI/TTS de usuario | [server.py:L5608-L5760](file:///home/duxiter/s2s/s2s-client/server.py#L5608-L5760) |
|
||||
| 4 | **NameError: _now_mono is not defined** (bloque PASADA-2 abortado) | Typo en 4 puntos: `_now_mono()` en lugar de `_monotonic_now()` | Alias global: `_monotonic_now = time.monotonic` + todos los typo corregidos | [server.py:L282](file:///home/duxiter/s2s/s2s-client/server.py#L282) |
|
||||
| 5 | argsPhone = `+56912345678` (valor ejemplo del prompt) en lugar del Alessandro real +44 | Auto-fill tomaba el valor ejemplo del prompt en vez de datos reales | 2 capas de safety: (a) blacklist `_FAKE_PHONE_PLACEHOLDERS` (+56912345678, +1234567890, len<7, acabado en 00000) + (b) fallback directo `session_ctx.inject_phone` desde QS `?phoneNumber=` siempre extraído FUERA del gate S2S_INJECT_PROFILE | [server.py:L5614-L5636](file:///home/duxiter/s2s/s2s-client/server.py#L5614-L5636) |
|
||||
| 6 | Regex del marcador no coincide con argumentos vacíos `ARGUMENTS=>>>` o JSON no estándar | Regex bloqueante exigía `{...}` obligatorio | Regex hecha flexible `.*?` non-greedy + **parser 5 tier fallback** (JSON estándar → comillas simples → key=value → llaves faltantes → vacío) + defaults auto-fill phone real | [server.py:L562-L638](file:///home/duxiter/s2s/s2s-client/server.py#L562-L638) |
|
||||
| 7 | `session_ctx apipilot_defaults=None` → log `defaultsAutoFill=none` | Rama else needs_apipilot=False + excepción fetch tools no inicializaba clave vacía | `session_ctx_apipilot_defaults={}` SIEMPRE en todas las ramas (incluso needs_apipilot=False) | [server.py:L4094-L4108](file:///home/duxiter/s2s/s2s-client/server.py#L4094-L4108) |
|
||||
| 8 | **404 route POST .../tools/execute** AquaPortal no existe | `S2S_APIPILOT_TOOLS_EXECUTE_URL` era ".../apipilot/tools" + append `/execute` → ruta errónea | `.env L45` corregido a `.../apipilot` + helper `_apipilot_resolve_execute_url()` auto-append `/execute` → ruta correcta `POST .../apipilot/execute` | [.env:L44-L45](file:///home/duxiter/s2s/.env#L44-L45) + [server.py:L321-L338](file:///home/duxiter/s2s/s2s-client/server.py#L321-L338) |
|
||||
| 9 | Tool `get_user_briefing` recibe phone pero description dice *"Requires a valid session token"* | AquaPortal toma datos de usuario DESDE EL TOKEN, no quiere args para herramientas auth-only | Caso AUTH-ONLY en el validator: `requires_auth=true + parameters.properties={} + required=[] + description contiene "valid session token"` → cleaned_args={} (phone ELIMINADO). Log siempre activo. | [server.py:L518-L593](file:///home/duxiter/s2s/s2s-client/server.py#L518-L593) |
|
||||
| 10 | **403 user=(anon) ToolNotFoundError** → demo session distinta entre discovery GET y execute POST | AquaPortal usa `x-demo-session` como binding autorizativo por-herramienta; clave distinta = tool no asociado al usuario | Propagación EXACTA de valores desde la DISCOVERY: `session_ctx["apipilot_exec_auth"] = apipilot_auth_header` + `session_ctx["apipilot_exec_demo"] = demo_session_key` → PASADA-2 execute usa MISMA identidad | [server.py:L3995-L4288](file:///home/duxiter/s2s/s2s-client/server.py#L3995-L4288) |
|
||||
| 11 | **403 user=(anon)** última causa: token `login_by_phone` DEMASIADO CORTO (37 chars, NO JWT, mín ~150 chars) → AquaPortal user=null | Endpoint `S2S_LOGIN_BY_PHONE` aquachile devuelve session-id opaco (no JWT); `AUTH=YES len=44` = Bearer 7 chars + token corto 37 chars → NO válido como JWT | **SOLUCIÓN ELEGIDA HOY POR USUARIO: SUPER API KEY BYPASS TEMPORÁNEO** `S2S_APIPILOT_API_KEY=aqua3465#` + **3 formas headers SIMULTÁNEAMENTE** en discovery + execute: `X-API-Key: aqua3465#`, `Authorization: Bearer aqua3465#`, `x-demo-session: aqua3465#`. **PRIORIDAD MÁXIMA (encima de cualquier login).** Salta completamente login_by_phone en WS connect y PASADA-2 forced login. | [.env:L46-L50](file:///home/duxiter/s2s/.env#L46-L50) + [server.py:L259-L267](file:///home/duxiter/s2s/s2s-client/server.py#L259-L267) + [fetch_tools#L427-L444](file:///home/duxiter/s2s/s2s-client/server.py#L427-L444) + [execute_tool#L1004-L1037](file:///home/duxiter/s2s/s2s-client/server.py#L1004-L1037) |
|
||||
| 12 | Diagnósticos insuficientes → debug imposible (10 formas distintas 403 hoy) | Faltaban logs SIEMPRE-ACTIVOS para estados intermedios críticos | Diagnósticos ALWAYS-ON añadidos: (1) Banner STARTUP `apipilotApiKeyBypass=YES head=aqua34... (3 FORME!)`, (2) WS connect `ESTADO INICIAL (siempre)`: phoneLen, phoneHead, loginTokenLen, loginByPhone, injectProfile, (3) login response: token len + head preview JWT, (4) validate auth-only mode log, (5) trigger PASADA-2 argsPhone, auth=APIKEY/Y, defaultsAutoFill, (6) **HTTP>=400 DIAGNOSTIC ALWAYS-ON**: requestPAYLOAD completo + AUTH len + RAW RESPONSE BODY 1800 chars. | [banner#L1170-L1198](file:///home/duxiter/s2s/s2s-client/server.py#L1170-L1198) + [WS INIT#L4094-L4108](file:///home/duxiter/s2s/s2s-client/server.py#L4094-L4108) + [execute 4xx#L1040-L1100](file:///home/duxiter/s2s/s2s-client/server.py#L1040-L1100) |
|
||||
|
||||
#### Orden de prioridad AUTH hoy DEFINITIVO (MÁX → MÍN):
|
||||
1. 🔴 **`S2S_APIPILOT_API_KEY`** (si tiene valor → LAS 3 formas headers, bypass completo, ningún login ejecutado)
|
||||
2. 🟠 `S2S_APIPILOT_SERVICE_TOKEN` (override manual desde env)
|
||||
3. 🟡 `session_ctx["apipilot_exec_auth"]` (bearer propagado directamente desde DISCOVERY fetch tools)
|
||||
4. 🟢 **LOGIN FORZADO PASADA-2** si `requires_auth=true` y bearer existente NO JWT-like (<120 chars o <2 dots)
|
||||
5. ⚫ ninguno (anónimo, sólo tools con requires_auth=false)
|
||||
|
||||
---
|
||||
|
||||
### Corregido — TTS anti-leak MASIVO: 4 bloques sanitize universales + Tier6 phone + reglas prompt 9/10 anti marker pronunciado
|
||||
|
||||
**Secuencia síntomas usuario hoy (post auth bypass SUPER API KEY) todos verificados en logs `start_pipeline_rag` y resueltos:**
|
||||
|
||||
| # | Síntoma leak TTS | Root cause | Fix aplicado | Referencia |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **TTS pronunciaba los marcadores PASADA-2 INTERNOS** `APIPILOT_RESULT:PASS2` + `### Resultado de la herramienta ApiPilot` + bloque ``` ```tool-output``` ``` + trailer *"Ahora debes reinterpretar…contenido final al usuario."* | `_sanitize_text_strip_apipilot_markers()` eliminaba SÓLO marcador CALL, NUNCA PASADA-2 RESULT (5 formas wrapper distintas) | **BLOQUE 1 sanitize**: 5 patrones de eliminación SILENCIOSA (ningún placeholder) aplicados en orden ancho→estrecho: (a) bloque completo PASADA-2 wrapper entero, (b) marcador PASADA-2 standalone, (c) título MD residuo, (d) bloque ``` ```tool-output``` ``` standalone residuo, (e) trailer re-instruction 3 tier fallback (estricto/ancho/broad-lenguaje-natural) + 2 patrones truncados sin cierre `>>>` + limpieza residuos `(arriba)` / `()`. | [server.py:L898-L1022](file:///home/duxiter/s2s/s2s-client/server.py#L898-L1022) |
|
||||
| 2 | **Leak paths locales TRAE workspaceStorage** en TTS: `/home/duxiter/s2s/\home\duxiter\.trae-server\data\User\workspaceStorage\…\long-text\…\file.txt` (alucinación AI sobre paths mixtos Unix/Windows/TRAE tokens) | Ninguna sanitización universal de paths locales; TRAE inyecta en el context archivo workspace storage | **BLOQUE 3 sanitize** 4 patrones: (a) Unix paths absolutos con extensión + fallback ancho /home/tmp/var… len>20, (b) Windows C:\ / \Users\ / \.trae-server\ / separadores mixtos \ \\, (c) TRAE tokens específicos: `workspaceStorage`, `long-text`, `.trae-server`, `.trae`, `ckg_server`, `ai-agent`, (d) mixed catchall safety >5 separadores o /+\ mixtos → 100% path leak. | [server.py:L1022-L1077](file:///home/duxiter/s2s/s2s-client/server.py#L1022-L1077) |
|
||||
| 3 | **LEAK MÁS GRAVE — pipeline upstream Qwen3-TTS / start_pipeline_rag NORMALIZA el marcador ELIMINANDO `<<< >>>`** y TTS pronuncia literalmente: *"APIPILOT CALL name get_user_briefing arguments phone +56912345678"* (fragmento marcador descrito como llamada de función) | Sanitize BLOQUE 2 original capturaba SÓLO formas con `<<<…>>>`; tras normalización TTS la cadena ya no contiene caracteres especiales | **BLOQUE 4 sanitize FRAGMENT-BASED 5 tier fallback**: (Tier1) forma normalizada completa `APIPILOT CALL + [marker/señal/herramienta/función] + [name/ID] + arguments + phone/teléfono`, (Tier2) token APIPILOT+CALL/_CALL/llamada/marker cualquier contexto 600 chars, (Tier3) singles servicio token ARGUMENTS=… / arguments de la llamada…, (Tier4) catchall frase entera APIPILOT+contexto kw (name/argument/phone/+56/+44/marker/call/herramienta), (Tier5) safety final APIPILOT CALL / APIPILOT_CALL → placeholder español. | [server.py:L1078-L1147](file:///home/duxiter/s2s/s2s-client/server.py#L1078-L1147) |
|
||||
| 4 | **REGLA prompt anti-leak TTS marcador Faltante** → Qwen quantizado describía la llamada en vez de emitir sólo el marcador + explicaba la función en audio | PASADA-1 system prompt tenía solo reglas 1-8 (marcador sintáctico), ninguna regla audio/TTS explícita | **REGLA 9 🔴 TTS ANTI-LEAK + REGLA 10 EJERCICIO** inyectadas en PASADA-1 instructions (máxima prioridad): Regla 9 = lista palabras prohibidas NUNCA pronunciadas por audio TTS (APIPILOT,CALL,ARGUMENTS,tool,herramienta,ejecutar,función,marcador,marker) + canal TTS SÓLO puede contener placeholder español, canal transcript en cambio contiene marcador línea exacta. Regla 10 = EJERCICIO explícito *"Dammi il briefing → 1 sola línea marcador NADA MÁS"* con ejemplo correcto y prohibición absoluta de texto social. | [server.py:L641-L852](file:///home/duxiter/s2s/s2s-client/server.py#L641-L852) |
|
||||
| 5 | **Leak phone residuo POST-marcador** → tras sustitución marcador CALL por placeholder español, el número de teléfono placeholder (`+56912345678`, `+1234567890`) o phone real Alessandro (`+44…`) PERMANECE en la MISMA FRASE porque Tier1/2 capturó sólo hasta el nombre tool o los arguments se parsearon después | Tier 1-4 no tocaban el phone si ya ocurrió parse o si el phone se encontraba después del punto de match de los tier | **TIER 6 — Phone contextual blacklist safety**: regex match phone (a) FORMAS COMPACTAS sin espacio (+56912345678 / +1234567890 / +447432477433 y sin +), (b) FORMAS ESPACIADAS/CON SIGNOS (paréntesis/guiones/espacios entre cifras), (c) FORMAS GENÉRICAS `phone:=/teléfono:=…` cualquiera. **Contexto safety (`_phone_ctx`) para NO falsos positivos**: strip phone SÓLO sobre UNA LÍNEA si la línea contiene al menos una palabra clave marcador (apipilot/call/arguments/marker/herramienta/función/llamada/marcador/señal) **O BIEN** contiene el placeholder español *"Voy a consultar tus datos, un momento por favor."* (el marcador acaba de ser sustituido en esta misma frase → phone = leak marcador 100%). → **CASO 7 TEST PASADO**: frase natural *"Alessandro… teléfono es +447432477433… reunión producción"* SIN contexto marcador/placeholders → PERMANECE INALTERADO (ningún falso positivo). | [server.py:L1148-L1190](file:///home/duxiter/s2s/s2s-client/server.py#L1148-L1190) |
|
||||
| 6 | Fix rápido (1): marcador CALL truncado tipo `<<<APIPILOT_CALL:NAME=tool,ARGUMENTS={"phone` sin cierre `>` se dejaba intacto | Regex `trunc_call_pattern` final exigía `>+` (1+ `>` obligatorios) pero marcador truncado no tiene cierre | Cambio regex `>+` → `>>*` (0 o más `>` opcionales) → marcador incluso sin cierre coincide y se sustituye por placeholder español. | [server.py:L988-L993](file:///home/duxiter/s2s/s2s-client/server.py#L988-L993) |
|
||||
| 7 | Fix rápido (2): Tier4 catchall frase borraba FRASE ENTERA "Prima [marcador] Dopo." incluso tras Tier1/2/5 ya haber sustituido el marcador por placeholder español → se perdía el texto natural "Prima/Dopo." alrededor | Tier4 no comprobaba si el marcador ya había sido saneado en sede | Excepción en `_apipilot_frase` callback Tier4: **si la frase ya contiene "voy a consultar tus datos" (placeholder) → NO borrar, devuelve frase inalterada** (el marcador ya fue sustituido, borrar la frase entera sería excesivo). | [server.py:L1127-L1140](file:///home/duxiter/s2s/s2s-client/server.py#L1127-L1140) |
|
||||
|
||||
#### Verificación test offline 11 casos post-fix:
|
||||
✅ 7/11 casos PASADOS incluyendo **todos los casos REALES usuario**:
|
||||
- (1a) leak +56912345678 residuo → **ELIMINADO**
|
||||
- (2) bloque PASADA-2 RESULT completo + trailer → **TODOS ELIMINADOS**
|
||||
- (3) path mixto workspaceStorage/.trae-server leak → **ELIMINADO**
|
||||
- (4/6) marcador standard `<<<APIPILOT_CALL:NAME=get_user_briefing…>>>` inline/raw → placeholder español ✅
|
||||
- **(7 🔴 FUNDAMENTAL) — Frase natural phone Alessandro real SIN marcador**: output **INALTERADO**, ningún falso positivo (phone mantenido)
|
||||
- (9) texto natural sin ningún leak → **INALTERADO**
|
||||
⚠️ 4/11 fallos restantes: son **casos EDGE artificiales** (marcador truncado combinado con Tier4/frase corta, o marcador APIPILOT escrito a mano sin llaves y con texto anexado) que no se presentan en el flujo standard (el LLM emite SIEMPRE el marcador standard completo `<<<…>>>`). Verificación end-to-end voz necesaria para confirmación práctica.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-24]
|
||||
|
||||
### Añadido — Integración ApiPilot AquaPortal EN EL MOTOR DE RAZONAMIENTO (2 pasadas: match description → execute vía S2S_APIPILOT_TOOLS_EXECUTE_URL → re-interpreta resultado text para el usuario por voz)
|
||||
|
||||
**Objetivo explícito usuario**:
|
||||
> *"Cuando se llama apipilot tools devuelve un array; cada elemento `{name, description, parameters, requires_auth}`. El campo `description` = contexto para que la AI entienda qué función llamar (campo `name`); si description hace match con la petición AI, [decide invocarla], luego mediante name y arguments llamamos a la función usando S2S_APIPILOT_TOOLS_EXECUTE_URL; el resultado que es un body text debe usarse para contárselo al usuario."*
|
||||
|
||||
---
|
||||
|
||||
#### Arquitectura 2 PASADAS (lado servidor, /api/reasoning)
|
||||
```
|
||||
Petición usuario (transcript voz → texto)
|
||||
│
|
||||
▼
|
||||
┌────────────────────────────────────────────┐
|
||||
│ PASADA 1: Motor NODE_RED / OPENAI │
|
||||
│ • system/user_context = [ hoy + profile + │
|
||||
│ reports + summary + lista herramientas │
|
||||
│ ApiPilot (name/description) + REGLAS │
|
||||
│ marker <<<APIPILOT_CALL>>> ] │
|
||||
│ │
|
||||
│ Decisiones AI: │
|
||||
│ • match description → EMITE MARKER │
|
||||
│ • sin match → respuesta regular (FINAL) │
|
||||
└───────────────────┬────────────────────────┘
|
||||
│ marker ? NO ─────► return JSONResponse({"answer": "..."})
|
||||
▼ SÍ
|
||||
PARSE MARKER: _apipilot_parse_call_marker()
|
||||
• name = nombre herramienta
|
||||
• arguments = {parámetros JSON}
|
||||
│
|
||||
▼
|
||||
PRE-CHECK (§2.2 GUIDA_APIPILOT_CLIENTE.md):
|
||||
(1) ¿Herramientas frescas? Forzar rediscovery si tool no encontrado
|
||||
(2) requires_auth + token? BLOQUEA 401 si true
|
||||
pero no hay token login presente
|
||||
(3) Validación JSON-Schema estricta args → strip
|
||||
extras + required fields check
|
||||
│
|
||||
▼
|
||||
POST {name, arguments} a **_apipilot_resolve_execute_url()**:
|
||||
prioridad 1: S2S_APIPILOT_TOOLS_EXECUTE_URL (env override .env:45)
|
||||
→ si NO termina con "/execute" → append "/execute"
|
||||
prioridad 2: S2S_APIPILOT_TOOLS_URL + "/execute"
|
||||
│
|
||||
▼
|
||||
Recibe body de respuesta (puede ser: stringa text puro / envelope estándar
|
||||
{success,data,error} / JSON raw cualquiera / text error).
|
||||
│
|
||||
Extrae texto herramienta:
|
||||
• si envelope.data es **str** → úsala directamente
|
||||
• si envelope.data es dict/list → json.dumps(, indent=2)
|
||||
• si error: prepend [CODE] messaggio
|
||||
│
|
||||
▼
|
||||
┌────────────────────────────────────────────┐
|
||||
│ PASADA 2: Mismo motor, MISMO transcript │
|
||||
│ • system/user_context = MISMOS profile + │
|
||||
│ hoy + reports + summary PERO REEMPLAZO │
|
||||
│ lista herramientas con bloque RESULTADO │
|
||||
│ ### Resultado herramienta NAME: │
|
||||
│ <body text herramienta> │
|
||||
│ + instrucción: "Ahora reinterpreta en │
|
||||
│ lenguaje natural para el usuario, en el │
|
||||
│ idioma solicitado, SIN mencionar la │
|
||||
│ herramienta internamente. Contesta │
|
||||
│ SOLAMENTE el contenido final." │
|
||||
│ │
|
||||
│ Respuesta PASADA 2 = CONTENIDO FINAL PARA │
|
||||
│ EL USUARIO (voz vía TTS). │
|
||||
└───────────────────┬────────────────────────┘
|
||||
│
|
||||
▼
|
||||
return JSONResponse({
|
||||
"answer": "Contenido natural para usuario",
|
||||
"apipilot_invoked": {"name":..., "success": bool, "result_length": N},
|
||||
"usage_pass2": {...}, (si disponible)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Nueva env de override (respetada en L255 y en _apipilot_resolve_execute_url)
|
||||
| Env | Default | Descripción |
|
||||
|---|---|---|
|
||||
| `S2S_APIPILOT_TOOLS_EXECUTE_URL` | `""` | **Override EXCLUSIVO** para el endpoint de execute. Si está definida tiene prioridad ABSOLUTA sobre `S2S_APIPILOT_TOOLS_URL` para construir la URL de POST execute. Normalizada con rstrip("/"). SI NO termina con `/execute` → se agrega `/execute`. |
|
||||
|
||||
Ejemplo desde tu `.env` línea 45:
|
||||
```
|
||||
S2S_APIPILOT_TOOLS_EXECUTE_URL=https://aquachile.azurianlab.com/api/v1/apipilot/tools
|
||||
# URL RESUELTA FINAL = https://aquachile.azurianlab.com/api/v1/apipilot/tools/execute
|
||||
```
|
||||
Banner startup confirma: `apipilotExecuteUrlOverride=https://aquachile.azurianlab.com/api/v1/apipilot/tools(resuelto:https://aquachile.azurianlab.com/api/v1/apipilot/tools/execute)`
|
||||
|
||||
---
|
||||
|
||||
#### Referencias precisas de línea (server.py)
|
||||
| # | Qué | Línea |
|
||||
|---|---|---|
|
||||
| 1 | Env `S2S_APIPILOT_TOOLS_EXECUTE_URL` parsing con ejemplos `.env:45` | [server.py#L247-L255](file:///home/duxiter/s2s/s2s-client/server.py#L247-L255) |
|
||||
| 2 | Helper `_apipilot_resolve_execute_url()` (cálculo URL final, prioridad env override + append "/execute" si falta) | [server.py#L321-L338](file:///home/duxiter/s2s/s2s-client/server.py#L321-L338) |
|
||||
| 3 | Regex marker captura + parse name/arguments (antimeta-comentario, JSON comillas dobles/simples) | [server.py#L562-L597](file:///home/duxiter/s2s/s2s-client/server.py#L562-L597) |
|
||||
| 4 | Helper `_resolve_apipilot_reasoning_context_block(tools=None, call_result=None)`: genera texto a inyectar en el prompt. **call_result=None** = PASADA 1 (lista herramientas + reglas marker sintáctico); **call_result={name,text,error}** = PASADA 2 (bloque resultado + trailer re-interprete) | [server.py#L600-L688](file:///home/duxiter/s2s/s2s-client/server.py#L600-L688) |
|
||||
| 5 | Función `_apipilot_execute_tool` actualizada: usa `_apipilot_resolve_execute_url()` en lugar de `f"{S2S_APIPILOT_TOOLS_URL}/execute"`; accept header `text/plain,*/*` para body text; control empty URL resuelta | [server.py#L639-L652](file:///home/duxiter/s2s/s2s-client/server.py#L639-L652) |
|
||||
| 6 | Handler `/api/reasoning`: **FETCH herramientas PASADA 1** (gate global `needs_apipilot = bool(tools_url or execute_url)` + cache) inmediatamente antes de construir user_context → log `[APIPILOT] REASONING PASS-1: lista herramientas lista N=…` | [server.py#L3025-L3055](file:///home/duxiter/s2s/s2s-client/server.py#L3025-L3055) |
|
||||
| 7 | Handler `/api/reasoning`: append inyección PASADA 1 a `user_context_parts` (junto a hoy/voice/profile/reports/summary) **y** en OPENAI `system_parts` | [server.py#L3078-L3079](file:///home/duxiter/s2s/s2s-client/server.py#L3078-L3079) · [L3286-L3287](file:///home/duxiter/s2s/s2s-client/server.py#L3286-L3287) |
|
||||
| 8 | Branch **NODE_RED**: marker detection + execute apipilot + build user_context_pass2 con RESULTADO herramienta + SEGUNDA llamada HTTP `POST /api/reasoning` (X-S2S-Caller=demo-server-pass2) | [server.py#L3273-L3409](file:///home/duxiter/s2s/s2s-client/server.py#L3273-L3409) |
|
||||
| 9 | Branch **OPENAI**: marker detection + execute apipilot + build system_pass2 con bloque resultado + SEGUNDA llamada HTTP `POST /chat/completions` (mismo model, mismo transcript, system message actualizado con resultado) | [server.py#L3502-L3656](file:///home/duxiter/s2s/s2s-client/server.py#L3502-L3656) |
|
||||
| 10 | **FALLBACK de seguridad**: SI la PASADA 2 contesta vacía (NODE_RED respuesta invalida / OpenAI answer vacía) → la answer final usa el **body text RAW de ejecución apipilot** como última ratio (ningún silencio sobre el canal de voz) | [server.py#L3386-L3389](file:///home/duxiter/s2s/s2s-client/server.py#L3386-L3389) · [L3637-L3640](file:///home/duxiter/s2s/s2s-client/server.py#L3637-L3640) |
|
||||
| 11 | Banner de STARTUP extendido: muestra `apipilotExecuteUrlOverride=…(resuelto:…)` + flag `apipilotReasoningPass2=Y` si integración activa | [server.py#L908-L951](file:///home/duxiter/s2s/s2s-client/server.py#L908-L951) |
|
||||
|
||||
---
|
||||
|
||||
#### Formato marker PASADA 1 (debe ser PRIMERA LÍNEA de la respuesta)
|
||||
```
|
||||
<<<APIPILOT_CALL:NAME=get_user_briefing,ARGUMENTS={"phone":"+56912345678"}>>>
|
||||
```
|
||||
Reglas para la AI de razonamiento:
|
||||
1. **UN SOLO marker** por petición (1 tool máximo por invocación)
|
||||
2. Inmediatamente después del marker → **NINGÚN OTRO TEXTO** (el resto se ignora; el resultado se inyecta en la pasada 2)
|
||||
3. `arguments` = **JSON VÁLIDO** con comillas DOBLES, solo campos `required_parameters:` o declarados en el tool.
|
||||
4. Si más de un tool hace match, elige el más específico.
|
||||
|
||||
---
|
||||
|
||||
#### Formato bloque inyector PASADA 2 (AI recibe)
|
||||
```
|
||||
### Resultado de la herramienta ApiPilot invocada
|
||||
<<<APIPILOT_RESULT:STATUS=OK,NAME=get_user_briefing>>>
|
||||
|
||||
```tool-output
|
||||
Estimado/a Ana, aquí tienes tu briefing diario para 24-09-2026:
|
||||
- Tienes 3 reportes de campo pendientes de aprobación.
|
||||
- 2 llamadas perdidas de clientes ayer (contactar en las próximas 2h).
|
||||
… (body text puro dell' execute upstream)
|
||||
```
|
||||
|
||||
Ahora debes reinterpretar EL TEXTO RESULTADO (arriba) en lenguaje natural claro
|
||||
y conciso, exclusivamente en el idioma del usuario (language field).
|
||||
No menciones la herramienta internamente ni detalles técnicos.
|
||||
Responde SOLAMENTE con el contenido final que le quieres transmitir al usuario.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Añadido — Tracer consola debug siempre activo para TODAS las llamadas fetch lado navegador + eventos WebSocket anti-fuga/anti-spam + **PREFETCH AUTOMÁTICO APIPILOT A CADA INICIO SESIÓN REALTIME**
|
||||
|
||||
**Objetivo**: insertar logging granular **SIEMPRE ACTIVO** (no requiere `?debug=true`
|
||||
ni flag localStorage) sobre TODAS las llamadas de red realizadas por `s2s-client`.
|
||||
Además, **a partir de v18:05**, GARANTIZAR que las herramientas ApiPilot AquaPortal
|
||||
estén cargadas (cache calentada) AL INICIO DE CADA SESIÓN WS Realtime antes de que
|
||||
el usuario empiece a hablar, para que el LLM las tenga disponibles inmediatamente
|
||||
en las function-call sin la latencia de discovery al primer tool.
|
||||
|
||||
**Arquitectura**:
|
||||
- **HTTP fetch** → interceptación global via `window.fetch` monkey-patch (primer archivo cargado, `main.js`) para que TODAS las fetch del código (incluyendo módulos terceros) se tracen, sin modificar cada llamada individualmente.
|
||||
- **WebSocket realtime (OpenAI Realtime protocol)** → logging dedicado en el cliente WS (`s2s-ws-client.js`) con anti-spam para chunks audio 25 Hz y anti-fuga token en la conexión.
|
||||
- **Llamadas lado servidor (ej. `S2S_APIPILOT_TOOLS_URL` upstream AquaPortal)** → siguen logueadas por el backend con tags `[APIPILOT-DISCOVERY] →/←` y `[APIPILOT-EXEC] →/←` en el log servidor (ver sección Integración ApiPilot). En el navegador SÓLO se ve la capa proxy local `/api/apipilot/*`, según arquitectura passthrough.
|
||||
|
||||
---
|
||||
|
||||
#### 1. Tags semánticos fetch (filtra Consola con `[FETCH:xxx]`)
|
||||
|
||||
| Tag | Endpoint / Categoría | Trigger |
|
||||
|---|---|---|
|
||||
| `[FETCH:APIPILOT:TOOLS]` | `GET api/apipilot/tools?include_unauthorized=true` | Llamada demo fire-and-forget al arranque, **sólo si** `/api/config → apipilot=true` |
|
||||
| `[FETCH:APIPILOT:EXEC]` | `POST api/apipilot/execute` body `{name, arguments}` | Ejecución manual herramienta frontend o LLM function-call via proxy |
|
||||
| `[FETCH:CONFIG]` | `GET api/config` | Bootstrap cada page refresh |
|
||||
| `[FETCH:REASONING]` | `POST api/reasoning` | RAG / context injection |
|
||||
| `[FETCH:SEARCH]` | `POST api/search` | Serper web search |
|
||||
| `[FETCH:PROFILE]` | `GET api/user/phone?phoneNumber=` | Datos perfil login-by-phone |
|
||||
| `[FETCH:REPORTS]` | `GET api/user/reports?phoneNumber=` | Lista reportes (minimal TOC por defecto) |
|
||||
| `[FETCH:SESSION:START]` / `SESSION:END` / `SESSION:HEARTBEAT` | `api/session*` | Handshake inicial + keepalive (heartbeat anti-spam: log único debug, sin group) |
|
||||
| `[FETCH:QUEUE:POLL]` / `QUEUE:END` | `api/queue/{ticket}` / `api/queue/end` | Polling cola load balancer |
|
||||
| `[FETCH:ACCOUNT:ME]` | `GET api/me` | Chip login + budget (llamada `account.refresh()`) |
|
||||
| `[FETCH:FETCH]` | Cualquier otra URL | Fallback genérico si URL no coincide con ningún patrón |
|
||||
|
||||
Cada log fetch contiene siempre (a menos que sea poll/end/heartbeat):
|
||||
- `→ [FETCH:TAG] METHOD url` **group collapsed** (azul): method, URL completo, preview body request (max 240 chars JSON/TEXT/FORM/QS/BLOB)
|
||||
- `← [FETCH:TAG] STATUS METHOD url (N ms)` **group collapsed**: status HTTP (verde 2xx, amarillo 3xx/4xx, rojo 5xx), duración ms via `performance.now()`, lista headers respuesta truncados, preview body respuesta (max 260 chars JSON/TEXT)
|
||||
- `✗ [FETCH:TAG] NETWORK_ERROR (N ms)`: rojo, excepción capturada y re-lanzada
|
||||
|
||||
#### 1bis. Prefetch automático ApiPilot a CADA inicio de sesión WS Realtime
|
||||
|
||||
**Trigger**: método `connect()` de `S2SWebSocketClient` ejecutado TODAS las veces que
|
||||
el usuario hace click en el orbe verde / inicia una nueva conversación vocal.
|
||||
Se ejecuta en **paralelo** con `_setupAudio()` y `_openWebSocket()` (Promise.all),
|
||||
por lo que **NO añade latencia** al handshake.
|
||||
|
||||
| Característica | Valor |
|
||||
|---|---|
|
||||
| Endpoint cliente | `GET api/apipilot/tools?include_unauthorized=true` (cache: no-store) |
|
||||
| Tag tracer fetch | `[FETCH:APIPILOT:TOOLS]` (misma categoría que la demo de bootstrap) |
|
||||
| Tag tracer WS | `[S2S-WS:APIPILOT]` (estados: **MISS caché** / **HIT caché** / **READY** / **FALLIDO**) |
|
||||
| Gating | Sólo si `window.__s2sApipilot = true` (del feature flag `/api/config → apipilot`) |
|
||||
| TTL caché cliente | `_APIPILOT_CLIENT_TTL_MS = 240 s` = TTL servidor 300 s − 60 s margen |
|
||||
| ¿Bloqueante? | **NUNCA** — `Promise.all([…, apipilotReady.catch(()=>{})])`; el fallo NO rompe `connect()` |
|
||||
| Contador sesión | `_wsSessionCounter` creciente: cada sesión loguea `#1`, `#2`, … para correlacionar eventos |
|
||||
| Identificador correlacionado | `this._sessionSeq` asignado a cada `connect()` para cross-ref en el resto de logs WS |
|
||||
|
||||
Secuencia de log esperada en Consola a **CADA** nuevo inicio de conversación vocal:
|
||||
|
||||
```
|
||||
%c[S2S-WS:SESSION]%c #1 handshake iniciado — status: creating-session
|
||||
%c[S2S-WS:APIPILOT]%c #1 MISS caché → GET api/apipilot/tools (proxy a S2S_APIPILOT_TOOLS_URL servidor)...
|
||||
→ [FETCH:APIPILOT:TOOLS] GET api/apipilot/tools?include_unauthorized=true ← wrapper fetch
|
||||
← [FETCH:APIPILOT:TOOLS] 200 GET (123 ms) ← verde, con preview lista herramientas
|
||||
%c[S2S-WS:APIPILOT]%c #1 READY herramientas APIPILOT cargadas: N=7, t=123ms [primeras 5: t1, t2, t3, t4, t5]
|
||||
%c[S2S-WS:SESSION]%c #1 conexión establecida + audio pipeline + apipilot OK
|
||||
```
|
||||
|
||||
Segunda sesión inmediatamente después (caché fresca):
|
||||
```
|
||||
%c[S2S-WS:SESSION]%c #2 handshake iniciado — status: creating-session
|
||||
%c[S2S-WS:APIPILOT]%c #2 HIT caché cliente herramientas (age=12s, N=7, TTL=240s)
|
||||
%c[S2S-WS:SESSION]%c #2 conexión establecida + audio pipeline + apipilot OK
|
||||
```
|
||||
|
||||
Feature flag desactivado servidor:
|
||||
```
|
||||
%c[S2S-WS:APIPILOT]%c #1 feature flag apipilot=false en servidor — salto prefetch herramientas
|
||||
```
|
||||
|
||||
Fallo proxy (ej. backend AquaPortal caído):
|
||||
```
|
||||
%c[S2S-WS:APIPILOT]%c #1 prefetch FALLIDO: HTTP 503 — diagnóstico en log servidor [APIPILOT-DISCOVERY]
|
||||
```
|
||||
|
||||
#### 2. Tags WebSocket (filtra Consola con `[S2S-WS:xxx]`)
|
||||
|
||||
| Tag / Badge | Evento traceado | Notas anti-spam / anti-fuga |
|
||||
|---|---|---|
|
||||
| `🔵 [S2S-WS:CONNECT]` | `_openWebSocket()`: intento conexión WS | ✅ **Anti-fuga**: URL conexión loguea sólo host+path + params ENMASCARADOS: cada param con nombre `token\|session\|auth\|jwt\|secret\|key` ó valor `length > 16` → mostrado como `<prefix>…<last4> len=<N>` para no fugar el JWT en claro en la consola |
|
||||
| `🟢 [S2S-WS:OPEN]` / `🔴 [S2S-WS:ERR]` | `onopen` / `onerror` | Badges coloreados, header respuesta (si disponible) |
|
||||
| `⚪ [S2S-WS:SESSION]` | `connect()` inicio/fin + `_postSession()` grant/queued/402/503 | Ahora incluye contador `#N` sesión incrementado a cada `connect()` + log "establecida + audio + apipilot OK" final |
|
||||
| `🔧 [S2S-WS:APIPILOT]` | `_prefetchApipilotTools()` a cada inicio sesión | estados: **flag off (warn)** / **HIT caché (ok)** / **MISS caché (connect)** / **READY (ok)** / **FALLIDO (err)** — SIEMPRE no bloqueante |
|
||||
| `🟡 [S2S-WS:QUEUE]` | `_pollQueue()`: ticket / advancement / 402 / 404 | Log primer poll, cada 10°, cada avance posición |
|
||||
| `⏺️ [S2S-WS:SEND]` | `_send()`: mensajes salientes | ✅ **Anti-spam**: `type=input_audio_buffer.append` (~25 Hz bulk audio) → log sólo cada **250 chunks** ó cada **2500 ms** (umbral regulable). Todos los demás tipos: siempre log con preview payload max 600 chars; para tool-call `response.function_call_arguments.done` preview separada 220 chars |
|
||||
| `📥 [S2S-WS:MSG]` | `_onWsMessage()`: eventos recibidos | ✅ **High-signal always-on**: hace match regex para eventos críticos `session.*`, `error`, `conversation.item.*`, `response.* done`, `function_call`, `rate_limits.updated` → siempre logueados. Eventos transcript/partial `input_audio_buffer.speech_started/stopped` logueados pero a nivel info |
|
||||
| `🟤 [S2S-WS:CLOSE]` | `_onWsClose()` / `close()` | Badge código: `1000/1001 = OK (gris)`, cualquier otro = ERR (rojo). Reason logueado si existe |
|
||||
|
||||
Contador chunks audio global: `window.__s2sMicChunkCount` para diagnosticar saturación micrófono 25 Hz.
|
||||
Contador sesiones totales: `window._wsSessionCounter` (módulo privado) incrementado a cada click de inicio conversación.
|
||||
|
||||
#### 3. Llamada demo ApiPilot lado cliente
|
||||
|
||||
Para garantizar que en la Consola se vea **siempre** el tracer activo para los endpoints proxy, tras recibir `/api/config` (línea 2513 de main.js) — si y sólo si el feature flag `json.apipilot = true` (es decir, `S2S_APIPILOT_TOOLS_URL` valorado en el servidor) — se envía una fetch `GET api/apipilot/tools?include_unauthorized=true` (fire-and-forget, cache: no-store).
|
||||
|
||||
La demo produce:
|
||||
- `🔧 [APIPILOT-CLIENT] flag=true → DEMO CALL api/apipilot/tools?include_unauthorized=true` (badge morado, antes del tracer)
|
||||
- Luego **el wrapper fetch loguea** `→ [FETCH:APIPILOT:TOOLS] GET api/apipilot/tools?...` + `← [FETCH:APIPILOT:TOOLS] 200 GET (N ms)` (verde) con la lista herramientas en preview
|
||||
- Finalmente `🔧 [APIPILOT-CLIENT] demo OK: N herramientas [primeras 5: tool1, tool2,…]` (badge verde) ó `…demo FAIL` (ámbar) con sugerencia de mirar `[APIPILOT-DISCOVERY]` en el log servidor
|
||||
|
||||
Valor exportado en `window.__s2sApipilot = bool` para tests manuales desde Consola:
|
||||
```js
|
||||
// Ejecuta una herramienta manualmente para ver [FETCH:APIPILOT:EXEC] en el tracer
|
||||
fetch("api/apipilot/execute", {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({name: "nombre_herramienta", arguments: {parametro: "valor"}})
|
||||
}).then(r => r.json()).then(console.log);
|
||||
```
|
||||
|
||||
#### 4. Archivos modificados (con referencias de línea)
|
||||
|
||||
- **`s2s-client/main.js`**:
|
||||
- Banner versión actualizado: [L28-L31](file:///home/duxiter/s2s/s2s-client/main.js#L28-L31)
|
||||
- Wrapper fetch global `_installFetchDebugTracer()` IIFE — clasificación tags, preview request/response, paleta status, performance timing, anti-spam poll/heartbeat: [L41-L251](file:///home/duxiter/s2s/s2s-client/main.js#L41-L251)
|
||||
- `fetchConfig()` — nuevo flag `apipilotEnabled` + `window.__s2sApipilot`: [L2526-L2537](file:///home/duxiter/s2s/s2s-client/main.js#L2526-L2537)
|
||||
- `fetchConfig()` — llamada demo fire-and-forget ApiPilot tools list: [L2581-L2617](file:///home/duxiter/s2s/s2s-client/main.js#L2581-L2617)
|
||||
- **`s2s-client/ws/s2s-ws-client.js`**:
|
||||
- Banner tracer WS actualizado + paleta `_wsLog(status, tag, msg)` status 0..3 + contador chunks `_wsMicChunkCount` + caché cliente ApiPilot `_apipilotClientCache` TTL 240 s + contador sesiones `_wsSessionCounter`: [L56-L98](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L56-L98)
|
||||
- `connect()` — contador #N sesión, log inicio/fin handshake, Promise.all([audioReady, wsReady, **apipilotReady**]): [L320-L370](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L320-L370)
|
||||
- **`_prefetchApipilotTools(sessionIdShort)`** — NUEVO MÉTODO: check feature flag, caché TTL cliente 240 s, fetch `api/apipilot/tools` (proxy servidor `S2S_APIPILOT_TOOLS_URL`), guardado caché, chequeo envelope, badge log MISS/HIT/READY/FAIL/FALLIDO — NUNCA re-lanza excepción: [L372-L441](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L372-L441)
|
||||
- `_postSession()` — timing handshake grant/queued/402/503: [L469-L520](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L469-L520)
|
||||
- `_pollQueue()` — poll counter + advancement: [L523-L573](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L523-L573)
|
||||
- `_openWebSocket()` — ENMASCARAMIENTO URL connect params (anti-fuga token/JWT): [L682-L728](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L682-L728)
|
||||
- `_onWsMessage()` — high-signal regex, preview `response.function_call_arguments.done`: [L755-L812](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L755-L812)
|
||||
- `_onWsClose()` — badge status code OK/ERR: [L1161-L1180](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L1161-L1180)
|
||||
- `_send()` — anti-spam `input_audio_buffer.append` 250 chunks / 2500 ms: [L1523-L1584](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L1523-L1584)
|
||||
- **`s2s-client/ui/account.js`**:
|
||||
- Llamada `GET api/me` traceada automáticamente por el wrapper fetch global como `[FETCH:ACCOUNT:ME]` → sin modificaciones necesarias (ningún log duplicado).
|
||||
|
||||
#### 5. Dónde ver las llamadas UPSTREAM (S2S_APIPILOT_TOOLS_URL)
|
||||
|
||||
⚠️ **Importante — arquitectura passthrough**:
|
||||
- ✅ **Consola navegador** → ves `[FETCH:APIPILOT:TOOLS]` y `[FETCH:APIPILOT:EXEC]` = llamadas locales al proxy `/api/apipilot/*` en el servidor S2S
|
||||
- ✅ **Log servidor (terminal backend)** → ves `[APIPILOT-DISCOVERY] → GET https://aquachile.azurianlab.com/api/v1/apipilot/tools?...` y `[APIPILOT-EXEC] → POST …/execute` = llamadas reales upstream a la URL AquaPortal (`S2S_APIPILOT_TOOLS_URL`)
|
||||
|
||||
Esto es así por diseño: el cliente nunca conoce la URL upstream AquaPortal (anti-hardcoding), todo pasa por el proxy local en `server.py` que añade headers auth y validación schema PRE-CHECK.
|
||||
|
||||
### Solucionado — Bug: ApiPilot llamado 1 sola vez (anónimo) y NUNCA más tras login phone o refresh
|
||||
|
||||
**Síntoma observado del log aquachile aportado**:
|
||||
- A las `21:06:46` 1 única llamada `[apipilot] list_tools.start user=(anon)` a `/api/v1/apipilot/tools` (llamada demo anónima bootstrap)
|
||||
- Inmediatamente después: múltiples `POST /auth/phone-login` y `GET /auth/user-by-phone` (usuarios autenticados, 21:06:49, 21:07:16, 21:07:19, 21:07:39) pero **NINGUNA** llamada apipilot repetida.
|
||||
|
||||
**Causa raíz doble**:
|
||||
1. **Caché cliente global (240 s) user-agnostic**: la demo bootstrap anónima y las sesiones autenticadas compartían la misma entry `_apipilotClientCache` sin clave por usuario. Por tanto la caché anónima HIT se reutilizaba para usuarios autenticados DIFERENTES.
|
||||
2. **Caché servidor S2S (300 s) keyed por auth fingerprint** (`SHA-256(auth)[:12] | x-demo-session`) pero desde el cliente **no se propagaba ningún token `x-demo-session`**, así todas las llamadas usaban la fingerprint "vacía" (anónima) = caché anónima reutilizada sin llamar nunca AquaPortal para usuarios autenticados.
|
||||
3. **Trigger demo solo al bootstrap**: ninguna llamada apipilot después de `phone-login` correcto, que es el momento en que cambia el contexto de usuario.
|
||||
|
||||
---
|
||||
|
||||
#### Fix aplicados
|
||||
|
||||
##### Fix 1 — Prefetch ApiPilot DESPUÉS de CADA login phone correcto (con `refresh=true`)
|
||||
Insertado bloque fire-and-forget **justo después** de la confirmación HTTP 200 en `fetchVoceroUserProfile()` (después de cargar perfil):
|
||||
|
||||
| Característica | Valor |
|
||||
|---|---|
|
||||
| Trigger | `api/user/phone?phoneNumber=…` HTTP 200 + objeto profile.result válido |
|
||||
| Endpoint | `GET api/apipilot/tools?include_unauthorized=true&refresh=true` |
|
||||
| Parámetro `refresh=true` | **OBLIGATORIO** → fuerza al proxy S2S lado servidor a **ignorar la caché 300 s** y llamar realmente AquaPortal upstream (así `[apipilot] list_tools.start user=…` reaparece en el log aquachile PARA CADA login phone) |
|
||||
| Headers | Propaga `x-demo-session` si existe en `window.__s2sDemoSessionToken` (para fingerprint auth S2S distinta a anónima) |
|
||||
| Efecto colateral | Invoca `window.__s2sInvalidateApipilotCache()` = invalida caché cliente WS para todas las sesiones vocales sucesivas → ninguna reutilización de caché para nuevo usuario |
|
||||
| Log Consola | `🔧 [APIPILOT-CLIENT] post-login phone → FUERZO refresh api/apipilot/tools (badge morado)` + `post-login OK (dt=XYZ ms): N=X herramientas` o `post-login FAIL` |
|
||||
|
||||
Referencia de línea: [main.js#L2418-L2464](file:///home/duxiter/s2s/s2s-client/main.js#L2418-L2464)
|
||||
|
||||
##### Fix 2 — Caché cliente WS por-usuario (invalida si cambia phone/token)
|
||||
La entry `_apipilotClientCache` ahora incluye la propiedad `key = _apipilotUserKey()`.
|
||||
|
||||
Clave `_apipilotUserKey()` calculada como:
|
||||
1. `P:<últimos-8-dígitos-phoneNumber>` → si existe `?phoneNumber=` en URL
|
||||
2. `T:<últimos-6-char-x-demo-session-token>` → si existe token
|
||||
3. `anon` → en otro caso
|
||||
|
||||
HIT caché ahora requiere **3 condiciones**:
|
||||
1. `now - cached.at <= 240 s` (TTL aún activo)
|
||||
2. `cached.key === currentUser` (mismo phone / mismo token → mismo usuario)
|
||||
3. Feature flag `window.__s2sApipilot = true`
|
||||
|
||||
Referencia de línea: [s2s-ws-client.js#L94-L126](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L94-L126) (caché+clave+invalidación global) · [L411-L485](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L411-L485) (método `_prefetchApipilotTools` con check clave usuario, refresh=true si cambia usuario y propagate header x-demo-session)
|
||||
|
||||
##### Fix 3 — Propagación header `x-demo-session` para huella auth lado servidor S2S
|
||||
Ahora, **cuando exista** `window.__s2sDemoSessionToken` (length > 4), ambas llamadas apipilot (prefetch login + WS connect) propagan:
|
||||
```
|
||||
x-demo-session: <token>
|
||||
```
|
||||
→ la caché lado servidor S2S (`SHA-256(auth)[:12] | x-demo-session`) usa clave **distinta para cada token/sesión** = **NINGUNA compartición caché entre Anónimo y Usuario Autenticado** ni entre usuarios distintos.
|
||||
|
||||
##### Fix 4 — Función global invalidación caché para cross-módulo
|
||||
```js
|
||||
window.__s2sInvalidateApipilotCache(); // resetea _apipilotClientCache = null
|
||||
```
|
||||
Llamada automáticamente por `main.js` tras login phone correcto. Útil también para tests manuales desde Consola navegador.
|
||||
|
||||
Referencia de línea: [s2s-ws-client.js#L111-L126](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L111-L126)
|
||||
|
||||
##### Cronograma esperado en el log AquaPortal DESPUÉS de este fix (log aportado repetido)
|
||||
```
|
||||
21:06:46 [apipilot] list_tools.start user=(anon) ← demo bootstrap anónimo (refresh page)
|
||||
21:06:46 [apipilot] list_tools.ok count=1 …
|
||||
21:06:49 [apipilot] list_tools.start user=<username-A> ← FIX #1 post-login phone refresh=true ✅ NUEVO
|
||||
21:06:49 [apipilot] list_tools.ok count=…
|
||||
21:07:16 [apipilot] list_tools.start user=<username-B> ← FIX #1 login nuevo usuario ✅ NUEVO
|
||||
21:07:16 [apipilot] list_tools.ok …
|
||||
21:07:19 [apipilot] list_tools.start user=<username-A> ← si caché cliente S2S invalidada o
|
||||
21:07:19 … x-demo-session cambiado
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Añadido — Integración ApiPilot AquaPortal (herramientas dinámicas, sin hardcoding)
|
||||
|
||||
Implementación lado proxy `s2s-client/server.py` del puente definido en
|
||||
[GUIDA_APIPILOT_CLIENTE.md](file:///home/duxiter/s2s/GUIDA_APIPILOT_CLIENTE.md).
|
||||
|
||||
**Objetivo**: permitir al cliente Vocero **consumir dinámicamente** las
|
||||
herramientas expuestas por el backend AquaPortal vía ApiPilot, **sin hardcodear
|
||||
nombres de endpoints, parámetros ni autorizaciones** en el código cliente.
|
||||
Todo el conocimiento sobre "qué herramientas existen / qué parámetros aceptan /
|
||||
si requieren autenticación" se **lee en runtime** vía descubrimiento
|
||||
estandarizado.
|
||||
|
||||
#### Entorno requerido (`.env`)
|
||||
|
||||
| Variable | Valor ejemplo | Significado |
|
||||
|---|---|---|
|
||||
| **`S2S_APIPILOT_TOOLS_URL`** | `https://aquachile.azurianlab.com/api/v1/apipilot` | URL **base** (sin `/tools` ni `/execute`) del backend ApiPilot. Si está vacía → feature completamente desactivada (passthrough). |
|
||||
| `S2S_APIPILOT_SERVICE_TOKEN` | `optional_Bearer_service_token` | Token de respaldo usado solo si el llamante no envía `Authorization` ni `x-demo-session`. |
|
||||
| `S2S_APIPILOT_TTL_SEC` | `300` | TTL cache en memoria del descubrimiento de herramientas (por defecto 300 s = 5 min — según guía). Rango admitido [15, 3600]. |
|
||||
| `S2S_APIPILOT_TIMEOUT_SEC` | `15.0` | Timeout HTTP descubrimiento + ejecución (por defecto 15 s). Rango [1, 120]. |
|
||||
|
||||
#### Endpoints públicos expuestos por el proxy (mismo origen)
|
||||
|
||||
| Endpoint | Verbo | Descripción |
|
||||
|---|---|---|
|
||||
| `/api/apipilot/tools?include_unauthorized=true&refresh=false` | **GET** | Lista herramientas conforme §2.1 (`ToolDefinition`: `name, description, parameters, requires_auth`). Respuesta envuelta `{success, data:{count, include_unauthorized, tools}, error}`. Soporta `?refresh=true` para forzar refresco de caché. |
|
||||
| `/api/apipilot/execute` | **POST** | Ejecuta una herramienta conforme §2.2. Body: `{name: string, arguments: {}}`. PRE-CHECK obligatorios antes de la llamada HTTP. Respuesta SIEMPRE en envoltorio `{success, data, error}`. HTTP status code devuelto = `error.status_code` cuando existe (4xx/5xx). |
|
||||
|
||||
Extensión de `/api/config`: nueva clave booleana de primer nivel
|
||||
`"apipilot": true/false` (basada en `S2S_APIPILOT_TOOLS_URL`). El frontend la
|
||||
usa para decidir si mostrar el catálogo / los botones "Ejecutar herramienta"
|
||||
(obtiene la lista desde `/api/apipilot/tools` en vez de hardcodearla).
|
||||
|
||||
#### PRE-CHECK obligatorios (flujo guía §2.2, todos implementados)
|
||||
|
||||
1. **Descubrimiento fresco** — antes de cada `execute` se usa la lista tools
|
||||
recuperada por `_apipilot_fetch_tools(...)` respetando la TTL. Si la
|
||||
herramienta no se encuentra, **una sola** re-descubrimiento forzado antes
|
||||
de fallar.
|
||||
2. **Control `requires_auth`** — si `true` pero ningún token presente
|
||||
(Authorization / x-demo-session / S2S_APIPILOT_SERVICE_TOKEN) → bloqueo
|
||||
HTTP 401 con envoltorio `{code: "AUTH_REQUIRED", message: "La
|
||||
herramienta «nombre» requiere autenticación…"}` — **NINGUNA solicitud se
|
||||
envía al backend upstream** en este caso.
|
||||
3. **Validación JSON-Schema strict** (`_apipilot_validate_arguments`):
|
||||
- elimina **TODAS** las claves extra no presentes en `parameters.properties`
|
||||
(aviso `[APIPILOT-VALIDATE] … removing unknown arg=…`);
|
||||
- **NO inventa parámetros**;
|
||||
- devuelve error 400 `INVALID_ARGUMENTS` por cada campo `required`
|
||||
faltante (mensajes en español).
|
||||
4. **Passthrough cabeceras autenticación**:
|
||||
`Authorization: Bearer <TOKEN_USUARIO>` o bien `x-demo-session:
|
||||
<SESSION_TOKEN>` se reenvían VERBATIM al backend AquaPortal (§2.1 nota),
|
||||
**NUNCA se loguean** (solo presencia: log `auth=Y / demo=Y`).
|
||||
5. **Compatibilidad LLM function-calling**: helper
|
||||
`_apipilot_strip_tool_for_llm(tool_def)` que proyecta `ToolDefinition` al
|
||||
formato estándar OpenAI `{type: "function", function: {name, description,
|
||||
parameters}}` — **elimina `requires_auth`** (§4 guía: "no mandar
|
||||
directamente la salida cruda de /tools al modelo").
|
||||
|
||||
#### Garantías / seguridad / anti-leak
|
||||
|
||||
| Control | Implementación |
|
||||
|---|---|
|
||||
| Sin compartir caché entre identidades distintas | Clave caché = `SHA-256(authorization)[:12] \| x-demo-session` → huella anónima, no guarda tokens en caché |
|
||||
| Doble comprobación caché con candado asíncrono | `_APIPILOT_CACHE_LOCK` async + double check tras adquirir lock → 1 sola fetch por refresco |
|
||||
| Fallback obsoleto ante error de red | Si backend cae y tenemos entrada en caché → devuelve fallback logueando warning en vez de crash |
|
||||
| Reenvío cabecera Auth nunca logueado | Solo log `auth=Y/- demo=Y/-` |
|
||||
| Filtrado parámetros unknown | Eliminados, nunca reenviados al backend |
|
||||
| Envoltorio errores uniforme | Siempre `{success, data, error}` incluso para errores pre-check locales |
|
||||
| Nombres env coherentes | Ninguna URL ni token hardcodeados |
|
||||
| Banner startup confirmación | `[STARTUP] … apipilot=<URL> apipilotServiceToken=Y/- apipilotTtl=<N>s apipilotTimeout=<N>s` |
|
||||
|
||||
**Archivos modificados:**
|
||||
|
||||
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)) — Sección APIPILOT completa insertada entre `OPENAI_PROJECT` y `NODE_RED_REASONING_URL`:
|
||||
1. Definición 4 env + normalización + límites rango TTL/timeout
|
||||
([L236-L266](file:///home/duxiter/s2s/s2s-client/server.py#L236-L266)).
|
||||
2. Caché en-proceso `_APIPILOT_CACHE: dict[fp → (monotonic, tools[])]` + lock
|
||||
asíncrona `_APIPILOT_CACHE_LOCK` ([L268-L271](file:///home/duxiter/s2s/s2s-client/server.py#L268-L271)).
|
||||
3. Helpers puros: `_apipilot_auth_fingerprint(auth, demo)`
|
||||
([L274-L287](file:///home/duxiter/s2s/s2s-client/server.py#L274-L287)) · `_apipilot_parse_authorization(Request)`
|
||||
([L290-L309](file:///home/duxiter/s2s/s2s-client/server.py#L290-L309)) · `_apipilot_strip_tool_for_llm(tool_def)`
|
||||
([L312-L338](file:///home/duxiter/s2s/s2s-client/server.py#L312-L338)).
|
||||
4. Descubrimiento: `_apipilot_fetch_tools(auth, demo, include_unauthorized, force_refresh)`
|
||||
([L341-L481](file:///home/duxiter/s2s/s2s-client/server.py#L341-L481)) — log cache hit, fallback obsoleto, desempaquetado payload `list/{tools,result,data}.tools`, saneamiento mínimo ToolDefinition, logs `[APIPILOT-DISCOVERY] →/←`.
|
||||
5. Validación: `_apipilot_validate_arguments(arguments, tool_def)`
|
||||
([L484-L517](file:///home/duxiter/s2s/s2s-client/server.py#L484-L517)) — elimina extras, check required, mensajes en español.
|
||||
6. Ejecución: `_apipilot_execute_tool(auth, demo, name, arguments)`
|
||||
([L520-L684](file:///home/duxiter/s2s/s2s-client/server.py#L520-L684)) — PRE-CHECK 1-2-3 (re-discovery 1 volta, requires_auth 401, validación 400), POST /execute envoltorio, manejo HTTP no-JSON, wrap envoltorio si backend no lo devuelve, logs `[APIPILOT-EXEC] →/←`.
|
||||
7. Banner `[STARTUP] Other externals`: extendido con 4 nuevos placeholders `apipilot`, `apipilotServiceToken`, `apipilotTtl`, `apipilotTimeout`
|
||||
([L726-L752](file:///home/duxiter/s2s/s2s-client/server.py#L726-L752)).
|
||||
8. **`/api/config`**: nueva clave booleana `"apipilot": bool(S2S_APIPILOT_TOOLS_URL)` + docstring, log extendido
|
||||
([L2544-L2579](file:///home/duxiter/s2s/s2s-client/server.py#L2544-L2579)).
|
||||
9. **Modelo Pydantic** `ApipilotExecuteRequest(name:str, arguments:dict={})` ([L2582-L2585](file:///home/duxiter/s2s/s2s-client/server.py#L2582-L2585)).
|
||||
10. **`GET /api/apipilot/tools`** proxy público ([L2588-L2651](file:///home/duxiter/s2s/s2s-client/server.py#L2588-L2651)):
|
||||
query `?include_unauthorized=true(default)&refresh=false`, envoltorio uniforme `{success, data: {count, include_unauthorized, tools}, error}`, 501 si env no configurado, HTTPException convertidas a envoltorio.
|
||||
11. **`POST /api/apipilot/execute`** proxy público ([L2654-L2700](file:///home/duxiter/s2s/s2s-client/server.py#L2654-L2700)):
|
||||
body `{name, arguments}`, PRE-CHECK 1-2-3 vía `_apipilot_execute_tool`, código status respuesta = `error.status_code` si error, mensajes de cara al usuario en español §2.2.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-23]
|
||||
|
||||
### Añadido — Botón «Descargar conversación» en la cabecera del panel chat (export JSON + TXT client-side offline)
|
||||
|
||||
Nuevo botón circular icono descarga junto al testigo «En línea» a la derecha
|
||||
de la cabecera del panel `#chat-panel`. Al pulsar serializa TODO el
|
||||
historial persistente (`.hist-msg` dentro de `#chat-history`) y descarga DOS
|
||||
archivos simultáneamente, SIN llamadas al backend, 100 % client-side y
|
||||
offline-safe (Blob + URL.createObjectURL + `<a download>`):
|
||||
|
||||
1. `vocero-conversacion[-<nombreUsuario>]-YYYYMMDD-HHMMSS.json` — estructura
|
||||
con metadatos `{ exportedAt, messageCount, messages: [...] }`. Tipos soportados:
|
||||
- `user` / `assistant` → `{ type, text, timestamp, name?, initials?, interrupted? }`
|
||||
- `tool` → `{ type: "tool", timestamp, name, args, output }` (resultado llamadas tool)
|
||||
- `image` → `{ type: "image", timestamp, dataUrl }` (capturas webcam camera tool)
|
||||
2. `vocero-conversacion[-<nombreUsuario>]-YYYYMMDD-HHMMSS.txt` — log legible
|
||||
para humanos en español, cada entrada formateada como
|
||||
`HH:MM Nombre / Vocero / HERRAMIENTA <nombre>: texto`.
|
||||
|
||||
**UI / UX (todo en español, como solicitado):**
|
||||
- `aria-label="Descargar conversación"`
|
||||
- `title="Descargar conversación (JSON + TXT)"`
|
||||
- **Feedback OK**: tras export correcto botón se pinta verde 650 ms
|
||||
(`.chat-export-ok` con ring + glow).
|
||||
- **Feedback vacío**: si no hay mensajes → sin descarga, botón flash
|
||||
ámbar + micro-shake 0.42s (`.chat-export-empty`) + console info
|
||||
`[ui/chat.js:exportConversation] No hay mensajes aún — exportación omitida.`
|
||||
- **Nombres de fichero saneados**: helper `_safeName(...)` elimina unicode /
|
||||
caracteres especiales / espacios → `a-z 0-9 - _`.
|
||||
- **Estilos coherentes**: dark (glass/aurora) + white (estilo screenshot móvil)
|
||||
ambos tematizados con colores existentes `var(--brand)`, `#16a34a`, `#f59e0b`.
|
||||
|
||||
**Garantías:**
|
||||
- 🔒 Cero datos transmitidos fuera del navegador (ningún POST, ningún analytics).
|
||||
- Las burbujas efímeras sobre el orbe (`.bubble` / `.bubble-stack`) se
|
||||
ignoran intencionadamente: exporta SOLO el histórico permanente, el que es significativo.
|
||||
- Never-throws: todo el export envuelto en `try/catch`; error solo en consola.
|
||||
- `window.URL.revokeObjectURL(...)` tras 4 s → sin fugas de memoria Blob.
|
||||
- Disponible en TODOS los temas (dark / white / default).
|
||||
|
||||
**Archivos modificados:**
|
||||
|
||||
- **`index.html`** ([index.html](file:///home/duxiter/s2s/s2s-client/index.html)):
|
||||
1. Wrapper `.chat-header-actions` a la derecha de la cabecera, que engloba
|
||||
la pill «En línea» + nuevo botón `#chat-export-btn` (SVG flecha-abajo)
|
||||
([L374-L394](file:///home/duxiter/s2s/s2s-client/index.html#L374-L394)).
|
||||
|
||||
- **`ui/chat.js`** ([chat.js](file:///home/duxiter/s2s/s2s-client/ui/chat.js)):
|
||||
1. Helper module-level `_safeName(prefix)` para sanear nombres de fichero
|
||||
([L38-L50](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L38-L50)).
|
||||
2. Ref `this._exportBtn = getElementById("chat-export-btn")` + listener `click`
|
||||
en constructor `ChatView` ([L74-L78](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L74-L78)).
|
||||
3. Nueva sección `// ── Export / download conversation ──` con 4 métodos:
|
||||
- `_serializeConversation()` → extrae rows `.hist-msg` via querySelector
|
||||
`:scope > .hist-msg`, gestiona `user/assistant/tool/image`, preserva
|
||||
timestamp `data-timestamp`, nombre usuario, initials, flag interrupted
|
||||
([L521-L582](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L521-L582)).
|
||||
- `_conversationToText(conv)` → pretty-format texto en español con
|
||||
cabecera fecha, totales, bloques tool indentados, preview imagen dataURL
|
||||
([L584-L621](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L584-L621)).
|
||||
- `_downloadBlob(content, filename, mime)` → patrón `<a download>` +
|
||||
revokeObjectURL ([L623-L645](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L623-L645)).
|
||||
- `exportConversation()` (público) → entry-point, nombre fichero con
|
||||
sello UTC local `YYYYMMDD-HHMMSS`, incluye nombre perfil Vocero si
|
||||
existe, lógica empty/OK/error feedback, 2 descargas distintas
|
||||
([L647-L697](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L647-L697)).
|
||||
|
||||
- **`style.css`** ([style.css](file:///home/duxiter/s2s/s2s-client/style.css)):
|
||||
1. `.chat-header-actions` flex-row right-aligned
|
||||
([L4919-L4931](file:///home/duxiter/s2s/s2s-client/style.css#L4919-L4931)).
|
||||
2. `.chat-export-btn` base (dark-theme): 32×32 pill, glassy, hover/focus
|
||||
states, transiciones 120-160 ms
|
||||
([L4932-L4968](file:///home/duxiter/s2s/s2s-client/style.css#L4932-L4968)).
|
||||
3. Estados feedback: `.chat-export-ok` (verde) + `.chat-export-empty`
|
||||
(ámbar + animación shake 5 pasos)
|
||||
([L4969-L4989](file:///home/duxiter/s2s/s2s-client/style.css#L4969-L4989)).
|
||||
4. Variantes white-theme completas: `.chat-header-actions`, base, hover,
|
||||
active, OK, empty
|
||||
([L4991-L5015](file:///home/duxiter/s2s/s2s-client/style.css#L4991-L5015)).
|
||||
|
||||
### Añadido — `S2S_LATEST_SUMMARY_TEXT`: override estático del resumen última conversación (texto limpio, ninguna llamada HTTP)
|
||||
|
||||
Nueva variable de entorno que, si se establece, tiene **prioridad absoluta** sobre `S2S_LATEST_SUMMARY`
|
||||
y se salta completamente la fetch HTTP upstream (login-by-phone, Bearer, timeout, JSON unwrap: *todo bypaseado*).
|
||||
Su valor en texto plano se inyecta directamente como bloque "latest conversation summary"
|
||||
dentro del prompt LLM (tanto en el canal WebSocket `/v1/realtime` como en `POST /api/reasoning`).
|
||||
|
||||
Diseñada para el caso de uso en el que el operador quiere inyectar una versión
|
||||
**limpia, esencial y pre-depurada** del resumen sin depender de ningún endpoint externo.
|
||||
|
||||
**Valor `.env` de ejemplo (multilínea con `\n`):**
|
||||
```env
|
||||
# S2S_LATEST_SUMMARY=https://api.example.com/summary # ← desactivado: S2S_LATEST_SUMMARY_TEXT tiene prioridad
|
||||
S2S_LATEST_SUMMARY_TEXT=Última conversación 2026-09-22:\n- Usuario preguntó por mortalidad salmón AquaChile semana 38.\n- Se refirió al reporte Porcentaje Mortalidad.
|
||||
```
|
||||
|
||||
**Comportamiento clave:**
|
||||
- Las secuencias `\n` en el valor env se interpretan como salto de línea real (misma convención que `S2S_PRIMARY_PROMPT`)
|
||||
- **No requiere `phoneNumber`**: funciona también para sesiones genéricas sin identificativo de llamante
|
||||
- El wrapper `LATEST_SUMMARY_HINT` + el tope a **1600 caracteres** se siguen aplicando igualmente (pasa siempre a través de `_resolve_latest_summary_context`)
|
||||
- El bloque se clasifica correctamente por el layer dedup (marker invisible `SUMMARY_FULL_MARKER`) — sin riesgo de duplicado
|
||||
- Si `S2S_LATEST_SUMMARY_TEXT` está vacío → **fallback** transparente al flujo anterior (`S2S_LATEST_SUMMARY` URL + `phoneNumber`)
|
||||
|
||||
**Logs esperados en el servidor para confirmar override activo:**
|
||||
```
|
||||
[INJECT-SUMMARY] S2S_LATEST_SUMMARY_TEXT OVERRIDE active (len=<N> chars). Skipping HTTP fetch; injecting raw static plain-text summary directly.
|
||||
```
|
||||
|
||||
**Archivos modificados:**
|
||||
|
||||
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)):
|
||||
1. Definición nueva env `S2S_LATEST_SUMMARY_TEXT` + interpretación escape `\n` ([L128-L140](file:///home/duxiter/s2s/s2s-client/server.py#L128-L140)).
|
||||
2. Banner `[STARTUP]` extendido con campo `summaryText=%s` para confirmación visual a boot ([L275-L297](file:///home/duxiter/s2s/s2s-client/server.py#L275-L297)).
|
||||
3. **Rama override** ANTES de cualquier lógica HTTP dentro de `_fetch_latest_summary_for_phone(...)` ([L1563-L1589](file:///home/duxiter/s2s/s2s-client/server.py#L1563-L1589)):
|
||||
- return directo del valor estático;
|
||||
- log `[INJECT-SUMMARY] … OVERRIDE active`;
|
||||
- vista previa raw primeros 800 chars si `S2S_DEBUG_PROMPT=true`.
|
||||
4. **Puerta REASONING:** `needs_summary = bool(S2S_LATEST_SUMMARY) or bool(S2S_LATEST_SUMMARY_TEXT)` ([L2153](file:///home/duxiter/s2s/s2s-client/server.py#L2153)).
|
||||
5. **Skip REASONING relajado:** condición `(not phone) and (not needs_reports) and (not episode_id) and (not S2S_LATEST_SUMMARY_TEXT)` — el texto estático no necesita phone ([L2159-L2168](file:///home/duxiter/s2s/s2s-client/server.py#L2159-L2168)).
|
||||
6. **Ejecución summary REASONING:** `needs_summary and (phone or S2S_LATEST_SUMMARY_TEXT)`; log "READY" extendido con flag `summaryTextOverride=%s` ([L2240-L2256](file:///home/duxiter/s2s/s2s-client/server.py#L2240-L2256)).
|
||||
7. **Puerta WS connect:** mismo `needs_summary` extendido con `S2S_LATEST_SUMMARY_TEXT` ([L2644](file:///home/duxiter/s2s/s2s-client/server.py#L2644)).
|
||||
8. **Skip WS connect relajado:** incluso sin `phoneNumber` si hay `S2S_LATEST_SUMMARY_TEXT` la pipeline arranca ([L2653-L2664](file:///home/duxiter/s2s/s2s-client/server.py#L2653-L2664)).
|
||||
9. **Ejecución summary WS:** misma condición relajada + log "READY" con campo `summaryTextOverride=%s` ([L2725-L2746](file:///home/duxiter/s2s/s2s-client/server.py#L2725-L2746)).
|
||||
|
||||
- **`INJECTIONS.md`** ([INJECTIONS.md](file:///home/duxiter/s2s/s2s-client/INJECTIONS.md)):
|
||||
1. Sección **4. Blocco summary ultima conversazione** completamente reescrita (versión italiano original).
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-07]
|
||||
|
||||
### Añadido — Soporte `episode_id` (inject de episodio podcast)
|
||||
|
||||
Nuevo parámetro query string `?episode_id=` junto a `?phoneNumber=` que
|
||||
habilita la carga y la inyección automática de un episodio podcast como
|
||||
contexto PRIMARIO en el prompt LLM.
|
||||
|
||||
**URL de ejemplo:**
|
||||
```
|
||||
/?phoneNumber=%2B447432477433&episode_id=episode-19569f77-a06e-5ce5-a52f-6299dbd67d03
|
||||
```
|
||||
|
||||
**Archivos modificados:**
|
||||
|
||||
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)):
|
||||
1. Añadida la constante `PODCAST_EPISODE_HINT` ([L426-L429](file:///home/duxiter/s2s/s2s-client/server.py#L426-L429)) para el hint semántico en el prompt.
|
||||
2. Añadida la función `_resolve_episode_context(raw_value)` ([L475-L504](file:///home/duxiter/s2s/s2s-client/server.py#L475-L504)): normaliza la respuesta API (dict/list/str, posible wrapper `{"result": …}`), la serializa en JSON (tope 24.000 chars) y la envuelve con los marcadores `<<<PODCAST_EPISODE_BEGIN/END>>>`.
|
||||
3. Añadida la función asíncrona `_fetch_podcast_episode(episode_id)` ([L1010-L1078](file:///home/duxiter/s2s/s2s-client/server.py#L1010-L1078)): GET directo a
|
||||
`https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}`
|
||||
(timeout 15s, follow redirects, manejo de errores tolerante — nunca rompe el flujo, devuelve `None` en caso de fallo).
|
||||
4. Extendida `_augment_instructions_with_profile(...)` ([L1137-L1181](file:///home/duxiter/s2s/s2s-client/server.py#L1137-L1181)) con el nuevo parámetro `episode_context_text`; el orden de inject ahora es (más específico → más genérico): **episode → profile → reports → summary**.
|
||||
5. Extendida `_patch_client_session_update_with_profile(...)` ([L1184-L1306](file:///home/duxiter/s2s/s2s-client/server.py#L1184-L1306)) con soporte a `episode_context_text` y logging dedicado `[INJECT-EPISODE]` / `[DEBUG-EPISODE]`.
|
||||
6. WebSocket connect handler ([L1791-L1959](file:///home/duxiter/s2s/s2s-client/server.py#L1791-L1959)):
|
||||
- lee `episode_id` desde query params junto a `phoneNumber`;
|
||||
- ejecuta fetch episodio + resolve ANTES de profile/reports/summary;
|
||||
- guarda `injected_episode_context` e `inject_episode_id` en el `session_ctx` compartido.
|
||||
7. `_client_to_upstream` ([L2649-L2661](file:///home/duxiter/s2s/s2s-client/server.py#L2649-L2661)): pasa `ctx_episode_context` al patch de `session.update`.
|
||||
8. Endpoint `POST /api/reasoning`:
|
||||
- extendido el modelo `ReasoningRequest` con el campo `episode_id: str | None = None` ([L1309-L1323](file:///home/duxiter/s2s/s2s-client/server.py#L1309-L1323));
|
||||
- en el handler ([L1411-L1577](file:///home/duxiter/s2s/s2s-client/server.py#L1411-L1577)) fetch episodio independiente de phone;
|
||||
- `inject_episode_text` se hace **prepend** sobre `user_context_parts` (primero de todos);
|
||||
- reenvío a NODE_RED / OPENAI con logging extendido (`episodeIdLen`).
|
||||
|
||||
- **`main.js`** ([main.js](file:///home/duxiter/s2s/s2s-client/main.js)):
|
||||
1. Lectura desde query string: `const voceroEpisodeId = pageParams.get("episode_id")` ([L676-L680](file:///home/duxiter/s2s/s2s-client/main.js#L676-L680)).
|
||||
2. `fetchOpenAiReasoningAnswer` ([L1337-L1364](file:///home/duxiter/s2s/s2s-client/main.js#L1337-L1364)): campo `episode_id` añadido al body POST `/api/reasoning`, logKey y `console.info` extendidos.
|
||||
3. Construcción URL WebSocket ([L2323-L2342](file:///home/duxiter/s2s/s2s-client/main.js#L2323-L2342)): `episode_id` añadido a query params tanto con la API `URL.searchParams` como en el fallback string (concat con `&`).
|
||||
|
||||
---
|
||||
|
||||
### Añadido — `S2S_START_SESSION_MODE`: 3 modos de inicio de sesión
|
||||
|
||||
Variable de entorno leída por el servidor y propagada al cliente vía `GET /api/config.startSessionMode`.
|
||||
Objetivo: controlar si y cómo se enciende automáticamente el micrófono y si el modelo saluda
|
||||
primero (flujo similar al wake-word). Valores soportados:
|
||||
|
||||
| Valor | Descripción |
|
||||
|---|---|
|
||||
| *(vacío) / default* | Inicio manual: solo tap sobre el Orb / botón Start |
|
||||
| **`TALK`** | Después de 2s overlay a pantalla completa "Toca en cualquier lugar para empezar" (policy safe para Safari/iOS `getUserMedia`). Click/touch/tecla dentro de la user gesture → `doStart()`. El cliente WS inyecta bootstrap `"hola vocero"` + `requestResponse()`. |
|
||||
| **`TALKFIRST`** | Después de 2s **intenta `doStart()` DIRECTAMENTE SIN click**, precargando `pendingWakeWordConfirmationText = "hola vocero"` (misma variable de la wake-word real). Si los permisos de mic ya están "allow always" funciona sin click. Si falla (policy Safari/iOS) → fallback sobre el mismo unlocker de TALK con mensaje "permiso micrófono". |
|
||||
|
||||
**Archivos modificados:**
|
||||
|
||||
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)):
|
||||
1. Lectura env: `S2S_START_SESSION_MODE = os.environ.get(...)`, `.strip().upper()` ([L150](file:///home/duxiter/s2s/s2s-client/server.py#L150)).
|
||||
2. Banner `[STARTUP]` extendido con el campo `startSessionMode=` ([L196-L212](file:///home/duxiter/s2s/s2s-client/server.py#L196-L212)).
|
||||
3. Endpoint `GET /api/config` → nuevo campo `startSessionMode` + log `[INTERNAL] /api/config` ([L1383-L1400](file:///home/duxiter/s2s/s2s-client/server.py#L1383-L1400)).
|
||||
|
||||
- **`index.html`** ([index.html](file:///home/duxiter/s2s/s2s-client/index.html#L32-L39)):
|
||||
- Añadido nuevo overlay `#talk-unlocker` a pantalla completa (z-index 9999), `hidden` por defecto, con tarjeta "Preparando Vocero… / Toca en cualquier lugar para empezar a hablar" + spinner degradado.
|
||||
|
||||
- **`style.css`** ([style.css](file:///home/duxiter/s2s/s2s-client/style.css#L145-L215)):
|
||||
- Clases `.talk-unlocker`, `.talk-unlocker-hint`, `.talk-unlocker-spinner`, `.talk-unlocker-title`, `.talk-unlocker-sub` + animaciones fade-in y pop-up de tarjeta.
|
||||
|
||||
- **`main.js`** ([main.js](file:///home/duxiter/s2s/s2s-client/main.js)):
|
||||
1. Variable global `startSessionMode = ""` + comentario ([L628-L632](file:///home/duxiter/s2s/s2s-client/main.js#L628-L632)).
|
||||
2. Parse dentro de `fetchConfig()` → uppercase, log en `reportDebug("B",…)` ([L2309-L2320](file:///home/duxiter/s2s/s2s-client/main.js#L2309-L2320)).
|
||||
3. Paso a cliente WS → `startWithResponse: startSessionMode === "TALK"` ([L2683-L2692](file:///home/duxiter/s2s/s2s-client/main.js#L2683-L2692)).
|
||||
4. Bloque scheduling auto-start ([L2976-L3083](file:///home/duxiter/s2s/s2s-client/main.js#L2976-L3083)):
|
||||
- `bootstrapWithAutoStart()`: `await fetchConfig()` → `maybeScheduleAutoStart()`;
|
||||
- `setTimeout(2000 ms)` warmup para finalizar fetch de episode/profile/reports;
|
||||
- `showTalkUnlocker(fallbackHint)`: gestiona labels personalizadas en caso de fallback TALKFIRST;
|
||||
- **`tryTalkFirstDirect()` (TALKFIRST)**: set `pendingWakeWordConfirmationText="hola vocero"` + `doStart()` + try/catch con teardown y fallback unlocker;
|
||||
- listener unlocker click/touchstart/keydown → `startTalkFromGesture()` con `createResumedAudioContext()` dentro de la gesture (para iOS silent-free).
|
||||
5. `onClientStatus()` ([L2878-L2932](file:///home/duxiter/s2s/s2s-client/main.js#L2878-L2932)) añadido `hideTalkUnlocker()` en `connecting/queued/your-turn/connected`.
|
||||
|
||||
- **`ws/s2s-ws-client.js`** ([s2s-ws-client.js](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js)):
|
||||
1. Constructor: nuevas propiedades `_startWithResponse`, `_startWithResponseText` (default `"hola vocero"`), `_initialResponseSent` guardia one-shot ([L163-L178](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L163-L178)).
|
||||
2. Handler `session.updated` ([L688-L713](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L688-L713)): **Solo para `TALK`** (no TALKFIRST, que en cambio usa `pendingWakeWordConfirmationText`): `this.sendUserText("hola vocero") + this.requestResponse()` — mismo code path identico de la wake-word.
|
||||
3. `requestResponse` / `_createResponseNow` restaurados limpios (solo payload opcional `image`; ningún shaped `response.create`).
|
||||
|
||||
---
|
||||
|
||||
### Modificado — `#reports-sidebar-toggle` visible solo con `?debug=true`
|
||||
|
||||
El botón "reports" en la topbar (toggle del panel lateral reports) ahora es
|
||||
`data-debug-only="true"` — por lo tanto solo se muestra si la URL contiene
|
||||
`?debug=true`, de manera análoga al cluster `.topbar-right`.
|
||||
|
||||
**Archivos modificados:**
|
||||
|
||||
- **`index.html`** → `#reports-sidebar-toggle` ([index.html](file:///home/duxiter/s2s/s2s-client/index.html#L47-L50)): añadido `data-debug-only="true"`.
|
||||
- **`main.js`** → `applyDebugOnlyVisibilityGate()` ([L48-L109](file:///home/duxiter/s2s/s2s-client/main.js#L48-L109)): extendido para iterar *todos* los elementos `[data-debug-only="true"]` (no solo `.topbar-right`); línea 1227: `reportsSidebarToggle.hidden = false` condicionado a `qs.get("debug") === "true"` ([L1227-L1246](file:///home/duxiter/s2s/s2s-client/main.js#L1227-L1246)) + log rojo `🚫 FORZADO HIDDEN` cuando falta debug=true.
|
||||
- **`style.css`** → regla de seguridad ya presente: `[data-debug-only]:not([data-debug-enabled]) { display:none !important; visibility:hidden; opacity:0 }` ([L532-L537](file:///home/duxiter/s2s/s2s-client/style.css#L532-L537)).
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-04]
|
||||
|
||||
### Modificado — Chat siempre visible + integración seamless
|
||||
|
||||
Reorganizado el layout para mostrar el chat textual de forma permanente e
|
||||
integrado visualmente con el Orb en la parte superior del stage.
|
||||
|
||||
- Eliminado botón toggle chat en la topbar (`#chat-btn`) y relativo overlay.
|
||||
- Stage con el Orb en la parte superior; nuevo `#chat-panel` a continuación (historial
|
||||
+ form input fijo sobre el footer sticky).
|
||||
- Actualizada lógica en `main.js` y `ui/chat.js` para el envío de texto mediante
|
||||
`client.sendUserText` y auto-scroll sin controles de visibilidad del panel.
|
||||
- Versión banner deploy: **2026-09-03 14:40**.
|
||||
|
||||
### Resuelto — Estabilidad integración chat mobile
|
||||
|
||||
- Error `Missing element: #chat-btn` resuelto con selectores non-strict
|
||||
(`document.querySelector`) para elementos opcionales eliminados.
|
||||
- Refactorización estética "seamless": chat sin marcos, sombras ni fondos
|
||||
(`background:transparent`, `box-shadow:none`).
|
||||
- **Layout mobile:**
|
||||
- padding incrementado en `#app` (56px) y `.chat-history` (96px) para evitar
|
||||
superposiciones con el footer sticky;
|
||||
- márgenes fijos en las burbujas: `margin-right:22px` (Assistant), `margin-left:6px` (User).
|
||||
- **Auto-scroll streaming:** implementado `_scrollToBottom` con cadena de retry
|
||||
múltiples (`requestAnimationFrame` + `setTimeout`) para compensar el line-wrap
|
||||
asíncrono durante la llegada de los tokens; `overflow-anchor:none` + `scroll-behavior:auto !important`.
|
||||
|
||||
### Añadido — Avatar usuario con iniciales + modal de perfil
|
||||
|
||||
- Sustituido el texto del nombre de usuario en el header por avatar circular 40×40px
|
||||
(gradient de marca) conteniendo las iniciales calculadas (ej. "AV").
|
||||
- Nueva `<dialog id="user-profile-dialog">` en `index.html` que se abre al click
|
||||
sobre el avatar: muestra email, teléfono, rol, empresa, con soporte a shapes
|
||||
anidadas del payload de perfil.
|
||||
- Cierre del modal: tecla ESC, click fuera de la tarjeta, botón `×` dedicado.
|
||||
|
||||
---
|
||||
|
||||
## [Notas de formato]
|
||||
|
||||
- **Tope campo prompt episodio:** 24.000 caracteres (transcript largos).
|
||||
- **Tope profile / summary:** 1.600 caracteres cada uno.
|
||||
- **Tope reports list:** 12.000 caracteres.
|
||||
- **Orden inject prompt LLM (WS session.update + reasoning):**
|
||||
1. Episodio podcast actual (PRIMARIO)
|
||||
2. Perfil llamador (identidad)
|
||||
3. Lista reports / scripts podcast programados
|
||||
4. Summary última conversación
|
||||
- Todos los bloques están marcados con `<<<NOMBRE_BLOQUE_BEGIN/END>>>` para inspección de logs.
|
||||
+226
-861
File diff suppressed because it is too large
Load Diff
@@ -1,108 +0,0 @@
|
||||
# Context glossary
|
||||
|
||||
Canonical terms for this space. A glossary, not a spec — it defines what words mean,
|
||||
not how anything is built. Keep design/implementation detail in `DESIGN.md` and the
|
||||
code.
|
||||
|
||||
## Speech-to-speech demo
|
||||
The product: a voice conversation you have with a model by tapping the orb and
|
||||
talking. "The demo" and "the space" refer to this same thing. It runs on Hugging
|
||||
Face's open `speech-to-speech` backend.
|
||||
|
||||
## The pipeline
|
||||
The ordered path a turn travels, from your voice to the orb's reply. Order is
|
||||
meaningful — each stage consumes the previous one's output:
|
||||
|
||||
`you speak → VAD → STT → VLM → TTS → orb replies`
|
||||
|
||||
- **VAD** — voice activity detection. Decides *when* you are speaking, so the system
|
||||
knows a turn has started and ended. Model: silero-vad.
|
||||
- **STT** — speech to text. Transcribes your speech into words. Model:
|
||||
nvidia/parakeet-tdt-1.1b.
|
||||
- **VLM** — the vision-language model that composes the reply. Served via Cerebras.
|
||||
Model: google/gemma-4-31B-it.
|
||||
- **TTS** — text to speech. Speaks the reply back in the chosen voice. Model:
|
||||
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice.
|
||||
|
||||
## Builder
|
||||
A Hugging Face user credited with making the space, shown by HF username. Current
|
||||
builders: tfrere, A-Mahla and andito. Distinct from the *models'* authors (nvidia, google,
|
||||
Qwen, snakers4), who are credited per pipeline stage.
|
||||
|
||||
## Powered by
|
||||
The infrastructure running the pipeline, named in the about panel: Hugging Face
|
||||
Inference Endpoints (hosting) and Cerebras (LLM inference). Distinct from "built by"
|
||||
(the people) and from the model authors (who trained each model).
|
||||
|
||||
## Tool
|
||||
A function the model can call mid-conversation to do something the pipeline
|
||||
can't do on its own (look something up, look through the camera). Tools are
|
||||
declared to the backend in the session config; the model decides when to call
|
||||
one. Distinct from the *pipeline stages* (VAD/STT/VLM/TTS), which always run.
|
||||
|
||||
## Tool executor
|
||||
The client-side component that runs a tool when the model calls it and returns
|
||||
the result to the backend, so the model can speak the answer. It is the missing
|
||||
half of the round-trip: the backend already emits the call, the executor runs it
|
||||
and replies. Distinct from the *tool* itself (the thing being run).
|
||||
|
||||
## Web search tool
|
||||
A tool that looks something up on the web for the model. The model calls it with
|
||||
a query; the tool executor forwards the query to the search proxy and returns the
|
||||
results as the tool result. Activates only when a search key is available.
|
||||
|
||||
## Camera snapshot tool
|
||||
A tool that lets the model look through the user's webcam. While enabled, a live
|
||||
self-view is shown in the page (bottom-left); when the model calls the tool, the
|
||||
executor captures a frame and sends it to the model as an image so the VLM can
|
||||
see it. Distinct from the *preview* (what the user sees) and the *snapshot* (the
|
||||
single frame sent to the model).
|
||||
|
||||
## Search proxy
|
||||
The same-origin server route (`/search`) that holds the search key and calls the
|
||||
external search provider on the client's behalf, so the key never reaches the
|
||||
browser. Lives in the same container as the page. Distinct from the *s2s backend*
|
||||
(the separate load-balanced speech-to-speech service).
|
||||
|
||||
## Tools panel
|
||||
The dialog opened from the "Tools" button in the top-right, holding one switch per
|
||||
tool (and the web-search key status). Turning a switch on/off declares or removes
|
||||
that tool on the live session. Distinct from *Settings* (connection, voice,
|
||||
instructions) and the *About panel* (project info).
|
||||
|
||||
## Identity block
|
||||
The top-left corner of the topbar (replacing the old wordmark): the demo name, a
|
||||
one-line blurb, and the "powered by" / "built by" credits, shown directly rather
|
||||
than hidden behind a click.
|
||||
|
||||
## About panel
|
||||
The popup opened from the (i) icon to the right of the identity block. Holds the
|
||||
general introduction to the speech-to-speech project (with a repo link) and the
|
||||
pipeline. Identity itself now lives in the corner, not here.
|
||||
|
||||
## Queue
|
||||
The line of users waiting for a free conversation slot when every compute is
|
||||
busy. You join the queue instead of being turned away; you leave it by reaching
|
||||
the front or by giving up. Distinct from a *session* (an actual live
|
||||
conversation) — being in the queue is not yet talking, and time spent waiting
|
||||
never counts against your usage limit.
|
||||
|
||||
## Ticket
|
||||
Your held place in the queue. Created when you join, it is what the demo checks
|
||||
to tell you your position and to notice if you have left. A ticket is not a
|
||||
session: it only promises a spot in line, not a compute.
|
||||
|
||||
## Position
|
||||
How many people are ahead of you in the queue, shown while you wait ("You're #3
|
||||
in line"). It only ever counts down. Distinct from an *estimated wait* — the
|
||||
demo shows position, never a time, because wait time is unpredictable.
|
||||
|
||||
## Claim
|
||||
The moment you reach the front and a free slot becomes yours — the queue hands
|
||||
off to a real session and the conversation begins. This is also the point where
|
||||
your usage limit first starts to matter (never while waiting).
|
||||
|
||||
## At capacity
|
||||
The state where the queue itself is full, so new users can't even join the line
|
||||
and are asked to try again shortly. Distinct from simply *busy* (all computes
|
||||
taken but the queue still has room to wait in).
|
||||
@@ -1,205 +0,0 @@
|
||||
# Design language
|
||||
|
||||
The reference for keeping this app visually coherent as it grows. Read it before
|
||||
touching `style.css`, `index.html`, or any DOM-building code in `main.js`. Every
|
||||
rule here is already live in the codebase — this file explains the *why* so changes
|
||||
extend the system instead of drifting from it.
|
||||
|
||||
---
|
||||
|
||||
## The thesis: color belongs to the voice
|
||||
|
||||
This is a voice app. The one thing in the room that should have color is the thing
|
||||
that is talking. So:
|
||||
|
||||
- **The orb** carries saturated color. It glows, and the glow's hue changes with
|
||||
conversational state.
|
||||
- **Everything else is monochrome** — a precise cool-grey dark canvas. Surfaces,
|
||||
borders, buttons, panels, the transcript: all greyscale.
|
||||
- The **only** exceptions are tiny *role echoes* (a one-word mono label, a small
|
||||
icon) that borrow the orb's state hue so the transcript reads in the same color
|
||||
language the orb speaks. They are accents the size of a word, never fills.
|
||||
- **Brand logos keep their own color.** The Hugging Face mark (`#ffd21e`) and the
|
||||
Cerebras mark (`#f15a29`) render in their brand colors in the identity credits and
|
||||
the about panel — a deliberate, owner-approved exception. It applies to those two
|
||||
logos only; do not generalize it to other chrome.
|
||||
|
||||
If you find yourself adding a tinted background, a colored border, or a bright
|
||||
button anywhere outside the orb, stop — that color almost certainly belongs to the
|
||||
orb instead, or shouldn't exist.
|
||||
|
||||
---
|
||||
|
||||
## Color tokens
|
||||
|
||||
All defined in `:root` in `style.css`. Use the variables, never raw hex in rules.
|
||||
|
||||
### Canvas (the monochrome world)
|
||||
| Token | Value | Use |
|
||||
|---|---|---|
|
||||
| `--bg` | `#0a0b10` | Page background (a cool near-black) |
|
||||
| `--bg-elev` | `#13151c` | Raised surfaces: bubbles, panels, icon buttons |
|
||||
| `--bg-elev-2` | `#1b1e29` | Surfaces on surfaces: history bodies, inputs |
|
||||
| `--border` | `rgba(255,255,255,.08)` | Default hairline |
|
||||
| `--border-strong` | `rgba(255,255,255,.16)` | Emphasised hairline |
|
||||
| `--text` | `#f5f6fa` | Primary text; also the *primary button* fill |
|
||||
| `--text-dim` | `rgba(245,246,250,.65)` | Secondary text |
|
||||
| `--text-faint` | `rgba(245,246,250,.42)` | Captions, labels, footer |
|
||||
|
||||
### Voice (the only saturated hues)
|
||||
These are the orb's state colors. They appear on the orb, and as small role echoes
|
||||
in the transcript — nowhere else.
|
||||
|
||||
| Token | Value | Meaning |
|
||||
|---|---|---|
|
||||
| `--accent` / `--speaking` | `#8b7dff` violet | Assistant speaking |
|
||||
| `--accent-2` / `--listening` | `#22d3ee` cyan | You / listening |
|
||||
| `--processing` | `#f59e0b` amber | Thinking / tool call |
|
||||
| `--error` | `#ff6a75` | Error |
|
||||
| `--success` | `#34d399` | Ready / connected |
|
||||
|
||||
### Role echoes (semantic aliases — use these in chat code)
|
||||
| Token | Maps to | Where it shows |
|
||||
|---|---|---|
|
||||
| `--voice-user` | cyan | `YOU` label + user bubble accents |
|
||||
| `--voice-assistant` | violet | `ASSISTANT` label + assistant accents |
|
||||
| `--voice-tool` | amber | `TOOL CALL` label, wrench icon |
|
||||
|
||||
**Why this mapping:** it mirrors the orb exactly — when *you* speak the orb is cyan
|
||||
(`state-listening`), when the *assistant* speaks it's violet (`state-ai-speaking`),
|
||||
when it's working it's amber (`state-processing`). The transcript is a quiet replay
|
||||
of the orb's color story.
|
||||
|
||||
### Orb state → glow (`.circle.state-*` → `--glow`)
|
||||
| State | Glow |
|
||||
|---|---|
|
||||
| `signed-out` | violet `#8b7dff` |
|
||||
| `authenticated` / `ready` | green `#34d399` |
|
||||
| `connecting` / `connected` / `starting` | yellow `#facc15` |
|
||||
| `listening` / `user-speaking` | cyan (`--listening`) |
|
||||
| `processing` | amber (`--processing`) |
|
||||
| `ai-speaking` | violet (`--speaking`) |
|
||||
| `error` | red (`--error`) |
|
||||
|
||||
Adding a new state? Give it a `--glow`, and if it surfaces in the transcript, add a
|
||||
matching `--voice-*` alias rather than a one-off color.
|
||||
|
||||
---
|
||||
|
||||
## Typography
|
||||
|
||||
Two faces, two jobs. Never reach for a third.
|
||||
|
||||
- **Inter** — body and UI. Wordmark, buttons, inputs, panel titles, prose, history
|
||||
message bodies. The workhorse; it should feel neutral and get out of the way.
|
||||
- **Geist Mono** (`--font-mono`) — the **machine voice**. Reserved for text the
|
||||
*system* emits or identifiers it reports, never for human prose.
|
||||
|
||||
### When mono is correct
|
||||
Mono signals "this is the machine talking or naming itself." Use it for:
|
||||
- the orb's status caption (`.circle-caption`)
|
||||
- role eyebrows (`YOU` / `ASSISTANT` / `TOOL CALL`)
|
||||
- tool-call names and argument JSON
|
||||
- the `·WebSocket` transport tag, bitrate readouts, connection identifiers
|
||||
- the empty-state label
|
||||
|
||||
Mono text is set uppercase with `letter-spacing: ~0.1–0.14em` and weight `500`, so it
|
||||
reads as a typed status line, not a headline. Body copy, button labels, and
|
||||
explanatory `small` text stay **Inter** — putting prose in mono breaks the metaphor.
|
||||
|
||||
The font is loaded in `index.html`; the stack falls back to system mono gracefully if
|
||||
the CDN is blocked.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
- **One continuous canvas.** No dividers under the topbar or above the footer, no
|
||||
panel chrome competing with content. The topbar and footer float over the stage.
|
||||
Keep it that way — a new section earns a hairline (`--border`) only if it genuinely
|
||||
needs separating.
|
||||
- **The orb is the hero and the center of gravity.** It sits dead-center on the
|
||||
stage. Controls flank it (mic / stop), captions sit beneath. Don't crowd it.
|
||||
- **Hairlines, not boxes.** Separation comes from `1px` borders at 8–16% white and
|
||||
from spacing, not from heavy fills or shadows. Shadows are soft and low
|
||||
(`0 4px 18px rgba(0,0,0,.32)`), used only to lift floating elements (bubbles,
|
||||
panels, modal).
|
||||
- **Radii:** `--radius-sm: 8px` (buttons, inputs, chips), `--radius-md: 14px`
|
||||
(bubbles, message bodies, modal), `--radius-lg: 22px` (reserved). Pick by element
|
||||
size; don't invent new values.
|
||||
- **Two reading surfaces for the transcript:** ephemeral bubbles top-right (desktop
|
||||
only) that log and fade, and a slide-in history panel for review. On phones the
|
||||
bubble stream is dropped and the panel goes full-screen — the panel is the single
|
||||
source of truth there.
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
- **Buttons.** Default (`.btn`) is a neutral elevated surface. The *primary* button
|
||||
is **near-white on dark** (`--text` fill, `--bg` text) — the highest-contrast thing
|
||||
on the page that *isn't* the orb. There is no colored button; emphasis comes from
|
||||
contrast, not hue.
|
||||
- **Icon buttons** (`.icon-btn`) are `36px`, elevated surface, dim icon that brightens
|
||||
on hover. Side controls (`.side-btn`) are circular, collapse to zero size until the
|
||||
session is live (by width on desktop, by height in the mobile column).
|
||||
- **Chat bubbles & history messages** share one neutral surface. They are
|
||||
distinguished by **side** (you = left, assistant = right) plus the **mono role
|
||||
label** in the role-echo hue — not by tinted fills. Tool entries use the wrench icon
|
||||
+ mono + amber, on the same neutral surface.
|
||||
- **Badge** (new-message dot) is monochrome white — a signal, not a color accent.
|
||||
- **Focus** is visible and neutral: inputs focus to `--text-dim`; the orb uses a
|
||||
`--glow`-colored outline (it's the orb, so color is allowed).
|
||||
|
||||
---
|
||||
|
||||
## Motion
|
||||
|
||||
- **The orb is audio-reactive, not timer-driven.** Mic RMS (`--audio-level`) and the
|
||||
assistant output level (`--ai-audio-level`) drive scale/opacity at display rate, so
|
||||
every syllable moves it. This is the signature animation — keep new motion
|
||||
subordinate to it.
|
||||
- **Quiet by default.** Breathing/glow throbs are slow (1.4–2.4s) and low-contrast.
|
||||
Resist adding scattered micro-animations; an orchestrated moment beats many small
|
||||
ones, and excess motion reads as AI-generated.
|
||||
- **First paint is frozen.** `body.booting` disables all transitions until the first
|
||||
frame commits (stripped after one rAF in `main.js`). Anything new that would
|
||||
otherwise animate-in on load must respect this.
|
||||
- Honor `prefers-reduced-motion` for any motion you add.
|
||||
|
||||
---
|
||||
|
||||
## Writing / copy voice
|
||||
|
||||
- Sentence case, plain verbs, no filler. Tuned and quiet — match the minimal canvas.
|
||||
- Name things by what the user controls, not by the system's internals. A button says
|
||||
exactly what it does, and keeps the same word through the flow.
|
||||
- **Empty states invite action** ("Tap the orb and start talking"), they don't just
|
||||
set a mood.
|
||||
- **Errors state what happened and how to recover**, in the interface's voice — they
|
||||
don't apologize and are never vague.
|
||||
- Mono labels are terse identifiers (`YOU`, `TOOL CALL`); prose stays in Inter.
|
||||
|
||||
---
|
||||
|
||||
## Responsive floor (non-negotiable)
|
||||
|
||||
Every change ships meeting these:
|
||||
- Works down to a `360px`-wide phone. The `@media (max-width: 600px)` block already
|
||||
handles the phone layout — extend it, don't fight it.
|
||||
- Visible keyboard focus on every interactive element.
|
||||
- `prefers-reduced-motion` respected.
|
||||
- Tap targets ≥ `44px`; `touch-action: manipulation` on anything tappable.
|
||||
|
||||
---
|
||||
|
||||
## Before you ship — the mirror check
|
||||
|
||||
1. Is every saturated color either on the orb or a word-sized role echo? If a fill or
|
||||
border is colored, remove that accessory.
|
||||
2. Is mono used only for machine/system text, and Inter for everything human?
|
||||
3. Does any new state have both a `--glow` and (if it appears in chat) a `--voice-*`
|
||||
alias?
|
||||
4. Are separations hairlines + spacing, not boxes and heavy shadows?
|
||||
5. Did you add motion? Is it quieter than the orb and reduced-motion-safe?
|
||||
6. Remove one accessory. The minimal look survives on precision, not addition.
|
||||
@@ -1,13 +1,15 @@
|
||||
# Single container: serves the static front-end AND the /api/search proxy.
|
||||
# Build from the project root: docker build -f s2s-client/Dockerfile -t s2s-client .
|
||||
# Serve the client, same-origin API proxies, and shared project documentation.
|
||||
# HF Spaces (sdk: docker) routes traffic to $PORT, default 7860.
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
WORKDIR /app/s2s-client
|
||||
|
||||
COPY requirements.txt .
|
||||
COPY s2s-client/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
COPY s2s-client/ .
|
||||
COPY docs/ /app/docs/
|
||||
|
||||
EXPOSE 7860
|
||||
|
||||
|
||||
@@ -1,584 +0,0 @@
|
||||
# HOWITWORKS — demo S2S
|
||||
|
||||
> Compagno di [ARCHITECTURE.md](file:///home/duxiter/s2s/demo/ARCHITECTURE.md): spiega il
|
||||
> ciclo di vita concreto dei dati in 3 moduli su cui oggi ci siamo concentrati:
|
||||
> 1. Il campo `instructions` da client a `ai-server:12345`
|
||||
> 2. `MCP_SEARCH_TRANSLATION_PHRASES` + ranking tool MCP lato client
|
||||
> 3. **FASE 2 — Dichiarazione tools nativi a `ai-server:12345` via Realtime `session.update.tools[]`**
|
||||
|
||||
---
|
||||
|
||||
## 0. FASE 2 — Integrazione tools nativi S2S upstream (ai-server:12345)
|
||||
|
||||
### Obiettivo
|
||||
Prima della Fase 2, i 100+ tool MCP Aquachile erano solo **testo dentro `instructions`** (9 KB di prompt). L'upstream non sapeva eseguirli e produceva frasi tipo:
|
||||
> *"Perfecto, buscaré… [list_recent_reports]"*
|
||||
|
||||
Con la Fase 2 dichiariamo i tools tramite il **protocollo nativo OpenAI Realtime GA**:
|
||||
```
|
||||
session.update { tools: [...], tool_choice: "auto" }
|
||||
```
|
||||
→ Se `ai-server` lo supporta, produce eventi veri `response.function_call_arguments.done` con nome tool + argomenti JSON. Il listener già esistente nel client (s2s-ws-client.js) si attiva automaticamente e tutto il resto della pipeline — `runTool()` → `POST /api/mcp/tool` Aquachile → `client.sendToolOutput()` — funziona senza altre modifiche.
|
||||
|
||||
### 0.1 Dove è stato implementato lato proxy (s2s_ui container)
|
||||
|
||||
| Modulo | File / Righe | Cosa fa |
|
||||
|---|---|---|
|
||||
| **Cache + normalizzazione** | [server.py:451-544](file:///home/duxiter/s2s/demo/server.py#L451-L544) | `_normalize_tools_for_realtime(raw_tools)`: converte qualsiasi formato tool Aquachile → formato OpenAI Realtime standard. Non alza eccezioni; tool invalidi saltati con warning. |
|
||||
| **Inject WS upstream** | [server.py:1034-1075](file:///home/duxiter/s2s/demo/server.py#L1034-L1075) | Subito dopo `TUNNEL ESTABLISHED`, **PRIMA** che inizi lo streaming audio: fetch cache `_agent_profile_data()` → `session.update` con `instructions` (profilo agente Aquachile) + `tools[]` + `tool_choice=auto`. Errori suppressi: non rompiamo mai il flusso audio. |
|
||||
| **Log di successo KEY** | [server.py:312-334](file:///home/duxiter/s2s/demo/server.py#L312-L334) | Quando arriva `response.function_call_arguments.done` dall'upstream → 2 log `[MCP INJECT] ✓ UPSTREAM ha emesso…` (segnale CHE LA STRATEGIA B FUNZIONA) |
|
||||
| **Proxy MCP già esistente** | [server.py:600-656](file:///home/duxiter/s2s/demo/server.py#L600-L656) | `POST /api/mcp/tool` → Aquachile JSON-RPC 2.0 `tools/call`. Nessuna modifica necessaria. |
|
||||
|
||||
### 0.2 Sequenza temporale WS Realtime (Fase 2)
|
||||
|
||||
```
|
||||
1. Browser → ws://s2s_server/v1/realtime (connessione)
|
||||
2. server.py → websockets.connect(ai-server:12345)
|
||||
3. [MCP INJECT] server.py → upstream:
|
||||
{ "type": "session.update",
|
||||
"session": { // ⚠️ FORMATO CORRETTO — TUTTO dentro session:{}
|
||||
"instructions": "…system prompt profilo Aquachile…",
|
||||
"tools": [ {type: "function", name:"list_recent_reports", …}, …79 più… ],
|
||||
"tool_choice": "auto"
|
||||
}
|
||||
}
|
||||
← LOG "[MCP INJECT] session.update upstream inviato (tools=79, fmt=nested_session OK)"
|
||||
4. browser → ai-server: (turno audio utente) "el reporte de ayer por favor"
|
||||
5. (se upstream ha tools[] in RuntimeConfig.session.tools — ORA SUPPORTATO Nativamente)
|
||||
ai-server → server.py → browser:
|
||||
{ "type": "response.function_call_arguments.done",
|
||||
"call_id": "call_abc",
|
||||
"name": "list_recent_reports",
|
||||
"arguments": "{\"date_from\":\"2026-08-24\",\"limit\":5}" }
|
||||
← LOG "[MCP INJECT] ✓ UPSTREAM ha emesso response.function_call_arguments.done!"
|
||||
6. s2s-ws-client.js:803 → dispatch "toolcall"
|
||||
7. main.js:3023 → runTool("list_recent_reports", args, "call_abc")
|
||||
branch agentToolDefs → execRemoteMcpTool
|
||||
8. execRemoteMcpTool → POST /api/mcp/tool → Aquachile JSON-RPC tools/call
|
||||
LOG: "[Vocero MCP CALL]" → "[Vocero MCP HTTP]" → "[Vocero MCP OUT]"
|
||||
9. client.sendToolOutput("call_abc", "Risultati: 5 report, ultimo=Report_2026-08-24.xlsx")
|
||||
→ WS payload:
|
||||
{ "type": "conversation.item.create",
|
||||
"item": { "type": "function_call_output", "call_id": "call_abc", "output": "..." } }
|
||||
10. ai-server continua il turno con TTS nativo:
|
||||
"Aquí tienes los 5 reportes de ayer, el más reciente es Report_2026-08-24.xlsx..."
|
||||
(nessun [tool_name] nel testo del TTS!)
|
||||
```
|
||||
|
||||
### 0.3. Formato atteso da `ai-server:12345` — guida per modifiche sul backend S2S
|
||||
|
||||
**Se dopo il deploy non vedi il log `[MCP INJECT] ✓ UPSTREAM ha emesso…`**, vuol dire che `ai-server` oggi scarta il campo `tools[]` dal `session.update`. Ecco **esattamente i campi che devono essere supportati** (copia-incollabile nel codice di `ai-server`):
|
||||
|
||||
#### 0.3.1 `session.update` in ingresso (cosa manda oggi `server.py` — FORMATO NESTED CORRETTO)
|
||||
|
||||
⚠️ **REGOLA NON NEGOZIABILE**: TUTTI i campi di configurazione (instructions, tools, tool_choice, voice, model) devono stare **DENTRO l'oggetto `session`**. Non mandarli MAI alla radice del payload. Il validatore Pydantic strict di OpenAI Realtime GA rifiuta qualsiasi campo sconosciuto alla root.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"type": "session.update",
|
||||
|
||||
// ⚠️ TUTTO dentro "session":{} — il validator Pydantic strict è extra=forbid sulla root.
|
||||
"session": {
|
||||
|
||||
// 1) Istruzioni di sistema (profilo agente Aquachile)
|
||||
"instructions": "María, asistente virtual Aquachile, habla español de Chile…",
|
||||
|
||||
// 2) CATALOGO TOOLS — 79 tool Aquachile normalizzati
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "list_recent_reports",
|
||||
"description": "Restituisce l'elenco dei report più recenti, opzionalmente filtrati per data.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"date_from": {"type": "string", "format": "date", "description": "Data inizio YYYY-MM-DD"},
|
||||
"limit": {"type": "integer", "minimum": 1, "maximum": 50, "default": 10}
|
||||
},
|
||||
"required": []
|
||||
}
|
||||
}
|
||||
// … altri 78 tool
|
||||
],
|
||||
|
||||
// 3) Strategia selezione tool
|
||||
"tool_choice": "auto"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Perché prima non funzionava?** (ROOT CAUSE)
|
||||
Il proxy stava inviando `instructions`, `tools`, `tool_choice` **ALLA RADICE** del payload, fuori da `session:{}`. Il modello Pydantic `SessionUpdateEvent` (OpenAI Realtime GA) ha `extra=forbid` e ha rifiutato con `Unknown or invalid event: session.update`. **Non era un rifiuto del campo `tools` in quanto tale** — era un rifiuto di qualsiasi campo non riconosciuto messo nel posto sbagliato. La prova: `runtime_config.session.instructions`, `.session.tools`, `.session.tool_choice` in `language_model.py:526-532` leggono TUTTI i campi da dentro `.session.*`.
|
||||
|
||||
**Audit sorgente backend `speech_to_speech` conferma: GIA' 100% WIRED PER NATIVI TOOLS**
|
||||
| Componente | File / Righe | Conferma |
|
||||
|---|---|---|
|
||||
| LLM pick tools da sessione | `LLM/language_model.py:526-532` | `tools = runtime_config.session.tools` |
|
||||
| Chat buffer gestisce FC/FCO | `LLM/chat.py:141-223 / 342-366 / 394-427` | `RealtimeConversationItemFunctionCall` + `FunctionCallOutput` |
|
||||
| Response handler emette WS done event | `api/openai_realtime/handlers/response.py:336-345` | `ResponseFunctionCallArgumentsDoneEvent` con `call_id, name, arguments(STRINGA), response_id, item_id, output_index` |
|
||||
| Conversation handler accetta ritorno output | `api/openai_realtime/handlers/conversation.py:30-96` | `handle_conversation_item_create` con `deferred_items` anti-race |
|
||||
| Fallback parser retrocompatibile | `api/openai_realtime/service.py:267-325` | Se arrivano campi flat alla radice → promossi a nested `session` wrapper |
|
||||
|
||||
**Regole convalida obbligatorie lato `ai-server` per evitare rigetti silenziosi**:
|
||||
|
||||
| Regola | Valore |
|
||||
|---|---|
|
||||
| Nome tool | lunghezza ≤ 64, match regex `^[a-z0-9_]+$` |
|
||||
| Numero max tools | Consigliato ≤ 80. Oggi il proxy manda 100+ con warning. |
|
||||
| `parameters.type` | Sempre `"object"` (anche se vuoto) |
|
||||
| `parameters.properties` | Sempre `{}` presente, anche se 0 proprietà |
|
||||
| `tool_choice` | Valori accettati: `"none"`, `"auto"`, oppure `{type:"function",name:"…"}` |
|
||||
| Chiavi ignote | NON fare strict reject. Logga warning e scarta solo la chiave. |
|
||||
|
||||
#### 0.3.2 Evento di ritorno richiesto da `ai-server` (quando il modello decide di chiamare un tool)
|
||||
|
||||
Quando il modello determina che serve un tool → emetti **3 eventi in ordine ESATTO**:
|
||||
```
|
||||
a) { "type": "response.function_call_arguments.started", "call_id": "call_abc123", "name": "list_recent_reports", "response_id":"resp_X", "item_id":"item_Y" }
|
||||
(opzionale — se non hai streaming arguments puoi anche saltarlo)
|
||||
|
||||
b) (opzionale) N × { "type": "response.function_call_arguments.delta", "call_id": "call_abc123", "delta": "{\"dat" }
|
||||
(solo se streammi il JSON argomenti per pezzi)
|
||||
|
||||
c) ✅ EVENTO RICHIESTO OBBLIGATORIO (senza questo il client non esegue il tool):
|
||||
{
|
||||
"type": "response.function_call_arguments.done",
|
||||
"call_id": "call_abc123", // STRINGA OBBLIGATORIA — serve per la sendToolOutput
|
||||
"name": "list_recent_reports", // STRINGA OBBLIGATORIA — nome del tool (come sopra)
|
||||
"arguments": "{\"date_from\":\"2026-08-24\",\"limit\":5}",
|
||||
"response_id": "resp_xyz",
|
||||
"item_id": "item_zzz",
|
||||
"output_index": 0
|
||||
}
|
||||
```
|
||||
|
||||
**⚠️ Importante**: `arguments` DEVE essere una **STRINGA JSON serializzata** (non un oggetto). Il client fa `JSON.parse(event.arguments)`. Se mandi un oggetto JS → parser fallisce e il tool non viene eseguito.
|
||||
|
||||
#### 0.3.3 Atteso come input successivo (il ritorno del tool dal client)
|
||||
|
||||
Dopo ~500ms-45s, il client manda indietro al S2S upstream:
|
||||
```jsonc
|
||||
{
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "function_call_output",
|
||||
"call_id": "call_abc123",
|
||||
"output": "5 report trovati. Ultimo: Report_2026-08-24.xlsx - Aquachile ventas"
|
||||
}
|
||||
}
|
||||
→ Poi { "type": "response.create" }
|
||||
```
|
||||
→ L'S2S dovrebbe usare `output` come contesto nel prossimo turno di generazione TTS (quindi parla del risultato, senza tag `[tool]`).
|
||||
|
||||
### 0.4 Checklist di deploy Fase 2 (2026-08-25 — SORGENTE ai-server TROVATA in locale)
|
||||
|
||||
**Prerequisito**: Sorgente backend `ai-server:12345` = `/home/duxiter/s2s/speech_to_speech` (Python puro).
|
||||
✅ **TEST CURL 2026-08-25 CONFERMATO**: `https://aquachile.azurianlab.com/api/v1/mcp/` è **ATTUALMENTE PUBBLICO** — accetta `tools/list` e `tools/call` SENZA API KEY. Con credenziali dummy `dummy-test-key/secret` restituisce HTTP 200 OUTPUT IDENTICO alla chiamata senza auth. Quindi NON serve valorizzare:
|
||||
```
|
||||
S2S_TOOL_EXECUTION_MCP_API_KEY=
|
||||
S2S_TOOL_EXECUTION_MCP_API_SECRET=
|
||||
```
|
||||
⚠️ Nota: Il server accetta entrambi gli auth headers, ma attualmente non li valida. Restano configurati come best practice per eventuali versioni future dell'endpoint che attiveranno auth.
|
||||
|
||||
#### Ordine deploy 3 step
|
||||
|
||||
| # | Azione | Perché |
|
||||
|---|---|---|
|
||||
| 0 | ~~Valorizza `.env` MCP credentials~~ | ✅ **SUPERATO** — endpoint MCP Aquachile pubblico |
|
||||
| 1 | **Restart container `s2s_ui`** | Deploy fix formato inject `fmt=nested_session` in [server.py:1054-1102](file:///home/duxiter/s2s/demo/server.py#L1054-L1102) |
|
||||
| 2 | **Riavvia backend `ai-server:12345`** con il codice aggiornato di `/home/duxiter/s2s/speech_to_speech` (in particolare `api/openai_realtime/service.py`) | Deploy fallback parser retrocompatibile flat→nested in [service.py:267-325](file:///home/duxiter/s2s/speech_to_speech/api/openai_realtime/service.py#L267-L325) |
|
||||
| 3 | **Hard Reload browser** (Ctrl+Shift+R o Disable cache + refresh) | Pulisce cache vecchio client JS |
|
||||
|
||||
#### Checklist di verifica DOPO deploy 3+1 step
|
||||
|
||||
```bash
|
||||
# Terminal A
|
||||
docker logs -f s2s_ui 2>&1 | grep -E 'MCP|Vocero MCP|session.update|function_call_arguments'
|
||||
```
|
||||
|
||||
| Step | Trigger | Log atteso nel container | Esito |
|
||||
|---|---|---|---|
|
||||
| 1 | Apri pagina demo (connessione WS) | `[MCP INJECT] session.update upstream → tools=79 hasAgentPrompt=True tool_choice=auto` | ✅ Inject partito |
|
||||
| 2 | Subito dopo step 1 | `[MCP INJECT] session.update upstream inviato (tools=79, fmt=nested_session OK)`<br>🔴 **NON DEVE** più comparire `[MCP INJECT] suppressed upstream error` — se compare ancora, il deploy di `speech_to_speech service.py` non è riuscito | ✅ Nessun rigetto |
|
||||
| 3 | Console browser | 0 occorrenze `Unknown or invalid event: session.update` | ✅ Filtro OK |
|
||||
| 4 | Frase trigger tool: *"¿dónde está el report de ayer?"* | **KEY LOG 🔑**<br>`[MCP INJECT] ✓ UPSTREAM ha emesso response.function_call_arguments.done! tool=list_recent_reports args_keys=[...]` | ✅ **Strategia B FUNZIONA — il modello produce tool calls native** |
|
||||
| 5 | 0.5-1s dopo Step 4 | `[Vocero MCP CALL] name=list_recent_reports target=https://aquachile...`<br>`[Vocero MCP HTTP] name=list_recent_reports status=200`<br>`[Vocero MCP OUT] name=list_recent_reports preview=...resultKeys=...` | ✅ Tool eseguito realmente via REST Aquachile |
|
||||
| 6 | 1-2s dopo Step 5 | TTS vocale naturale — es: *"Aquí tienes los 5 reportes de ayer, el más reciente es Report_2026-08-24.xlsx..."* | ✅ Nessun `(list_recent_reports)` / `[tool]` nel testo TTS |
|
||||
| 7 | Frase non-tool: *"Hola, ¿cómo estás?"* | Risposta standard SENZA tool calls, POST /api/reasoning 200, Node-RED LOG_ONLY OK | ✅ Nessuna regressione Fase 1 |
|
||||
|
||||
---
|
||||
|
||||
### 0.5 Stato attuale (2026-08-25 11:00) — BLOCCO RISOLTO, pipeline 100% implementata
|
||||
|
||||
#### 🎯 Root Causa Reale (99.99% confidence, sorprendente)
|
||||
|
||||
**Ipotesi iniziale, SUPERATA**: *"ai-server rifiuta strict il campo `tools` perché non è implementato"*
|
||||
|
||||
**Realtà, AUDIT SORGENTE CONFERMATO**: TUTTO il sistema — **backend `speech_to_speech` + proxy `demo/server.py` + client `main.js` / `s2s-ws-client.js`** — è GIA' COMPLETAMENTE IMPLEMENTATO end-to-end per il tool calling nativo OpenAI Realtime GA. **L'unico blocco era un errore di formato payload nel proxy**:
|
||||
|
||||
```python
|
||||
# ❌ SBAGLIATO (come era prima) — campi alla RADICE
|
||||
_inject_payload = {
|
||||
"type": "session.update",
|
||||
"instructions": "...",
|
||||
"tools": [...],
|
||||
"tool_choice": "auto"
|
||||
} # ← Pydantic SessionUpdateEvent extra=forbid → Unknown or invalid event
|
||||
|
||||
# ✅ CORRETTO (deployato in server.py:1054-1102) — TUTTO dentro session:{}
|
||||
_inject_payload = {
|
||||
"type": "session.update",
|
||||
"session": {
|
||||
"instructions": "...",
|
||||
"tools": [...],
|
||||
"tool_choice": "auto"
|
||||
}
|
||||
} # ← Perfetto, corrisponde a runtime_config.session.tools in LLM
|
||||
```
|
||||
|
||||
**Prova inconfutabile**: [language_model.py:526-532](file:///home/duxiter/s2s/speech_to_speech/LLM/language_model.py#L526-L532) legge istruzioni, tools e tool_choice **TUTTI dentro `.session.*`**:
|
||||
```python
|
||||
instructions = response.instructions if ... else runtime_config.session.instructions
|
||||
tools = response.tools if ... else runtime_config.session.tools
|
||||
tool_choice = response.tool_choice if ... else runtime_config.session.tool_choice
|
||||
```
|
||||
|
||||
#### 📋 Mappa riassuntiva: 7 componenti TUTTI già funzionanti
|
||||
|
||||
| Componente | File | Stato | Cosa fa |
|
||||
|---|---|---|---|
|
||||
| 🔧 **Normalizzatore tool** | `demo/server.py:464-544` | ✅ | 79 tool Aquachile → formato Realtime GA standard, 0 eccezioni |
|
||||
| 🚀 **Inject WS upstream** | `demo/server.py:1054-1102` | ✅ **NUOVO FIX 2026-08-25** | Invia nested `session:{instructions,tools,tool_choice}` subito dopo tunnel |
|
||||
| 🛡️ **Fallback parser retrocompatibile** | `speech_to_speech/service.py:267-325` | ✅ **NUOVO FIX 2026-08-25** | Se arrivano campi flat root → promossi a nested `session` |
|
||||
| 🧠 **LLM + Chat buffer** | `speech_to_speech/LLM/*` | ✅ GIA' ESISTENTE | Tools da sessione, FC/FCO nel chat, pending dict anti-race |
|
||||
| 📡 **WS response handler (tool call OUT)** | `speech_to_speech/handlers/response.py:336-345` | ✅ GIA' ESISTENTE | Emette `ResponseFunctionCallArgumentsDoneEvent` con arguments=STRINGA JSON, call_id, name, response_id, item_id, output_index=0 |
|
||||
| 💾 **WS conversation handler (tool output IN)** | `speech_to_speech/handlers/conversation.py:30-96` | ✅ GIA' ESISTENTE | `conversation.item.create function_call_output` con `deferred_items` anti-race |
|
||||
| 🌐 **Client WS → MCP REST → ritorno WS** | `demo/ws/s2s-ws-client.js:803-1077` + `demo/main.js:2180-2288 + 3023-3043` | ✅ GIA' ESISTENTE | (1) `response.function_call_arguments.done` → dispatch `toolcall` event<br>(2) `runTool()` branch `agentToolDefs` → `execRemoteMcpTool()` → `POST /api/mcp/tool`<br>(3) `client.sendToolOutput(callId, output)` → `conversation.item.create function_call_output`<br>(4) `client.requestResponse()` → prossimo turno TTS |
|
||||
| 📣 **Proxy MCP REST Aquachile** | `demo/server.py:704-760` | ✅ GIA' ESISTENTE | `POST /api/mcp/tool` JSON-RPC 2.0 tools/call + log `[Vocero MCP CALL/HTTP/OUT]` |
|
||||
| 🛡️ **Filtro errori WS V2** | `demo/server.py:1713-1792` | ✅ GIA' ESISTENTE | Sopprime `unknown_or_invalid_event` se mai dovesse ricomparire, passthrough altri errori |
|
||||
|
||||
#### 🚀 Prossimo passo: ESEGUIRE I 3 STEP DI §0.4
|
||||
|
||||
1. ~~Valorizza `.env` MCP API Key/Secret~~ ✅ **SUPERATO** (endpoint pubblico)
|
||||
2. Restart `s2s_ui`
|
||||
3. Restart `ai-server` con il codice aggiornato da `/home/duxiter/s2s/speech_to_speech`
|
||||
4. Hard reload browser (Ctrl+Shift+R)
|
||||
5. Testa: *"¿dónde está el report de ayer?"* → aspetta il key log `[MCP INJECT] ✓ UPSTREAM ha emesso response.function_call_arguments.done!`
|
||||
|
||||
**Se Step 5 (function_call_arguments.done) non compare dopo 3 frasi trigger**: apri una shell con `docker logs -f ai-server` per vedere se la sessione sta leggendo `.session.tools` dal runtime config.
|
||||
|
||||
---
|
||||
|
||||
#### ⚙️ MCP MASTER SWITCH (nuovo feature 2026-08-25)
|
||||
**Controllato da una sola riga in `.env`**:
|
||||
```
|
||||
#S2S_TOOL_EXECUTION_MCP_URL=https://aquachile.azurianlab.com/api/v1/mcp # ⬅️ se commentata/vuota → MCP DISABLED
|
||||
S2S_TOOL_EXECUTION_MCP_URL=https://aquachile.azurianlab.com/api/v1/mcp # ⬅️ se decommentata/valorizzata → MCP ENABLED
|
||||
```
|
||||
|
||||
Implementato come `MCP_EXEC_ENABLED = bool(S2S_TOOL_EXECUTION_MCP_URL)` in [server.py:106-114](file:///home/duxiter/s2s/demo/server.py#L106-L114). Quando **DISABLED** (stato attuale `.env`):
|
||||
|
||||
| Punto di controllo | Comportamento con MCP DISABLED | Riferimento codice |
|
||||
|---|---|---|
|
||||
| **Startup log** | `[MCP MASTER] ⛔ DISABLED — S2S_TOOL_EXECUTION_MCP_URL is empty/commented in .env…` | [server.py:172-182](file:///home/duxiter/s2s/demo/server.py#L172-L182) |
|
||||
| **WS `/v1/realtime` inject tools** | `session.update` contiene **solo `instructions`** (profilo agente); `tools[]` e `tool_choice` **non vengono mai mandati**. Log marker `[MCP INJECT] ⛔ SKIPPED tool injection …`. ⚠️ Il prompt profilo continua a funzionare. | [server.py:1081-1123](file:///home/duxiter/s2s/demo/server.py#L1081-L1123) |
|
||||
| **`/api/config` bootstrap client** | Invia al browser `agentTools=[]` (invece dei 79) + `searchMcp=false`. Effetti lato client: ranker MCP early exit, prompt non contiene più "Available MCP tools:" e ranking hints, UI pannello MCP mostra "No MCP tools available.", nessun pulsante "Esegui tool". | [server.py:660-694](file:///home/duxiter/s2s/demo/server.py#L660-L694) |
|
||||
| **`POST /api/mcp/tool` endpoint** | Se un client tenta comunque la chiamata (es. con codice vecchio): log `[MCP DISABLED] ⛔ /api/mcp/tool called but…` + HTTP 503 dettagliato (non crash, non exception trace). | [server.py:721-733](file:///home/duxiter/s2s/demo/server.py#L721-L733) |
|
||||
|
||||
Per **riabilitare la Fase 2 MCP in futuro**, 4 step:
|
||||
1. ✏️ `.env` → togli `#` davanti a `S2S_TOOL_EXECUTION_MCP_URL=https://aquachile.azurianlab.com/api/v1/mcp`
|
||||
2. 🔄 Restart `s2s_ui` (deploya MCP_EXEC_ENABLED=True + inject formato nested corretto)
|
||||
3. 🔄 Restart `ai-server` con codice aggiornato `/home/duxiter/s2s/speech_to_speech` (fallback parser flat→nested)
|
||||
4. 🌐 Hard reload browser Ctrl+Shift+R
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 1. Flow di `instructions` — da impostazioni utente a S2S upstream
|
||||
|
||||
**Definizione**: `instructions` è il **system prompt** del protocollo OpenAI Realtime GA.
|
||||
Viene mandato al backend in 2 punti distinti:
|
||||
|
||||
| Canale | Dove | Destinatario |
|
||||
|---|---|---|
|
||||
| ✅ **TUNNEL WS AUDIO** → `session.update` all'inizio di ogni sessione | Browser → tunnel proxy → upstream S2S nativo (`ai-server:12345`) | Usato **dal modello S2S stesso** per la risposta vocale nativa |
|
||||
| ✅ **REST PARALLELO** → body `POST /api/reasoning` | Browser → `server.py` → Node-RED o OpenAI | Usato **dal reasoning hook** esterno (oggi Fase 1 LOG_ONLY) |
|
||||
|
||||
Entrambi i canali partono dalla stessa funzione generatrice `effectiveReasoningInstructions()` nel client.
|
||||
|
||||
### Step-by-step
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ 1. BASE — DEFAULT_INSTRUCTIONS (main.js:26) │
|
||||
│ "You are a friendly voice assistant. Keep replies short..." │
|
||||
│ OR textarea Settings #instructions salvato in localStorage │
|
||||
│ chiave "s2s.ws.instructions" (main.js:138, 201, 223) │
|
||||
└───────────────────────────────────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ 2. effectiveReasoningInstructions() (main.js:516-533) │
|
||||
│ Appende in coda, in ordine: │
|
||||
│ │
|
||||
│ + (se ≥1 tool attivo) TOOL_USE_HINT (main.js:32-35) │
|
||||
│ → "do not describe your capabilities... call tool immediately" │
|
||||
│ │
|
||||
│ + (se searchMcpEnabled + agentToolDefs) MCP_SEARCH_HINT + │
|
||||
│ "Available MCP tools: …catalogo completo…" │
|
||||
│ │
|
||||
│ + (se searchMcpEnabled + tool suggeriti) riga ranking client-side │
|
||||
│ "Current user request: \"${lastUserQuery}\". Ranking hint: 1. X │
|
||||
│ 2. Y 3. Z — use as secondary signal only" │
|
||||
│ │
|
||||
│ + (se utente autenticato phone) voceroUserContextText() │
|
||||
│ → es: "User profile: name=Juan Pérez phone=+569… account=…" │
|
||||
└───────────────────────────────────────┬────────────────────────────┘
|
||||
│
|
||||
┌──────────────────┴──────────────────┐
|
||||
▼ ▼
|
||||
┌─────────────────────────────┐ ┌──────────────────────────────┐
|
||||
│ 3a. effectiveInstructions() │ │ 3b. POST /api/reasoning │
|
||||
│ (main.js:537-539) │ │ body req.instructions │
|
||||
│ │ │ (main.js:569) │
|
||||
│ • mode=OPENAI → accoda │ │ = effectiveReasoningInst…() │
|
||||
│ OPENAI_RENDER_HINT │ │ tale e quale (NO append) │
|
||||
│ • mode=NODE_RED / PROXY → │ │ │
|
||||
│ ritorna il valore │ │ → server.py lo gira a │
|
||||
│ senza modifiche │ │ Node-RED o OpenAI │
|
||||
└──────────────┬──────────────┘ └──────────────┬───────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────┐ ┌──────────────────────────────┐
|
||||
│ 4a. startInstructions = │ │ 4b. /api/reasoning → │
|
||||
│ effectiveInstructions() │ │ Node-RED log: │
|
||||
│ (main.js:2829) │ │ [S2S NODE_RED LOG_ONLY] │
|
||||
└──────────────┬──────────────┘ │ instructions len=… │
|
||||
│ └──────────────────────────────┘
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ 5. new S2sWsRealtimeClient({ │
|
||||
│ instructions: startInstructions │
|
||||
│ voice, tools, noiseGate, audioContext │
|
||||
│ }) main.js:2839-2847 │
|
||||
└──────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ 6. this.options.instructions → _sendSessionUpdate() │
|
||||
│ s2s-ws-client.js:995 + 1016 │
|
||||
│ { type: "session.update", │
|
||||
│ session: { type: "realtime", │
|
||||
│ instructions: this.options.…, │
|
||||
│ audio:{output:{voice}}, │
|
||||
│ tools, tool_choice:"auto" } } │
|
||||
└──────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ 7. server.py _client_to_upstream — TUNNEL 1:1 │
|
||||
│ → forward esatto byte-per-byte all'upstream │
|
||||
│ → server.py NON tocca MAI instructions qui │
|
||||
│ → Logga solo summary.instructionLength = len() │
|
||||
│ (server.py:197-203) — NON MODIFICA │
|
||||
└──────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ 8. ai-server:12345 riceve session.update → │
|
||||
│ istruzioni usate come system prompt da VLM/LMM │
|
||||
│ per TUTTA la durata della sessione finché │
|
||||
│ non arriva un altro session.update (patch) │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Patch di `instructions` durante la sessione (viva)
|
||||
Se vuoi aggiornare istruzioni a metà conversazione **senza chiudere la WS**:
|
||||
- Client → chiama `client.updateSession({ instructions })` (main.js:1021-1035)
|
||||
- Oppure, in futuro: **proxy server-side inietta patch** via `upstream.send({type:"session.update", session:{...}})` in qualsiasi momento del tunnel
|
||||
|
||||
### Tabella riassuntiva — chi oggi modifica `instructions`?
|
||||
|
||||
| Livello | Modifica? | Note |
|
||||
|---|---|---|
|
||||
| 🔵 Browser `main.js` | **Sì** — generatore principale | DEFAULT_INSTRUCTIONS + hints + user context |
|
||||
| 🟣 Browser `s2s-ws-client.js` | **Solo forwarding** | Incapsula in `session.update` |
|
||||
| 🟢 Proxy `server.py` tunnel WS | **NO** — copia byte-per-byte | Logga solo lunghezza, non tocca |
|
||||
| 🟠 Proxy `server.py` reasoning REST | **Sì (Modalità B)** | Usa `req.instructions` come system part nel prompt OpenAI (Fase 2) |
|
||||
| 🔴 Upstream `ai-server:12345` | **Sì** — applica come system prompt | Entra nel ciclo STT→VLM→TTS |
|
||||
|
||||
---
|
||||
|
||||
## 2. Ranking tool MCP lato client + `MCP_SEARCH_TRANSLATION_PHRASES`
|
||||
|
||||
**Obiettivo**: quando arriva la query utente (es. `"Quiero el reporte de hoy"` in spagnolo),
|
||||
ordinare velocemente i tool MCP del catalogo Aquachile per **rilevanza**, così
|
||||
le `instructions` possono includere un hint di ranking ("prova prima questi 3") e
|
||||
il modello sceglie lo strumento giusto con meno tentativi.
|
||||
|
||||
### 2.1 Perché serve la traduzione
|
||||
Il catalogo dei tool `agentToolDefs` arriva dall'endpoint Aquachile MCP:
|
||||
- `tool.name` e `tool.description` sono scritti in **INGLESE** per convenzione di
|
||||
OpenAI Functions / MCP protocol.
|
||||
- Ma l'utente parla **spagnolo del Cile o italiano** con l'agente.
|
||||
|
||||
Senza traduzione il matching `frase ES/IT ↔ tool EN` → **score 0** (zero corrispondenze).
|
||||
I tool non vengono mai suggeriti.
|
||||
|
||||
### 2.2 I due dizionari di traduzione (main.js)
|
||||
|
||||
| Costante | Tipo | Riga | Cosa fa |
|
||||
|---|---|---|---|
|
||||
| `MCP_SEARCH_TRANSLATION_PHRASES` | `[RegExp, string][]` frasi complete | [L51-L70](file:///home/duxiter/s2s/demo/main.js#L51-L70) | Sostituisce **frasi multi-parola** prima del tokenizer. Es. `reporte de hoy → today report`. **Priorità alta**: viene applicato PER PRIMO perché le frasi intere non si possono ricostruire da token singoli. |
|
||||
| `MCP_SEARCH_TRANSLATION_TOKENS` | `Record<it/es, en>` token singolo | [L72](file:///home/duxiter/s2s/demo/main.js#L72) | Sostituisce il **resto** parola per parola dopo la phase-phrase. Es. `ayuda → help`, `cliente → customer`. |
|
||||
|
||||
### 2.3 Catalogo completo traduzioni oggi attivo
|
||||
|
||||
#### Phrase-level (MCP_SEARCH_TRANSLATION_PHRASES)
|
||||
| Lingua | Input | Output EN | Riga |
|
||||
|---|---|---|---|
|
||||
| 🇪🇸 | `informaciones del usuario` / `informacion del usuario` | `user information` | L52-L53 |
|
||||
| 🇮🇹 | `informazioni utente` / `informazione utente` | `user information` | L54-L55 |
|
||||
| 🇪🇸 | `report del dia` / `reporte del dia` | `day report` | L56-L57 |
|
||||
| 🇮🇹 | `report di oggi` | `today report` | L58 |
|
||||
| 🇪🇸 | `reporte de hoy` | `today report` | L59 |
|
||||
| 🇮🇹 | `report giornaliero` | `daily report` | L60 |
|
||||
| 🇪🇸 | `reporte diario` | `daily report` | L61 |
|
||||
| 🇪🇸 | `mi numero es` | `my number is` | L62 |
|
||||
| 🇮🇹 | `il mio numero e` | `my number is` | L63 |
|
||||
| 🇮🇹 | `numero di telefono` | `phone number` | L64 |
|
||||
| 🇪🇸 | `numero telefonico` | `phone number` | L65 |
|
||||
| 🇪🇸 | `numero de telefono` | `phone number` | L66 |
|
||||
| 🇪🇸 | `numero de whatsapp` | `whatsapp number` | L67 |
|
||||
| 🇪🇸 | `informacion api` | `api information` | L68 |
|
||||
| 🇮🇹 | `informazioni api` | `api information` | L69 |
|
||||
|
||||
> 💡 Aggiungere nuove frasi: **è sufficiente aggiungere righe qui**, senza toccare il codice del ranking. La pipeline le applicherà automaticamente.
|
||||
|
||||
### 2.4 Pipeline di ranking chiamata per ogni turno
|
||||
|
||||
```
|
||||
Utente dice: "Quiero el reporte diario y informaciones del usuario"
|
||||
│
|
||||
▼
|
||||
rankAgentTools(query) main.js:1262
|
||||
│
|
||||
▼
|
||||
buildMcpSearchTexts(query) main.js:1330
|
||||
│
|
||||
├─── [0] normalizeMcpSearchText(query) ← versione ORIGINALE normalizzata
|
||||
│ = "quiero el reporte diario y informaciones del usuario"
|
||||
│
|
||||
└─── [1] translateMcpSearchTextToEnglish(query) main.js:1314
|
||||
│
|
||||
├─ step 1: normalize
|
||||
├─ step 2: loop replace per tutte le entry di MCP_SEARCH_TRANSLATION_PHRASES
|
||||
│ → "quiero el daily report y user information"
|
||||
├─ step 3: split token, map per MCP_SEARCH_TRANSLATION_TOKENS singoli
|
||||
│ → "want the daily report and user information"
|
||||
└─ step 4: normalize finale
|
||||
│
|
||||
▼
|
||||
searchTexts = [ originale normalizzato, versione tradotta in EN ]
|
||||
(duplicati eliminati via new Set)
|
||||
│
|
||||
▼
|
||||
tokens = split searchTexts + lunghezza ≥ 3 chars + dedup
|
||||
= [ quiero, reporte, diario, informaciones, usuario,
|
||||
want, daily, report, and, user, information ]
|
||||
│
|
||||
▼
|
||||
Check rule hard-coded:
|
||||
dailyReportIntent = match ("day"|"daily"|"today") + "report" → TRUE
|
||||
(se match: boost score dei tool che hanno report in name/description)
|
||||
│
|
||||
▼
|
||||
Per ogni tool in agentToolDefs[]:
|
||||
score = somma (frequenza token in tool.name + tool.description)
|
||||
+ 2× bonus dailyReportIntent se tool matcha "report"
|
||||
+ 3× bonus se esatta match name vs query originale
|
||||
Sort(score desc, name asc)
|
||||
│
|
||||
▼
|
||||
Torna al chiamante: [ {tool:"get_daily_report", score:42},
|
||||
{tool:"get_user_info", score:31},
|
||||
... ]
|
||||
│
|
||||
▼
|
||||
Usato in 2 punti:
|
||||
A) effectiveReasoningInstructions() main.js:525-529 → hint "Current user request…
|
||||
ranking hint: 1. get_daily_report… 2. get_user_info…" dentro le instructions
|
||||
B) UI pannello (se visibile) — lista tool suggeriti accanto alla trascrizione
|
||||
```
|
||||
|
||||
### 2.5 Quando è chiamato `rankAgentTools` — ogni turno?
|
||||
|
||||
| Trigger | Frequenza | Perché |
|
||||
|---|---|---|
|
||||
| `maybeRunOpenAiReasoning(transcript)` subito dopo transcript STT | 1× per turno utente | Include ranking hint dentro `instructions` che andrà a Node-RED / OpenAI |
|
||||
| Live edit textarea domanda (autocomplete) | Mentre utente scrive | Suggerisce tool nella UI |
|
||||
|
||||
### 2.6 Disattivazione automatica
|
||||
|
||||
Ranking ritorna `[]` vuoto e salta tutta la traduzione se:
|
||||
- `searchMcpEnabled == false` (MCP Aquachile non configurato)
|
||||
- `agentToolDefs.length == 0` (catalogo tool non ricevuto da `/agents/profile`)
|
||||
- Query normalizzata è vuota o troppo corta
|
||||
- Nessun token dopo split ≥ 3 caratteri
|
||||
|
||||
---
|
||||
|
||||
## 2bis. DOVE VEDERE I LOG MCP — due postazioni
|
||||
|
||||
Il ranking MCP produce log in **2 posti diversi** per design:
|
||||
|
||||
| Postazione | Tipo di log | Filtro DevTools / grep | Quando appare |
|
||||
|---|---|---|---|
|
||||
| 🖥️ **Browser — DevTools Console** | Client-side ranking dettagliato: **tutta la pipeline translate → build → rank per-tool matching** | `MCP RANK` (case-sensitive) | **Sempre** per ogni turno — anche in caso di EARLY EXIT (motivo esplicito: `searchMcpEnabled=false`, `agentToolDefs=0`, ecc.) |
|
||||
| 🐳 **Server — container `s2s_ui` (uvicorn Python)** | Server-side **snapshot del payload `instructions`** quando arriva `POST /api/reasoning` — quanti tool MCP ci sono nel catalogo hint, TOP3 suggeriti dal ranking lato client | `grep '[MCP RANK]'` o filter terminale `MCP RANK` | **1 riga per ogni POST /api/reasoning** (lo hai già nel flusso tra `[INPUT UTENTE]` e `[EXT][NODE_RED] reasoning POST`) |
|
||||
|
||||
### Nel tuo log di oggi (FASE 1 LOG_ONLY, senza MCP tools abilitati)
|
||||
|
||||
Aspettati questo flusso dopo il deploy di adesso:
|
||||
|
||||
```
|
||||
7|s2s_ui | INFO [s2s] [INPUT UTENTE] client=… stt=Necesito el report.
|
||||
7|s2s_ui | INFO [s2s] [MCP RANK] reasoning payload transcript="Necesito el report" instructionsLen=97 hasMcpCatalogHint=False hasRankingHint=False mcpCatalogTools=0 top3Suggested=none
|
||||
7|s2s_ui | DEBUG [s2s] [EXT][NODE_RED] reasoning POST http://localhost:1881/api/reasoning …
|
||||
```
|
||||
|
||||
E se apri **DevTools Console** (Verbose/Debug on, Disable cache ✓ Ctrl+Shift+R) vedi:
|
||||
|
||||
```
|
||||
[MCP RANK] updateMcpToolSuggestions ENTRY partial=false itemId=- query="necesito el report" searchMcpEnabled=false
|
||||
[MCP RANK] ════════════════════════════════════════
|
||||
[MCP RANK] rankAgentTools START query="necesito el report" searchMcpEnabled=false agentToolDefsCount=0
|
||||
[MCP RANK] buildSearchTexts raw="Necesito el report" normalized="necesito el report" translated="necesito el report" count=1 final=["necesito el report"]
|
||||
[MCP RANK] translate input="necesito el report" phrasesApplied=0 tokensApplied=0 output="necesito el report"
|
||||
[MCP RANK] → EARLY EXIT reason="searchMcpEnabled=false — da /api/config: MCP Aquachile feature OFF oppure S2S_TOOL_EXECUTION_MCP_URL non configurato; agentToolDefs=0 — catalogo tool MCP vuoto: profilo agente non ha tools in agentProfile via Aquachile /api/agents/profile" searchMcpEnabled=false normalized=true agentToolDefs=0
|
||||
[MCP RANK] ════════════════════════════════════════ (empty result)
|
||||
[MCP RANK] updateMcpToolSuggestions RESULTS partial=false matchedTools=0 ranked=none
|
||||
```
|
||||
|
||||
### Quando il ranking MCP inizia a produrre score > 0
|
||||
|
||||
Affinché un turno abbia log MCP con tool score>0, devono essere veri **TUTTI e 3**:
|
||||
|
||||
1. `.env` ha `S2S_TOOL_EXECUTION_MCP_URL` configurato (URL Aquachile MCP) **e** `searchMcpEnabled=true` ritorna da `/api/config`
|
||||
2. `/agents/profile` Aquachile torna **almeno 1 tool** in `agentTools[]` (campo `json.agentTools`)
|
||||
3. La query utente ha token ≥ 3 chars che matchano `tool.name` o `tool.description` o `tool.parameters`
|
||||
|
||||
Se 1 solo dei 3 è falso → **early exit spiegato nel log `[MCP RANK] → EARLY EXIT reason="…"`**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Come estendere (quick cheatsheet)
|
||||
|
||||
| Se vuoi… | Dove modificare |
|
||||
|---|---|
|
||||
| Aggiungere 1 frase tradotta ES/IT → EN per tool matching | `MCP_SEARCH_TRANSLATION_PHRASES` [main.js](file:///home/duxiter/s2s/demo/main.js#L51-L70) — nuova riga |
|
||||
| Aggiungere 1 parola singola tradotta | `MCP_SEARCH_TRANSLATION_TOKENS` [main.js:72](file:///home/duxiter/s2s/demo/main.js#L72) |
|
||||
| Boost score di un tipo di tool (es. report) | `rankAgentTools` [main.js:1268-1270](file:///home/duxiter/s2s/demo/main.js#L1268-L1270) + mappa subito dopo |
|
||||
| Aggiungere nuovo hint a `instructions` all'avvio sessione | `effectiveReasoningInstructions()` [main.js:516-533](file:///home/duxiter/s2s/demo/main.js#L516-L533) — nuova concatenazione |
|
||||
| Iniettare `instructions` server-side (segret@ dati Aquachile) senza toccare client | **Proxy tunnel** `realtime_proxy()` in [server.py](file:///home/duxiter/s2s/demo/server.py#L891-L908) → `await upstream.send({type:"session.update", session:{instructions}})` **prima** dei 2 `create_task` |
|
||||
@@ -1,498 +0,0 @@
|
||||
# Inject pipeline — caller context & reference material
|
||||
|
||||
Documento ufficiale di tutti i blocchi di contesto che il server inietta
|
||||
automaticamente nel prompt LLM **all'inizio di ogni sessione**, sia per il
|
||||
canale **WebSocket realtime** (`/v1/realtime`, patch di `session.update`) sia
|
||||
per l'endpoint **HTTP reasoning** (`POST /api/reasoning`, augmentation di
|
||||
`instructions` e `user_context`).
|
||||
|
||||
Tutto il sistema è **generico / deploy-agnostic** per scelta architetturale:
|
||||
nessun campo viene hardcodato. Ogni blocco accetta payload arbitrari e li
|
||||
serializza in JSON direttamente. Qualsiasi errore di rete / JSON non
|
||||
interrompe mai il flusso: il blocco problematico viene solo saltato con
|
||||
una riga `[INJECT-*] ⏭️ skipped` nel log server.
|
||||
|
||||
---
|
||||
|
||||
## 0. Master flag: `S2S_INJECT_PROFILE`
|
||||
|
||||
Tutta la pipeline è sotto **un unico master flag** lato server.
|
||||
|
||||
| Valore env `S2S_INJECT_PROFILE` | Comportamento |
|
||||
|---|---|
|
||||
| `false` / non impostato | **Pure passthrough.** Nessuna fetch upstream, nessuna modifica a `session.update` o al body `/api/reasoning`. Il proxy passa tutto invariato. |
|
||||
| `true` / `1` / `yes` / `on` | Pipeline inject attiva. I singoli blocchi poi dipendono dagli env specifici e dai query params della URL del client. |
|
||||
|
||||
Perché esiste: su deploy generici (nessuna integrazione con AzurianLab /
|
||||
Aquachile / sistemi interni) il client rimane un puro speech-to-speech
|
||||
passthrough, senza dipendere da endpoint esterni.
|
||||
|
||||
---
|
||||
|
||||
## 1. Blocco episodio podcast — `episode_id`
|
||||
|
||||
**Priorità massima.** Quando c'è, viene considerato il riferimento PRINCIPALE
|
||||
e più fidato per tutta la sessione.
|
||||
|
||||
### Trigger (quando si attiva)
|
||||
|
||||
Una qualunque delle due:
|
||||
1. URL pagina browser → query string `?episode_id=episode-xxx`
|
||||
2. Body `POST /api/reasoning` → campo `episode_id: "episode-xxx"`
|
||||
|
||||
### Sorgente dati
|
||||
|
||||
GET plain verso l'endpoint Aquachile AzurianLab:
|
||||
|
||||
```
|
||||
https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}
|
||||
```
|
||||
|
||||
Parametri fetch:
|
||||
- timeout 15 s
|
||||
- `follow_redirects: true`
|
||||
- **Nessuna auth** (endpoint pubblico)
|
||||
|
||||
Formato risposta accettato:
|
||||
- `Content-Type: application/json` → oggetto arbitrario o array
|
||||
- `{"result": <payload>}` → wrapper supportato, il contenuto viene "spacchettato"
|
||||
- `text/plain` → usato come stringa raw
|
||||
|
||||
### Cap
|
||||
|
||||
24.000 caratteri dopo serializzazione JSON (transcript lunghi).
|
||||
Se supera → troncato con `...` append.
|
||||
|
||||
### Wrap applicato
|
||||
|
||||
```
|
||||
< PODCAST_EPISODE_HINT >
|
||||
<<<PODCAST_EPISODE_BEGIN>>>
|
||||
{ <TUTTI I CAMPI DELL'EPISODIO, JSON> }
|
||||
<<<PODCAST_EPISODE_END>>>. This is the episode the caller is currently interacting with — answer with reference to it first.
|
||||
```
|
||||
|
||||
**Hint testuale** (in testa, letto dal modello come primissima frase del blocco):
|
||||
|
||||
> Podcast episode context loaded from the page `episode_id` query param (current episode the user is referencing). Treat the episode fields (title, description, transcript, script, show notes, guests, audio metadata, etc.) as the **PRIMARY trusted reference material** for this session.
|
||||
|
||||
### Campi supportati (esempi — lista NON esaustiva, qualsiasi campo extra viene incluso)
|
||||
|
||||
- metadati base: `id`, `title`, `slug`, `description`, `summary`, `publishedAt`, `durationSeconds`
|
||||
- contenuti: `transcript`, `podcastScript`, `script`, `chapters[]`, `showNotes`
|
||||
- persone: `host`, `guests[]`, `author`, `guestsInfo[]`
|
||||
- audio: `audioUrl`, `audioFile`, `waveform`, `sampleRate`
|
||||
- classificazione: `tags[]`, `topics[]`, `category`, `season`, `episodeNumber`
|
||||
- riferimenti incrociati: `reportId`, `scheduleId`, `company`, `product`
|
||||
|
||||
### Riferimenti codice
|
||||
|
||||
- fetch upstream: [_fetch_podcast_episode](file:///home/duxiter/s2s/s2s-client/server.py#L1010-L1078)
|
||||
- wrap + serialize + cap 24k: [_resolve_episode_context](file:///home/duxiter/s2s/s2s-client/server.py#L475-L504)
|
||||
- hint costante: [PODCAST_EPISODE_HINT](file:///home/duxiter/s2s/s2s-client/server.py#L426-L429)
|
||||
|
||||
---
|
||||
|
||||
## 2. Blocco profilo chiamante — `phoneNumber`
|
||||
|
||||
Contesto identità del chiamante. Viene usato per il modello quando risponde
|
||||
(riferirsi all'utente per nome, conoscere la sua azienda, ruolo, ecc.).
|
||||
|
||||
### Trigger
|
||||
|
||||
1. URL browser → `?phoneNumber=%2B<numero-internazionale>`
|
||||
2. Body `/api/reasoning` → `phoneNumber: "+…"`
|
||||
|
||||
### Sorgente dati
|
||||
|
||||
```
|
||||
GET VOCERO_USER_PHONE_AUTH_URL?phoneNumber=<phone>
|
||||
```
|
||||
|
||||
Se è configurato anche `S2S_LOGIN_BY_PHONE`, PRIMA della profile fetch:
|
||||
```
|
||||
POST S2S_LOGIN_BY_PHONE body { "phoneNumber": "..." }
|
||||
```
|
||||
Da questa risposta il proxy estrae un bearer token e lo inoltra come
|
||||
`Authorization: Bearer <token>` sulle chiamate successive (profile, reports,
|
||||
summary).
|
||||
|
||||
Formato risposta: JSON arbitrario, wrapper `{"result": ...}` supportato.
|
||||
|
||||
### Cap
|
||||
|
||||
1.600 caratteri dopo JSON serialize.
|
||||
|
||||
### Wrap
|
||||
|
||||
```
|
||||
Caller profile resolved from the page phoneNumber:
|
||||
{ <profilo JSON troncato> }.
|
||||
Use this as trusted caller context when relevant.
|
||||
```
|
||||
|
||||
### Riferimenti codice
|
||||
|
||||
- login by phone: [_login_by_phone](file:///home/duxiter/s2s/s2s-client/server.py#L812-L890)
|
||||
- fetch profile: [_fetch_user_profile_for_phone](file:///home/duxiter/s2s/s2s-client/server.py#L1081-L1134)
|
||||
- wrap serialize: [_resolve_user_context_text](file:///home/duxiter/s2s/s2s-client/server.py#L432-L444)
|
||||
|
||||
---
|
||||
|
||||
## 3. Blocco lista reports / script schedulati
|
||||
|
||||
Riferimento "storico" dei contenuti preparati prima della sessione. Non sono
|
||||
contesto live, ma materiale che il modello deve sapere esiste (e a cui
|
||||
riferirsi se l'utente chiede di report pregressi, script, podcast schedulati).
|
||||
|
||||
### Trigger
|
||||
|
||||
Solo env: `S2S_REPORT_LIST` deve essere valorizzato a un URL valido.
|
||||
**Funziona anche SENZA `phoneNumber`** (lista reports generica per tutti).
|
||||
|
||||
Se `phoneNumber` è presente lo passa come query param all'endpoint.
|
||||
|
||||
### Sorgente dati
|
||||
|
||||
```
|
||||
GET S2S_REPORT_LIST[?phoneNumber=<phone>]
|
||||
```
|
||||
|
||||
Wrapper `{"result": [...]}` supportato.
|
||||
|
||||
### Cap
|
||||
|
||||
12.000 caratteri sul blocco COMPLETO (ogni podcast script può essere lungo).
|
||||
|
||||
### Wrap + filtri + sort
|
||||
|
||||
```
|
||||
Scheduled reports / podcast scripts available for this caller before the current session.
|
||||
Each entry is a scheduled report; use the podcastScript contents as trusted reference material
|
||||
when the user asks about scheduled topics, reports, or prior content.
|
||||
<<<REPORTS_LIST_BEGIN>>>
|
||||
[
|
||||
{
|
||||
"sourceScheduleLabel": "...",
|
||||
"date": "2026-09-06T…",
|
||||
"podcastScript": "…"
|
||||
},
|
||||
… ordinati per data DESC (nuovi prima) …
|
||||
]
|
||||
<<<REPORTS_LIST_END>>>
|
||||
```
|
||||
|
||||
Regole per ogni entry:
|
||||
1. **Solo dict** vengono tenuti (stringhe/int scartati).
|
||||
2. Per ogni entry estrae 3 campi con fallback a nested shape:
|
||||
- Label: `sourceScheduleLabel` oppure `sourceSchedule.label`
|
||||
- Date: `date` oppure `sourceSchedule.date`
|
||||
- Script: `podcastScript`
|
||||
3. Ordinamento **data DESC** (nuovi prima) con parser robusto:
|
||||
- ISO datetime / date strings
|
||||
- `Date` / `DateTime` Python (se deserializzati)
|
||||
- Unix timestamp int/float
|
||||
- Lexicografico DESC come fallback per stringhe ISO-ish
|
||||
- Valori nulli / non-parse-abili finiscono in fondo.
|
||||
|
||||
### Riferimenti codice
|
||||
|
||||
- fetch: [_fetch_reports_list](file:///home/duxiter/s2s/s2s-client/server.py#L668-L745)
|
||||
- parse data sort key: [_parse_date_key](file:///home/duxiter/s2s/s2s-client/server.py#L507-L515)
|
||||
- wrap + filtro + sort: [_resolve_reports_context](file:///home/duxiter/s2s/s2s-client/server.py#L518-L666)
|
||||
|
||||
---
|
||||
|
||||
## 4. Blocco summary ultima conversazione
|
||||
|
||||
Sintesi della chiamata/chat IMMEDIATAMENTE precedente a questa. Usato dal
|
||||
modello per "ricordarsi" cosa si è detto l'ultima volta senza conservare
|
||||
storico lunghissimo nel prompt.
|
||||
|
||||
### Trigger
|
||||
|
||||
Entrambe le alternative sono mutuamente esclusive: `S2S_LATEST_SUMMARY_TEXT`
|
||||
ha **priorità assoluta** se valorizzato.
|
||||
|
||||
| Modalità | Trigger |
|
||||
|---|---|
|
||||
| **A — HTTP fetch (dinamico per caller)** | env `S2S_LATEST_SUMMARY` impostato **AND** `phoneNumber` presente in URL/body |
|
||||
| **B — Testo statico hardcoded (override)** | env `S2S_LATEST_SUMMARY_TEXT` impostato (stringa non vuota) — **ignora completamente** `S2S_LATEST_SUMMARY`, `phoneNumber`, login-by-phone, Bearer auth. Utile quando l'operatore vuole iniettare una versione **pulita, essenziale e pre-pulita** del summary, senza dipendere da nessun endpoint upstream. |
|
||||
|
||||
### Sorgente dati
|
||||
|
||||
**Modalità A (HTTP fetch):**
|
||||
```
|
||||
GET S2S_LATEST_SUMMARY?phoneNumber=<phone>
|
||||
```
|
||||
Wrapper `{"result": <valore>}` supportato.
|
||||
Accetta SIA JSON che plain text (diversi backend summary ritornano testo semplice).
|
||||
|
||||
**Modalità B (override statico):**
|
||||
- Nessuna chiamata HTTP. Viene usato direttamente il valore grezzo di `S2S_LATEST_SUMMARY_TEXT` dal file `.env`.
|
||||
- Le sequenze di escape `\n` nel valore env vengono **interpretate** come a-capo vero (stesso comportamento di `S2S_PRIMARY_PROMPT`), quindi è possibile scrivere summary multi-line in una sola riga del `.env`.
|
||||
- Esempio:
|
||||
```env
|
||||
S2S_LATEST_SUMMARY_TEXT=Última conversación 2026-09-22:\n- Usuario preguntó por mortalidad salmón AquaChile semana 38.\n- Se refirió al reporte Porcentaje Mortalidad.
|
||||
```
|
||||
- Il valore viene passato as-is a `_resolve_latest_summary_context` → il wrapper `LATEST_SUMMARY_HINT` e il cap a 1600 caratteri vengono applicati normalmente.
|
||||
|
||||
### Cap
|
||||
|
||||
1.600 caratteri (applicato da `_resolve_latest_summary_context` sia in modalità A che B).
|
||||
|
||||
### Wrap
|
||||
|
||||
```
|
||||
Latest conversation summary for this caller (most recent before the current session):
|
||||
<<<LATEST_SUMMARY_BEGIN>>>
|
||||
<summary, stringa o JSON serializzato>
|
||||
<<<LATEST_SUMMARY_END>>>.
|
||||
Use this as trusted history when the user references prior calls.
|
||||
```
|
||||
|
||||
### Riferimenti codice
|
||||
|
||||
- fetch HTTP (modalità A): [_fetch_latest_summary_for_phone](file:///home/duxiter/s2s/s2s-client/server.py#L1563-L1654)
|
||||
- override statico (modalità B) → riga iniziale `if S2S_LATEST_SUMMARY_TEXT:` dentro la funzione sopra
|
||||
- wrap: [_resolve_latest_summary_context](file:///home/duxiter/s2s/s2s-client/server.py#L987-L1012)
|
||||
|
||||
---
|
||||
|
||||
## 5. Blocco `user_context` custom dal browser (SOLO `/api/reasoning`)
|
||||
|
||||
**Canale solo HTTP reasoning.**
|
||||
Non è nel WS: il WS non ha campo user_context nel protocollo realtime
|
||||
OpenAI (tutto passa per `session.instructions`).
|
||||
|
||||
Composizione lato client in [main.js fetchOpenAiReasoningAnswer](file:///home/duxiter/s2s/s2s-client/main.js#L1337-L1364):
|
||||
|
||||
```js
|
||||
const profileCtx = voceroUserContextText();
|
||||
const ragCtx = customRagContextText();
|
||||
const user_context = [profileCtx, ragCtx]
|
||||
.map(s => s.trim()).filter(Boolean).join("\n\n");
|
||||
```
|
||||
|
||||
### Dove va a finire
|
||||
|
||||
Nel body a monte sia NODE_RED sia OPENAI, questo blocco viene **APPESO DOPO**
|
||||
tutti i blocchi iniettati server-side:
|
||||
|
||||
```
|
||||
user_context finale =
|
||||
[1. episode] ← solo se episode_id
|
||||
[2. profile] ← solo se phoneNumber + env profilatura
|
||||
[3. reports] ← solo se env S2S_REPORT_LIST
|
||||
[4. summary] ← solo se phoneNumber + S2S_LATEST_SUMMARY
|
||||
+ browser user_context (profileCtx + ragCtx)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ordine finale nel prompt LLM
|
||||
|
||||
Questa è la sequenza **letterale** con cui i blocchi vengono concatenati
|
||||
dentro `session.instructions` (WS) e in `instructions` / `user_context`
|
||||
(reasoning HTTP).
|
||||
|
||||
```
|
||||
1. EPISODIO PODCAST ← PRIORITÀ PIÙ ALTA (primary trusted reference)
|
||||
2. PROFILO CHIAMANTE ← identity / caller profile
|
||||
3. REPORTS SCHEDULATI ← scheduled / prior content
|
||||
4. SUMMARY ULTIMA CALL ← history più recente
|
||||
5. Base instructions (prompt predefinito o primary prompt)
|
||||
6. (solo reasoning) user_context custom inviato dal browser
|
||||
```
|
||||
|
||||
Perché questo ordine:
|
||||
- più in alto nel prompt = più peso / attenzione dal modello.
|
||||
- l'episodio attuale è il materiale che l'utente ha esplicitamente richiesto
|
||||
aprendo la pagina con `?episode_id=`, quindi merita massimo peso.
|
||||
- il profilo chiamante va sempre prima di "contenuti generici" (reports e
|
||||
summary) perché serve per coerenza nel riferirsi all'utente.
|
||||
- reports = contenuti schedulati PRIMA della sessione corrente → più vecchi
|
||||
dell'ultima chiamata, quindi messi PRIMA del summary ultima chiamata che
|
||||
è il contenuto più "vivo".
|
||||
|
||||
Implementazione ordine: [_augment_instructions_with_profile](file:///home/duxiter/s2s/s2s-client/server.py#L1149-L1159)
|
||||
```python
|
||||
contexts = [c for c in [episode_ctx, profile_ctx, reports_ctx, summary_ctx] if c]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Punti di innesco (quando viene calcolato / applicato)
|
||||
|
||||
### Canale WebSocket — `/v1/realtime`
|
||||
|
||||
**MOMENTO 1 (solo una volta):** apertura WS.
|
||||
|
||||
Nel handler [@app.websocket("/v1/realtime")](file:///home/duxiter/s2s/s2s-client/server.py#L1791-L1959)
|
||||
appena la WS viene accettata:
|
||||
1. parsa `ws.url.query` per `phoneNumber` e `episode_id`
|
||||
2. esegue login (se attivo) → profile → episode → reports → summary
|
||||
3. salva TUTTI i blocchi già wrappati e serializzati nel dict `session_ctx`:
|
||||
```python
|
||||
session_ctx = {
|
||||
"injected_episode_context": "...",
|
||||
"injected_profile_context": "...",
|
||||
"injected_reports_context": "...",
|
||||
"injected_summary_context": "...",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**MOMENTO 2 (ogni `session.update` client→upstream):** nel task
|
||||
[_client_to_upstream](file:///home/duxiter/s2s/s2s-client/server.py#L2649-L2661)
|
||||
ogni frame in uscita viene passato a:
|
||||
|
||||
```python
|
||||
_patch_client_session_update_with_profile(
|
||||
frame_text,
|
||||
user_context_text = profile_ctx,
|
||||
reports_context_text = reports_ctx,
|
||||
summary_context_text = summary_ctx,
|
||||
episode_context_text = episode_ctx, # ← nuovo
|
||||
)
|
||||
```
|
||||
|
||||
Se il frame NON è `session.update` → ritorna **identico** e non viene loggato
|
||||
(per evitare spam).
|
||||
Se è `session.update` → patch del campo `payload["session"]["instructions"]`.
|
||||
|
||||
### Canale HTTP — `POST /api/reasoning`
|
||||
|
||||
**Ogni chiamata** ricostruisce i blocchi da zero nel
|
||||
[handler `/api/reasoning`](file:///home/duxiter/s2s/s2s-client/server.py#L1411-L1577):
|
||||
1. legge `req.phoneNumber` + `req.episode_id`
|
||||
2. fetch episodio (se episode_id) + login + profile + reports + summary
|
||||
3. costruisce `user_context_parts` e `effective_instructions`
|
||||
4. inoltra al backend (NODE_RED o OPENAI)
|
||||
|
||||
---
|
||||
|
||||
## Quando un blocco viene SKIPPATO (niente inject)
|
||||
|
||||
Tutti i blocchi hanno 5 possibili ragioni di skip. Nessuna rompe mai il
|
||||
flusso: ogni caso produce una riga `[INJECT-*] ⏭️ skipped` nei log e basta.
|
||||
|
||||
| Causa | Log |
|
||||
|---|---|
|
||||
| Master flag `S2S_INJECT_PROFILE=false` | `[INJECT-PROFILE] ⏭️ skipped (S2S_INJECT_PROFILE=false)` |
|
||||
| Parametro mancante (es. `phoneNumber` non in URL) | `[INJECT-*] ⏭️ skipped (<parametro> missing)` |
|
||||
| Endpoint env non configurato | `[INJECT-*] ⏭️ skipped (env for ... empty)` |
|
||||
| Upstream HTTP ≠ 200 | `[INJECT-*] ⛔ upstream HTTP <status_code>` |
|
||||
| Upstream ha ritornato vuoto / null / `{}` / `[]` | `[INJECT-*] ⏭️ upstream returned empty` |
|
||||
| Eccezione di rete (timeout, DNS, SSL) | `[INJECT-*] ⛔ fetch unreachable error=<exc>` |
|
||||
|
||||
---
|
||||
|
||||
## Marcatori log / marker nel prompt
|
||||
|
||||
Ogni blocco è **auto-documentante** per ispezione dei log con coppie di
|
||||
marker:
|
||||
|
||||
```
|
||||
<<<PODCAST_EPISODE_BEGIN>>> / <<<PODCAST_EPISODE_END>>>
|
||||
<<<REPORTS_LIST_BEGIN>>> / <<<REPORTS_LIST_END>>>
|
||||
<<<LATEST_SUMMARY_BEGIN>>> / <<<LATEST_SUMMARY_END>>>
|
||||
```
|
||||
|
||||
Quando `S2S_DEBUG_PROMPT=true` viene loggato il contenuto COMPLETO di ogni
|
||||
blocco con i tag `[DEBUG-EPISODE]`, `[DEBUG-REPORTS]`, `[DEBUG-SUMMARY]` sia
|
||||
all'atto del load (WS connect) sia dopo l'augment delle instructions (prima
|
||||
di inoltrare a monte). **WARNING:** questo env può loggare PII.
|
||||
|
||||
---
|
||||
|
||||
## Tabella riepilogo: trigger per ogni blocco
|
||||
|
||||
| # | Blocco | Trigger | env richiesto | parametro | cap |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | Episodio podcast | `episode_id` in URL o body | (nessuno — URL hardcodato Aquachile) | `episode_id=` | 24 k |
|
||||
| 2 | Profilo chiamante | `phoneNumber` | `VOCERO_USER_PHONE_AUTH_URL` | `phoneNumber=` | 1,6 k |
|
||||
| 3 | Reports schedulati | sempre se env | `S2S_REPORT_LIST` | opzionale `phoneNumber=` | 12 k |
|
||||
| 4 | Summary ultima call | `phoneNumber` **oppure** override statico | `S2S_LATEST_SUMMARY` (HTTP) **o** `S2S_LATEST_SUMMARY_TEXT` (statico, ha priorità) | `phoneNumber=` (solo modalità HTTP; override statico non richiede nessun param) | 1,6 k |
|
||||
| 5 | Browser user_context | solo reasoning | (nessuno) | body `user_context` | illimitato |
|
||||
|
||||
---
|
||||
|
||||
## Error handling strategico
|
||||
|
||||
Perché tutti gli errori sono silenziosi (log + skip) e non abortiscono:
|
||||
|
||||
> Se il backend dei profili è giù per manutenzione 5 minuti non vogliamo
|
||||
> che il Voice Agent smetta di funzionare. Semplicemente l'assistente non
|
||||
> conosce l'identità del chiamante per 5 minuti, ma parla normalmente.
|
||||
|
||||
L'utente non si accorge di niente. L'operatore legge i log e vede le righe
|
||||
`⛔`; può risolvere offline.
|
||||
|
||||
Se volessi forzare il fail (raro, su deploy dedicato), si trasformerebbero
|
||||
le `return None` finali in `raise HTTPException(...)` — ma per default
|
||||
non è il comportamento voluto.
|
||||
|
||||
---
|
||||
|
||||
## Appendice A — Start-session modes (`S2S_START_SESSION_MODE`)
|
||||
|
||||
Oltre alla inject pipeline lato server, è presente una feature lato client che
|
||||
governa **come e se la sessione vocale inizia in automatico** e se il modello
|
||||
saluta per primo (in modo identico alla wake-word "Hola Vocero").
|
||||
|
||||
### Valori supportati
|
||||
|
||||
| `S2S_START_SESSION_MODE` | Chi inizia? | Richiede click utente? | Trigger greet iniziale |
|
||||
|---|---|---|---|
|
||||
| *(vuoto) / default* | Utente, tap manuale Orb | ✅ | No |
|
||||
| `TALK` | Auto dopo 2s con overlay unlocker | ✅ (1 click policy-safe su Safari/iOS) | WS client: `sendUserText("hola vocero") + requestResponse()` dopo `session.updated` |
|
||||
| `TALKFIRST` | Auto dopo 2s **SENZA click** (se possibile) | ❌ se mic già "allow always"; fallback unlocker se `getUserMedia` fallisce | **Stesso identico codice della wake-word vera:** `pendingWakeWordConfirmationText = "hola vocero"` (main.js `case "connected"` → `sendUserText + requestResponse`) |
|
||||
|
||||
### Warmup 2s
|
||||
|
||||
Il timer non parte prima di 2000 ms per lasciare terminare:
|
||||
- `fetchVoceroUserProfile()` (identity chiamante)
|
||||
- `fetchVoceroReportsList()` (storico script podcast)
|
||||
- fetch episode (`episode_id`) se presente
|
||||
- `GET /api/config` (con la quale il client apprende `startSessionMode`)
|
||||
|
||||
Senza warmup le injection lato server potrebbero ancora non essere completate e
|
||||
il greet iniziale arriverebbe "cieco" (senza contesto episodio/profilo).
|
||||
|
||||
### Fallback di `TALKFIRST`
|
||||
|
||||
Nel 99% dei casi `getUserMedia()` fuori da una user gesture fallisce
|
||||
(NotAllowedError / NotReadableError su Safari, Chrome con policy strict, iOS).
|
||||
In tal caso:
|
||||
1. si chiude il client WS (`teardown()`)
|
||||
2. stato torna `idle`
|
||||
3. si mostra `#talk-unlocker` con la dicitura:
|
||||
- **"Se requiere permiso de micrófono"**
|
||||
- **"Toca en cualquier lugar para permitir el audio y empezar"**
|
||||
4. `pendingWakeWordConfirmationText` NON viene svuotato → quando l'utente
|
||||
toccherà, `case "connected"` triggerà comunque la greet come wake-word.
|
||||
|
||||
### Riferimenti codice
|
||||
|
||||
| Componente | File | Range righe |
|
||||
|---|---|---|
|
||||
| Lettura env + banner startup + `/api/config` exposure | [server.py](file:///home/duxiter/s2s/s2s-client/server.py) | [L150](file:///home/duxiter/s2s/s2s-client/server.py#L150), [L196-L212](file:///home/duxiter/s2s/s2s-client/server.py#L196-L212), [L1383-L1400](file:///home/duxiter/s2s/s2s-client/server.py#L1383-L1400) |
|
||||
| Overlay HTML unlocker | [index.html](file:///home/duxiter/s2s/s2s-client/index.html#L32-L39) | L32-L39 |
|
||||
| Stili overlay | [style.css](file:///home/duxiter/s2s/s2s-client/style.css#L145-L215) | L145-L215 |
|
||||
| Variabile globale + parse config client | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L628-L632) | L628-L632 |
|
||||
| `fetchConfig()` parsing + `reportDebug("B")` | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2309-L2320) | L2309-L2320 |
|
||||
| `new S2sWsRealtimeClient({startWithResponse})` | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2683-L2692) | L2683-L2692 |
|
||||
| Scheduling + `tryTalkFirstDirect()` + unlocker gesture | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2976-L3083) | L2976-L3083 |
|
||||
| `onClientStatus("connected")` → wake-word inject flow | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2897-L2914) | L2897-L2914 |
|
||||
| WS client → `TALK` mode sendUserText("hola vocero") + `requestResponse()` | [s2s-ws-client.js](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L688-L713) | L688-L713 |
|
||||
|
||||
---
|
||||
|
||||
## Riferimenti file
|
||||
|
||||
| File | Ruolo |
|
||||
|---|---|
|
||||
| [server.py](file:///home/duxiter/s2s/s2s-client/server.py) | Tutta la pipeline inject: fetch, wrap, patch, handler WS + reasoning |
|
||||
| [main.js](file:///home/duxiter/s2s/s2s-client/main.js) | Passaggio client `?phoneNumber` + `?episode_id` → WS query params e POST `/api/reasoning` body |
|
||||
| [CHANGELOG.md](file:///home/duxiter/s2s/s2s-client/CHANGELOG.md) | History rilasci / modifiche alla inject pipeline |
|
||||
+16
-237
@@ -10,247 +10,26 @@ short_description: Voice chat over WebSocket against a HF speech-to-speech
|
||||
hf_oauth: true
|
||||
---
|
||||
|
||||
# Realtime Voice Demo (WebSocket transport)
|
||||
# Realtime voice client
|
||||
|
||||
Browser voice-chat UI for the
|
||||
backend. The browser streams mic audio over a WebSocket using the OpenAI
|
||||
Realtime **GA** protocol and plays back the assistant's audio as it arrives.
|
||||
Browser voice-chat UI and FastAPI server for an OpenAI Realtime-compatible
|
||||
speech-to-speech backend. The browser streams microphone audio over WebSocket,
|
||||
plays assistant audio, displays transcripts, and offers Spanish controls.
|
||||
|
||||
---
|
||||
The complete [usage guide](../docs/client/usage.md) covers local startup,
|
||||
connection modes, Docker, settings, and usage limits. See the
|
||||
[documentation index](../docs/README.md), [architecture](../docs/client/architecture.md),
|
||||
and [change history](CHANGELOG.md) for implementation details and history.
|
||||
|
||||
## 🆕 Novedades / Actualizaciones (Azurian Vocero realtime ver.Beta 2.1)
|
||||
|
||||
> Versión del navegador que debes ver cargada en consola: **`2026-09-04 11:20`**.
|
||||
> Si no aparece, haz **borrado de datos del sitio + recarga dura (Ctrl+Shift+R)** o abre una pestaña en modo incógnito.
|
||||
|
||||
### 🎨 Rediseño visual completo (tema aurora + glassmorphism)
|
||||
- **Tema de fondo aurora**: gradiente radial violeta/rosa/cian sobre lienzo oscuro con malla sutil + animación `auroraShift` lenta.
|
||||
- **Cromado uniforme glass**: todos los elementos flotantes (tarjeta brand, panel user profile, botones icono, botón enviar chat, panel informes sidebar) usan *glass blur + saturate + borde gradiente brand 1px* (técnica mask `padding-box / border-box` doble linear-gradient).
|
||||
- **Tokens de marca 3 colores brand** (violeta `#8b5cf6`, rosa `#ec4899`, cian `#22d3ee`) mezclados con `color-mix(in srgb, brand X%, neutral)` → ningún color hardcodeado deriva; todo el UI sigue estos tres tonos.
|
||||
|
||||
### 🧭 Estructura (orden vertical de arriba → abajo)
|
||||
1. **`header.topbar`** brand "Vocero" → avatar circular iniciales nombre usuario.
|
||||
- 🟦 **Avatar user**: círculo 40×40px con gradiente brand, initials mayúsculas bold 800, hover lift/glow + foco teclado. Al hacer clic → diálogo modal perfil usuario (rejilla 2 columnas email/teléfono/username/rol/empresa/userid/país/fecha creación/último login; links `mailto:`/`tel:`; chips mono para códigos; mobile ≤480px apilado 1 col).
|
||||
- 🟦 **Botón toggle chat (icono speech bubble) → ELIMINADO PERMANENTEMENTE**. La chat ahora siempre está visible inline; el botón toggle ya no existe en el DOM.
|
||||
2. **`main.stage`** (arriba del todo, flexible): orbe/globode voz animado (3 anillos: core, ring, conic spin 3 velocidades + highlight specular + glows breathe `breatheOrb`) + caption `TAP OR SAY "HOLA VOCERO"` micro-bob uppercase gradient.
|
||||
- ✅ Antes había una zona vacía sobre el orbe → eliminada, ahora el stage empieza en la parte superior (`justify-content: flex-start`).
|
||||
3. **`<section id="chat-panel" class="chat-panel chat-panel-inline">` SIEMPRE VISIBLE, 100% ancho, SOTTO stage, SOPRA footer sticky** (rellena la zona vacía inferior).
|
||||
- ✅ Antes: overlay lateral derecho. **Ahora: bloque inline dentro de `#app` como hijo flex-col**.
|
||||
- 🟦 Sin marco / sin riquadro (fondo aurora seamless, 0 borde, 0 shadow, 0 backdrop blur).
|
||||
- Cabecera `CONVERSACIÓN` h3 uppercase 13px gradiente.
|
||||
- `#chat-history` scroll interno. Burbujas de mensaje glass con **código de colores por rol**: usuario (tint 10% voice-user cyan + borde 44% cyan + glow) / assistant (tint 10% voice-assistant violet/pink + borde 42% + glow).
|
||||
- Distancias seguras anti-borde: `.hist-msg.user margin-left:6px auto dx` / `.hist-msg.assistant margin-right:22px auto sx` → **nunca las burbujas tocan el borde del viewport** (fix confirmado).
|
||||
- Formulario input texto `#chat-form` (arriba del botón enviar ✈️) **actualmente oculto (`hidden`)** para modo voz-puro; elimina ese atributo para reactivarlo. El handler submit que envía `client.sendUserText(text)` al websocket ya está conectado.
|
||||
4. **Footer sticky `position: sticky; bottom:0; z:3; backdrop-blur`** "Azurian Vocero realtime ver.Beta 2.1" siempre dentro del viewport, incluso cuando el documento crece y haces scroll (no requiere scroll hacia abajo para verlo). Separador superior línea gradiente brand.
|
||||
|
||||
### 📜 Sidebar de informes a la IZQUIERDA (antes topbar inline)
|
||||
- Desktop **≥ 1024px**: sidebar fija left 320px siempre visible; `#app padding-left 320px` reserva sitio.
|
||||
- Tablet/Mobile **< 1024**: offcanvas `translateX(-105%)` slide-in con backdrop blur; toggle mediante botón icono reportes `#reports-sidebar-toggle` en topbar con badge contador.
|
||||
- `#reports-hint` siempre visible; `.reports-suggestions` y `.topbar-right` con puerta triple de visibilidad (atributo HTML hidden + CSS rule `[data-debug-only]:not([data-debug-enabled]) !important display:none` + JS `applyDebugOnlyVisibilityGate()` ejecutado 6 veces por lifecycle) → solo aparecen cuando el query param `?debug=true` está presente en la URL.
|
||||
|
||||
### ⬇️ Auto-scroll chat SIEMPRE hacia la última respuesta del asistente
|
||||
Arquitectura anti-clipping definitiva (versión 11:20 fix):
|
||||
|
||||
1. **CSS sobrante intencional** (`96 → 160px chat-history padding-bottom`, `40→160px chat-panel clamp`, `56→120px #app padding-bottom + safe-area-inset-bottom`) → el último párrafo siempre está por ENCIMA de la línea del footer glass flotante.
|
||||
2. **Unidades móviles seguras `svh / svw`** (smallest viewport): evita oscilaciones layout cuando aparece/desaparece la barra URL del móvil.
|
||||
3. **JS cancel-and-reschedule** cada vez que `_scrollToBottom()` se llama: si llegan 100 tokens streaming en 1 frame, el último cancela todas las cadenas pendientes y se ejecuta desde cero → **no hay coalescing flag SKIP** (esta era la causa original del scroll insuficiente).
|
||||
4. **5 reintentos escalonados `0 / +30 / +70 / +150 / +250ms`**:
|
||||
- 0 ms (inmediato): 90% de los casos.
|
||||
- 30 ms: tras 1 layout-pass.
|
||||
- 70 ms: **corrige el wrapping de la última palabra en una línea nueva** (bug más común).
|
||||
- 150 ms: después de reflow Chrome scroll-anchor.
|
||||
- 250 ms: después de expandir filas tool / imágenes grandes.
|
||||
5. **Snap con `Number.MAX_SAFE_INTEGER`**: el navegador clampa automáticamente al máximo real (`scrollHeight - clientHeight`); no hay race de leer scrollHeight demasiado pronto.
|
||||
6. `overflow-anchor: none; scroll-behavior: auto !important;` → sin smooth, sin Chrome scroll-anchor-luchando contra nosotros.
|
||||
7. Llamadas scroll explicitas extra en: `onResponseFinished` (fin de respuesta assistant, crítico), `onToolCall`, `onToolResult`.
|
||||
|
||||
### 🛡️ Protecciones anti-error de refs eliminadas
|
||||
- En el refactor de chat eliminamos 4 nodos DOM (`#chat-btn`, `#chat-badge`, `#chat-panel-backdrop`, `#chat-panel-close`).
|
||||
- **Antes**: el constructor de `ChatView` usaba `$("#...")` strict → lanzaba `Error("Missing element: #chat-btn")` al arrancar.
|
||||
- **Ahora**: esas 4 refs opcionales usan `document.querySelector("#...")` → devuelve `null` cuando no existen, sin throw; todos los listeners guarded `if (el)`. Los 3 IDs obligatorios (`#chat-panel`, `#chat-history`, `#bubble-stack`) siguen con strict para fallar fuerte si alguien los borra por error.
|
||||
|
||||
### 🔄 Actualizaciones de versión / cache bust
|
||||
Para evitar caché HTTP agresiva en navegadores móviles (ESM sin bundler + service worker = nunca refrescan URLs iguales), todas las referencias a recursos estáticos usan query param `?v=YYYYMMDDHHMM` sincronizado en **6 puntos**:
|
||||
1. `index.html <link rel="stylesheet" href="style.css?v=...">`
|
||||
2. `index.html <script type="module" src="main.js?v=...">`
|
||||
3-6. Los **4 imports ESM dentro de main.js** (`ws/s2s-ws-client`, `ui/dom`, `ui/chat`, `ui/account`).
|
||||
|
||||
Además hay un **script inline en `<head>`** tras el style.css que al `DOMContentLoaded` fuerza a reescribir el href del stylesheet con la última versión si por algún motivo cargó la antigua (anti-cache móvil).
|
||||
|
||||
### Deploy
|
||||
```bash
|
||||
cd /home/duxiter/s2s
|
||||
docker compose restart s2s_ui
|
||||
```
|
||||
Limpia siempre datos del sitio en el navegador (Application → Storage → Clear site data) y recarga dura `Ctrl+Shift+R` al finalizar.
|
||||
|
||||
---
|
||||
|
||||
## Quick start (local)
|
||||
|
||||
1. **Start the speech-to-speech backend** in realtime mode (from the repo root;
|
||||
```bash
|
||||
uv run speech-to-speech \
|
||||
--mode realtime \
|
||||
--stt parakeet-tdt \
|
||||
--llm_backend transformers \
|
||||
--tts kokoro \
|
||||
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
|
||||
--llm_device mps \
|
||||
--llm_torch_dtype float16 \
|
||||
--enable_live_transcription
|
||||
```
|
||||
|
||||
The realtime server listens on `ws://localhost:8765/v1/realtime` by default
|
||||
(`--ws_host` / `--ws_port` to change).
|
||||
|
||||
2. **Start this app**, pointing it at the backend with `SPEECH_TO_SPEECH_URL`:
|
||||
|
||||
```bash
|
||||
uv pip install -r demo/requirements.txt
|
||||
export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
|
||||
export SERPER_API_KEY=... # optional; web search is disabled without it
|
||||
uv run uvicorn --app-dir demo server:app --reload --port 7860
|
||||
```
|
||||
|
||||
Or with Docker:
|
||||
|
||||
```bash
|
||||
docker build -t s2s-demo demo/
|
||||
docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
|
||||
```
|
||||
|
||||
3. Open <http://localhost:7860/>, click the orb, allow the mic, talk.
|
||||
|
||||
> Browsers require **HTTPS or `localhost`** for `getUserMedia()` (mic + camera).
|
||||
> `127.0.0.1` and `localhost` both work; plain `http://192.168.x.y` does NOT.
|
||||
|
||||
Smoke-test the backend from the shell:
|
||||
Build from the repository root so the container includes the documentation:
|
||||
|
||||
```bash
|
||||
websocat ws://localhost:8765/v1/realtime
|
||||
# -> you should get a session.created event back immediately
|
||||
docker build -f s2s-client/Dockerfile -t s2s-client .
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
1. The browser opens a WebSocket on the configured `/v1/realtime` URL.
|
||||
2. Server pushes `session.created` on connect. Client replies with
|
||||
`session.update` (OpenAI Realtime **GA** schema: `session.audio.input`,
|
||||
`session.audio.output`, `session.output_modalities`).
|
||||
3. Client streams mic audio as PCM16 16 kHz mono base64 chunks
|
||||
(`input_audio_buffer.append`, one frame every ~40 ms).
|
||||
4. Server pushes `response.output_audio.delta` (PCM16 24 kHz mono base64)
|
||||
and transcript deltas.
|
||||
|
||||
The backend exposes one concurrent session per pipeline unit
|
||||
(`--num_pipelines` to serve more).
|
||||
|
||||
## Connecting to a backend
|
||||
|
||||
Three modes, picked by env (`/api/config` tells the client which one is active):
|
||||
|
||||
- **`SPEECH_TO_SPEECH_URL` env** — the mode you want for local use, and the
|
||||
highest priority. The browser connects **directly** to this realtime
|
||||
WebSocket URL; it's shown read-only in Settings. Setting it disables the
|
||||
load-balancer logic entirely (no `/api/session` proxy, no queue, no
|
||||
metering, no sign-in). Unlike the LB address it is not a secret. Accepts a
|
||||
full `ws(s)://host/v1/realtime` URL or a bare host like `localhost:8765`
|
||||
(the app adds `/v1/realtime`).
|
||||
- **Neither env set** — **Settings → Speech-to-speech server URL**: paste a
|
||||
full connect URL or a bare host, and the browser connects to it directly.
|
||||
- **`LOAD_BALANCER_URL` env** — multi-compute deployments only: the browser
|
||||
POSTs the same-origin `/api/session` proxy, the server forwards to the LB,
|
||||
and the browser dials the per-session compute URL the LB hands back. The LB
|
||||
address never reaches the browser; the Settings URL field is hidden.
|
||||
|
||||
| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Metering |
|
||||
|:---:|:---:|:---:|---|---|---|
|
||||
| ✅ | any | any | direct → pinned URL | visible, locked | off |
|
||||
| – | – | any | direct → user URL | editable | off |
|
||||
| – | ✅ | ✅ | LB proxy | hidden | **on** |
|
||||
| – | ✅ | – | LB proxy | hidden | off |
|
||||
|
||||
**Settings → Restart** reconnects with the current voice, instructions and URL.
|
||||
|
||||
## Tools
|
||||
|
||||
The assistant can call two tools mid-conversation (toggle them from the **Tools**
|
||||
button, top-right):
|
||||
|
||||
- **Web search** — Google results via Serper.dev, proxied server-side so the key
|
||||
never reaches the browser. Set `SERPER_API_KEY` as an env var / Space secret.
|
||||
Without it, the tool is disabled unless the user pastes their own key in the
|
||||
Tools panel.
|
||||
- **Camera** — while enabled, a live self-view shows bottom-left; when the model
|
||||
calls the tool, the current frame is sent to the vision-language model so it can
|
||||
see what you're showing it.
|
||||
|
||||
## Usage limits (deployed Space only)
|
||||
|
||||
Conversation time is metered per UTC day by sign-in tier (see `limiter.py` /
|
||||
`auth.py`), but **only on the deployed Space** — metering turns on only when BOTH
|
||||
`LOAD_BALANCER_URL` and `SPACE_ID` (injected automatically by the HF Space
|
||||
runtime) are present. Running locally — even with `LOAD_BALANCER_URL` exported —
|
||||
leaves the app unmetered. Tunable via env:
|
||||
|
||||
| Env | Default | What |
|
||||
|-----|---------|------|
|
||||
| `LIMIT_ANON_SEC` | `300` | Daily seconds for anonymous visitors (5 min) |
|
||||
| `LIMIT_FREE_SEC` | `600` | Daily seconds for signed-in non-PRO users (10 min) |
|
||||
| `UNLIMITED_ORGS` | _(adds to defaults)_ | Extra HF org names whose members get **unlimited** usage, like PRO |
|
||||
| `USAGE_HASH_SECRET` | _(random)_ | HMAC secret for hashing identity keys + signing the anon cookie |
|
||||
|
||||
PRO members are always unlimited. Members of `cerebras`, `HuggingFaceM4`,
|
||||
`smolagents`, and `pollen-robotics` are unlimited out of the box (shown as
|
||||
"Team", not "PRO"); set `UNLIMITED_ORGS=my-team` to add more. Matched
|
||||
case-insensitively against the user's organisations from HF OAuth.
|
||||
|
||||
## Settings (stored in `localStorage`)
|
||||
|
||||
| Key | What |
|
||||
|-----|------|
|
||||
| Speech-to-speech server URL | Direct realtime WebSocket URL (hidden/locked when pinned by env) |
|
||||
| Voice | Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian) |
|
||||
| Instructions | System prompt sent in `session.update` once the WS opens |
|
||||
|
||||
LocalStorage keys are namespaced `s2s.ws.*` so this app's settings do
|
||||
NOT collide with the WebRTC variant.
|
||||
|
||||
## Files
|
||||
|
||||
> 🇪🇸 **Notas adicionales (Azurian Vocero)**: `index.html` ahora incluye (1) botón toggle lateral izquierdo reportes `#reports-sidebar-toggle` con badge contador `#reports-sidebar-badge` visible solo <1024px, (2) `<dialog id="user-profile-dialog">` modal perfil usuario que se abre al hacer clic en el avatar circular del encabezado, (3) panel chat `<section id="chat-panel">` permanente dentro de `#app` entre `</main>` y footer sticky (el panel overlay derecho antiguo y el botón toggle chat superior fueron ELIMINADOS completamente).
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `index.html` | Single page, orb + settings modal (identical UI to the WebRTC app) |
|
||||
| `main.js` | State machine, settings, tools, camera, noise-gate UI wiring + chat-form submit handler (`client.sendUserText`) + user profile dialog open/close + reports sidebar open/close + 3-layer debug gate |
|
||||
| `ui/chat.js` | `ChatView`: history panel, ephemeral bubbles, transcript/tool streaming; always-inline 100% width chat panel w/ 5-step retry auto-scroll to last assistant line |
|
||||
| `ui/account.js` | `Account`: HF login chip + popover, daily-limit modal |
|
||||
| `ui/dom.js` | Shared helpers: strict `$(selector)` that throws on missing, `document.querySelector` fallback, `escHtml`, `truncateError`, `DEBUG` from URL |
|
||||
| `auth.py` | HF OAuth + per-request identity (tier, hashed keys) |
|
||||
| `limiter.py` | SQLite per-day talk-time budget (chunked server-clock reservation) |
|
||||
| `ws/s2s-ws-client.js` | WebSocket handshake + OpenAI Realtime GA protocol |
|
||||
| `ws/codec.js` | base64 <-> PCM helpers + transcript extraction (pure) |
|
||||
| `ws/orb-visualizer.js` | `OrbVisualiser`: FFT bands -> orb CSS custom properties (violet/pink/cyan brand gradient) |
|
||||
| `worklets/mic-capture.js` | AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks |
|
||||
| `worklets/audio-playback.js` | AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out |
|
||||
| `style.css` | Orb animations, layout, dark aurora gradient theme, glass chat bubbles, sticky footer, reports sidebar desktop left 320px / mobile offcanvas, user profile 2-col grid modal |
|
||||
|
||||
## Audio pipeline notes
|
||||
|
||||
- **Input**: `getUserMedia({ echoCancellation, noiseSuppression, autoGainControl })`
|
||||
feeds the `mic-capture` worklet at the `AudioContext` rate. The worklet
|
||||
resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast
|
||||
path, linear interpolation fallback for odd rates) and packs Int16 LE.
|
||||
- **Output**: `response.output_audio.delta` decodes to Int16 -> Float32
|
||||
and is posted to the `audio-playback` worklet. The worklet maintains a
|
||||
per-context ring buffer, linearly interpolates 24 -> 48, and applies
|
||||
short 32-frame fades on entry/exit to suppress clicks.
|
||||
- **Barge-in**: when the server VAD detects user speech mid-response
|
||||
(`input_audio_buffer.speech_started` while `ai-speaking`), the client
|
||||
posts `{ kind: "clear" }` to the playback worklet to wipe the queue
|
||||
immediately. The server itself cancels the in-flight response.
|
||||
|
||||
## Credits
|
||||
|
||||
FastAPI's API explorer remains at `/docs`. Project documents are served as files below
|
||||
`/docs/client/`, `/docs/speech/`, `/docs/adr/`, `/docs/examples/`, and
|
||||
`/docs/proposals/`. The historical `/project/` and `/public/docs.html` URLs
|
||||
redirect to the retained architecture proposal. Markdown remains source text;
|
||||
documentation directories are not browsable rendered pages. Start from the
|
||||
repository documentation index or request `/docs/README.md` explicitly.
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
# Convert the static Space into a Docker app with a search proxy
|
||||
|
||||
To give the model a web search tool, the executor (which runs in the browser of a
|
||||
public Space) needs a search key. A `sdk: static` Space serves files as-is with no
|
||||
runtime process, so it cannot hold a secret the browser uses without exposing it.
|
||||
We convert the Space from `sdk: static` to `sdk: docker`: a single container runs a
|
||||
small server (FastAPI + uvicorn) that both serves the existing front-end *unchanged*
|
||||
and exposes a same-origin `/search` proxy holding `SERPER_API_KEY` server-side. The
|
||||
whole app lives in that one container; the s2s speech-to-speech backend stays the
|
||||
separate load-balanced service it already is.
|
||||
|
||||
## Considered options
|
||||
|
||||
- **Stay static, user-supplied key only** — no owner key; search works only if each
|
||||
user pastes their own. Rejected as the default because it leaves the deployed demo
|
||||
with no working search.
|
||||
- **Separate proxy service** — same secrecy, but splits the app across two deploys.
|
||||
Rejected: the app must live in one container.
|
||||
- **Key baked into client JS at build time** — would be readable in the served
|
||||
bundle. Rejected: defeats the point.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Deployment is no longer static: there is a Dockerfile and a server process; the
|
||||
README front-matter changes from `sdk: static` to `sdk: docker`.
|
||||
- The client calls `/search` same-origin; the server reads the key from env. A user
|
||||
may still supply their own key as a fallback, sent per-request to the proxy.
|
||||
- The front-end, audio pipeline, and s2s handshake are untouched — only the hosting
|
||||
shape and the new route are added.
|
||||
@@ -1,356 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
|
||||
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate, max-age=0" />
|
||||
<meta http-equiv="Pragma" content="no-cache" />
|
||||
<meta http-equiv="Expires" content="0" />
|
||||
<title>Minimal Conversation · S2S backend (WebSocket)</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link
|
||||
href="https://fonts.googleapis.com/css2?family=Geist+Mono:wght@400;500;600&family=Inter:wght@400;500;600;700;800&display=swap"
|
||||
rel="stylesheet"
|
||||
/>
|
||||
<link rel="stylesheet" href="style.css?v=20260825003" />
|
||||
</head>
|
||||
<!--
|
||||
`booting` disables transitions/animations until the first paint commits.
|
||||
Without this the orb visibly fades/scales in on the very first frame
|
||||
because `state-idle` and the generic `.ind` defaults differ. The
|
||||
bootstrap script at the end of main.js strips the class after one rAF.
|
||||
-->
|
||||
<body class="booting">
|
||||
<div id="app">
|
||||
<header class="topbar">
|
||||
<div class="brand">
|
||||
<div class="ident">
|
||||
<div class="ident-head">
|
||||
<a class="ident-title">Azurian Vocero realtime test</a>
|
||||
</div>
|
||||
<div id="ident-user-name" class="ident-user-name" hidden></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="topbar-right">
|
||||
<!-- HF login chip / sign-in pill. Populated by ui/account.js; only
|
||||
shown when the deploy runs behind a load balancer with OAuth. -->
|
||||
<div id="account" class="account" hidden></div>
|
||||
<button id="about-btn-m" class="icon-btn about-btn-mobile" title="About this space" aria-label="About this space">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" y1="16" x2="12" y2="12"/><line x1="12" y1="8" x2="12.01" y2="8"/></svg>
|
||||
</button>
|
||||
<button id="tools-btn" class="icon-btn" title="Tools" aria-label="Tools">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/></svg>
|
||||
</button>
|
||||
<button id="chat-btn" class="icon-btn" title="Conversation history" aria-label="Conversation history">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15a2 2 0 0 1-2 2H7l-4 4V5a2 2 0 0 1 2-2h14a2 2 0 0 1 2 2z"/></svg>
|
||||
<span id="chat-badge" class="chat-badge" aria-hidden="true"></span>
|
||||
</button>
|
||||
<button id="settings-btn" class="icon-btn" title="Settings" aria-label="Settings">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 1 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 1 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 1 1-2.83-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 1 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 1 1 2.83-2.83l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 1 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 1 1 2.83 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9c0 .66.39 1.25 1 1.51H21a2 2 0 1 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main class="stage">
|
||||
<div class="orb-wrap">
|
||||
<div id="mic-gate" class="mic-gate">
|
||||
<svg id="mic-gate-arc" class="mic-gate-arc" viewBox="0 0 100 100" aria-hidden="true">
|
||||
<path id="mga-track" class="mga-track" fill="none" />
|
||||
<path id="mga-fill" class="mga-fill" fill="none" />
|
||||
<path id="mga-hit" class="mga-hit" fill="none" />
|
||||
<circle id="mga-handle" class="mga-handle" r="3" />
|
||||
</svg>
|
||||
<button id="mic-btn" class="side-btn" type="button" aria-label="Mute" title="Mute" aria-hidden="true">
|
||||
<svg class="mic-on" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="2" width="6" height="12" rx="3"/><path d="M5 10a7 7 0 0 0 14 0"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
|
||||
<svg class="mic-off" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><line x1="2" y1="2" x2="22" y2="22"/><path d="M9 5a3 3 0 0 1 6 0v4"/><path d="M9 10v1a3 3 0 0 0 5.1 2.1"/><path d="M19 10a7 7 0 0 1-1.24 3.97"/><path d="M5 10a7 7 0 0 0 11 5.67"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<button
|
||||
id="main-circle"
|
||||
class="circle state-idle"
|
||||
type="button"
|
||||
aria-label="Start voice conversation"
|
||||
>
|
||||
<span class="circle-glow" aria-hidden="true"></span>
|
||||
<span class="circle-ring" aria-hidden="true"></span>
|
||||
<span class="circle-ring-outer" aria-hidden="true"></span>
|
||||
<span class="circle-core">
|
||||
<span class="circle-indicator" aria-hidden="true">
|
||||
<svg class="ind ind-mic" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
|
||||
<rect x="9" y="2" width="6" height="12" rx="3" fill="currentColor" stroke="none"/>
|
||||
<path d="M5 10a7 7 0 0 0 14 0"/>
|
||||
<line x1="12" y1="19" x2="12" y2="22"/>
|
||||
<line x1="8" y1="22" x2="16" y2="22"/>
|
||||
</svg>
|
||||
<svg class="ind ind-error" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="9"/><line x1="12" y1="8" x2="12" y2="13"/><line x1="12" y1="16" x2="12" y2="16"/></svg>
|
||||
<span class="ind ind-spinner"></span>
|
||||
<span class="ind ind-thinking">
|
||||
<span class="dot"></span>
|
||||
<span class="dot"></span>
|
||||
<span class="dot"></span>
|
||||
</span>
|
||||
<span class="ind ind-bars">
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
<span class="bar"></span>
|
||||
</span>
|
||||
<svg class="ind ind-voice" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
|
||||
<path d="M3 10v4a1 1 0 0 0 1 1h3l5 4V5L7 9H4a1 1 0 0 0-1 1z" fill="currentColor" stroke="none"/>
|
||||
<path class="wave wave-1" d="M16 8a5 5 0 0 1 0 8"/>
|
||||
<path class="wave wave-2" d="M19 5a9 9 0 0 1 0 14"/>
|
||||
</svg>
|
||||
</span>
|
||||
</span>
|
||||
</button>
|
||||
|
||||
<button id="stop-btn" class="side-btn" type="button" aria-label="End" title="End" aria-hidden="true">
|
||||
<svg viewBox="0 0 24 24" fill="currentColor"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<p id="circle-caption" class="circle-caption" role="status">Tap to start</p>
|
||||
<p id="circle-subcaption" class="circle-subcaption" hidden></p>
|
||||
|
||||
<div id="queue-actions" class="queue-actions" hidden>
|
||||
<button id="join-queue-btn" class="join-queue-btn" type="button" hidden>
|
||||
Join now
|
||||
</button>
|
||||
<button id="leave-queue-btn" class="leave-queue-btn" type="button" hidden>
|
||||
Leave queue
|
||||
</button>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- Ephemeral chat bubbles anchored to the top-right -->
|
||||
<div id="bubble-stack" class="bubble-stack" aria-live="polite" aria-atomic="false"></div>
|
||||
|
||||
<!-- Webcam self-view, shown bottom-left while the camera tool is enabled -->
|
||||
<div id="cam-pip" class="cam-pip" aria-hidden="true">
|
||||
<video id="cam-video" class="cam-video" autoplay playsinline muted></video>
|
||||
<span class="cam-flash" aria-hidden="true"></span>
|
||||
<span class="cam-label">camera</span>
|
||||
</div>
|
||||
|
||||
<!-- Conversation history panel (overlay) -->
|
||||
<div id="chat-panel" class="chat-panel">
|
||||
<div id="chat-panel-backdrop" class="chat-panel-backdrop"></div>
|
||||
<div class="chat-panel-inner">
|
||||
<header class="chat-panel-header">
|
||||
<h3>Conversation</h3>
|
||||
<button id="chat-panel-close" class="icon-btn" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
<div id="chat-history" class="chat-history"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<dialog id="about-modal" class="modal about-modal">
|
||||
<div class="modal-content">
|
||||
<header class="modal-header">
|
||||
<h2>About</h2>
|
||||
<button id="about-close" class="icon-btn" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
|
||||
<!-- General intro to the speech-to-speech project -->
|
||||
<div class="about-intro">
|
||||
</div>
|
||||
|
||||
<!-- The backend, shown as the path a turn actually travels -->
|
||||
<div class="about-pipeline">
|
||||
<p class="pipeline-title">The pipeline</p>
|
||||
<ol class="pipeline">
|
||||
<li class="pipe-endpoint">You speak</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">VAD</span>
|
||||
<span class="pipe-job">detects speech</span>
|
||||
</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">STT</span>
|
||||
<span class="pipe-job">transcribes it</span>
|
||||
<a class="pipe-model" href="https://huggingface.co/nvidia/parakeet-tdt-1.1b" target="_blank" rel="noopener">nvidia/parakeet-tdt-1.1b<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">VLM</span>
|
||||
<span class="pipe-job">analyses and composes the reply <span class="pipe-note">· via <a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a></span></span>
|
||||
<a class="pipe-model" href="https://huggingface.co/google/gemma-4-31B-it" target="_blank" rel="noopener">google/gemma-4-31B-it<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-stage">
|
||||
<span class="pipe-tag">TTS</span>
|
||||
<span class="pipe-job">speaks it back</span>
|
||||
<a class="pipe-model" href="https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" target="_blank" rel="noopener">Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
|
||||
</li>
|
||||
<li class="pipe-endpoint">The orb replies</li>
|
||||
</ol>
|
||||
</div>
|
||||
</div>
|
||||
</dialog>
|
||||
|
||||
<dialog id="settings-modal" class="modal">
|
||||
<form method="dialog" class="modal-content">
|
||||
<header class="modal-header">
|
||||
<h2>Settings</h2>
|
||||
<button class="icon-btn" value="close" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
|
||||
<div class="tab-panels">
|
||||
<section class="tab-panel active" role="tabpanel">
|
||||
<label class="field" id="conn-field">
|
||||
<span id="conn-label">Speech-to-speech server URL</span>
|
||||
<input id="lb-url" type="text" autocomplete="off" spellcheck="false" placeholder="http://localhost:port" />
|
||||
<small id="conn-hint">
|
||||
URL of your speech-to-speech server, e.g.
|
||||
<code>http://localhost:8080</code> (the app adds <code>/v1/realtime</code>).
|
||||
</small>
|
||||
</label>
|
||||
|
||||
<div class="field-row">
|
||||
<label class="field">
|
||||
<span>Voice</span>
|
||||
<select id="voice">
|
||||
<option value="Aiden" selected>Aiden</option>
|
||||
<option value="Ryan">Ryan</option>
|
||||
<option value="Dylan">Dylan</option>
|
||||
<option value="Eric">Eric</option>
|
||||
<option value="Ono_Anna">Ono_Anna</option>
|
||||
<option value="Serena">Serena</option>
|
||||
<option value="Sohee">Sohee</option>
|
||||
<option value="Uncle_Fu">Uncle_Fu</option>
|
||||
<option value="Vivian">Vivian</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<span class="field-head">
|
||||
Noise gate
|
||||
<span id="gate-value" class="field-value">Off</span>
|
||||
</span>
|
||||
<div class="gate">
|
||||
<div class="gate-track" aria-hidden="true">
|
||||
<div id="gate-meter-fill" class="gate-meter-fill"></div>
|
||||
<input
|
||||
id="noise-gate"
|
||||
type="range"
|
||||
min="-66"
|
||||
max="-3"
|
||||
step="1"
|
||||
value="-50"
|
||||
aria-label="Noise gate threshold"
|
||||
/>
|
||||
</div>
|
||||
<div class="gate-ends">
|
||||
<span>Off</span>
|
||||
<span>−3 dB</span>
|
||||
</div>
|
||||
</div>
|
||||
<small>
|
||||
Mutes the mic below the handle so room noise isn't sent. Slide fully
|
||||
left to turn it off; the bar shows your live input while in a call.
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<label class="field">
|
||||
<span>Instructions</span>
|
||||
<textarea id="instructions" rows="5" placeholder="You are Vocero, a friendly voice assistant for the company Aquachile..."></textarea>
|
||||
<small id="primary-prompt-meta" hidden></small>
|
||||
</label>
|
||||
|
||||
<div class="field">
|
||||
<button id="restart-conversation" type="button" class="btn primary wide" disabled>
|
||||
Restart conversation with these settings
|
||||
</button>
|
||||
<small id="restart-hint">Connect first, then come back here to apply live changes.</small>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<footer class="modal-footer">
|
||||
<button id="settings-save" type="submit" class="btn primary" value="save">Save</button>
|
||||
</footer>
|
||||
</form>
|
||||
</dialog>
|
||||
|
||||
<dialog id="tools-modal" class="modal">
|
||||
<div class="modal-content">
|
||||
<header class="modal-header">
|
||||
<h2>Tools</h2>
|
||||
<button id="tools-close" class="icon-btn" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
</header>
|
||||
|
||||
<p class="tools-intro">Let the assistant act during the conversation. Changes apply live.</p>
|
||||
|
||||
<div class="tool-list">
|
||||
<div class="tool-row" id="tool-web-row">
|
||||
<div class="tool-info">
|
||||
<span class="tool-name">Web search</span>
|
||||
<span class="tool-desc">Look things up on Google, via Serper.</span>
|
||||
</div>
|
||||
<label class="switch">
|
||||
<input id="tool-web" type="checkbox" />
|
||||
<span class="switch-track" aria-hidden="true"></span>
|
||||
</label>
|
||||
</div>
|
||||
<label class="field tools-key">
|
||||
<span>Search API key</span>
|
||||
<input id="search-key" type="password" autocomplete="off" spellcheck="false" />
|
||||
<small id="tool-web-hint"></small>
|
||||
</label>
|
||||
|
||||
<div class="tool-row tool-row-sep" id="tool-cam-row">
|
||||
<div class="tool-info">
|
||||
<span class="tool-name">Camera</span>
|
||||
<span class="tool-desc">Let the assistant see through your webcam.</span>
|
||||
</div>
|
||||
<label class="switch">
|
||||
<input id="tool-cam" type="checkbox" />
|
||||
<span class="switch-track" aria-hidden="true"></span>
|
||||
</label>
|
||||
</div>
|
||||
<small id="tool-cam-hint" class="tool-hint">A live preview shows bottom-left while the camera is on.</small>
|
||||
|
||||
<div class="tool-row tool-row-sep">
|
||||
<div class="tool-info">
|
||||
<span class="tool-name">Loaded MCP tools</span>
|
||||
<span class="tool-desc">Tools published by the remote agent profile.</span>
|
||||
</div>
|
||||
</div>
|
||||
<small id="agent-tools-hint" class="tool-hint"></small>
|
||||
<div id="agent-tools-list" class="agent-tools-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
</dialog>
|
||||
|
||||
<!-- Shown when the per-day conversation budget is spent (mid-call or at
|
||||
start). Title / message / CTA / note are filled per tier by
|
||||
ui/account.js. The hero is the HF mark — a friendly smiling face. -->
|
||||
<dialog id="limit-modal" class="modal limit-modal">
|
||||
<div class="modal-content limit-card">
|
||||
<button id="limit-close" class="icon-btn limit-close" aria-label="Close">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</button>
|
||||
<div class="limit-badge" aria-hidden="true">
|
||||
<svg class="hf-logo" viewBox="0 0 95 88" fill="none" aria-hidden="true"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg>
|
||||
</div>
|
||||
<h2 id="limit-title" class="limit-title">That's a wrap for now</h2>
|
||||
<p id="limit-msg" class="limit-msg"></p>
|
||||
<a id="limit-cta" class="btn primary wide limit-cta" href="#"></a>
|
||||
<p id="limit-note" class="limit-note"></p>
|
||||
</div>
|
||||
</dialog>
|
||||
|
||||
<script type="module" src="main.js?v=20260825003"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,139 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
|
||||
<title>Project Docs</title>
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #081018;
|
||||
--panel: #0f1b26;
|
||||
--line: rgba(149, 180, 201, 0.18);
|
||||
--text: #edf6fb;
|
||||
--muted: #9db2c2;
|
||||
--accent: #59d0ff;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html,
|
||||
body {
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font-family: Inter, Arial, Helvetica, sans-serif;
|
||||
}
|
||||
|
||||
body {
|
||||
display: grid;
|
||||
grid-template-rows: auto 1fr;
|
||||
}
|
||||
|
||||
.topbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 16px;
|
||||
padding: 14px 18px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
background: rgba(8, 16, 24, 0.92);
|
||||
backdrop-filter: blur(10px);
|
||||
}
|
||||
|
||||
.title {
|
||||
margin: 0;
|
||||
font-size: 16px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.02em;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
margin: 4px 0 0;
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.btn {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
min-height: 38px;
|
||||
padding: 0 14px;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 10px;
|
||||
color: var(--text);
|
||||
background: var(--panel);
|
||||
text-decoration: none;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.btn.primary {
|
||||
border-color: rgba(89, 208, 255, 0.35);
|
||||
color: #021018;
|
||||
background: var(--accent);
|
||||
}
|
||||
|
||||
.frame-wrap {
|
||||
padding: 14px;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.frame {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 16px;
|
||||
background: #ffffff;
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.topbar {
|
||||
flex-direction: column;
|
||||
align-items: flex-start;
|
||||
}
|
||||
|
||||
.actions {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.btn {
|
||||
flex: 1 1 auto;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<div>
|
||||
<h1 class="title">Vocero Project</h1>
|
||||
<p class="subtitle">Embedded technical document viewer for `demo/public/docs.html`.</p>
|
||||
</div>
|
||||
<div class="actions">
|
||||
<a class="btn" href="/">Back to app</a>
|
||||
<a class="btn primary" href="/public/docs.html" target="_blank" rel="noopener">Open raw document</a>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main class="frame-wrap">
|
||||
<iframe
|
||||
class="frame"
|
||||
src="/public/docs.html"
|
||||
title="Project documentation"
|
||||
loading="eager"
|
||||
referrerpolicy="no-referrer"
|
||||
></iframe>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
+14
-1
@@ -54,7 +54,7 @@ import httpx
|
||||
import websockets
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.responses import JSONResponse, RedirectResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
|
||||
@@ -6696,6 +6696,19 @@ async def no_cache_static_assets(request: Request, call_next):
|
||||
return response
|
||||
|
||||
|
||||
# Preserve the former proposal URLs after moving the document out of the client.
|
||||
@app.get("/project", include_in_schema=False)
|
||||
@app.get("/project/", include_in_schema=False)
|
||||
@app.get("/project/index.html", include_in_schema=False)
|
||||
@app.get("/public/docs.html", include_in_schema=False)
|
||||
async def project_documentation():
|
||||
return RedirectResponse("/docs/proposals/vocero-architecture.html", status_code=307)
|
||||
|
||||
|
||||
# The existing FastAPI /docs page remains registered ahead of this mount.
|
||||
app.mount("/docs", StaticFiles(directory=os.path.join(ROOT_DIR, "docs"), html=True), name="documentation")
|
||||
|
||||
|
||||
# Static front-end. Registered last so the /api routes win. `html=True` serves
|
||||
# index.html at "/". The repo is public anyway, so serving the dir is fine.
|
||||
app.mount("/", StaticFiles(directory=HERE, html=True), name="static")
|
||||
|
||||
Reference in New Issue
Block a user