valenti/vocero-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: valenti/vocero-client:e529435980e3db91ebc710fc232333995cc4939c
Branches Tags
valenti/vocero-client:main
...
compare: valenti/vocero-client:cdbff5cb0ecb116979e1ebf49bf275fa3f529662
Branches Tags
valenti/vocero-client:main
5 Commits
e529435980 ... cdbff5cb0e
Author SHA1 Message Date
alvaro cdbff5cb0e fix: standardize Spanish user-facing copy 2026-10-05 10:24:33 -03:00
alvaro dfc1f663c7 refactor: standardize developer text and identifiers in English 2026-10-05 10:13:25 -03:00
alvaro 6b8c27fd76 docs: consolidate English documentation under root docs 2026-10-05 10:11:35 -03:00
alvaro b91762a927 chore: remove debug markdown files 2026-10-05 09:35:49 -03:00
alvaro e0b64dfb99 chore: remove old directory 2026-10-05 09:35:49 -03:00
62 changed files with 3396 additions and 7455 deletions
Expand all files Collapse all files
.dockerignore
+19
View File
@@ -0,0 +1,19 @@
.git/
.agents/
.codex/
.aws/
**/.env
**/.env.*
!**/.env.example
**/secrets/
**/__pycache__/
**/*.pyc
**/.venv/
**/venv/
**/node_modules/
**/.node-red/
**/.cache/
**/*.log
**/uploads/
**/sessions/
debug_files/
.gitignore
+1
View File
@@ -139,3 +139,4 @@ speech_to_speech/*.log
speech_to_speech/sessions/
speech_to_speech/.cache/
s2s-reasoning-engine/*.log
debug_files
README.md
+50
View File
@@ -0,0 +1,50 @@
# Vocero speech-to-speech
Vocero provides a browser voice interface, a Python speech pipeline, and an
optional Node-RED reasoning service. The interface defaults to Spanish;
documentation, source identifiers, comments, and internal instructions use English.
| Component | Purpose |
|---|---|
| [s2s-client](s2s-client) | Browser UI, FastAPI server, realtime WebSocket proxy, search and ApiPilot bridges |
| [speech_to_speech](speech_to_speech) | Queue-driven VAD, speech recognition, language model, and speech synthesis pipeline |
| [s2s-reasoning-engine](s2s-reasoning-engine) | Optional Node-RED HTTP reasoning and conversation logging flows |
Start with [client setup and usage](docs/client/usage.md), then read the
[architecture](docs/client/architecture.md) and [documentation index](docs/README.md).
The speech backend needs its own installed model and runtime dependencies;
`s2s-client/requirements.txt` installs only the UI server dependencies.
## Run the browser client
From the repository root, point the client at an existing realtime backend:
```bash
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
```
Open <http://localhost:8420/> and allow microphone access. The launcher creates
`.venv`, installs missing UI requirements, loads the root `.env` if present, and
starts the server. For the local WebSocket proxy instead of a direct browser
connection, configure `S2S_PROXY_TARGET` and leave `SPEECH_TO_SPEECH_URL` unset.
To build the UI container, use the repository root as the Docker build context so
the root documentation is included:
```bash
docker build -f s2s-client/Dockerfile -t s2s-client .
docker run --rm -p 7860:7860 \
-e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client
```
The direct WebSocket URL is resolved by the user's browser. For a server-side
proxy deployment, supply a backend address reachable from the container instead.
## Project references
- [Change history](s2s-client/CHANGELOG.md)
- [Realtime protocol and pipeline](docs/speech/realtime-api.md)
- [Context injection](docs/client/context-injection.md)
- [Tool calling](docs/client/tool-calling.md)
- [Troubleshooting](docs/client/troubleshooting.md)
- [Historical architecture proposal](docs/proposals/vocero-architecture.html)
_old_to_del/index copy.html
-401
View File
@@ -1,401 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate, max-age=0" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
<title>Minimal Conversation · S2S backend (WebSocket)</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Geist+Mono:wght@400;500;600&family=Inter:wght@400;500;600;700;800&display=swap"
rel="stylesheet"
/>
<link rel="stylesheet" href="style.css?v=20260902001" />
</head>
<!--
`booting` disables transitions/animations until the first paint commits.
Without this the orb visibly fades/scales in on the very first frame
because `state-idle` and the generic `.ind` defaults differ. The
bootstrap script at the end of main.js strips the class after one rAF.
-->
<body class="booting">
<div id="app">
<header class="topbar">
<div class="brand">
<div class="ident">
<div class="ident-head">
<a class="ident-title">Azurian Vocero realtime ver.Beta 2.0</a>
</div>
<div id="ident-user-name" class="ident-user-name" hidden></div>
<div id="reports-hint" class="reports-hint" hidden></div>
</div>
</div>
<div class="topbar-right">
<!-- HF login chip / sign-in pill. Populated by ui/account.js; only
shown when the deploy runs behind a load balancer with OAuth. -->
<div id="account" class="account" hidden></div>
<button id="about-btn-m" class="icon-btn about-btn-mobile" title="About this space" aria-label="About this space">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" y1="16" x2="12" y2="12"/><line x1="12" y1="8" x2="12.01" y2="8"/></svg>
</button>
<button id="tools-btn" class="icon-btn" title="Tools" aria-label="Tools">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/></svg>
</button>
<button id="rag-btn" class="icon-btn" title="Custom RAG context" aria-label="Custom RAG context">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"/><path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2z"/></svg>
</button>
<button id="chat-btn" class="icon-btn" title="Conversation history" aria-label="Conversation history">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15a2 2 0 0 1-2 2H7l-4 4V5a2 2 0 0 1 2-2h14a2 2 0 0 1 2 2z"/></svg>
<span id="chat-badge" class="chat-badge" aria-hidden="true"></span>
</button>
<button id="settings-btn" class="icon-btn" title="Settings" aria-label="Settings">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 1 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 1 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 1 1-2.83-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 1 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 1 1 2.83-2.83l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 1 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 1 1 2.83 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9c0 .66.39 1.25 1 1.51H21a2 2 0 1 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"/></svg>
</button>
</div>
</header>
<main class="stage">
<div class="orb-wrap">
<div id="mic-gate" class="mic-gate">
<svg id="mic-gate-arc" class="mic-gate-arc" viewBox="0 0 100 100" aria-hidden="true">
<path id="mga-track" class="mga-track" fill="none" />
<path id="mga-fill" class="mga-fill" fill="none" />
<path id="mga-hit" class="mga-hit" fill="none" />
<circle id="mga-handle" class="mga-handle" r="3" />
</svg>
<button id="mic-btn" class="side-btn" type="button" aria-label="Mute" title="Mute" aria-hidden="true">
<svg class="mic-on" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="2" width="6" height="12" rx="3"/><path d="M5 10a7 7 0 0 0 14 0"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
<svg class="mic-off" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><line x1="2" y1="2" x2="22" y2="22"/><path d="M9 5a3 3 0 0 1 6 0v4"/><path d="M9 10v1a3 3 0 0 0 5.1 2.1"/><path d="M19 10a7 7 0 0 1-1.24 3.97"/><path d="M5 10a7 7 0 0 0 11 5.67"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
</button>
</div>
<button
id="main-circle"
class="circle state-idle"
type="button"
aria-label="Start voice conversation"
>
<span class="circle-glow" aria-hidden="true"></span>
<span class="circle-ring" aria-hidden="true"></span>
<span class="circle-ring-outer" aria-hidden="true"></span>
<span class="circle-core">
<span class="circle-indicator" aria-hidden="true">
<svg class="ind ind-mic" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<rect x="9" y="2" width="6" height="12" rx="3" fill="currentColor" stroke="none"/>
<path d="M5 10a7 7 0 0 0 14 0"/>
<line x1="12" y1="19" x2="12" y2="22"/>
<line x1="8" y1="22" x2="16" y2="22"/>
</svg>
<svg class="ind ind-error" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="9"/><line x1="12" y1="8" x2="12" y2="13"/><line x1="12" y1="16" x2="12" y2="16"/></svg>
<span class="ind ind-spinner"></span>
<span class="ind ind-thinking">
<span class="dot"></span>
<span class="dot"></span>
<span class="dot"></span>
</span>
<span class="ind ind-bars">
<span class="bar"></span>
<span class="bar"></span>
<span class="bar"></span>
<span class="bar"></span>
<span class="bar"></span>
</span>
<svg class="ind ind-voice" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="M3 10v4a1 1 0 0 0 1 1h3l5 4V5L7 9H4a1 1 0 0 0-1 1z" fill="currentColor" stroke="none"/>
<path class="wave wave-1" d="M16 8a5 5 0 0 1 0 8"/>
<path class="wave wave-2" d="M19 5a9 9 0 0 1 0 14"/>
</svg>
</span>
</span>
</button>
<button id="stop-btn" class="side-btn" type="button" aria-label="End" title="End" aria-hidden="true">
<svg viewBox="0 0 24 24" fill="currentColor"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>
</button>
</div>
<p id="circle-caption" class="circle-caption" role="status">Tap to start</p>
<p id="circle-subcaption" class="circle-subcaption" hidden></p>
<div id="queue-actions" class="queue-actions" hidden>
<button id="join-queue-btn" class="join-queue-btn" type="button" hidden>
Join now
</button>
<button id="leave-queue-btn" class="leave-queue-btn" type="button" hidden>
Leave queue
</button>
</div>
</main>
</div>
<!-- Ephemeral chat bubbles anchored to the top-right -->
<div id="bubble-stack" class="bubble-stack" aria-live="polite" aria-atomic="false"></div>
<!-- Webcam self-view, shown bottom-left while the camera tool is enabled.
HIDDEN BY DEFAULT TODAY: riattivare togliendo hidden + scommentando listener in main.js -->
<div id="cam-pip" class="cam-pip" aria-hidden="true" hidden>
<video id="cam-video" class="cam-video" autoplay playsinline muted></video>
<span class="cam-flash" aria-hidden="true"></span>
<span class="cam-label">camera</span>
</div>
<!-- Conversation history panel (overlay) -->
<div id="chat-panel" class="chat-panel">
<div id="chat-panel-backdrop" class="chat-panel-backdrop"></div>
<div class="chat-panel-inner">
<header class="chat-panel-header">
<h3>Conversation</h3>
<button id="chat-panel-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<div id="chat-history" class="chat-history"></div>
</div>
</div>
<dialog id="about-modal" class="modal about-modal">
<div class="modal-content">
<header class="modal-header">
<h2>About</h2>
<button id="about-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<!-- General intro to the speech-to-speech project -->
<div class="about-intro">
</div>
<!-- The backend, shown as the path a turn actually travels -->
<div class="about-pipeline">
<p class="pipeline-title">The pipeline</p>
<ol class="pipeline">
<li class="pipe-endpoint">You speak</li>
<li class="pipe-stage">
<span class="pipe-tag">VAD</span>
<span class="pipe-job">detects speech</span>
</li>
<li class="pipe-stage">
<span class="pipe-tag">STT</span>
<span class="pipe-job">transcribes it</span>
<a class="pipe-model" href="https://huggingface.co/nvidia/parakeet-tdt-1.1b" target="_blank" rel="noopener">nvidia/parakeet-tdt-1.1b<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-stage">
<span class="pipe-tag">VLM</span>
<span class="pipe-job">analyses and composes the reply <span class="pipe-note">· via <a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a></span></span>
<a class="pipe-model" href="https://huggingface.co/google/gemma-4-31B-it" target="_blank" rel="noopener">google/gemma-4-31B-it<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-stage">
<span class="pipe-tag">TTS</span>
<span class="pipe-job">speaks it back</span>
<a class="pipe-model" href="https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" target="_blank" rel="noopener">Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-endpoint">The orb replies</li>
</ol>
</div>
</div>
</dialog>
<dialog id="settings-modal" class="modal">
<form method="dialog" class="modal-content">
<header class="modal-header">
<h2>Settings</h2>
<button class="icon-btn" value="close" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<div class="tab-panels">
<section class="tab-panel active" role="tabpanel">
<label class="field" id="conn-field">
<span id="conn-label">Speech-to-speech server URL</span>
<input id="lb-url" type="text" autocomplete="off" spellcheck="false" placeholder="http://localhost:port" />
<small id="conn-hint">
URL of your speech-to-speech server, e.g.
<code>http://localhost:8080</code> (the app adds <code>/v1/realtime</code>).
</small>
</label>
<div class="field-row">
<label class="field">
<span>Voice</span>
<select id="voice">
<option value="Aiden" selected>Aiden</option>
<option value="Ryan">Ryan</option>
<option value="Dylan">Dylan</option>
<option value="Eric">Eric</option>
<option value="Ono_Anna">Ono_Anna</option>
<option value="Serena">Serena</option>
<option value="Sohee">Sohee</option>
<option value="Uncle_Fu">Uncle_Fu</option>
<option value="Vivian">Vivian</option>
</select>
</label>
</div>
<div class="field">
<span class="field-head">
Noise gate
<span id="gate-value" class="field-value">Off</span>
</span>
<div class="gate">
<div class="gate-track" aria-hidden="true">
<div id="gate-meter-fill" class="gate-meter-fill"></div>
<input
id="noise-gate"
type="range"
min="-66"
max="-3"
step="1"
value="-50"
aria-label="Noise gate threshold"
/>
</div>
<div class="gate-ends">
<span>Off</span>
<span>−3 dB</span>
</div>
</div>
<small>
Mutes the mic below the handle so room noise isn't sent. Slide fully
left to turn it off; the bar shows your live input while in a call.
</small>
</div>
<label class="field">
<span>Instructions</span>
<textarea id="instructions" rows="5" placeholder="You are Vocero, a friendly voice assistant for the company Aquachile..."></textarea>
<small id="primary-prompt-meta" hidden></small>
</label>
<div class="field">
<button id="restart-conversation" type="button" class="btn primary wide" disabled>
Restart conversation with these settings
</button>
<small id="restart-hint">Connect first, then come back here to apply live changes.</small>
</div>
</section>
</div>
<footer class="modal-footer">
<button id="settings-save" type="submit" class="btn primary" value="save">Save</button>
</footer>
</form>
</dialog>
<dialog id="tools-modal" class="modal">
<div class="modal-content">
<header class="modal-header">
<h2>Tools</h2>
<button id="tools-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<p class="tools-intro">Let the assistant act during the conversation. Changes apply live.</p>
<div class="tool-list">
<div class="tool-row" id="tool-web-row">
<div class="tool-info">
<span class="tool-name">Web search</span>
<span class="tool-desc">Look things up on Google, via Serper.</span>
</div>
<label class="switch">
<input id="tool-web" type="checkbox" />
<span class="switch-track" aria-hidden="true"></span>
</label>
</div>
<label class="field tools-key">
<span>Search API key</span>
<input id="search-key" type="password" autocomplete="off" spellcheck="false" />
<small id="tool-web-hint"></small>
</label>
<!-- Camera tool row — HIDDEN BY DEFAULT TODAY.
Per riattivarlo in futuro: togli hidden da div + small,
scommenta listener toolCamSwitch + funzioni enableCamera in main.js. -->
<div class="tool-row tool-row-sep" id="tool-cam-row" hidden>
<div class="tool-info">
<span class="tool-name">Camera</span>
<span class="tool-desc">Let the assistant see through your webcam.</span>
</div>
<label class="switch">
<input id="tool-cam" type="checkbox" />
<span class="switch-track" aria-hidden="true"></span>
</label>
</div>
<small id="tool-cam-hint" class="tool-hint" hidden>A live preview shows bottom-left while the camera is on.</small>
<div class="tool-row tool-row-sep">
<div class="tool-info">
<span class="tool-name">Loaded MCP tools</span>
<span class="tool-desc">Tools published by the remote agent profile.</span>
</div>
</div>
<small id="agent-tools-hint" class="tool-hint"></small>
<div id="agent-tools-list" class="agent-tools-list"></div>
</div>
</div>
</dialog>
<dialog id="rag-modal" class="modal">
<div class="modal-content">
<header class="modal-header">
<h2>Custom RAG context</h2>
<button id="rag-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<p class="tools-intro">
Incolla qui testo, documenti, o estratti RAG che vuoi siano disponibili
per la sessione vocale corrente. Il testo viene aggiunto sia alle
istruzioni del LLM sia al campo <code>user_context</code> del body
<code>/api/reasoning</code>.
</p>
<label class="field">
<span>RAG context (testo libero, puro)</span>
<textarea
id="rag-textarea"
rows="12"
spellcheck="false"
placeholder="Incolla o scrivi qui il contesto RAG che il modello deve usare durante la risposta. Es: dati cliente, policy, estratti documentali, ecc."
></textarea>
<small id="rag-hint" class="tool-hint">
Nessun RAG context caricato. Premi &quot;Invia nella sessione&quot; per applicare le modifiche.
</small>
</label>
<footer class="modal-footer">
<button id="rag-clear" class="btn secondary" type="button" value="clear">Svuota</button>
<div style="flex: 1"></div>
<button id="rag-submit" class="btn primary" type="button" value="submit">Invia nella sessione</button>
</footer>
</div>
</dialog>
<!-- Shown when the per-day conversation budget is spent (mid-call or at
start). Title / message / CTA / note are filled per tier by
ui/account.js. The hero is the HF mark — a friendly smiling face. -->
<dialog id="limit-modal" class="modal limit-modal">
<div class="modal-content limit-card">
<button id="limit-close" class="icon-btn limit-close" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
<div class="limit-badge" aria-hidden="true">
<svg class="hf-logo" viewBox="0 0 95 88" fill="none" aria-hidden="true"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg>
</div>
<h2 id="limit-title" class="limit-title">That's a wrap for now</h2>
<p id="limit-msg" class="limit-msg"></p>
<a id="limit-cta" class="btn primary wide limit-cta" href="#"></a>
<p id="limit-note" class="limit-note"></p>
</div>
</dialog>
<script type="module" src="main.js?v=20260902001"></script>
</body>
</html>
debug-mcp-audio-stutter.md
-45
View File
@@ -1,45 +0,0 @@
# Debug Session: mcp-audio-stutter
- **Status**: [OPEN]
- **Issue**: Piccole interruzioni audio quando una risposta MCP lunga viene eseguita o reiniettata nella conversazione live.
## Reproduction Steps
1. Avvia la UI realtime.
2. Esegui una richiesta che attiva un tool MCP con output grande.
3. Ascolta la risposta audio dell'assistente.
4. Verifica se compaiono micro-interruzioni durante o subito dopo l'iniezione del risultato MCP.
## Hypotheses & Verification
| ID | Hypothesis | Likelihood | Effort | Evidence |
|----|------------|------------|--------|----------|
| A | Inviare al modello il payload MCP quasi completo degrada la fluidità della risposta audio anche senza bloccare il main thread. | High | Low | Confirmed |
| B | Esiste una race: il fallback MCP decide il turno prima che `response.created` arrivi, quindi la prima response speculativa parte comunque e viene tagliata. | High | Medium | Confirmed |
| C | La chat/UI fa render pesanti quando arriva il risultato MCP grande. | Medium | Medium | Rejected |
| D | I log/debug del payload MCP grande aggravano il problema sul main thread. | Medium | Low | Rejected |
| E | L'attacco audio di ogni frase soffre di jitter iniziale perché il playback parte senza prebuffer sul primo chunk disponibile. | High | Medium | Confirmed |
## Log Evidence
- `execRemoteMcpTool` riceve un result JSON di `5738` byte e produce un output testo di `3262` caratteri senza tempi anomali.
- `built live prompt from MCP output` mostra `promptLength` ~`3400` e `buildMs: 0`.
- `rendered tool result row` e `rendered MCP tool result in chat` mostrano `renderMs: 0-1`.
- `sending conversation.item.create payload` mostra `payloadLength` ~`3790` e `stringifyMs: 0`.
- Quindi serializzazione, render UI e stringify non spiegano i micro-tagli.
- Nel run post-fix `sendUserText` mostra `responseActive: false`, `openResponses: 0`, `createInFlight: false`: non è un classico barge-in su response già attiva.
- Il sintomo residuo si manifesta solo sulla prima frase speculativa; questo è coerente con una race prima di `response.created`.
- Il browser può decidere il fallback MCP prima che il client abbia un `responseId` da silenziare: per questo ora viene marcata anche la **next response** come sopprimibile.
- L'ultimo sintomo riferito dall'utente cambia ancora: le micro-interruzioni si sentono sulle prime parole di ogni frase, non solo su quella speculativa.
- Questo è coerente con il worklet `audio-playback` che parte subito sul primo chunk ricevuto, senza accumulare buffer iniziale.
## Minimal Fix
- Accodata la `sendUserText()` del payload MCP quando esiste una response attiva.
- Flush del testo differito solo dopo `response.done`, prima del `response.create` già in coda.
- Riassunto/troncamento del payload MCP prima di reiniettarlo nella conversazione live.
- Ridotto il delay del fallback MCP a `150ms` per match forti e senza argomenti obbligatori.
- Soppressione della **next speculative response** se il fallback MCP vince il turno prima che `response.created` sia arrivato.
- Introdotto un piccolo prebuffer di avvio (`90ms`) nel worklet `audio-playback` prima di iniziare ogni frase.
- Obiettivo: evitare sia il barge-in quando presente, sia il carico eccessivo sul modello con JSON troppo grande.
- Obiettivo aggiuntivo: ridurre la finestra in cui il modello può iniziare una frase speculativa prima che parta il tool MCP.
- Obiettivo finale: evitare il clipping/jitter sulle prime parole di ogni frase.
## Verification Conclusion
- Pre-fix: il browser non mostrava alcun collo di bottiglia locale; il prompt inviato al modello restava però molto grande.
- Post-fix: il testo MCP viene accodato se c'è una risposta attiva, il modello riceve solo un summary compatto, il fallback parte quasi subito, la next response speculativa viene silenziata se necessario e il playback aspetta un buffer iniziale prima di iniziare ogni frase.
debug-mcp-tool-not-called.md
-75
View File
@@ -1,75 +0,0 @@
# Debug Session: mcp-tool-not-called
Status: [OPEN]
## Symptom
- The voice assistant says it will call an MCP tool, but no MCP tool call is observed.
- No useful runtime log or call attempt is visible from the current instrumentation.
## Expected
- When the model decides to use an MCP tool, the client should emit a tool-call event, execute the MCP proxy request, and produce visible debug evidence for each stage.
## Hypotheses
1. The realtime model is not actually emitting `response.function_call_arguments.done`, so the spoken promise to use a tool is plain text, not a real function call.
2. The MCP tools are not present in the effective session state at the moment of the request, so the model cannot legally call them.
3. The session instructions are updated too late or with the wrong content, so the model receives guidance text but not usable tool declarations.
4. The browser receives a tool-call event, but the handler path fails before logging or before `execRemoteMcpTool()` starts.
5. The MCP proxy request starts, but the current logs do not surface pre-call or failure details, making the attempt invisible.
## Evidence Plan
- Instrument session bootstrap, tool declaration updates, transcript-based MCP ranking, tool-call event handling, and MCP proxy fetch lifecycle.
- Reproduce with a prompt that should match one MCP tool by description.
- Compare whether the model emits only assistant transcript text or a real tool-call event.
## Notes
- First code change in this session must remain instrumentation only.
## Evidence Summary
- `api/config` loads correctly with `searchMcpEnabled=true`, `agentToolCount=16`, and a non-empty primary prompt.
- Transcript-time MCP ranking is active and produces ranked candidates such as `get_api_info`, `get_report_detail`, and `get_public_day_report` for relevant spoken requests.
- No `response.function_call_arguments.done` event was recorded in the current reproduction.
- No `toolcall` event, no `runTool()` start, and no MCP proxy fetch were recorded in the current reproduction.
## Hypothesis Status
| ID | Hypothesis | Status | Evidence |
|----|------------|--------|----------|
| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | The log file contains transcript/ranking events only; no `function_call_arguments.done` entry was produced in the reproduced turn. |
| B | MCP tools are missing from runtime state | PARTIAL | `api/config` proves the remote profile exposes 16 tools, but the current run lacks direct evidence from live `session.update`. |
| C | MCP guidance reaches the session too late for the same voice turn | LIKELY | Ranking is computed from the live user transcript; current code only pushes updated instructions on the final transcript, which can miss the active turn. |
| D | The browser gets the toolcall event but fails before execution | REJECTED (current run) | There is no `toolcall` event at all in the current reproduction. |
| E | The MCP proxy request starts but is invisible | REJECTED (current run) | No pre-fetch or post-fetch MCP proxy log was emitted. |
## Minimal Fix
- Push the computed MCP-guidance instructions to the live session as soon as ranked tool suggestions appear on partial user transcripts, instead of waiting only for the final transcript.
- Track the last live instruction snapshot to avoid redundant `session.update` spam.
- Add client-construction instrumentation so the next run proves which tools/instructions are present at session start.
## Post-Fix Evidence
- The live session is created with 17 tools and non-empty instructions.
- During matching turns, the client now narrows the tool set down to the strongest MCP candidates and sends `tool_choice="required"`.
- Example evidence from the latest run:
- `get_api_info` is selected as the top candidate with strong score (`29`).
- The client sends `setTools(..., "required")` with `["get_api_info", ...]`.
- The client sends updated instructions immediately after.
- Even with `tool_choice="required"` and a narrowed tool list, no `response.function_call_arguments.done` event is emitted.
- Therefore no browser `toolcall`, no `runTool()`, and no MCP proxy request can happen.
## Updated Hypothesis Status
| ID | Hypothesis | Status | Evidence |
|----|------------|--------|----------|
| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | Even after forcing `tool_choice="required"` and narrowing the tool list, no `function_call_arguments.done` event appears. |
| B | MCP tools are missing from runtime state | REJECTED | The client sends 17 initial tools and later sends narrowed MCP-only tool lists, including `tool_choice="required"`. |
| C | MCP guidance reaches the session too late for the same voice turn | REJECTED | The client now pushes guidance and tool policy during partial transcripts, before the turn finishes. |
| D | The browser gets the toolcall event but fails before execution | REJECTED | No upstream function-call event is emitted at all. |
| E | The MCP proxy request starts but is invisible | REJECTED | The MCP proxy is never reached because the model never initiates a tool call. |
## Current Root Cause
- The speech-to-speech realtime model path in the current backend stack is not performing actual function calling for these MCP tools in this conversation flow, even when tools are declared and `tool_choice="required"` is sent.
- The spoken assistant text is therefore behaving like a promise in natural language, not a real tool invocation.
## Backend Branch B
- Added relay-side instrumentation in `demo/server.py` to report:
- Outbound client text events forwarded to upstream, including `session.update` summaries with `toolCount`, `toolNames`, `toolChoice`, and instruction length.
- Inbound upstream text events forwarded back to the client, including `session.updated`, `response.function_call_arguments.done`, and `error` summaries when present.
- Binary frame forwarding in both directions and client disconnects.
- No new runtime evidence has been collected from this instrumentation yet; a fresh reproduction is required with the patched relay active.
docs/README.md
+35
View File
@@ -0,0 +1,35 @@
# Documentation
All maintained documentation is in English. Spanish examples represent actual
user input or spoken output. Historical proposals are identified explicitly and
do not define the current implementation.
## Browser client
- [Usage and configuration](client/usage.md)
- [Architecture and source map](client/architecture.md)
- [Reasoning modes](client/reasoning.md)
- [Context injection](client/context-injection.md)
- [Tool calling](client/tool-calling.md)
- [Design language](client/design.md)
- [Glossary](client/glossary.md)
- [Troubleshooting](client/troubleshooting.md)
- [Change history](../s2s-client/CHANGELOG.md)
## Speech backend
- [Language model backends](speech/llm.md)
- [Speech-to-text backends](speech/stt.md)
- [Text-to-speech backends](speech/tts.md)
- [Realtime API, tools, and interruption handling](speech/realtime-api.md)
## Decisions, proposals, and examples
- [Example files and usage](examples/README.md)
- [ADR 0001: Docker hosting and search proxy](adr/0001-docker-space-with-search-proxy.md)
- [Historical Vocero architecture proposal](proposals/vocero-architecture.html)
- [Agent-profile example](examples/agent-profile.json): an illustrative profile
with agent instructions and tool definitions; this is not the current
`/api/config` response or an automatically loaded runtime configuration.
- [Reference audio](examples/reference-audio.wav): an optional voice-cloning
example described in the [TTS guide](speech/tts.md).
s2s-client/docs/adr/0001-docker-space-with-search-proxy.md → docs/adr/0001-docker-space-with-search-proxy.md
+6 -2
View File
@@ -1,11 +1,15 @@
# Convert the static Space into a Docker app with a search proxy
Status: accepted. This records the original hosting decision; the current route
is `/api/search`, and the current container also includes the root documentation.
See [client usage](../client/usage.md) for the root-context Docker build.
To give the model a web search tool, the executor (which runs in the browser of a
public Space) needs a search key. A `sdk: static` Space serves files as-is with no
runtime process, so it cannot hold a secret the browser uses without exposing it.
We convert the Space from `sdk: static` to `sdk: docker`: a single container runs a
small server (FastAPI + uvicorn) that both serves the existing front-end *unchanged*
and exposes a same-origin `/search` proxy holding `SERPER_API_KEY` server-side. The
and exposes a same-origin `/api/search` proxy holding `SERPER_API_KEY` server-side. The
whole app lives in that one container; the s2s speech-to-speech backend stays the
separate load-balanced service it already is.
@@ -23,7 +27,7 @@ separate load-balanced service it already is.
- Deployment is no longer static: there is a Dockerfile and a server process; the
README front-matter changes from `sdk: static` to `sdk: docker`.
- The client calls `/search` same-origin; the server reads the key from env. A user
- The client calls `/api/search` same-origin; the server reads the key from env. A user
may still supply their own key as a fallback, sent per-request to the proxy.
- The front-end, audio pipeline, and s2s handshake are untouched — only the hosting
shape and the new route are added.
docs/client/architecture.md
+98
View File
@@ -0,0 +1,98 @@
# Client architecture
The [browser client](../../s2s-client) combines a voice UI with a FastAPI server.
The speech pipeline and optional Node-RED reasoning service run separately.
Runtime configuration comes from environment variables and the root `.env`.
```mermaid
flowchart LR
Browser[Browser UI]
Server[Client FastAPI server]
Speech[Realtime speech backend]
Reasoning[Node-RED or HTTP reasoning provider]
Tools[ApiPilot backend]
Browser -->|HTTP configuration and proxies| Server
Browser <-->|WebSocket via local proxy| Server
Server <-->|Realtime events and audio| Speech
Browser <-->|Optional direct connection| Speech
Server -->|Optional transcript reasoning| Reasoning
Server -->|Tool discovery and execution| Tools
```
## Connection modes
`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
session/queue proxy, or the user can enter a direct URL in settings. See
[usage](usage.md) for the configuration matrix.
The local realtime proxy relays audio and events in both directions. It also
patches selected `session.update` instructions, handles ApiPilot marker-based
tool execution, filters internal text, and can suppress speculative/tool-marker
audio. It is not an unconditional byte-for-byte tunnel.
## Source map
| Source | Responsibility |
|---|---|
| [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes |
| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning |
| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar |
| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion |
| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll |
| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs |
| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers |
| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog |
| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction |
| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state |
| [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier |
| [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation |
| [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging |
## Server routes
| Route | Purpose |
|---|---|
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
| `POST /api/reasoning` | Node-RED or OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
| `GET /api/user/phone` | Configured caller profile lookup |
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
| `POST /api/search` | Serper web search proxy |
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
| `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation |
The root serves the UI. `/docs` remains the FastAPI API explorer, while project
documentation files are served in the `/docs/` subdirectories. Markdown remains
source text, and directories without `index.html` are not browsable pages.
Static HTML, JS and CSS
responses receive `Cache-Control: no-store` during development.
## Audio path
1. The browser captures microphone audio with echo cancellation, noise
suppression and automatic gain-control settings.
2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at 24 kHz is decoded and sent to
[audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers,
resamples and fades playback edges. Input speech can clear playback for barge-in.
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
registration structure intact when editing them. The speech backend's thread
and cancellation model is documented in [realtime API](../speech/realtime-api.md).
## Reasoning and context
The browser triggers external reasoning on completed user transcripts only in
`OPENAI` or `NODE_RED` mode. Node-RED `log_only=true` responses leave native speech
output running; an answer-producing response uses the external-answer path.
The server can independently forward assistant text to Node-RED for paired
conversation logs. These HTTP operations supplement the realtime audio path.
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
[tool calling](tool-calling.md) for the conditions and limitations of each path.
docs/client/context-injection.md
+132
View File
@@ -0,0 +1,132 @@
# Context injection
Context reaches the assistant through browser-built instructions, server-patched
realtime session instructions, and HTTP reasoning context. The implementation is
in [server.py](../../s2s-client/server.py),
[main.js](../../s2s-client/main.js), and the
[WebSocket client](../../s2s-client/ws/s2s-ws-client.js).
## Independent controls
`S2S_INJECT_PROFILE` defaults to false and gates server loading of caller,
episode, report and prior-conversation context. It also controls whether the
browser appends caller profile text to its realtime instructions.
It is not a master switch for every context source. Operator-supplied RAG text
and configured ApiPilot discovery work independently. `S2S_INJECT_TODAY`
defaults to true; date and voice-output guards are built on paths that invoke
instruction augmentation or HTTP context assembly. A WebSocket session with no
profile/episode/report/summary or ApiPilot block skips proxy augmentation.
Direct browser connections bypass server-side WebSocket injection entirely.
| Setting | Default | Effect |
|---|---|---|
| `S2S_INJECT_PROFILE` | false | Enable server-resolved caller, episode and history context |
| `S2S_INJECT_REPORTS` | false | Allow scheduled-report fetch/injection; also requires the profile gate |
| `S2S_REPORTS_INJECT_MODE` | `minimal` | `minimal`: labels/dates; `full`: scripts; `off`/`none`: no report prompt block |
| `S2S_REPORTS_QUICK_HINT_MAX` | 5 | Keep at most this many newest reports, clamped to 1–50 |
| `S2S_INJECT_TODAY` | true | Add the current server-local date guard when its context builder runs |
| `S2S_DEBUG_PROMPT` | false | Log complete prompt/context text instead of only summaries |
## Context sources
| Block | Trigger and source | Size/behavior |
|---|---|---|
| Podcast episode | `episode_id` page/query/body value and profile gate | Up to 24,000 characters of serialized episode content |
| Caller profile | `phoneNumber` and `VOCERO_USER_PHONE_AUTH_URL` | Up to 1,600 serialized characters |
| Scheduled reports | `S2S_REPORT_LIST`, profile gate and report gate | Newest N labels/dates by default; full mode includes scripts with a 12,000-character block limit |
| Latest conversation | `S2S_LATEST_SUMMARY_TEXT`, or `S2S_LATEST_SUMMARY` with a phone number | Up to 1,600 characters; static text overrides HTTP fetch |
| Custom RAG | Operator applies text through the RAG dialog | Up to 12,000 characters; appended in browser instructions and HTTP `user_context` |
| ApiPilot discovery | Configured `S2S_APIPILOT_TOOLS_URL` | Tool names/descriptions/auth requirements and call-marker rules, independent of the profile gate |
### Episode
The current episode fetch uses the deployment-specific URL:
```text
https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}
```
It follows redirects with a 15-second timeout, accepts JSON or plain text,
unwraps a `result` field, and includes arbitrary episode fields. The prompt
labels it as the primary reference for the user's current episode. The endpoint
is hardcoded in the fetch helper rather than selected through an environment
variable.
### Caller profile and login
For caller-specific context, the proxy can first post
`{"phoneNumber":"+56912345678"}` to `S2S_LOGIN_BY_PHONE`, extract a bearer token,
and forward it to subsequent profile/history requests. Profile retrieval uses
`GET VOCERO_USER_PHONE_AUTH_URL?phoneNumber=...` with a 10-second timeout and
accepts an object, optionally wrapped in `result`.
Encode the leading plus sign in page links, for example
`?phoneNumber=%2B56912345678`. Caller information is also fetched through
`/api/user/phone` for the UI. An available profile can update live browser
instructions when profile injection is enabled.
### Scheduled reports
`S2S_INJECT_REPORTS=false` skips report fetching and prompt content, including
the table of contents. When enabled, the report endpoint may be called with or
without a phone number. The resolver keeps dictionary entries, reads
`sourceScheduleLabel` or `sourceSchedule.label`, reads `date` or
`sourceSchedule.date`, and optionally keeps `podcastScript` in full mode.
Entries are sorted newest first using ISO/date/timestamp handling and an
ISO-like string fallback. Empty/unparseable dates sort last. The newest-item
limit applies before constructing the prompt block. `minimal` exposes only the
labels and dates; `full` explicitly opts into script content.
### Latest conversation summary
`S2S_LATEST_SUMMARY_TEXT` takes priority over `S2S_LATEST_SUMMARY` and needs no
phone number or network request. Literal `\n` sequences in the environment value
become newlines. Otherwise the proxy fetches
`S2S_LATEST_SUMMARY?phoneNumber=...` and accepts text or JSON with an optional
`result` wrapper. Both sources pass through the same cap and history wrapper.
### Custom RAG
The operator can paste reference material and explicitly apply it to the
session. The browser appends it to instructions even with profile injection
disabled, includes it in HTTP `user_context`, and updates an open realtime
session. This context does not require an external retrieval endpoint.
## Application timing and order
At local WebSocket connect, the server resolves enabled context and stores it in
`session_ctx`. Each client `session.update` then passes through
`_patch_client_session_update_with_profile()`. Other client frame types are not
modified by this helper. The broader relay still performs ApiPilot handling and
text/audio filtering.
The current augmentation function **keeps base instructions first**, then
appends surviving voice-output guards, date context, episode, profile, reports,
and summary blocks. ApiPilot discovery is appended separately. If a block type
is already present in the base, deduplication may omit its server candidate.
This describes string assembly, not a guarantee of model attention or priority.
Every HTTP reasoning request rebuilds enabled context. Its `user_context`
assembly orders voice guard, date, episode, profile, reports, summary, ApiPilot
and browser-provided context, then deduplicates recognized types and strips
internal markers. Server instruction augmentation remains separate, so avoid
assuming the WebSocket and HTTP prompts have identical ordering.
Context markers support internal deduplication and diagnostics; ordinary
profile/report/episode/history sentinels are stripped before final augmentation.
ApiPilot call/catalog markers have their own handling. Treat the source helpers
as the reference for the exact final payload.
## Failure behavior and inspection
Optional context-fetch errors are logged and the affected context is skipped.
The speech conversation normally continues without that reference material.
This does not mean all API calls ignore errors: missing reasoning configuration,
invalid tool arguments and tool authorization can return explicit errors.
Inspect `/api/config` for active gates, the actual outbound `session.update` in
browser DevTools, and `[INJECT-*]`/`[DEBUG-PROMPT]` server logs. Full prompt
logging includes caller data and conversation text. See
[troubleshooting](troubleshooting.md) for a step-by-step diagnostic flow.
docs/client/design.md
+90
View File
@@ -0,0 +1,90 @@
# Design language
This guide describes the current browser UI. Use
[style.css](../../s2s-client/style.css) as the token/selector reference and
[index.html](../../s2s-client/index.html) for the theme-specific structure.
Earlier documentation described a monochrome-only dark UI; the current design
also uses aurora gradients, glass surfaces, and a separate light theme.
## Themes and tokens
`S2S_THEME=white` selects the white/blue interface. Empty or `dark` selects the
dark aurora interface. The browser sets `html[data-theme]` after configuration
loads; an initial visibility guard prevents the wrong theme flashing on first
paint. Extend the existing theme selectors when adding components.
| Token group | Dark theme values/purpose |
|---|---|
| Canvas | `--bg: #07080d`; translucent `--bg-elev` and `--bg-elev-2` surfaces |
| Glass | `--glass` and `--glass-strong` for translucent floating elements |
| Text | `--text`, `--text-dim`, `--text-faint` |
| Borders | `--border` and `--border-strong` hairlines |
| Brand | `--brand: #8b5cf6`, `--brand-2: #ec4899`, `--brand-3: #22d3ee` |
| Radius | `--radius-sm: 10px`, `--radius-md: 16px`, `--radius-lg: 26px`, `--radius-xl: 34px` |
| Shadows | `--shadow-soft`, `--shadow-glow-brand`, `--shadow-glass` |
Prefer existing variables and `color-mix` over new unrelated colors. The dark
theme's orb is the strongest visual focus, while gradients and restrained role
tints also appear in the surrounding interface. The white theme changes
surfaces, typography contrast, controls and chat presentation through its own
selector overrides rather than a second stylesheet.
## Voice state
| State | Dark-theme color |
|---|---|
| Listening/user speaking | Cyan (`--listening`) |
| Assistant speaking | Violet (`--speaking`) |
| Processing | Amber (`--processing`) |
| Ready/authenticated | Green (`--success`) |
| Error | Red (`--error`) |
`--voice-user`, `--voice-assistant` and `--voice-tool` connect transcript roles
to the same state vocabulary. When adding a state, define its existing control,
caption and transcript behavior together. Light-theme controls can use their
blue palette while retaining recognizable state feedback.
## Typography and copy
Use Inter (`--font-sans`) for human prose, buttons, settings and transcript
bodies. Geist Mono (`--font-mono`) is reserved for machine status, role labels,
tool identifiers and diagnostic JSON. Font stacks have system fallbacks.
Visible labels, accessibility text, permission instructions and recovery
messages default to Spanish. Source identifiers, comments, diagnostic logs and
internal model instructions use English. Exact model/voice/protocol identifiers
and proper names are preserved. Spanish wake phrases are user-input literals.
Use sentence case and concrete verbs. Empty states should explain the next
action; errors should say what failed and how to recover. Keep internal system
details out of ordinary user flows.
## Layout and components
The page uses a column layout: header, voice stage, always-visible inline
conversation, and sticky footer. The conversation is no longer the old
toggleable right overlay. Preserve space below the latest message so the sticky
footer does not obscure it; chat autoscroll retries after layout/reflow changes.
In the dark theme, the reports sidebar is fixed on the left at desktop widths
of at least 1024px. On smaller screens it becomes an off-canvas panel with a
toggle and backdrop. The white theme hides the sidebar, its toggle, and hint.
The user avatar opens a profile dialog; profile fields collapse from two
columns on narrow screens. Some controls are intentionally gated by `?debug=true`.
Use the existing button, icon-button, modal, switch and chat-message classes.
The dark theme uses glass bubbles with role tints, borders and restrained glow;
the light theme uses its white/blue message and avatar treatment. Keep required
chat DOM elements intact; optional historical overlay controls may be absent.
## Motion and accessibility
Microphone RMS and assistant audio levels drive the voice visualization.
Breathing, glow and decorative gradients should remain subordinate to speech
feedback. `body.booting` suppresses initial transitions; new animation should
respect that state and `prefers-reduced-motion`.
For changes, verify both themes, narrow mobile layouts down to 360px, keyboard
focus, dialog dismissal, and readable contrast. Prefer touch targets of at least
44px and respect safe-area padding. A CSS guideline is not proof that every
existing control already meets the target.
docs/client/glossary.md
+40
View File
@@ -0,0 +1,40 @@
# Context glossary
These terms explain the product and its conversation flow. Implementation
details belong in [architecture](architecture.md) and the source. Model/provider
names shown in a deployment are configuration examples, not definitions.
| Term | Meaning |
|---|---|
| Speech-to-speech client | The browser interface used to start a voice conversation, display transcripts, and play the assistant's reply |
| Pipeline | The ordered backend path: VAD → STT → LLM → TTS |
| VAD | Voice activity detection; determines when a user speech segment starts and stops |
| STT | Speech to text; transcribes an audio utterance |
| LLM | Language model; generates response text and optional tool calls |
| VLM | Vision-language model; a language model with image support when that capability is configured |
| TTS | Text to speech; synthesizes response audio with the configured voice/model |
| Realtime backend | The service that processes audio and exposes the `/v1/realtime` WebSocket protocol |
| Client server/proxy | The local FastAPI service serving the page and optional HTTP/WebSocket bridges |
| Builder | A person/team credited with creating the interface or deployment, distinct from a model's authors |
| Powered by | Hosting or inference infrastructure credited by the deployment |
| Tool | A callable operation used on demand; distinct from the pipeline stages that process ordinary turns |
| Native tool declaration | A function name, description and JSON Schema sent in `session.tools` |
| Tool executor | Code that receives a tool call, executes it and returns the result to the model |
| Web search | The browser-supported native tool whose request goes through the server's `/api/search` proxy |
| Search proxy | Same-origin server route holding a deployment search key and calling the external provider |
| ApiPilot | The dynamic-tool discovery/execution bridge, with both HTTP endpoints and model-text call markers |
| Call marker | Internal structured text such as `<<<APIPILOT_CALL:...>>>` used by the ApiPilot bridge; separate from native function-call events |
| Tools panel | Controls for available browser tools and search-key configuration |
| Settings | Connection, voice, audio and instruction controls persisted in browser storage |
| Identity block | The header's product/user identity, separate from configuration dialogs |
| About panel | Introduction and project/pipeline credits |
| Caller profile | Information looked up using the page's `phoneNumber`, used for identity display and optionally model context |
| RAG context | Reference material explicitly applied by the operator for the current session |
| Episode context | Content loaded for the page's `episode_id`, treated as the current episode reference |
| Queue | Users waiting for an available compute slot; waiting is not yet an active conversation |
| Ticket | A held queue position returned by the load balancer |
| Position | The displayed place in the queue; distinct from a time estimate |
| Claim | The transition from a waiting ticket to an allocated realtime session and usage reservation |
| At capacity | A full waiting queue that cannot accept another ticket |
| Barge-in | User speech interrupts an active assistant response, coordinating playback clearing and backend cancellation |
| Log-only reasoning | External transcript processing that observes/logs the turn without replacing native speech output |
docs/client/reasoning.md
+106
View File
@@ -0,0 +1,106 @@
# Reasoning modes
Reasoning determines the assistant's response text. The realtime backend always
provides the speech channel; optional HTTP reasoning can observe a transcript or
produce a separate answer. Set `S2S_REASONING_TYPE` to select the mode.
| Mode | Response owner | HTTP hook |
|---|---|---|
| `PROXY` (default) | Realtime backend | Browser does not call `/api/reasoning`; that endpoint returns 409 |
| `NODE_RED` | Realtime backend for `log_only=true`; external answer otherwise | Server forwards to `S2S_REASONING_NODE_RED_URL/api/reasoning` |
| `OPENAI` | OpenAI-compatible Chat Completions response | Server calls `OPENAI_BASE_URL/chat/completions` |
The mode is normalized case-insensitively. `NODERED` and `NODE-RED` resolve to
`NODE_RED`; unknown values resolve to `PROXY`.
## Shared instruction assembly
In [main.js](../../s2s-client/main.js), `effectiveReasoningInstructions()` starts
with `pinnedPrompt || settings.instructions`. `/api/config` pins
`S2S_PRIMARY_PROMPT` when non-empty; otherwise the settings prompt or
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
profile to choose this base prompt.
The browser adds caller context when `injectProfile` is enabled, custom RAG text
when the operator has applied it, and an available ApiPilot catalog block.
`effectiveInstructions()` adds `OPENAI_RENDER_HINT` only in `OPENAI` mode.
The WebSocket client packages these instructions in `session.update.session`;
the local proxy may then add server-resolved context. Details belong in
[context injection](context-injection.md).
## Native realtime reasoning (`PROXY`)
The realtime backend receives session instructions and enabled native tools,
then processes each voice turn through STT, model generation and TTS.
Native function-call events are executed by the browser. ApiPilot also has a
separate marker-based path handled by the local proxy. These are distinct
protocols, described in [tool calling](tool-calling.md).
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
instructions still apply, but server-side session context and marker execution
require the local `/v1/realtime` route.
## Node-RED reasoning
Configure a Node-RED base URL reachable by the UI server:
```env
S2S_REASONING_TYPE=NODE_RED
S2S_REASONING_NODE_RED_URL=http://localhost:1881
```
`maybeRunOpenAiReasoning()` handles both external modes despite its historical
function name. It runs after a complete user transcript and posts a body with
`transcript`, `instructions`, `language`, `user_context`, `phoneNumber`, and
`episode_id`. The server builds context and forwards the reasoning fields to
Node-RED with a 30-second timeout.
The checked-in Node-RED flow exposes `POST /api/reasoning` and
`POST /api/log/system-response`. Its reasoning response uses `log_only=true` and
an empty answer. The browser returns early in that case, preserving the native
backend's response. Assistant-text logging is forwarded separately by the relay
and is not an audio transport.
An answer-producing flow must return a non-empty `answer` and omit `log_only` or
set it to `false`. That activates the external-answer behavior below.
## OpenAI-compatible HTTP reasoning
```env
S2S_REASONING_TYPE=OPENAI
OPENAI_API_KEY=YOUR_API_KEY
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_REASONING_MODEL=gpt-4.1-mini
```
`OPENAI_ORGANIZATION` and `OPENAI_PROJECT` are optional headers. The model value
is a runtime default, not a recommendation about provider availability.
The server combines instructions and context into model messages, requests a
final answer, and returns `answer` with available usage metadata. ApiPilot can
append discovery instructions, execute a detected call marker, and request a
second response based on the tool result. The endpoint does not forward the
browser's native `session.tools` catalog as OpenAI Chat Completions `tools`.
For a non-log-only answer, the browser cancels/suppresses the active native
response and appends the external assistant text. It uses browser
`SpeechSynthesis` when available. Its compatibility fallback sends
`[VOCERO_RENDER_REPLY]` plus the answer to the realtime backend and requests a
response. The render hint asks the model to read only the answer after that
marker. This instruction is model behavior, not a guarantee of exact playback.
External reasoning adds an HTTP operation to each completed transcript and has
separate provider token usage. It is not the speech pipeline's `responses-api`
or `chat-completions` LLM backend setting; those configure the model inside the
realtime backend instead.
## Where to change behavior
| Goal | Source |
|---|---|
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
| Change browser context assembly | `effectiveReasoningInstructions()` in `main.js` |
| Change HTTP provider messages | `reasoning()` in `server.py` |
| Change external-answer rendering | `maybeRunOpenAiReasoning()` in `main.js` |
| Change Node-RED processing | `s2s-reasoning-engine/flows.json` |
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
docs/client/tool-calling.md
+137
View File
@@ -0,0 +1,137 @@
# Tool calling
The application has two tool mechanisms: native realtime function calling and
ApiPilot calls detected from model text. A spoken promise to fetch something is
not evidence that either mechanism executed a tool.
## Native realtime function calls
The browser sends tool declarations inside the `session` object:
```json
{
"type": "session.update",
"session": {
"type": "realtime",
"tools": [
{
"type": "function",
"name": "web_search",
"description": "Search the web for information.",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"]
}
}
],
"tool_choice": "auto"
}
}
```
Keep configuration inside `session`, not at the event root. The backend has a
legacy flat-payload compatibility path, but new clients should use the nested
shape. The current browser's `activeToolDefs()` declares `web_search` when
enabled and a deploy/user search key is available. It does not load the old
remote agent-profile MCP catalog into native session tools.
For a native call, the backend emits:
```json
{
"type": "response.function_call_arguments.done",
"call_id": "call_example",
"name": "web_search",
"arguments": "{\"query\":\"example query\"}"
}
```
`arguments` is a JSON-encoded string. The
[WebSocket client](../../s2s-client/ws/s2s-ws-client.js) parses it and dispatches
`toolcall`; [main.js](../../s2s-client/main.js) runs the supported executor.
Web search uses `POST /api/search`, keeping the deployment's Serper key on the
server. Tool output returns to the model as:
```json
{
"type": "conversation.item.create",
"item": {
"type": "function_call_output",
"call_id": "call_example",
"output": "Tool result text"
}
}
```
The client requests follow-up generation with `response.create` when the
result should be spoken. A `function_call_output` item alone does not start
generation. See the [realtime API guide](../speech/realtime-api.md) for local
model `<code>` extraction and native remote-provider tool calls.
## ApiPilot discovery and execution
`S2S_APIPILOT_TOOLS_URL` enables the current dynamic-tool bridge. The backend
discovers tools through its `/tools` route and executes through `/execute`.
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
server exposes:
- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
`data.tools`, `data.count`, and `data.include_unauthorized`.
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
a `success/data/error` envelope.
Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
`requires_auth`. Execution looks up the current discovery cache, refreshes once
if the tool is missing, and applies an authorization gate and argument
validation before forwarding. Caller `Authorization` and
`x-demo-session` can be forwarded; configured service credentials and
login-by-phone provide additional server paths. Display `error.message` to the
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.
The discovery cache defaults to 300 seconds server-side
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
also keeps a per-user catalog cache. ApiPilot configuration is independent of
`S2S_INJECT_PROFILE`.
## Marker-based two-pass reasoning
The browser and proxy add an ApiPilot catalog as **instruction text**, including
tool names, descriptions, authentication requirements and required parameters.
The model is instructed to emit a call marker when it chooses a matching tool:
```text
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
```
Tool names are exact identifiers and must not be translated. The local realtime
proxy accumulates assistant text, detects the marker, validates and executes
the ApiPilot call, then injects the result as context and requests a new
response. The HTTP reasoning route has analogous discovery/execution and
second-answer handling. Internal marker text and its speculative audio are
filtered from user output where those handlers run.
This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
discovery is not the same as declaring a native `session.tools` array. Direct
browser connections bypass the local relay's marker executor even if the
browser includes catalog instructions.
## Retired documentation assumptions
The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
Those APIs/rankers are absent from the current client/server path. The retained
agent-profile JSON is an illustrative historical/example shape. Add or modify
current ApiPilot tool definitions in the provider; do not restore obsolete
endpoints solely to match older documentation.
## Verify a tool turn
1. Confirm `/api/config` reports the intended feature and connection mode.
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
separate evidence.
3. For native calls, find `response.function_call_arguments.done`, then the
browser `toolcall` and executor request.
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
5. Confirm the tool result reaches the model and follow-up text/audio reaches
the browser. A spoken acknowledgement alone does not complete the flow.
docs/client/troubleshooting.md
+115
View File
@@ -0,0 +1,115 @@
# Troubleshooting
Diagnose configuration, protocol events, execution and playback separately.
Use [architecture](architecture.md), [tool calling](tool-calling.md) and
[context injection](context-injection.md) to identify which path is active.
## Check configuration and connection first
Inspect the server configuration without printing secret environment files:
```bash
curl -fsS http://localhost:8420/api/config | jq
```
Confirm `s2sUrl`, `lb`, `reasoningType`, `injectProfile`, `apipilot`, `theme`
and `startSessionMode`. A direct WebSocket URL bypasses the local relay's
server-side context injection and marker execution. For a local proxy, confirm
the server can resolve and reach `S2S_PROXY_TARGET`.
In browser DevTools, inspect the WebSocket handshake and `session.created`,
`session.update` and error events. Check that instructions and native tools are
inside `session`. A cached browser module can disagree with current server
code; inspect served asset versions and reload with cache disabled when needed.
## The assistant promises a tool but nothing happens
Follow the actual execution mechanism:
1. For a native tool, confirm it appears in the outbound `session.tools` array.
2. Find `response.function_call_arguments.done` with a tool name, `call_id`, and
JSON-string arguments.
3. Confirm the browser dispatched `toolcall` and the executor sent its request.
4. Confirm `function_call_output` and any required `response.create` were sent.
5. For ApiPilot, inspect discovery, marker detection, authorization/schema
validation, execution, and second-pass result injection instead.
A spoken sentence such as “Voy a buscar el reporte” does not satisfy step 2.
If the function-call event is absent, inspect model/backend tool generation
before debugging a browser executor that has not been invoked.
Earlier local debug notes recorded a reproduction where tools and required
tool policy were sent but the model produced only spoken text. Those notes
refer to the retired MCP ranking path; they do not establish a current ApiPilot
failure. Both local investigation notes remain marked `OPEN`. Their useful
lesson is to collect evidence at each boundary and reproduce against current
code before claiming a root cause is resolved.
## The assistant misses caller or report context
- Verify the page has an encoded `phoneNumber` or the intended `episode_id`.
- Inspect `/api/user/phone` and context-fetch logs for HTTP/JSON failures.
- Confirm profile injection is enabled; a visible profile alone does not prove
realtime instructions contain it.
- Reports need `S2S_INJECT_REPORTS=true`; `minimal` sends only labels and dates.
A historical script will not be in the prompt in that mode.
- Static summary text overrides the HTTP summary endpoint.
- Inspect the final `session.update` rather than only its browser base prompt.
The proxy can append and deduplicate context after the browser sends it.
`S2S_DEBUG_PROMPT=true` enables complete prompt logs, including caller information
and conversation text. Use summaries when full payloads are unnecessary. The
browser's application functions are module-scoped; do not assume names such as
`effectiveInstructions()` are available directly on `window` in DevTools.
## Short gaps or clipped words around tool results
The retained audio investigation found very low local serialization/render
times for one large tool result, while still observing audio interruptions.
That evidence ruled out a UI-render bottleneck for that reproduction; it does
not rule out rendering costs for all future payloads.
Useful checks are:
- Measure tool-result size, prompt length, serialization time and UI render time.
- Inspect whether result injection overlaps an active or pending response.
- Check whether a speculative response starts before its `response.created`
reaches the browser. Cancellation based only on an existing response ID can
miss that window.
- Inspect deferred text/response queues and whether they flush after
`response.done` rather than interrupting the active turn.
- Inspect playback prebuffering and queue underflow, especially when the first
words of every sentence are affected.
- Confirm oversized tool results are summarized or capped on the relevant
path rather than assuming every tool payload gets the same treatment.
The historical investigation introduced a 90ms playback prebuffer and reduced
MCP prompt reinjection. The old 150ms MCP fallback timer is historical, not a
current ApiPilot configuration knob. Verify the live worklet and current relay
before changing audio timing based on those observations.
## External reasoning and startup
`/api/reasoning` returning 409 is expected in `PROXY` mode. In `NODE_RED`, check
that `S2S_REASONING_NODE_RED_URL` is reachable from the server and whether the
response is `log_only=true`. An empty answer is valid only in log-only mode.
In `OPENAI`, check the configured endpoint/model/key and provider logs; HTTP
reasoning does not use the native session's tool array.
For `TALK`/`TALKFIRST`, the two-second warmup does not override browser
microphone/audio permissions. If automatic startup fails, use the unlocker
gesture and inspect the permission error rather than increasing the timer
without evidence.
## Read-only source checks
From the repository root, useful syntax checks are:
```bash
node --check s2s-client/main.js
node --check s2s-client/ws/s2s-ws-client.js
python -m py_compile s2s-client/server.py
```
Syntax checks do not validate microphone access, network services, model output,
or audio quality. Collect a fresh browser/backend reproduction for those paths.
docs/client/usage.md
+153
View File
@@ -0,0 +1,153 @@
# Client usage and configuration
The browser voice client connects to an existing OpenAI Realtime-compatible
speech backend. The UI server can also proxy WebSocket traffic, search,
reasoning, caller context, ApiPilot tools and load-balancer requests.
## Local startup
From the repository root:
```bash
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
```
The launcher loads the root `.env`, creates `.venv` if needed, installs the
client requirements when the dependency check fails, and runs uvicorn on port
8420. `UI_PORT` overrides the UI port. Because `.env` is sourced by the script,
values in that file can replace variables supplied on the command line.
To install and start the UI manually:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r s2s-client/requirements.txt
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime \
.venv/bin/python -m uvicorn --app-dir s2s-client server:app --port 8420
```
Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
access, and speak. Microphone access requires HTTPS or a local browser origin;
use HTTPS when accessing the UI from another machine.
The speech pipeline has separate model/runtime dependencies and no complete
dependency manifest in this checkout. In an already prepared speech
environment, a local realtime server can be started with:
```bash
python -m speech_to_speech.s2s_pipeline \
--mode realtime \
--stt parakeet-tdt \
--llm_backend transformers \
--tts kokoro \
--model_name Qwen/Qwen3-4B-Instruct-2507 \
--llm_device cuda \
--llm_torch_dtype float16 \
--enable_live_transcription
```
Select backends and devices supported by the host; the example model is not a
fixed deployment requirement. The default realtime endpoint is
`ws://localhost:8765/v1/realtime`, configured through `--ws_host` and `--ws_port`.
`--num_pipelines` controls the number of isolated concurrent realtime sessions.
See [LLM](../speech/llm.md), [STT](../speech/stt.md), and [TTS](../speech/tts.md).
## Connection modes
| Configuration | Browser connection | Settings URL | Load-balancer features |
|---|---|---|---|
| `SPEECH_TO_SPEECH_URL` set | Direct to pinned URL | Locked | Disabled |
| `S2S_PROXY_TARGET` set, no direct URL | Same-origin `/v1/realtime` proxy | Locked | Disabled |
| `LOAD_BALANCER_URL` set, neither above set | Per-session compute URL from `/api/session` | Hidden | Queue enabled; metering requires `SPACE_ID` too |
| None set | User-provided realtime URL | Editable | Disabled |
Direct URLs may be full `ws(s)://.../v1/realtime` addresses or host values that
the browser normalizes. A direct URL must be reachable **from the browser**.
`S2S_PROXY_TARGET` must be reachable **from the UI server**. HTTPS deployments
need a secure browser WebSocket URL.
The launcher supplies `ws://ai-server:12345/v1/realtime` as a proxy-target
default when no value is configured. That hostname must resolve from the UI
server; it is not a universally available local service.
## Docker
Use the project root as the build context:
```bash
docker build -f s2s-client/Dockerfile -t s2s-client .
docker run --rm -p 7860:7860 \
-e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client
```
For the proxy path, replace the direct setting with a server-reachable
`S2S_PROXY_TARGET`. If the upstream is another container, attach the UI to the
appropriate existing network and use that service's hostname. An upstream
bound only to host loopback is not reachable via an arbitrary container name.
The image includes `s2s-client` and root `docs`. `/docs` is still the FastAPI API
explorer; documentation lives under `/docs/client/`, `/docs/speech/`,
`/docs/adr/`, `/docs/examples/`, and `/docs/proposals/`. Legacy `/project/` and
`/public/docs.html` redirect to the historical proposal. Markdown is served as a
source file rather than rendered through a separate documentation site. Request
file paths such as `/docs/README.md`; directories without `index.html` are not
browsable pages.
## Settings and tools
Voice, editable system instructions and connection settings are kept in browser
`localStorage` under `s2s.ws.*` keys. A non-empty `S2S_PRIMARY_PROMPT` pins the
instruction field. Restart applies connection/voice changes to a new session.
Model voice names are runtime identifiers and stay unchanged across UI
translations.
Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
tools panel. The server proxies requests through `/api/search`; its own key is
not returned to the browser. Dynamic ApiPilot tools are configured through
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).
Operator RAG text is applied through the RAG dialog and updates live
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
include `episode_id=...`. See [context injection](context-injection.md) for
enabled data sources and limits.
## Theme and session startup
`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
the dark aurora theme. Theme-specific sections remain hidden until configuration
resolves, with a fallback to avoid an invisible page after config failure.
`S2S_START_SESSION_MODE` controls initial microphone/session startup:
| Value | Behavior |
|---|---|
| Empty/default | User taps to start |
| `TALK` | After a 2-second warmup, show an audio unlocker; a user gesture starts the session and greeting |
| `TALKFIRST` | After a 2-second warmup, attempt automatic start and inject the Spanish wake phrase; microphone failure falls back to the unlocker |
The warmup is a timer, not proof that all upstream context requests have
finished. Browser permission and audio policies can still require a gesture.
The greeting uses the same `hola vocero` input as wake-word handling.
## Queue and daily usage
Load-balancer mode proxies session acquisition and queue polling. Waiting does
not reserve conversation time; a daily budget is reserved when a compute slot
is granted. The server enables metering only with both `LOAD_BALANCER_URL` and
`SPACE_ID` and with neither pinned direct/proxy mode active.
| Environment setting | Default | Purpose |
|---|---|---|
| `LIMIT_ANON_SEC` | 300 | Anonymous seconds per UTC day |
| `LIMIT_FREE_SEC` | 600 | Signed-in non-PRO seconds per UTC day |
| `UNLIMITED_ORGS` | Adds to built-in organizations | Additional unlimited organization members |
| `USAGE_HASH_SECRET` | Random if unset | Identity-key hashing and anonymous cookie signing |
PRO members are unlimited. Built-in unlimited organizations are `cerebras`,
`HuggingFaceM4`, `smolagents`, and `pollen-robotics`; organization matching is
case-insensitive. See [auth.py](../../s2s-client/auth.py) and
[limiter.py](../../s2s-client/limiter.py) for the current enforcement behavior.
For external transcript processing, configure the selected
[reasoning mode](reasoning.md). For connection, context and audio diagnostics,
use [troubleshooting](troubleshooting.md).
docs/examples/README.md
+16
View File
@@ -0,0 +1,16 @@
# Examples
- [agent-profile.json](agent-profile.json) is an illustrative agent-profile
response with a dynamic tool catalog. Its model instructions are written in
English and request Spanish speech through both instructions and `language`.
Tool names and schemas illustrate the original API response; this file is not
loaded by the application and is not a live API contract.
- [reference-audio.wav](reference-audio.wav) is the existing optional
voice-cloning sample, moved out of the implementation directory. CustomVoice
models do not require it. When testing a compatible Base/voice-cloning model,
supply its path explicitly with `--qwen3_tts_ref_audio
docs/examples/reference-audio.wav` from the project root and use the matching
transcription required by that model. See [the TTS guide](../speech/tts.md).
These examples contain no deployment credentials and are not default runtime
configuration. Use an authorized reference voice for your own deployment.
sample.json → docs/examples/agent-profile.json
+2 -2
View File
@@ -1,7 +1,7 @@
{
"agent": {
"name": "Aquachile Voice Agent",
"description": "Asistente de voz de Aquachile para soporte, informacion operativa y guia conversacional en tiempo real.",
"description": "Aquachile voice assistant for support, operational information, and real-time conversational guidance.",
"model": "gpt-realtime",
"voice": "alloy",
"language": "es",
@@ -564,7 +564,7 @@
"operations",
"voice"
],
"instructions": "Eres el asistente de voz oficial de Aquachile. Debes hablar siempre en espanol, con tono profesional, claro, cordial y seguro. Responde de forma breve pero util. Haz una sola pregunta por turno cuando necesites aclaraciones. No inventes datos, precios, fechas, disponibilidad ni politicas. Si necesitas informacion verificable, usa primero las herramientas disponibles. Si una accion requiere confirmacion del usuario, pidela antes de continuar. Resume siempre la accion o conclusion al final de cada respuesta. Si el usuario cambia de tema, adapta el contexto sin perder claridad. Si no tienes datos suficientes, dilo explicitamente y propone el siguiente paso mas util. Nunca respondas en italiano o ingles salvo que el usuario lo pida de forma explicita.",
"instructions": "You are Aquachile's official voice assistant. Always speak Spanish in a professional, clear, friendly, and confident tone. Keep replies brief and useful. Ask one question per turn when clarification is needed. Do not invent data, prices, dates, availability, or policies. Use the available tools first when you need verifiable information. If an action requires user confirmation, ask before proceeding. Summarize the action or conclusion at the end of each reply. If the user changes the subject, adapt the context while staying clear. If you lack sufficient information, say so explicitly and suggest the most useful next step. Never reply in Italian or English unless the user explicitly asks.",
"turnDetection": {
"type": "server_vad"
}
speech_to_speech/TTS/ref_audio.wav → docs/examples/reference-audio.wav
View File
docs/proposals/vocero-architecture.html
+272
View File
File diff suppressed because one or more lines are too long
docs/speech/llm.md
+162
View File
@@ -0,0 +1,162 @@
# Language model backends
## Available LLM backends (`--llm_backend`)
Runtime-supported values in [s2s_pipeline.py](../../speech_to_speech/s2s_pipeline.py):
- `transformers` → `language_model.py` (Transformers backend)
- `mlx-lm` → `language_model.py` (MLX backend)
- `responses-api` → `responses_api_language_model.py`
- `chat-completions` → `chat_completions_language_model.py`
Run the commands below from the repository root in an environment with the speech
pipeline dependencies installed. This checkout has no package manifest or complete
speech dependency installer; the UI requirements alone do not install the pipeline.
## Usage
### 1) Transformers (`--llm_backend transformers`)
- Handler: `LanguageModelHandler`
- Typical use: local GPU/CPU inference using Hugging Face Transformers
- Backend-specific args prefix: `--llm_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
```bash
python -m speech_to_speech.s2s_pipeline \
--llm_backend transformers \
--model_name Qwen/Qwen3-4B-Instruct-2507 \
--llm_device cuda \
--llm_torch_dtype float16 \
--llm_gen_max_new_tokens 128
```
Common options:
- `--llm_gen_min_new_tokens`
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`
### 2) MLX-LM (`--llm_backend mlx-lm`)
- Handler: `LanguageModelHandler`
- Typical use: Apple Silicon local inference
- Backend-specific args prefix: same as Transformers (`--llm_*`)
```bash
python -m speech_to_speech.s2s_pipeline \
--llm_backend mlx-lm \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
--llm_device mps \
--llm_gen_max_new_tokens 128
```
Common options:
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`
### 3) OpenAI-compatible API (`--llm_backend responses-api`)
- Handler: `ResponsesApiModelHandler`
- Typical use: remote model serving via OpenAI-compatible endpoints
- Backend-specific args prefix: `--responses_api_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
```bash
python -m speech_to_speech.s2s_pipeline \
--llm_backend responses-api \
--model_name gpt-5.4-mini \
--responses_api_api_key YOUR_API_KEY \
--responses_api_base_url https://api.example.com/v1 \
--responses_api_stream true
```
Common options:
- `--chat_size`
- `--init_chat_prompt`
- `--user_role`
### 4) Chat Completions (`--llm_backend chat-completions`)
- Handler: `ChatCompletionsLanguageModelHandler`
- Uses the provider's `/chat/completions` API rather than `/responses`
- Inherits the same `--responses_api_base_url`, `--responses_api_api_key`,
`--responses_api_stream`, and `--responses_api_disable_thinking` flags
- Adds `--responses_api_reasoning_effort` for providers that support that setting
The current handler removes `tools` and `tool_choice` from outgoing provider
requests as a compatibility workaround. Native tool calling therefore differs
from the `responses-api` backend; see the [realtime guide](realtime-api.md).
```bash
python -m speech_to_speech.s2s_pipeline \
--llm_backend chat-completions \
--model_name YOUR_MODEL \
--responses_api_base_url https://api.example.com/v1 \
--responses_api_api_key YOUR_API_KEY
```
The default LLM backend is `responses-api`. Model names in this guide are examples;
select a model available in your environment or from your provider.
## LLM Behavior
When STT is set to language auto-detection (`--language auto`), LLM handlers can receive `(text, language_code)` and prepend a language control instruction like:
- `Please reply to my message in <language>.`
This helps the assistant respond in the detected language. The behavior is opt-in via `--enable_lang_prompt` (shared across all backends); it defaults to `False`.
## Setup
### CUDA setup
```bash
python -m speech_to_speech.s2s_pipeline \
--llm_backend transformers \
--model_name microsoft/Phi-3-mini-4k-instruct
```
### Local Mac setup
```bash
python -m speech_to_speech.s2s_pipeline \
--local_mac_optimal_settings \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16
```
`--local_mac_optimal_settings` already sets `--llm_backend mlx-lm` and will default the model to `mlx-community/Qwen3-4B-Instruct-2507-bf16` if not overridden.
### Realtime (OpenAI-compatible) setup
Run the server in realtime mode, then connect with the [browser client](../client/usage.md):
```bash
# 1. Start the pipeline in realtime mode
python -m speech_to_speech.s2s_pipeline \
--mode realtime \
--llm_backend mlx-lm \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
--ws_host 0.0.0.0 \
--ws_port 8765
# 2. Point the browser client at ws://localhost:8765/v1/realtime.
# See the browser-client usage guide for startup commands.
```
For realtime mode on Apple Silicon, set the desired MLX/STT/TTS/device options
explicitly. The current `--local_mac_optimal_settings` helper forces
`mode="local"`, so combining it with `--mode realtime` does not preserve realtime
mode.
### Remote API setup
```bash
python -m speech_to_speech.s2s_pipeline \
--llm_backend responses-api \
--model_name gpt-5.4-mini \
--responses_api_api_key YOUR_API_KEY
```
docs/speech/realtime-api.md
+230
View File
@@ -0,0 +1,230 @@
# Realtime API
## High-level architecture
The server runs as a handler thread inside the existing pipeline. A FastAPI/uvicorn instance serves the WebSocket endpoint at `/v1/realtime`, while the same queue-driven handler chain (VAD, STT, LLM, TTS) processes audio in parallel threads.
```mermaid
flowchart LR
subgraph client [Client]
WS["WebSocket /v1/realtime"]
end
subgraph server [Realtime Server]
Router["WebSocket Router"]
Service["RealtimeService"]
Config["RuntimeConfig"]
end
subgraph pipeline [Pipeline Threads]
VAD["VAD"]
STT["STT"]
TN["TranscriptionNotifier"]
LLM["LLM"]
Proc["LMOutputProcessor"]
TTS["TTS"]
end
WS -- "client events (JSON)" --> Router
Router -- "parse / dispatch" --> Service
Service -- "PCM chunks" --> VAD
Service -- "session config" --> Config
Config -. "read by" .-> VAD & LLM & TTS
VAD -- "speech segments" --> STT
STT -- "transcript" --> TN
TN -- "transcript + notify" --> LLM
LLM -- "text + tools" --> Proc
Proc -- "clean text" --> TTS
TTS -- "PCM audio" --> Router
Proc -- "assistant_text / tools" --> Router
VAD -- "speech_started/stopped" --> Router
TN -- "transcription events" --> Router
Router -- "server events (JSON)" --> WS
```
**Key flow:**
1. **Inbound audio**: Client sends `input_audio_buffer.append` with base64 PCM. `RealtimeService` decodes, resamples to 16 kHz, splits into 512-sample chunks, and puts them on the `recv_audio_chunks_queue` for VAD.
2. **Speech detection**: VAD detects speech boundaries and emits `speech_started` / `speech_stopped` events on the `text_output_queue`. Full utterance audio goes to STT.
3. **Transcription**: STT output passes through `TranscriptionNotifier`, which taps the transcript for `transcription.delta` / `transcription.completed` events before forwarding to the LLM.
4. **Generation**: The LLM generates text (and optional tool calls). `LMOutputProcessor` splits the output: clean text goes to TTS, and `assistant_text` + tool call dicts go to the `text_output_queue`.
5. **Outbound audio**: TTS writes PCM chunks to `send_audio_chunks_queue`. The router's async `_send_loop` drains both queues, encoding PCM as `response.output_audio.delta` events and translating internal messages into protocol events.
6. **Session config**: `session.update` events deep-merge into `RuntimeConfig`, which is a shared Pydantic model read by VAD (turn detection thresholds), LLM (instructions, tools), and TTS (voice) at processing time.
---
## Supported OpenAI Realtime Events
### Client -> Server
| Event | Description |
|---|---|
| `input_audio_buffer.append` | Stream base64 PCM audio. Decoded, resampled to 16 kHz, and chunked for the VAD. |
| `session.update` | Deep-merge session config (instructions, tools, voice, turn detection, audio format). |
| `conversation.item.create` | Inject `input_text` or `function_call_output` into the LLM context without triggering generation. |
| `response.create` | Trigger LLM generation. Supports per-response `instructions` and `tool_choice` overrides. |
| `response.cancel` | Cancel the in-progress response and re-enable listening. |
### Server -> Client
| Event | Description |
|---|---|
| `session.created` | Sent on connection with current session config. |
| `error` | Protocol errors (`session_limit_reached`, `unknown_or_invalid_event`, `invalid_session_type`, `conversation_already_has_active_response`, etc.) |
| `input_audio_buffer.speech_started` | VAD detected user speech. |
| `input_audio_buffer.speech_stopped` | End of user speech segment. |
| `conversation.item.created` | Acknowledges injected `input_text` from `conversation.item.create`. |
| `conversation.item.input_audio_transcription.delta` | Streaming partial transcript (when live transcription is enabled). |
| `conversation.item.input_audio_transcription.completed` | Final transcript for the user turn (with duration usage). |
| `response.created` | Emitted on the first outbound audio chunk (response is `in_progress`). |
| `response.output_audio.delta` | Base64 PCM audio chunk from TTS. |
| `response.output_audio.done` | Audio stream complete for the current output item. |
| `response.output_audio_transcript.done` | Full assistant text transcript for the turn. |
| `response.function_call_arguments.done` | Tool call with `call_id`, `name`, and JSON `arguments`. |
| `response.done` | Response finished (`completed`, `cancelled` with reason `turn_detected` or `client_cancelled`). |
---
## Tool Calling Design
Tool calling works through two distinct paths depending on the LLM backend, but both converge to the same wire protocol for the client.
### Local LLM path (`LanguageModelHandler` -- transformers / mlx-lm)
Tools defined in `session.update` are converted to `FunctionTool` objects. Each tool's JSON Schema `parameters` are turned into a Python `inspect.Signature` (via `signature_from_schema`), and `to_code_prompt()` renders a human-readable `def name(...): """docstring"""` block.
These tool signatures are injected into the system prompt using a Jinja2 template (`tool_prompt.py`) that instructs the model to wrap tool calls in `<code>...</code>` delimiters:
```
<code>
function_name(arg_name_1=value1, arg_name_2='string_value')
</code>
```
After generation, `_extract_tools` uses a regex to find `<code>` blocks, then `extract_function_calls_from_text` parses each `name(kwargs)` call and validates it against registered tools. Valid calls become `ResponseFunctionToolCall` dicts with generated `call_id`s.
### Remote API paths
`ResponsesApiModelHandler` passes tools natively to `client.responses.create`.
The API returns structured `function_call` items. Per-response `tool_choice`
overrides are read from `response.create`.
`ChatCompletionsLanguageModelHandler` also contains conversion for Chat
Completions tool-call results, but its current `_request()` removes `tools` and
`tool_choice` before the HTTP call to avoid a provider rejection. That backend
does not currently declare native tools to its provider. Do not assume its
result parsing implies working native tool selection.
### Common output path
Both handlers yield `(text, language_code, tools)` tuples. `LMOutputProcessor` forwards clean text to TTS and puts `{"type": "assistant_text", "text": ..., "tools": [...]}` on the `text_output_queue`. The router's `_send_loop` translates these into:
- `response.output_audio_transcript.done` for the text
- `response.function_call_arguments.done` for each tool call
### Tool result flow
1. Client executes the tool and sends `conversation.item.create` with `type: "function_call_output"` and `output: "<result>"`
2. `RealtimeService` appends the tool output to the chat context and emits `conversation.item.created`; this does not trigger generation.
3. If the tool result needs to be spoken to the user, such as camera/search/data results, the client sends `response.create` to trigger follow-up generation.
4. For fire-and-forget robot actions such as dance, emotion, head movement, stop, or idle tools, the client can stop after `conversation.item.created`; the assistant should already have spoken the natural lead-in before the tool call.
---
## Interruption Handling
Barge-in (user speaks while the assistant is playing audio) is handled cooperatively between the VAD, the `_send_loop`, and the LLM/TTS handlers via a shared `CancelScope` object (`cancel_scope.py`).
### CancelScope design
`CancelScope` replaces the old two-signal pattern (`cancel_response` Event + `discard_stale_output` boolean) with a single object that manages:
- **Generation counter** (`cancel_scope.generation`): pipeline threads (LLM, TTS) capture the current generation at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token. When `cancel()` is called, the generation increments and all prior generations become stale -- no timing games required.
- **Discard flag** (`cancel_scope.discarding`): set by `cancel()`, checked by the async `_send_loop` to drop output from superseded generations that arrives between `cancel()` and `response_done()`. Cleared by `response_done(generation)` (only when the sentinel's generation matches the discarded or current one -- sentinels from unrelated older generations are ignored), by `new_response()` on an explicit `response.create`, or by `reset()` on session claim/release.
Pipeline output is **generation-tagged**: `AudioOutput` chunks and `AssistantTextEvent`s carry a `cancel_generation` field stamped by the handler that produced them. The send loop's `_generation_is_discardable` drops an item if its generation is stale, or if `discarding` is set and the item is not from the current generation. Output from the *current* generation always passes through, so a fresh response is never swallowed by a lingering discard window (e.g. a superseded speculative turn whose TTS never emitted a `__RESPONSE_DONE__` sentinel).
```mermaid
sequenceDiagram
participant User
participant VAD
participant SendLoop as _send_loop
participant LLM
participant TTS
participant Client
Note over TTS,Client: Response active or pending (in_response / response_pending)
User->>VAD: speaks
VAD->>SendLoop: speech_started on text_output_queue
SendLoop->>Client: input_audio_buffer.speech_started
SendLoop->>Client: response.done (status=cancelled, reason=turn_detected)
SendLoop->>SendLoop: cancel_scope.cancel() (gen++ & discarding=True)
SendLoop->>SendLoop: flush output_queue + text_output_queue
SendLoop->>SendLoop: response_playing.clear()
LLM->>LLM: is_stale(gen) → True, aborts generation
TTS->>TTS: is_stale(gen) → True, aborts generation
TTS->>SendLoop: __RESPONSE_DONE__ (tagged with gen)
SendLoop->>SendLoop: cancel_scope.response_done(gen) (discarding=False)
Note over VAD,Client: Pipeline is now processing the new user utterance
```
**Step by step:**
1. **VAD detects speech**: puts a `SpeechStartedEvent` on `text_output_queue`.
2. **`_send_loop` processes text events first** (priority over audio): translates `speech_started` into protocol events. If an active response was in progress, `RealtimeService.dispatch_pipeline_event` emits `response.output_audio.done` + `response.done` with `status="cancelled"` and `reason="turn_detected"`.
3. **Cancel + queue flush**: if a response is active (`in_response`) *or* pending (`response_pending` -- accepted `response.create`, no audio yet), and interrupts are enabled (see step 4), the send loop calls `cancel_scope.cancel()` (increments generation, enables discard), clears `response_pending`, drains `output_queue` (preserving `__RESPONSE_DONE__` sentinels) and `text_output_queue` (preserving user-side events: `speech_stopped`, partial/completed transcriptions, token usage), then clears `response_playing`.
4. **Interrupt gating**: the cancel only fires if the `SpeechStartedEvent.interrupt_response` flag is set *and* the session config allows it (`turn_detection.interrupt_response`, read via `RuntimeConfig.interrupt_response_enabled`, default true). When disabled, user speech during a response is transcribed but the response keeps playing.
5. **LLM/TTS cancellation**: handlers capture `gen = cancel_scope.generation` at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token, aborting early when stale.
6. **Discard guard**: while `cancel_scope.discarding` is True, the send loop drops audio chunks and assistant text whose `cancel_generation` is not current (see `_generation_is_discardable` above). The guard clears when a `__RESPONSE_DONE__` with a matching generation arrives (via `cancel_scope.response_done(gen)`), or when an explicit `response.create` starts a new response (`cancel_scope.new_response()`).
7. **Client-initiated cancel**: `response.cancel` calls `cancel_scope.cancel()` (only if a response was active), flushes both queues with the same preservation rules, triggers `finish_response(status="cancelled", reason="client_cancelled")`, re-enables `should_listen`, and clears `response_playing`.
8. **Spurious cancel safety**: if no response is active, `cancel_scope.cancel()` is not called, preventing the discard guard from being set without a `__RESPONSE_DONE__` to clear it.
---
## Testing
Run these examples from the repository root in an environment with the speech
dependencies installed. See [LLM](llm.md), [STT](stt.md), and [TTS](tts.md) for
backend-specific options; this checkout does not include a package manifest.
### Local LLM with Transformers
```bash
python -m speech_to_speech.s2s_pipeline \
--mode realtime \
--stt parakeet-tdt \
--llm_backend transformers \
--tts kokoro \
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
### Local LLM with MLX-LM
```bash
python -m speech_to_speech.s2s_pipeline \
--mode realtime \
--stt parakeet-tdt \
--llm_backend mlx-lm \
--tts kokoro \
--model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
### Remote LLM with OpenAI-compatible API
```bash
python -m speech_to_speech.s2s_pipeline \
--mode realtime \
--stt parakeet-tdt \
--llm_backend responses-api \
--tts kokoro \
--model_name "openai/gpt-oss-20b:groq" \
--responses_api_base_url "https://router.huggingface.co/v1" \
--responses_api_api_key "$HF_TOKEN" \
--responses_api_stream \
--enable_live_transcription
```
docs/speech/stt.md
+166
View File
@@ -0,0 +1,166 @@
# Speech-to-text backends
This document summarizes the speech-to-text implementations in
[speech_to_speech/STT](../../speech_to_speech/STT), including language support,
language abbreviations, and pipeline options. Run examples from the repository
root in an environment with the speech dependencies installed.
## Available STT Modes (`--stt`)
- `whisper` → `STT/whisper_stt_handler.py`
- `whisper-mlx` → `STT/lightning_whisper_mlx_handler.py`
- `mlx-audio-whisper` → `STT/mlx_audio_whisper_handler.py`
- `faster-whisper` → `STT/faster_whisper_handler.py`
- `parakeet-tdt` → `STT/parakeet_tdt_handler.py`
- `paraformer` → `STT/paraformer_handler.py`
## Language Support by Handler
### 1) Whisper (`--stt whisper`)
- Handler: `WhisperSTTHandler`
- Language input flag: `--language` (from shared Whisper args)
- Supports fixed language (e.g. `en`) or `auto`
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- Detects language from token output
- If detected language is outside the supported list, it falls back to the previous language
### 2) Lightning Whisper MLX (`--stt whisper-mlx`)
- Handler: `LightningWhisperSTTHandler`
- Uses same shared `--language` argument as Whisper
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- If `--language auto`, model auto-detects each utterance
- If detected language is unsupported, falls back to last supported language
### 3) MLX Audio Whisper (`--stt mlx-audio-whisper`)
- Handler: `MLXAudioWhisperSTTHandler`
- Model flag: `--mlx_audio_whisper_model_name`
- Language still comes from shared `--language` flag (wired in pipeline)
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- Uses fixed language unless `--language auto`
- Falls back to last known supported language when needed
### 4) Faster-Whisper (`--stt faster-whisper`)
- Handler: `FasterWhisperSTTHandler`
- Language flag: `--faster_whisper_stt_gen_language`
- Default language: `en`
- Note:
- This handler passes generation kwargs directly to `faster_whisper.WhisperModel.transcribe(...)`
- Effective language coverage depends on selected Faster-Whisper/OpenAI Whisper model
### 5) Parakeet TDT (`--stt parakeet-tdt`)
- Handler: `ParakeetTDTSTTHandler`
- Language flag: `--parakeet_tdt_language` (optional)
- Supports auto language detection when language not specified
- Declared supported language list (25 European languages):
- `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `pl`, `ru`, `uk`, `cs`, `sk`, `hu`, `ro`, `bg`, `hr`, `sl`, `sr`, `da`, `no`, `sv`, `fi`, `et`, `lv`, `lt`
- Backend behavior:
- On macOS/MPS: MLX (`mlx-community/parakeet-tdt-0.6b-v3`)
- On CUDA/CPU: nano-parakeet (`nvidia/parakeet-tdt-0.6b-v3`)
### 6) Paraformer (`--stt paraformer`)
- Handler: `ParaformerSTTHandler`
- Model flag: `--paraformer_stt_model_name`
- Default model: `paraformer-zh`
- No dedicated language flag in current args class
- Practical support:
- Depends on selected FunASR model checkpoint
- Default setup is Chinese-oriented (`zh`)
## Language Abbreviations (ISO-style codes seen in STT handlers)
| Code | Language |
|---|---|
| `en` | English |
| `fr` | French |
| `es` | Spanish |
| `zh` | Chinese |
| `ja` | Japanese |
| `ko` | Korean |
| `hi` | Hindi |
| `de` | German |
| `pt` | Portuguese |
| `pl` | Polish |
| `it` | Italian |
| `nl` | Dutch |
| `ru` | Russian |
| `uk` | Ukrainian |
| `cs` | Czech |
| `sk` | Slovak |
| `hu` | Hungarian |
| `ro` | Romanian |
| `bg` | Bulgarian |
| `hr` | Croatian |
| `sl` | Slovenian |
| `sr` | Serbian |
| `da` | Danish |
| `no` | Norwegian |
| `sv` | Swedish |
| `fi` | Finnish |
| `et` | Estonian |
| `lv` | Latvian |
| `lt` | Lithuanian |
| `auto` | Per-utterance automatic language detection |
## Usage Examples
### Whisper (Transformers)
```bash
python -m speech_to_speech.s2s_pipeline --stt whisper --language es
python -m speech_to_speech.s2s_pipeline --stt whisper --language auto
```
### Whisper MLX (LightningWhisperMLX)
```bash
python -m speech_to_speech.s2s_pipeline --stt whisper-mlx --language auto --device mps
```
### MLX Audio Whisper
```bash
python -m speech_to_speech.s2s_pipeline --stt mlx-audio-whisper \
--mlx_audio_whisper_model_name mlx-community/whisper-large-v3-turbo \
--language auto
```
### Faster-Whisper
```bash
python -m speech_to_speech.s2s_pipeline --stt faster-whisper \
--faster_whisper_stt_model_name large-v3 \
--faster_whisper_stt_gen_language en
```
### Parakeet TDT
```bash
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt --parakeet_tdt_device auto
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt --parakeet_tdt_language es
```
With live transcription (MLX or CUDA/nano-parakeet backend):
```bash
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt \
--enable_live_transcription \
--live_transcription_update_interval 0.25
```
### Paraformer
```bash
python -m speech_to_speech.s2s_pipeline --stt paraformer --paraformer_stt_model_name paraformer-zh
```
docs/speech/tts.md
+151
View File
@@ -0,0 +1,151 @@
# Text-to-speech backends
## Available TTS modes (`--tts`)
Runtime-supported values in [s2s_pipeline.py](../../speech_to_speech/s2s_pipeline.py):
- `chatTTS` → `chatTTS_handler.py`
- `facebookMMS` → `facebookmms_handler.py`
- `pocket` → `pocket_tts_handler.py`
- `kokoro` → `kokoro_handler.py`
- `qwen3` → `qwen3_tts_handler.py`
Run examples from the repository root in an environment with the speech
dependencies installed. The UI requirements do not install TTS dependencies.
## Usage
### 1) ChatTTS (`--tts chatTTS`)
Primary args prefix: `--chat_tts_*`
```bash
python -m speech_to_speech.s2s_pipeline \
--tts chatTTS \
--chat_tts_device cuda \
--chat_tts_stream true \
--chat_tts_chunk_size 512
```
### 2) Facebook MMS (`--tts facebookMMS`)
Primary args prefix: `--facebook_mms_*` plus `--tts_language`
```bash
python -m speech_to_speech.s2s_pipeline \
--tts facebookMMS \
--facebook_mms_device cuda \
--tts_language en
```
This handler maps STT language codes (e.g. `en`, `fr`, `es`) to MMS model suffixes (e.g. `eng`, `fra`, `spa`) and reloads the model on language changes.
### 3) Pocket TTS (`--tts pocket`)
Primary args prefix: `--pocket_tts_*`
```bash
python -m speech_to_speech.s2s_pipeline \
--tts pocket \
--pocket_tts_voice jean \
--pocket_tts_device cpu \
--pocket_tts_sample_rate 16000
```
Available preset voices include:
`alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`, `azelma`.
### 4) Kokoro (`--tts kokoro`)
Primary args prefix: `--kokoro_*`
```bash
python -m speech_to_speech.s2s_pipeline \
--tts kokoro \
--kokoro_device auto \
--kokoro_voice bm_fable \
--kokoro_lang_code b
```
Behavior:
- Uses MLX backend on Apple Silicon (`mlx-community/Kokoro-82M-bf16`)
- Uses native kokoro pipeline otherwise (`hexgrad/Kokoro-82M`)
- Can auto-switch voice/language based on STT language code mapping
### 5) Qwen3-TTS (`--tts qwen3`)
Primary args prefix: `--qwen3_tts_*`
```bash
python -m speech_to_speech.s2s_pipeline \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_device cuda \
--qwen3_tts_backend ggml \
--qwen3_tts_speaker Aiden \
--qwen3_tts_language auto \
--qwen3_tts_non_streaming_mode True
```
Behavior:
- Uses `faster-qwen3-tts` on non-macOS platforms, defaulting to the GGML backend. Pass `--qwen3_tts_backend torch` to use the CUDA-graphs backend instead.
- Uses `mlx-audio` on Apple Silicon and auto-maps `Qwen/...` model IDs to `mlx-community/...`, defaulting to the `6bit` MLX variant unless the model name already pins a suffix.
- Supports MLX quantization overrides on Apple Silicon via `--qwen3_tts_mlx_quantization bf16|4bit|6bit|8bit`.
- Keeps the existing voice-clone/custom-voice/voice-design handler flow intact.
- Defaults to the CustomVoice model with speaker `Aiden`, so no reference audio is required. Voice-clone/base models can still use `--qwen3_tts_ref_audio`.
Install notes for Linux GGML:
- The default PyPI `qwentts-cpp-python` wheel targets CUDA 12.8.
- If that wheel does not match your CUDA runtime, install one of the Hugging Face wheelhouse builds before installing `speech-to-speech`.
```bash
pip install "qwentts-cpp-python==0.3.1+cu130" \
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130
pip install speech-to-speech
```
Available wheelhouse directories include `cu124`, `cu128`, `cu130`, and `cpu`.
Example for Apple Silicon using the default 6-bit MLX variant:
```bash
python -m speech_to_speech.s2s_pipeline \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_speaker Aiden
```
You can override the default and select `bf16`, `4bit`, or `8bit` explicitly:
```bash
python -m speech_to_speech.s2s_pipeline \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_mlx_quantization 4bit \
--qwen3_tts_speaker Aiden
```
For voice-cloning examples, an optional reference clip is retained at
[docs/examples/reference-audio.wav](../examples/reference-audio.wav). Supply an
appropriate base/voice-cloning model and `--qwen3_tts_ref_audio` explicitly;
CustomVoice mode does not need this clip. The repository does not include a TTS
benchmark script.
## Setup
### Low-latency GPU setup
```bash
python -m speech_to_speech.s2s_pipeline \
--stt whisper \
--tts pocket
```
### Apple Silicon setup
```bash
python -m speech_to_speech.s2s_pipeline \
--local_mac_optimal_settings
```
`--tts pocket` and `--tts kokoro` are also valid options on macOS.
reasoning.md
-608
View File
@@ -1,608 +0,0 @@
# Reasoning & Context Injection — Architettura Completa
Questo documento descrive **come funziona il "reasoning"** (il layer di intelligenza che decide cosa rispondere) nell'app Vocero/S2S e **in quali punti puoi iniettare contesto** (identità dell'agente, istruzioni, strumenti MCP, profilo chiamante, dati extra, ecc.).
Esistono **due modalità di reasoning** distinte:
| Modalità | Variabile env | Chi fa il "pensiero" | Flusso |
|---|---|---|---|
| **PROXY** *(default)* | `S2S_REASONING_TYPE=PROXY` | Il server realtime upstream (`ai-server`, Vocero, o qualunque backend dietro `S2S_PROXY_TARGET`) | Browser → WebSocket `/v1/realtime` → upstream (prompt + tools inviati in `session.update`) |
| **OPENAI** *(reasoning esterno)* | `S2S_REASONING_TYPE=OPENAI` | Una chiamata HTTP separata a OpenAI Chat Completions (es. `gpt-4.1-mini`) | Browser → `POST /api/reasoning` → OpenAI → risposta reiniettata nel realtime come messaggio utente forzato |
In **entrambe** le modalità le informazioni di contesto vengono assemblate lato client in un'unica stringa "istruzioni" (`effectiveReasoningInstructions`) e passate poi in modo diverso.
---
## 1. Panoramica del flusso completo (uno sguardo dall'alto)
```
┌─────────────────────────────────────────────────────────────────────────┐
│ /home/duxiter/s2s/.env │
│ S2S_PRIMARY_PROMPT S2S_AGENT_PROFILE S2S_REASONING_TYPE │
│ S2S_TOOL_EXECUTION_MCP_URL S2S_USER_PHONE_AUTH_URL │
│ OPENAI_API_KEY / BASE_URL / REASONING_MODEL / ORG / PROJECT │
│ S2S_PROXY_TARGET S2S_SEARCH_MCP etc. │
└───────────────────────────┬─────────────────────────────────────────────┘
│ caricato da start_ui.sh
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ demo/server.py (backend Python FastAPI) │
│ │
│ fetchConfig() GET /api/config ←─────── GET /api/user/phone │
│ │ │ (se URL param phone) │
│ ├─ primaryPrompt (priorità 1: Aquachile /agents/profile │
│ │ priorità 2: S2S_PRIMARY_PROMPT env) │
│ ├─ agentProfile (URL di dove abbiamo preso il prompt) │
│ ├─ agentTools[] (catalogo MCP Aquachile, enabled=True) │
│ ├─ reasoningType = "PROXY" oppure "OPENAI" │
│ └─ searchMcp, s2sUrl, lb, auth, search, ecc. │
│ │
│ POST /api/reasoning (solo se reasoningType=OPENAI) │
│ body → httpx POST {OPENAI_BASE_URL}/chat/completions │
│ │
│ POST /api/mcp/tool │
│ body → httpx POST {S2S_TOOL_EXECUTION_MCP_URL} + chiavi API │
│ │
│ WS /v1/realtime (proxy a S2S_PROXY_TARGET) │
│ client ↔ proxy ↔ upstream: logga [INPUT UTENTE]/[OUTPUT AGENTE] │
└───────────────────────────┬─────────────────────────────────────────────┘
│ fetch su caricamento pagina
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ demo/main.js (frontend browser) │
│ │
│ pinnedPrompt = primaryPrompt (da /api/config) — read-only se valoriz.│
│ agentToolDefs[] = agentTools (da /api/config) │
│ reasoningType = "PROXY" | "OPENAI" │
│ voceroUserPhoneNumber = new URL(location).searchParams.phoneNumber │
│ │ │
│ └─► async fetchVoceroUserProfile() │
│ GET api/user/phone?phoneNumber=… → voceroUserProfile{} │
│ │
│ Quando l'utente clicca l'orb → doStart(): │
│ startInstructions = effectiveInstructions() │
│ startTools = activeToolDefs() │
│ │ │
│ └─► new S2sWsRealtimeClient({instructions, tools, …}) │
│ │ │
│ ├─► session.update al realtime upstream │
│ │ (se reasoningType=PROXY → qui pensa) │
│ │ │
│ └─► event "transcript" (user, partial=false) │
│ │ │
│ └─► maybeRunOpenAiReasoning() │
│ (solo reasoningType=OPENAI) │
│ → POST /api/reasoning → answer │
│ → sendUserText(renderPrompt) │
│ → requestResponse() │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## 2. Assemblaggio del prompt: `effectiveReasoningInstructions()`
La funzione **centrale** che costruisce il blocco di testo che verrà dato al modello è in [main.js:516-533](file:///home/duxiter/s2s/demo/main.js#L516-L533):
```js
function effectiveReasoningInstructions() {
const base = pinnedPrompt || settings.instructions;
let instructions = base;
if (activeToolDefs().length) instructions += TOOL_USE_HINT;
if (searchMcpEnabled && agentToolDefs.length) {
instructions += MCP_SEARCH_HINT;
instructions += ` Available MCP tools: ${buildMcpToolCatalogSummary(agentToolDefs)}`;
}
const suggested = topSuggestedAgentTools(3);
if (searchMcpEnabled && suggested.length) {
instructions += ` Current user request: "${lastUserQuery}". Client-side ranking hint only: …`;
}
const userContext = voceroUserContextText();
if (userContext) instructions += userContext;
return instructions;
}
```
Ordine di concatenazione (dalla priorità più alta alla più bassa):
| # | Fonte | Dove viene da | Note |
|---|---|---|---|
| 1 | **`pinnedPrompt`** | `S2S_AGENT_PROFILE` (`instructions` campo Aquachile) → fallback `S2S_PRIMARY_PROMPT` env → fallback `settings.instructions` (localStorage) → fallback `DEFAULT_INSTRUCTIONS` | **Identità e personalità principale dell'agente**. Quando `pinnedPrompt` è presente, il campo `instructions` di Settings è disabilitato in UI (read-only). |
| 2 | **`TOOL_USE_HINT`** | Costante in linea [main.js:32-36](file:///home/duxiter/s2s/demo/main.js#L32-L36) | Aggiunto se c'è **almeno 1 tool attivo**: "*When the user's request calls for one of your tools, do not describe your capabilities or say you can do it and wait for another turn. Instead, say a brief acknowledgement… and call the tool right away in the same response.*" |
| 3 | **`MCP_SEARCH_HINT`** + catalogo completo tools | Condizionato a `S2S_SEARCH_MCP=true` in env | Istruisce il modello a leggere l'intero catalogo MCP e scegliere lo strumento migliore. Poi **appende come testo TUTTI i tool MCP** (`buildMcpToolCatalogSummary`) con nome, descrizione e parametri obbligatori. |
| 4 | **Suggerimento tool client-side (top 3)** | `topSuggestedAgentTools(3)` basato su `lastUserQuery` | Messaggio "Client-side ranking hint only: 1. get_day_report: … 2. … 3. …". Usato **solo come hint secondario**, il modello deve sempre controllare il catalogo completo prima. |
| 5 | **`voceroUserContextText()`** | Profile restituito da `/api/user/phone?phoneNumber=…` | Stringa: `" Caller profile resolved from the page phoneNumber: {JSON profilo}. Use this as trusted caller context when relevant."` (troncato a 1600 char) |
Poi la funzione **`effectiveInstructions()`** ([main.js:537-539](file:///home/duxiter/s2s/demo/main.js#L537-L539)) prende il risultato e:
- Se `reasoningType=OPENAI` → **appende `OPENAI_RENDER_HINT`**: istruisce il realtime a **rispondere solo ai messaggi che iniziano con il marker `[VOCERO_RENDER_REPLY]`**, in modalità "leggi e basta" (niente tool, niente ragionamento). Questo è il segreto che permette di scrivere la frase pensata da OpenAI e farla ripetere fedelmente in TTS.
- Se `reasoningType=PROXY` → ritorna le istruzioni tal quali.
### Quando viene ricalcolato e reinviato al modello
- **Alla connessione** in `doStart()`: `startInstructions = effectiveInstructions()` → `new S2sWsRealtimeClient({instructions: startInstructions, …})`.
- **Durante la chiamata**, in risposta a:
- ogni **trascrizione utente** parziale o finale (linee [1122-1128](file:///home/duxiter/s2s/demo/main.js#L1122-L1128)): `pushLiveToolPolicy()` + `pushLiveInstructions(effectiveInstructions())` → `client.updateSession({ instructions })`
- quando arriva il **profilo chiamante da phone auth** ([2388-2390](file:///home/duxiter/s2s/demo/main.js#L2388-L2390)): stesso `pushLiveInstructions`
- quando l'utente **modifica le impostazioni** nel form e salva.
---
## 3. I due motori di reasoning, nel dettaglio
### 3.1 Modalità **PROXY** (default — consigliata con Aquachile)
```
S2S_REASONING_TYPE=PROXY (o qualsiasi valore diverso da "OPENAI")
S2S_PROXY_TARGET=ws://ai-server:12345/v1/realtime # o qualunque endpoint WS realtime
```
In questa modalità, **l'upstream realtime è l'unico a "pensare"**. Tutta l'intelligenza risiede nel backend puntato da `S2S_PROXY_TARGET` (es. Vocero, un wrapper realtime custom, un server che parla OpenAI Realtime API).
Flusso **testuale** di quello che succede a ogni avvio:
1. `doStart()` calcola `startInstructions = effectiveInstructions()` (tutto il blocco §2).
2. Costruisce `startTools = activeToolDefs()` = tool browser (web_search, camera_snapshot) **+** `agentToolDefs` (catalogo MCP Aquachile).
3. Passa `instructions` e `tools` al costruttore di `S2sWsRealtimeClient`.
4. Quando il WS si connette e arriva `session.created`, il client manda `session.update` con:
```json
{
"type": "realtime",
"instructions": "<tutte le istruzioni assemblate>",
"tools": [ { "type":"function", "name":"…", "description":"…", "parameters":{…} }, … ],
"tool_choice": "auto",
"audio": { "output": { "voice": "Aiden" } }
}
```
5. **TUTTO IL RESTO è SULL'UPSTREAM**: L'upstream decide quando fare tool call, quando parlare, quando ascoltare. Il browser è solo uno "schermo+microfono+telecomando".
6. Quando l'upstream fa `conversation.item.input_audio_transcription.completed` il log server stampa `[INPUT UTENTE] …`.
7. Quando l'upstream fa `response.*_transcript.done` / `response.content_part.done` ecc. il log server stampa `[OUTPUT AGENTE] …`.
**Vantaggi:**
- Un solo round trip per tutto (latenza più bassa per voce→voce).
- L'upstream può usare qualsiasi modello/rete interna (non per forza OpenAI).
- Le MCP tool call sono orchestrate dallo stesso server che fa STT/TTS → meno desincronizzazioni.
**Svantaggi:**
- Tutto il "pensiero" è black-box sull'upstream; non puoi vedere i token del thinking né usare un modello diverso senza cambiare upstream.
---
### 3.2 Modalità **OPENAI** (reasoning HTTP separato)
```
S2S_REASONING_TYPE=OPENAI
OPENAI_API_KEY=sk-… # obbligatorio
OPENAI_BASE_URL=https://api.openai.com/v1 # default
OPENAI_REASONING_MODEL=gpt-4.1-mini # default
OPENAI_ORGANIZATION=… # opzionale
OPENAI_PROJECT=… # opzionale
```
In questa modalità il **ragionamento vero e proprio** non viene fatto dall'upstream realtime. Invece:
1. L'upstream fa solo **STT** (converte voce in testo) e **TTS** (converte testo in voce). Agisce come un "dumb" audio I/O.
2. **Non appena arriva una trascrizione utente COMPLETA** (evento `transcript`, `role=="user"`, `partial==false`), `maybeRunOpenAiReasoning(transcript, itemId)` viene triggerato.
3. Questo chiama `fetchOpenAiReasoningAnswer(transcript)` ([main.js:563-586](file:///home/duxiter/s2s/demo/main.js#L563-L586)) che fa:
```
POST /api/reasoning
body = {
transcript: "quanto è il report giornaliero?",
instructions: effectiveReasoningInstructions(), // stesse di PROXY
language: "es-CL" o vuoto,
user_context: voceroUserContextText(), // duplicato per sicurezza
}
```
4. Il server (endpoint in [server.py:635-710](file:///home/duxiter/s2s/demo/server.py#L635-L710)) a sua volta fa:
```
POST {OPENAI_BASE_URL}/chat/completions
body = {
model: OPENAI_REASONING_MODEL,
messages: [
{ role:"system",
content: "{instructions}\n\n{user_context}\nReply only with final answer. Language=…" },
{ role:"user", content: transcript }
]
}
```
5. Riceve `answer` (stringa semplice, pensata per essere "parlata"), logga `[Vocero REASONING] transcript=… answer=…` e la restituisce al client.
6. Il client costruisce `renderPrompt = "[VOCERO_RENDER_REPLY]\n" + answer` e lo **manda come se fosse un testo scritto dall'utente** al realtime:
```
client.sendUserText(renderPrompt);
client.requestResponse();
```
7. L'upstream realtime, a causa di `OPENAI_RENDER_HINT` nel prompt, **vede il marker `[VOCERO_RENDER_REPLY]`** e quindi:
- NON usa tool, NON fa ragionamento proprio, NON modifica la frase.
- Si limita a leggere la parte dopo il marker in TTS.
8. Risultato: l'utente sente "la voce dell'agente" (TTS dell'upstream) ma il testo **è stato scritto da OpenAI** in `/chat/completions`.
Log server corrispondenti:
```
[EXT][OPENAI] reasoning POST https://api.openai.com/v1/chat/completions model=gpt-4.1-mini systemLen=… transcriptLen=…
[EXT][OPENAI] reasoning RESP status=200 len=…
[EXT][OPENAI] reasoning USAGE prompt=1234 completion=89 total=1323
INFO [s2s] [Vocero REASONING] transcript=… answer=…
```
**Vantaggi:**
- Scegli tu **qualsiasi modello compatibile Chat Completions** (anche Azure, Together, self-hosted se l'endpoint parla lo stesso protocollo).
- Hai **usage e log del reasoning separati** da quelli dell'audio.
- Puoi aggiungere facilmente post-processing, cache o fallback.
**Svantaggi:**
- Due round trip invece di uno → un po' più di latenza totale (~200-800 ms).
- Più costoso in token: `instructions` + contesto utente vengono mandati **due volte** (una a OpenAI, una al realtime).
- Le tool call che il modello OpenAI vorrebbe fare non possono essere usate direttamente — in OPENAI mode `chat/completions` NON ha accesso a `tools` (al momento l'endpoint `POST /api/reasoning` non inoltra il catalogo MCP a OpenAI). **Quindi in OPENAI mode, le MCP Aquachile vengono ancora passate al realtime (che in questa modalità non "pensa" ma solo reinderizza il testo), quindi non sono usabili.**
> ⚠️ Importante: se usi Aquachile con il suo ecosistema MCP, **la modalità PROXY è quella corretta da usare**. OPENAI mode è pensata per scenari "solo testo + voce" senza tool complessi.
---
## 4. TUTTI i modi per iniettare contesto all'agente, dal più semplice al più avanzato
I punti di injection sono ordinati per **priorità di applicazione** (chi "vince" in caso di conflitto) e per **facilità di modifica**.
---
### 🅿️ Punto 1 — `.env` (variabili di ambiente, configurazione statica)
File: [.env](file:///home/duxiter/s2s/.env)
Caricato in memoria da [start_ui.sh](file:///home/duxiter/s2s/start_ui.sh#L8-L12) (`source "$ROOT_DIR/.env"`) prima di avviare uvicorn.
| Variabile | Tipo | Cosa fa all'agente |
|---|---|---|
| **`S2S_PRIMARY_PROMPT`** | testo lungo | **Prompt backup quando Aquachile è irraggiungibile o `S2S_AGENT_PROFILE` non è impostato.** Sarà il valore di `primaryPrompt` in `/api/config` se `_extract_agent_prompt()` torna stringa vuota. Supporta `\n` → convertiti in newline. |
| **`S2S_AGENT_PROFILE`** | URL | Endpoint `GET` (es. Aquachile: `https://aquachile.azurianlab.com/api/v1/agents/profile`). Deve ritornare JSON `{ agent: { instructions: "...", tools: [ {name, description, parameters, enabled}, ... ] } }`. **Ha priorità su `S2S_PRIMARY_PROMPT`** — vedi `primary_prompt = agent_prompt or S2S_PRIMARY_PROMPT` in [server.py:517](file:///home/duxiter/s2s/demo/server.py#L517). |
| **`S2S_TOOL_EXECUTION_MCP_URL`** | URL | Endpoint `POST` per **eseguire** le tool call MCP (stesso backend Aquachile). Riceve `{server, name, arguments}` e viene chiamata ogni volta che il modello invoca uno strumento. |
| `S2S_TOOL_EXECUTION_MCP_API_KEY` / `S2S_TOOL_EXECUTION_MCP_API_SECRET` | stringhe segrete | Intestazioni `X-API-KEY` e `X-API-SECRET` aggiunte a ogni chiamata MCP. |
| **`S2S_USER_PHONE_AUTH_URL`** | URL | Endpoint `GET` (con `?phoneNumber=…`) che restituisce il profilo del chiamante. Se l'URL della UI contiene `?phoneNumber=+56…`, questo endpoint viene chiamato e il JSON risultante viene appeso al prompt di sistema (vedi Punto 4). |
| **`S2S_SEARCH_MCP=true\|false`** | bool | Quando `true`, appende `MCP_SEARCH_HINT` + il catalogo COMPLETO dei tool MCP dentro le `instructions`. Fondamentale se vuoi che il modello sappia che esistono 17 strumenti e possa scegliere tra quelli. |
| **`S2S_REASONING_TYPE`** | `PROXY` \| `OPENAI` | Motore di reasoning (§3). Normalizzato con `_normalize_reasoning_type()` (case-insensitive). |
| **`OPENAI_API_KEY` / `BASE_URL` / `REASONING_MODEL`** | stringhe | Solo per mode OPENAI. `BASE_URL` serve anche per compatibilità Azure/Together (basta che parli /chat/completions). |
| **`S2S_PROXY_TARGET`** | URL WS | Upstream realtime che fa STT/TTS + (in mode PROXY) il reasoning vero e proprio. Di solito `ws://ai-server:12345/v1/realtime`. |
| `S2S_PRELOAD_DATA_ENDPOINT`, `SERPER_API_KEY`, `LOAD_BALANCER_URL`, `SPEECH_TO_SPEECH_URL` | varie | Non iniettano testo nel prompt direttamente, ma abilitano altre funzionalità (PowerBI preload, web search Serper, queue & limiter, direct URL fissa). |
---
### 🅿️ Punto 2 — **Aquachile `/api/v1/agents/profile`** (contesto Aziendale + Tools)
URL impostato da `S2S_AGENT_PROFILE`. Fetchato lato server in [_agent_profile_data()](file:///home/duxiter/s2s/demo/server.py#L357-L397) con:
- cache 60 secondi globale (`_agent_profile_cached`), lock asincrona
- timeout 10 s
- chiamata GET senza header speciali (aggiungibili se serve a mano nel codice)
**Formato JSON atteso** (si adatta in modo tollerante):
```jsonc
{
"agent": {
// ↓ usato come prompt principale (priority #1)
"instructions": "Eres el asistente de voz oficial de Aquachile. Te llamas Aquachile Assistant…",
// ↓ fallback se instructions non esiste
"description": "…",
"tools": [
{
"name": "get_day_report",
"description": "Obtiene el reporte de producción diario de una planta…",
"parameters": {
"type": "object",
"properties": {
"center": {"type": "string", "description": "Nombre de la planta"}
},
"required": []
},
"enabled": true // ↓ se missing o true → incluso; se false → ignorato
},
// … altri 16 strumenti
]
}
}
```
Parser corrispondente:
- [_extract_agent_prompt()](file:///home/duxiter/s2s/demo/server.py#L405-L424) — cerca in ordine: `agent.instructions` → `agent.description` → `instructions` radice → `description` radice.
- [_extract_agent_tools()](file:///home/duxiter/s2s/demo/server.py#L427-L437) — cerca `agent.tools[]`, filtra quelli con `enabled` != `false`.
Tutto il resto del payload viene ignorato. Se vuoi estrarre altri campi (es. `agent.name`, `agent.greeting`, `agent.language`) puoi aggiungerne l'estensione in `_extract_agent_prompt` per concatenarli, oppure creare un nuovo campo nel dizionario di ritorno e poi usarlo in `/api/config`.
**Quando modificare questo endpoint piuttosto che altre parti:**
- Quando il contenuto cambia spesso (ogni giorno: nuovi report, nuove tipologie di domanda).
- Quando vuoi che **tutte le installazioni** dell'UI aggiornino all'istante senza rebuild/restart del server.
- Quando vuoi rilasciare un **nuovo tool MCP**: basta aggiungerlo all'array `tools[]` e comparirà ovunque entro 60 s (prossima invalidazione della cache).
---
### 🅿️ Punto 3 — **URL parameters** della pagina (contesto per-chiamata)
Analizzati all'avvio di `main.js` ([riga 410](file:///home/duxiter/s2s/demo/main.js#L410)):
```js
const pageParams = new URLSearchParams(window.location.search);
const voceroUserPhoneNumber = (pageParams.get("phoneNumber") || "").trim();
```
Attualmente è gestito un solo param:
| Parametro URL | Tipo | Effetto |
|---|---|---|
| `?phoneNumber=+56912345678` | stringa E.164 | Triggera `fetchVoceroUserProfile()` → `GET /api/user/phone?phoneNumber=…` → endpoint Aquachile `S2S_USER_PHONE_AUTH_URL` → profilo JSON appenduto al prompt (vedi Punto 4). |
È molto semplice estendere la lista: aggiungi una riga tipo:
```js
const plantId = (pageParams.get("plant") || "").trim();
```
e poi inietta il valore in modo statico (in `voceroUserContextText()` o in un pezzo di testo a parte in `effectiveReasoningInstructions`) o fai una fetch al tuo endpoint.
---
### 🅿️ Punto 4 — **Profilo chiamante `/api/user/phone`** (contesto per-caller)
End-to-end:
```
URL: https://<app>/?phoneNumber=+56912345678
│
▼ fetchVoceroUserProfile() [main.js:2370-2398]
GET api/user/phone?phoneNumber=%2B56912345678
│
▼ user_by_phone() [server.py:539-573]
GET {S2S_USER_PHONE_AUTH_URL}?phoneNumber=%2B56912345678
│
▼
Risposta Aquachile (es.)
{
"id": 987,
"name": "Don Manuel",
"role": "Jefe de Planta",
"center": "Puerto Montt",
"email": "manuel@aquachile.cl",
"authorizations": ["report:daily", "report:weekly"]
}
│
▼ { "result": <payload> } al browser
│
▼ voceroUserProfile = <payload> ; voceroUserProfileLoaded = true
│
▼ voceroUserContextText() [main.js:454-458]
" Caller profile resolved from the page phoneNumber: {name:\"Don Manuel\",…}. Use this as trusted caller context when relevant."
│
└─► concatenato ALLA FINE di effectiveReasoningInstructions
```
Una volta che il profilo è arrivato, viene chiamato `pushLiveInstructions(effectiveInstructions())` → l'istruzione sul realtime viene **aggiornata in volo** anche se la conversazione è già iniziata. Quindi anche se il profilo impiega 2 secondi ad arrivare, le battute successive alla ricezione beneficeranno del contesto.
**Quando usarlo:** passare dati per-caller (ruolo, permessi, centro, nome, lingua preferita, reporting line, storico chiamate recenti ecc.). Tieni conto che il JSON viene serializzato in chiaro nel prompt (max 1600 caratteri), quindi non passare PII sensibili in chiaro.
---
### 🅿️ Punto 5 — **Catalogo MCP Aquachile** (contesto come "abilità" dell'agente)
Fonte: `agent.tools[]` dal profilo Aquachile (Punto 2), poi in `agentToolDefs` lato client.
Entrano nel prompt in due modi:
1. **Come `tools` (dichiarazione formale OpenAI Realtime function-call)** : `S2sWsRealtimeClient({tools: activeToolDefs()})` → arrivano all'upstream come `session.tools[]` con schema JSON Schema. Questo permette all'upstream di **decidere di chiamarli** tramite `response.create_response + function_call_arguments.done`.
2. **Come testo nel prompt vero e proprio** : `Available MCP tools: get_day_report - Obtiene el reporte… Required parameters: []; get_metrics_summary - …. ` (solo se `S2S_SEARCH_MCP=true`).
Quando poi il modello effettivamente **chiama** un tool MCP, il client:
- emette l'evento `toolcall`
- il server chiama `POST /api/mcp/tool` ([server.py](file:///home/duxiter/s2s/demo/server.py#L576…)):
```
POST {S2S_TOOL_EXECUTION_MCP_URL}
headers: X-API-KEY, X-API-SECRET, Content-Type: application/json
body: {"server": "mcp", "name": "get_day_report", "arguments": "{\"center\":\"Puerto Montt\"}"}
```
- Il risultato torna al client e viene rinviato all'upstream come `conversation.item` di tipo `function_call_output`. Il modello legge l'output e prosegue il turn.
---
### 🅿️ Punto 6 — **Settings utente (localStorage)** (solo se prompt non pinned)
Se `pinnedPrompt` è stringa vuota (né da Aquachile, né da env), allora in `effectiveReasoningInstructions()` la base è `settings.instructions` che viene da:
```js
// main.js loadSettings: riga 201
instructions: localStorage.getItem("s2s.ws.instructions") || DEFAULT_INSTRUCTIONS
```
Nel form Settings (pannellino in alto a destra) l'utente può editare il testo "System instructions" e salvarlo → va in localStorage.
⚠️ **Attenzione**: se è presente un prompt "pinned" (da Aquachile o env), il campo nel Settings diventa **read-only** (non è modificabile). Puoi sempre verificare lo stato aprendo il pannello.
---
### 🅿️ Punto 7 — **Aggiungere contesto custom via codice (patch diretta)**
Se vuoi iniettare dati specifici (es. l'ID di una campagna, il turno di lavoro corrente, la stazione meteorologica più vicina, ecc.) senza creare un nuovo endpoint esterno, hai **3 punti comodi** dove patchare con poche righe:
#### 7a. Aggiungere un pezzo di testo statico alle `instructions`
Modifica direttamente `effectiveReasoningInstructions()` in [main.js:516-533](file:///home/duxiter/s2s/demo/main.js#L516-L533), aggiungendo una sezione. Esempio:
```js
function effectiveReasoningInstructions() {
const base = pinnedPrompt || settings.instructions;
let instructions = base;
if (activeToolDefs().length) instructions += TOOL_USE_HINT;
if (searchMcpEnabled && agentToolDefs.length) { … }
const suggested = topSuggestedAgentTools(3);
if (searchMcpEnabled && suggested.length) { … }
const userContext = voceroUserContextText();
if (userContext) instructions += userContext;
// ↓ aggiungi contesto custom QUI
instructions += `\n\nTurno actual: ${TURNO_CORRIENTE}. Campaña: ${CAMPAIGN_ID}. Planta: ${PLANTA_ID}.`;
return instructions;
}
```
Funziona in **entrambe** le modalità (PROXY e OPENAI).
#### 7b. Inviare un messaggio pre-filla a inizio conversazione
Dopo che il client è connesso e pronto (evento `ready` o `status=="connected"`), chiama:
```js
client.sendUserText(
"Contexto adicional de esta llamada (no responder a esto directamente): " +
"El usuario está llamando desde la planta Puerto Montt. Su último reporte " +
"fue el día 2026-08-17 y mostró un 95% de cumplimiento. El turno actual " +
"es el turno noche. Usa esta información cuando respondas."
);
// (opzionale) poi client.requestResponse() se vuoi che l'agente saluti subito
```
Il messaggio entra nella history del modello (coda `conversation.items[]`) ed è visibile per il resto della conversazione.
#### 7c. Patchare `/api/config` lato server per ritornare un campo custom e usarlo dal client
Aggiungi in [server.py ~riga 518-531](file:///home/duxiter/s2s/demo/server.py#L518-L531):
```python
out = {
...,
"myCustomContext": {"turno": current_shift(), "planta": os.environ.get("PLANTA_ID", "")},
}
```
Poi lato client in `fetchConfig()` (circa riga 2408):
```js
myCustomContext = json.myCustomContext || null;
```
e usa il valore come in 7a. Utile quando il contesto lo conosci solo lato server (es. viene da DB aziendale o da variabili env del deploy).
#### 7d. Aggiungere un parametro HTTP `user_context` custom alla chiamata `/api/reasoning` (solo OPENAI mode)
L'endpoint già accetta il campo `user_context` nel body della request — è già popolato da `voceroUserContextText()`, ma puoi aggiungere ulteriore testo concatenandolo nel client o nel server.
Oppure estendi `ReasoningRequest` in [server.py:346-350](file:///home/duxiter/s2s/demo/server.py#L346-L350) con un nuovo campo BaseModel (es. `turno: str = ""`, `planta: str = ""`) e poi usalo in `system_parts.append(...)` dentro `def reasoning()`.
---
## 5. Verifiche e Debug — come controllare "cosa vede veramente l'agente"
### 5a. Log del server (cosa arriva all'esterno)
Su `docker logs -f s2s_ui` (o sull'output di `start_ui.sh`) puoi monitorare TUTTE le chiamate esterne. Filtri utili:
```bash
# Solo contesto caricato all'avvio (profilo Aquachile)
grep "\[EXT\]\[AQUACHILE\]" server.log
# Solo ragionamento OpenAI (se mode OPENAI)
grep "\[EXT\]\[OPENAI\]" server.log
# Dialogo parlato (TESTO della chat)
grep -E "INPUT UTENTE|OUTPUT AGENTE" server.log
# Tutte le chiamate MCP
grep "\[EXT\]\[AQUACHILE\].*mcp.*tools/call" server.log
# /api/config (quello che arriva al browser come initial setup)
grep "/api/config returned" server.log
# Autenticazione chiamante
grep "Vocero USER AUTH" server.log
```
### 5b. Debug lato client (cosa viene assemblato)
Apri la console del browser (F12) e, dopo che la pagina ha caricato e ti sei connesso, puoi eseguire:
```js
// Controlla il tipo di reasoning attivo
> reasoningType
"PROXY" // oppure "OPENAI"
// Stringa effettiva di instruzioni (copiala/incolala per ispezione manuale)
> copy(effectiveInstructions())
// Testo che è stato passato all'ultimo session.update
> liveInstructionsSnapshot.length
32456
// Elenco dei tool attivi che l'agente "vede"
> activeToolDefs().map(t => t.name)
(19) ['web_search', 'camera_snapshot', 'get_day_report', 'get_weekly_report', 'get_monthly_report', 'get_report_detail', … ]
// Profilo chiamante caricato
> voceroUserProfileLoaded, voceroUserProfile
true, { id: 987, name: "Don Manuel", … }
// Top 3 suggeriti per l'ultima domanda
> topSuggestedAgentTools(3).map(x => `${x.tool.name}:${x.score}`)
["get_day_report:38", "get_metrics_summary:22", "get_report_detail:15"]
```
### 5c. Cosa viene mandato a OpenAI in modalità OPENAI
Basta guardare il log `[EXT][OPENAI] reasoning POST … systemLen=… transcriptLen=…` e saprai già il numero di token di sistema. Se vuoi vedere l'esatto system message, puoi aggiungere temporaneamente:
```python
# dentro def reasoning() dopo la costruzione di body
logger.debug("Reasoning system message PREVIEW:\n%s",
body["messages"][0]["content"][:3000])
```
---
## 6. Priorità effettiva di tutte le sorgenti (chi vince se c'è conflitto)
Tabella riassuntiva che risponde a: *"Se due punti dicono cose diverse, quale vince?"*
| Cosa | Vincitore (priorità max → min) |
|---|---|
| **Identità / sistema prompt** | 1. `agent.instructions` da Aquachile `S2S_AGENT_PROFILE` → 2. `S2S_PRIMARY_PROMPT` env → 3. `settings.instructions` (localStorage utente) → 4. `DEFAULT_INSTRUCTIONS` hardcoded |
| **Catalogo MCP** | Aquachile `agent.tools[]` (unico fornitore). Gli unici tool aggiuntivi sono `web_search` e `camera_snapshot` (browser-native). |
| **Hint su come usare i tool** | Sempre `TOOL_USE_HINT` fisso se `tools>0`; se `searchMcpEnabled` si aggiunge `MCP_SEARCH_HINT` + catalogo testuale completo. |
| **Hint tool ranking (top 3)** | Calcolato lato client per ogni battuta; prompt dice "secondary signal, full catalog wins" |
| **Profilo chiamante** | `S2S_USER_PHONE_AUTH_URL` + URL param `phoneNumber` (nessun altro percorso attuale). |
| **Modalità di reasoning** | 1. `S2S_REASONING_TYPE` env normalizzato (`OPENAI` o default `PROXY`) |
| **URL endpoint upstream** | 1. `SPEECH_TO_SPEECH_URL` pinned direct → 2. `LOAD_BALANCER_URL` LB + queue → 3. URL inserito manualmente in Settings (solo allowDirect) |
---
## 7. Troubleshooting: "L'agente non usa il contesto che mi aspetto"
Checklist da fare in ordine:
1. **`/api/config` cosa restituisce?**
```
curl -s http://localhost:8000/api/config | python -m json.tool
```
Verifica: `primaryPrompt` è valorizzato e lungo? `agentTools` quanti elementi ha? `reasoningType` è quello che ti aspetti?
2. **Aquachile profile risponde?** Guarda `[EXT][AQUACHILE] agent_profile` nel log. Status 200? Cache HIT o MISS? `PARSED promptLen=… toolCount=…` ha i numeri attesi?
3. **Il browser ha `phoneNumber` nella URL?** Apri DevTools → Network e cerca `user/phone`. C'è una richiesta? Torna 200? Che JSON ritorna? Il `result` viene inserito in `voceroUserProfile` (check console).
4. **Hai cambiato qualcosa nel prompt in Aquachile ma non l'aggiornamento non si vede?** Il profilo ha cache 60 secondi. Aspetta o riavvia il server (la cache è in memoria).
5. **In modalità PROXY i tool MCP non vengono chiamati:**
- Verifica che il **tool catalogo testuale** sia nel prompt (controlla `copy(effectiveInstructions())` e cerca la stringa `Available MCP tools`).
- Verifica che `S2S_SEARCH_MCP=true`.
- Guarda nella console del browser `[Vocero MCP MATCH]` e `[Vocero MCP RANK]` — se nessuno score è sopra la soglia (default 20), l'auto-invocazione lato client non scatta; il modello potrebbe comunque chiamarlo via function-call standard.
6. **In modalità OPENAI non sente le MCP:** Corretto — attualmente `/api/reasoning` NON manda tools a OpenAI. Vedi §3.2. Se vuoi chiamare tool in OPENAI mode devi spezzare la catena: OpenAI deve dire quale tool chiamare, tu lo chiami via `/api/mcp/tool`, e poi reinietti il risultato in un nuovo reasoning turn. O, più semplicemente, usa la modalità PROXY quando servono i tool Aquachile.
7. **Output lungo o ripetitivo:** Inserisci in coda al prompt un blocco tipo: `Limita la risposta a 2 frasi e parla in modo naturale. Non ripetere la domanda dell'utente.` — funziona in entrambe le modalità.
---
## 8. Riferimenti rapidi (file e linee)
| Concetto | File e righe |
|---|---|
| `.env` caricato | [start_ui.sh](file:///home/duxiter/s2s/start_ui.sh#L8-L12) |
| Variabili env parsate lato server | [server.py](file:///home/duxiter/s2s/demo/server.py#L71-L120) |
| `ReasoningRequest` body schema | [server.py:346-350](file:///home/duxiter/s2s/demo/server.py#L346-L350) |
| Fetch profilo Aquachile + cache 60 s | [_agent_profile_data](file:///home/duxiter/s2s/demo/server.py#L357-L397) |
| Parse istruzioni + tools dal JSON Aquachile | [_extract_agent_prompt](file:///home/duxiter/s2s/demo/server.py#L405-L424), [_extract_agent_tools](file:///home/duxiter/s2s/demo/server.py#L427-L437) |
| `/api/config` (bootstrap verso client) | [server.py:509-536](file:///home/duxiter/s2s/demo/server.py#L509-L536) |
| `/api/user/phone` (profilo chiamante) | [server.py:539-573](file:///home/duxiter/s2s/demo/server.py#L539-L573) |
| Esecuzione tool MCP (inoltro ad Aquachile) | [server.py /api/mcp/tool](file:///home/duxiter/s2s/demo/server.py#L576-L633) |
| `/api/reasoning` (OpenAI chat completions proxy) | [server.py:635-710](file:///home/duxiter/s2s/demo/server.py#L635-L710) |
| Proxy WS realtime, INPUT/OUTPUT text log | [_client_to_upstream](file:///home/duxiter/s2s/demo/server.py#L1112-L1171), [_upstream_to_client](file:///home/duxiter/s2s/demo/server.py#L1174-L1255) |
| `fetchConfig()` lato client | [main.js:2401-2439](file:///home/duxiter/s2s/demo/main.js#L2401-L2439) |
| `fetchVoceroUserProfile()` phone auth | [main.js:2370-2398](file:///home/duxiter/s2s/demo/main.js#L2370-L2398) |
| `effectiveReasoningInstructions()` assemblaggio prompt | [main.js:516-533](file:///home/duxiter/s2s/demo/main.js#L516-L533) |
| `effectiveInstructions()` (prompt per realtime, + OPENAI marker hint) | [main.js:537-539](file:///home/duxiter/s2s/demo/main.js#L537-L539) |
| Tool hint + MCP search hint costanti | [main.js:32-42](file:///home/duxiter/s2s/demo/main.js#L32-L42) |
| `voceroUserContextText()` caller profile injection | [main.js:454-458](file:///home/duxiter/s2s/demo/main.js#L454-L458) |
| `fetchOpenAiReasoningAnswer()` POST /api/reasoning client | [main.js:563-586](file:///home/duxiter/s2s/demo/main.js#L563-L586) |
| `maybeRunOpenAiReasoning()` trigger da transcript + inject `[VOCERO_RENDER_REPLY]` | [main.js:592-625](file:///home/duxiter/s2s/demo/main.js#L592-L625) |
| `pushLiveInstructions()` / `pushLiveToolPolicy()` update sessione live | [main.js:628-649](file:///home/duxiter/s2s/demo/main.js#L628-L649) |
| `doStart()` avvio sessione + invio istruzioni/tools iniziali | [main.js:2739-2788](file:///home/duxiter/s2s/demo/main.js#L2739-L2788) |
| Gestore transcript utente → trigger reasoning + ranking MCP | [main.js:2814-2821](file:///home/duxiter/s2s/demo/main.js#L2814-L2821) |
s2s-client/ARCHITECTURE.md
-188
View File
@@ -1,188 +0,0 @@
# Demo S2S — schema di funzionamento
> Documento orientativo del modulo `demo/` (UI + proxy S2S + reasoning). Non sostituisce
> il codice: serve a trovare il proprio posto quando si apre il progetto.
---
## 1. Cosa c'è in `demo/`
Il modulo fa **tre cose contemporaneamente**:
| Ruolo | Cosa fa | Entry point |
|---|---|---|
| **HTTP server statico + API** | Serve l'UI, offre endpoint REST, gira un WebSocket proxy | [server.py](file:///home/duxiter/s2s/demo/server.py) |
| **Voice UI (browser)** | Orba, mic, transcriptioni, audio playback, pannelli | [index.html](file:///home/duxiter/s2s/demo/index.html) + [main.js](file:///home/duxiter/s2s/demo/main.js) |
| **Reasoning hook** | Inoltra transcript utente a un motore di ragionamento esterno (NODE_RED o OpenAI) | [server.py](file:///home/duxiter/s2s/demo/server.py#L658-L834) `/api/reasoning` |
La configurazione (URL backend S2S, tipo di reasoning, chiavi API) viene presa dal file `.env` nella root del progetto.
---
## 2. Architettura in un diagramma
```
┌──────────────────────────────────────────────────┐
│ BROWSER (demo UI) │
│ [index.html] → [main.js] (ESM type=module) │
│ │ ▲ │
│ │ audio + WS events │ transcript chunks │
│ ▼ │ │
│ ┌──────────────────────────────────────────┐ │
│ │ s2s-ws-client.js → WebSocket /v1/realtime │ │
│ │ + codec.js + orb-visualizer.js │ │
│ └──────────────────────────────────────────┘ │
└─────────────────────────┬────────────────────────┘
│ WebSocket /v1/realtime
▼
┌───────────────────────────────────────────────┐
│ server.py (FastAPI) │
│ │
│ ┌──────────────────────────────────────┐ │
│ │ realtime_proxy ← TUNNEL → upstream │ │
│ │ (WS relay puro, copia byte-per-byte)│ │
│ │ _client_to_upstream ─┐ │ │
│ │ _upstream_to_client ─┤ session_ctx │ │
│ │ session_ctx{user,id} │ shared dict │ │
│ └───────────────────────┴──────────────┘ │
│ │
│ endpoint REST: │
│ GET / → index.html │
│ GET /api/config → env flags │
│ POST /api/reasoning → NODE_RED / │
│ OPENAI │
│ POST /api/auth/* → Aquachile │
│ │
└──────┬──────────────────────────────┬─────────┘
│ POST /api/reasoning │ fire-and-forget
▼ ▼
┌──────────────────────┐ ┌───────────────────────────────┐
│ Node-RED (port 1881)│ │ Node-RED (port 1881) │
│ POST /api/reasoning │ │ POST /api/log/system-response │
│ log user transcript │ │ log assistant TTS + pairing │
└──────────────────────┘ └───────────────────────────────┘
^
│ forward 4 eventi realtime:
│ audio_transcript.done
│ content_part.done
│ output_item.done role=assistant
│ conversation.item.created role=assistant
```
> ### 📝 Nota importante sul flusso AUDIO con l'attuale `.env`
>
> Con `.env` attuale che ha `S2S_PROXY_TARGET=ws://ai-server:12345/v1/realtime`:
>
> ✅ **Sì, TUTTO l'AUDIO utente (PCM16 16kHz mono, eventi `input_audio_buffer.append`) arriva a `ws://ai-server:12345/v1/realtime`** — ma **non direttamente**: passa SEMPRE per il **tunnel 1:1 WebSocket di `server.py`** (`_client_to_upstream`), che copia byte-per-byte senza modifiche.
>
> ✅ Allo stesso modo l'AUDIO risposta (TTS, `response.output_audio.delta`) ritorna da `ai-server:12345` → tunnel `_upstream_to_client` → browser → playback nativo.
>
> ❌ **Cosa invece NON va a `ai-server:12345`** (canale HTTP separato parallelo, non tocca l'audio):
> - Il **transcript STT** viene mandato via HTTP a `http://localhost:1881/api/reasoning` (Node-RED, hook LOG_ONLY oggi)
> - Il **testo risposta TTS** viene mandato via HTTP a `http://localhost:1881/api/log/system-response` (Node-RED, forward fire-and-forget)
> - Chiamate **auth / profilo Aquachile** verso `aquachile.azurianlab.com`
>
> Questi 3 flussi REST sono **aggiuntivi** e **non alterano minimamente il tunnel audio**: browser ↔ server.py ↔ `ai-server:12345` rimane il percorso esclusivo per voce e risposta.
---
## 3. Cose importanti di [server.py](file:///home/duxiter/s2s/demo/server.py)
### File → ruoli
- **`/`** → StaticFast `StaticFiles` su `demo/` (index.html + assets)
- **`@app.middleware http` no_cache_static_assets** → `Cache-Control: no-store` per `.js/.html/.css/.mjs/.wasm` per sconfiggere cache browser durante sviluppo
- **`@app.get /api/config`** → manda al browser variabili env (`S2S_REASONING_TYPE`, `S2S_PROXY_TARGET`, auth, features). Il client decida da sé cosa mostrare in UI.
- **`@app.post /api/reasoning` ([L658-L834](file:///home/duxiter/s2s/demo/server.py#L658-L834))** → cuore del reasoning. Due modalità, scelta da env `S2S_REASONING_TYPE`:
- **Modalità A — NODE_RED**: inoltra `POST {NODE_RED_URL}/api/reasoning` timeout 30s. Accetta `log_only=true` + `answer=""` (Fase 1).
- **Modalità B — OPENAI**: chiamata diretta `POST $OPENAI_BASE_URL/chat/completions` (Fasi successive).
- **`@app.websocket /v1/realtime` ([L874-L980](file:///home/duxiter/s2s/demo/server.py#L874-L980))** → **tunnel S2S nativo**. Non traduce messaggi: copia + registra + forwarda. 2 task paralleli:
- `_client_to_upstream(ws, upstream, client_ip, session_ctx)`: browser → AI server
- `_upstream_to_client(ws, upstream, client_ip, session_ctx)`: AI server → browser
- `session_ctx = {last_user_transcript, last_item_id}` **dizionario mutabile condiviso** per accoppiare input↔output
- **`async def _forward_agent_text_to_nodered` ([L837-L873](file:///home/duxiter/s2s/demo/server.py#L837-L873))** → helper fire-and-forget. Tutti errori → `logger.debug` (non spezza il tunnel WS).
- **`_upstream_to_client` hook eventi ([L1305-L1500](file:///home/duxiter/s2s/demo/server.py#L1305-L1500))**: per ogni evento output agente avvia `asyncio.create_task(_forward_agent_text_to_nodered(...))` → non blocca lo streaming audio.
- **Sidecar Python locali**: [auth.py](file:///home/duxiter/s2s/demo/auth.py) (Aquachile profile+user+phone), [limiter.py](file:///home/duxiter/s2s/demo/limiter.py) (rate limiter).
### Ordine di chiamata in un turno vocale
1. Browser clicca orb → `ws /v1/realtime` → `realtime_proxy()` accetta
2. Browser manda `session.update` → `input_audio_buffer.append` in loop
3. Upstream genera `conversation.item.input_audio_transcription.completed` → server cattura → **`session_ctx["last_user_transcript"] = transcript`**
4. `_upstream_to_client` vede transcript STT completo → server invoca in parallelo **`POST /api/reasoning`** con quel transcript
5. Upstream manda `response.output_audio.delta` audio chunks → tunnelati diretti al browser → playback nativo
6. Upstream manda `response.output_audio_transcript.done` → server cattura → `create_task(_forward_agent_text_to_nodered({tts, source_event, user_transcript, …}))` → Node-RED SYSTEM_RESPONSE log
7. Audio finisce → turno chiuso
---
## 4. Cose importanti del lato client ([main.js](file:///home/duxiter/s2s/demo/main.js))
### Moduli ESM importati (tutti con `?v=VERSIONE` per cache-bust)
```
main.js
├─ dom.js → helper DOM (create, qs, bubble, ecc.)
├─ chat.js → pannello messaggi + appendAssistantText
├─ account.js → pannello auth + profilo utente
└─ ws/s2s-ws-client.js
├─ codec.js → codec pcm + base64
└─ orb-visualizer.js → animazione orba su eventi volume/state
```
### Funzioni chiave
| Funzione | Ruolo |
|---|---|
| `fetchOpenAiReasoningAnswer(transcript)` | `POST /api/reasoning`, ritorna `{answer, log_only, raw}` |
| `maybeRunOpenAiReasoning(transcript, itemId)` | guardia "ho già processato questo turn?" (`processedReasoningTurns`). Se `result.log_only === true` → **early return, zero side-effect**. Altrimenti branch standard (Fasi successive). |
| `cancelActiveResponse()` | Fasi successive: ferma risposta corrente prima di inject custom |
| `appendAssistantText(answer)` / `sendUserText(renderMarker)+requestResponse` | Fasi successive: sostituisce testo risposta backend nativo |
| `speechSynthesis.speak(utterance)` | Fasi successive: TTS *browser* per answer custom, invece del backend nativo |
### Regole per non rompere Fase 1
- Se `S2S_REASONING_TYPE=NODE_RED` **e** Node-RED torna `log_only=true` → il client non tocca nulla. Zero `cancelActiveResponse`, zero `speechSynthesis.speak`, zero `conversation.item.create` manuali. La voce è quella del backend S2S nativo.
- **Cache busting triplo**: `index.html` meta tag → `Cache-Control no-store` middleware → query-string `?v=20260825003` su ogni import.
---
## 5. Node-RED (Fase 1 — LOG_ONLY)
Risiede in [`s2s-reasoning-engine/flows.json`](file:///home/duxiter/s2s/s2s-reasoning-engine/flows.json). Espone 2 endpoint completamente indipendenti:
| Endpoint | Payload | Output Node-RED |
|---|---|---|
| `POST /api/reasoning` | `{transcript, language, phoneNumber, …}` | `[S2S NODE_RED LOG_ONLY] ts=… lang=… phone=… transcript="..."` |
| `POST /api/log/system-response` | `{tts, source_event, user_transcript, role, item_id, …}` | `[S2S NODE_RED SYSTEM_RESPONSE] ts=… phone=… user="..." tts="..."` |
Entrambi ritornano 200 `{log_only:true, intent:"..."}`.
---
## 6. Worklets (non toccare)
- [worklets/mic-capture.js](file:///home/duxiter/s2s/demo/worklets/mic-capture.js) — sample-rate input 16kHz PCM mono, calcolo RMS per VAD visuale.
- [worklets/audio-playback.js](file:///home/duxiter/s2s/demo/worklets/audio-playback.js) — riproduzione audio 24kHz server side.
> Nota: i worklets vengono caricati via `AudioWorklet.addModule()`, che usa script-context **non-ESM**. Non aggiungere `export function` a questi file: rompono il caricamento.
---
## 7. Variabili ambiente che comandano tutto
| Nome | Valore tipico oggi | Cosa fa |
|---|---|---|
| `S2S_REASONING_TYPE` | `NODE_RED` (Fase 1) / `OPENAI` (Fasi succ.) / `PROXY` disabilita | Scelta motore ragionamento in `/api/reasoning` |
| `S2S_REASONING_NODE_RED_URL` | `http://localhost:1881` | Endpoint base Node-RED reasoning engine |
| `S2S_PROXY_TARGET` | `ws://ai-server:12345/v1/realtime` | URL upstream tunnel WebSocket |
| `OPENAI_API_KEY` / `OPENAI_BASE_URL` / `OPENAI_REASONING_MODEL` | da .env | Usati in Modalità B OPENAI |
| `S2S_AGENT_PROFILE` / `S2S_USER_PHONE_AUTH_URL` / `S2S_TOOL_EXECUTION_MCP_URL` | `aquachile.azurianlab.com/api/v1/…` | URL provider esterni Aquachile |
| `LOG_LEVEL` | `INFO` / `DEBUG` | Livello uvicorn |
---
## 8. Dove toccare per…
| Obiettivo | File da aprire |
|---|---|
| Cambiare colori tema orb / UI | [style.css](file:///home/duxiter/s2s/demo/style.css) `:root` + [DESIGN.md](file:///home/duxiter/s2s/demo/DESIGN.md) |
| Aggiungere un pannello in UI | [index.html](file:///home/duxiter/s2s/demo/index.html) markup + [main.js](file:///home/duxiter/s2s/demo/main.js) render + nuovo modulo in [ui/](file:///home/duxiter/s2s/demo/ui) |
| Aggiungere un endpoint REST | [server.py](file:///home/duxiter/s2s/demo/server.py), dopo le route esistenti |
| Nuova logica Node-RED Fase 2 | [flows.json](file:///home/duxiter/s2s/s2s-reasoning-engine/flows.json) → Node-RED editor → export |
| Passare da LOG_ONLY a produzione di answer | Flip in flows.json `log_only=false` + riempire `answer`, il branch `result.log_only` del client si spegne automaticamente |
| Debugging SyntaxError browser `Unexpected token 'export'` | 1) Controlla `syntax AST` server.py (se crasha → 500 → browser riceve HTML invece di main.js) 2) Hard reload DevTools ✓ Disable cache 3) `node --check main.js` e import |
s2s-client/CHANGELOG.es.md
-859
View File
@@ -1,859 +0,0 @@
# Changelog — s2s-client (Azurian Vocero)
Todos los cambios relevantes del proyecto están documentados en este archivo.
Formato basado en [Keep a Changelog](https://keepachangelog.com/es-ES/1.1.0/).
---
## [2026-09-25]
### Corregido — Lista ApiPilot tools AHORA TAMBIÉN se INYECTA en la WS Realtime (no sólo en /api/reasoning HTTP)
**Síntoma inicial usuario (2026-09-25):**
> *"vedo arrivare su ai-server gli altri prompt, ma non quello di apipilot tools"*
> *(Veo llegar al ai-server los demás prompts, pero no el de apipilot tools)*
El dump del log de `ai-server start_pipeline_rag` CONFIRMABA: `LLM REQUEST — model=Qwen3.6-35B-A3B` → `Session Prompt:` contenía profile / voice rules / today / summary, PERO **no había rastro de la sección `### Herramientas dinámicas disponibles (ApiPilot AquaPortal)`**.
#### Root cause (3 bugs actuando juntos):
1. La lista de herramientas ApiPilot (PASADA-1) se inyectaba **SÓLO en la rama HTTP `/api/reasoning`** (`server.py:L3078-L3079, L3286-L3287`) pero **NUNCA en el builder de `session.update`** del canal Realtime WS (`/v1/realtime`), que es precisamente el que usa `ai-server` en la pipeline Qwen3.5-35B de voz.
2. El gate `S2S_INJECT_PROFILE` dejaba fuera ApiPilot en la rama `=false`.
3. Ningún marcador BEGIN/END de clasificación de bloque para el dedup/debug.
#### Fix aplicado hoy (server.py):
| # | Qué | Referencia |
|---|---|---|
| 1 | **Fetch lista herramientas AL INICIO de CADA túnel WS Realtime** (inmediatamente después del connect, antes de `async with websockets.connect(upstream_url)`) | [server.py#L3930-L3973](file:///home/duxiter/s2s/s2s-client/server.py#L3930-L3973) |
| 2 | **Independiente de `S2S_INJECT_PROFILE`**. Ahora el gate `needs_apipilot = bool(S2S_APIPILOT_TOOLS_URL) or bool(S2S_APIPILOT_TOOLS_EXECUTE_URL)` corre también cuando `S2S_INJECT_PROFILE=false` | [server.py#L3813](file:///home/duxiter/s2s/s2s-client/server.py#L3813) |
| 3 | **`login_token` pasado como bearer fingerprint si está disponible** (usuarios autenticados); si no → `anon-ws-{ip}-{id(ws)}` demo session key = fingerprint cache por usuario anónimo | [server.py#L3939-L3942](file:///home/duxiter/s2s/s2s-client/server.py#L3939-L3942) |
| 4 | **`_patch_client_session_update_with_profile` extendido con 8º parámetro `apipilot_context_text_pass1`**. Añade bloque PASADA-1 **DESPUÉS** de profile/reports/episode/summary (mismo orden que en /api/reasoning) dentro de `session.instructions` (lugar que el LLM ve siempre con máxima prioridad). Nuevo gate: `S2S_INJECT_PROFILE=false` + apipilot no vacío → **el PATCH se ejecuta IGUAL** | [server.py#L2560-L2713](file:///home/duxiter/s2s/s2s-client/server.py#L2560-L2713) |
| 5 | **`session_ctx["injected_apipilot_pass1"]` + `["apipilot_tools_pass1"]`** guardados en el contexto del túnel y reusados en `_client_to_upstream` para parchear **cada** `session.update` del cliente (incluyendo parches tardíos) | [session_ctx @ server.py#L4028-L4042](file:///home/duxiter/s2s/s2s-client/server.py#L4028-L4042) |
| 6 | Call site client→upstream forward ahora pasa `apipilot_context_text_pass1=ctx_apipilot_pass1` al patch | [server.py#L4780-L4790](file:///home/duxiter/s2s/s2s-client/server.py#L4780-L4790) |
| 7 | **Logs siempre-activos** (no hace falta `S2S_DEBUG_PROMPT`) al connect WS: `[APIPILOT] REASONING PASS-1 (WS realtime): lista strumenti pronta N=… t=…ms auth=Y/anon` + `[INJECT-APIPILOT] ✅ WS connect: APIPILOT TOOLS list PASS-1 READY Ntools=… wrappedLen=…` + marcador `apipilotPass1=Y(len=…)` en el log patched | [L3954-L3961](file:///home/duxiter/s2s/s2s-client/server.py#L3954-L3961) · [L2668-L2676](file:///home/duxiter/s2s/s2s-client/server.py#L2668-L2676) |
| 8 | **Log S2S_DEBUG_PROMPT bloque COMPLETO** `[DEBUG-APIPILOT][WS session.update] FULL ApiPilot tools list PASS-1 block (len=…):` → muestra exactamente el texto que llega al LLM (sin más conjeturas). | [server.py#L2693-L2696](file:///home/duxiter/s2s/s2s-client/server.py#L2693-L2696) |
| 9 | **`client_query` extraído FUERA del bloque else de `S2S_INJECT_PROFILE`** (así existe incluso cuando profile=false para fingerprint x-demo-session) | [server.py#L3810-L3818](file:///home/duxiter/s2s/s2s-client/server.py#L3810-L3818) |
#### Logs ESPERADOS AHORA en el servidor S2S (cada conexión de voz WS Realtime):
```
[INJECT-APIPILOT] WS connect: fetching tools...
[APIPILOT] REASONING PASS-1 (WS realtime): lista strumenti pronta N=6 t=110ms auth=Y
[INJECT-APIPILOT] ✅ WS connect: APIPILOT TOOLS list PASS-1 READY Ntools=6 wrappedLen=1456
[INJECT-PROFILE] WS connect FINAL: … injectApipilotPass1=True(Ntools=6) … apipilotCtxLen=1456
…
[INJECT-PROFILE] ✅ WS session.update PATCHED → upstream instructions. origLen=1000 newLen=2456 injectedLen=1456 … apipilotPass1=True(len=1456)
```
Y en el log **ai-server `start_pipeline_rag`**, dentro de `>>> PROMPT: … Session Prompt:` al final, APARECERÁ AHORA:
```
<<<APIPILOT_TOOLS_LIST_BEGIN>>>
### Herramientas dinámicas disponibles (ApiPilot AquaPortal)
REGLAS EJECUCIÓN OBLIGATORIAS:
1. … (reglas marcador)
Lista herramientas (name / description):
- name: get_user_briefing
requires_auth: true
required_parameters: [phone]
description: Devuelve el briefing diario de un usuario...
<<<APIPILOT_TOOLS_LIST_END>>>
```
---
### Corregido — 12+ fixes secuenciales PASADA-2 marker CALL, sanitize, flujo auth, SUPER API KEY bypass (25/09/2026)
**Secuencia de bugs corregidos hoy, después del inject PASADA-1 WS (todos verificados con py_compile OK):**
| # | Bug síntoma | Root cause | Fix | Referencia |
|---|---|---|---|---|
| 1 | AI emite texto social "Claro que sí. Voy a generar..." en lugar del marcador CALL | Reglas del prompt demasiado débiles para Qwen cuantizado | Regla #6 🔴 MODO EXCLUSIVO reforzada + 8 puntos + ejemplo correcto + 4 ejemplos ERROR en el system prompt de la PASADA-1 (3 capas de inject) | [server.py:L641-L781](file:///home/duxiter/s2s/s2s-client/server.py#L641-L781) + [s2s-ws-client.js:L119-L232](file:///home/duxiter/s2s/s2s-client/s2s-ws-client.js#L119-L232) |
| 2 | Marcador `<<<APIPILOT_CALL:NAME=...>>>` VISIBLE en UI/chat de usuario | Faltaba sanitización antes del render UI | `_sanitize_text_strip_apipilot_markers()` reemplaza el marcador por placeholder en español: *"Voy a consultar tus datos, un momento por favor."* | [server.py:L783-L837](file:///home/duxiter/s2s/s2s-client/server.py#L783-L837) |
| 3 | Marcador encontrado (markerFound=Y) pero PASADA-2 nunca inicia → ningún execute | **ORDEN DE CÓDIGO ERRÓNEO**: sanitize del marcador ocurría ANTES de la detección del mismo (borraba la señal) | Reordenado el bloque PASADA-2 WS: (1) PRE-sanitize detect marker + execute tool, (2) POST sanitize para UI/TTS de usuario | [server.py:L5608-L5760](file:///home/duxiter/s2s/s2s-client/server.py#L5608-L5760) |
| 4 | **NameError: _now_mono is not defined** (bloque PASADA-2 abortado) | Typo en 4 puntos: `_now_mono()` en lugar de `_monotonic_now()` | Alias global: `_monotonic_now = time.monotonic` + todos los typo corregidos | [server.py:L282](file:///home/duxiter/s2s/s2s-client/server.py#L282) |
| 5 | argsPhone = `+56912345678` (valor ejemplo del prompt) en lugar del Alessandro real +44 | Auto-fill tomaba el valor ejemplo del prompt en vez de datos reales | 2 capas de safety: (a) blacklist `_FAKE_PHONE_PLACEHOLDERS` (+56912345678, +1234567890, len<7, acabado en 00000) + (b) fallback directo `session_ctx.inject_phone` desde QS `?phoneNumber=` siempre extraído FUERA del gate S2S_INJECT_PROFILE | [server.py:L5614-L5636](file:///home/duxiter/s2s/s2s-client/server.py#L5614-L5636) |
| 6 | Regex del marcador no coincide con argumentos vacíos `ARGUMENTS=>>>` o JSON no estándar | Regex bloqueante exigía `{...}` obligatorio | Regex hecha flexible `.*?` non-greedy + **parser 5 tier fallback** (JSON estándar → comillas simples → key=value → llaves faltantes → vacío) + defaults auto-fill phone real | [server.py:L562-L638](file:///home/duxiter/s2s/s2s-client/server.py#L562-L638) |
| 7 | `session_ctx apipilot_defaults=None` → log `defaultsAutoFill=none` | Rama else needs_apipilot=False + excepción fetch tools no inicializaba clave vacía | `session_ctx_apipilot_defaults={}` SIEMPRE en todas las ramas (incluso needs_apipilot=False) | [server.py:L4094-L4108](file:///home/duxiter/s2s/s2s-client/server.py#L4094-L4108) |
| 8 | **404 route POST .../tools/execute** AquaPortal no existe | `S2S_APIPILOT_TOOLS_EXECUTE_URL` era ".../apipilot/tools" + append `/execute` → ruta errónea | `.env L45` corregido a `.../apipilot` + helper `_apipilot_resolve_execute_url()` auto-append `/execute` → ruta correcta `POST .../apipilot/execute` | [.env:L44-L45](file:///home/duxiter/s2s/.env#L44-L45) + [server.py:L321-L338](file:///home/duxiter/s2s/s2s-client/server.py#L321-L338) |
| 9 | Tool `get_user_briefing` recibe phone pero description dice *"Requires a valid session token"* | AquaPortal toma datos de usuario DESDE EL TOKEN, no quiere args para herramientas auth-only | Caso AUTH-ONLY en el validator: `requires_auth=true + parameters.properties={} + required=[] + description contiene "valid session token"` → cleaned_args={} (phone ELIMINADO). Log siempre activo. | [server.py:L518-L593](file:///home/duxiter/s2s/s2s-client/server.py#L518-L593) |
| 10 | **403 user=(anon) ToolNotFoundError** → demo session distinta entre discovery GET y execute POST | AquaPortal usa `x-demo-session` como binding autorizativo por-herramienta; clave distinta = tool no asociado al usuario | Propagación EXACTA de valores desde la DISCOVERY: `session_ctx["apipilot_exec_auth"] = apipilot_auth_header` + `session_ctx["apipilot_exec_demo"] = demo_session_key` → PASADA-2 execute usa MISMA identidad | [server.py:L3995-L4288](file:///home/duxiter/s2s/s2s-client/server.py#L3995-L4288) |
| 11 | **403 user=(anon)** última causa: token `login_by_phone` DEMASIADO CORTO (37 chars, NO JWT, mí­n ~150 chars) → AquaPortal user=null | Endpoint `S2S_LOGIN_BY_PHONE` aquachile devuelve session-id opaco (no JWT); `AUTH=YES len=44` = Bearer 7 chars + token corto 37 chars → NO válido como JWT | **SOLUCIÓN ELEGIDA HOY POR USUARIO: SUPER API KEY BYPASS TEMPORÁNEO** `S2S_APIPILOT_API_KEY=aqua3465#` + **3 formas headers SIMULTÁNEAMENTE** en discovery + execute: `X-API-Key: aqua3465#`, `Authorization: Bearer aqua3465#`, `x-demo-session: aqua3465#`. **PRIORIDAD MÁXIMA (encima de cualquier login).** Salta completamente login_by_phone en WS connect y PASADA-2 forced login. | [.env:L46-L50](file:///home/duxiter/s2s/.env#L46-L50) + [server.py:L259-L267](file:///home/duxiter/s2s/s2s-client/server.py#L259-L267) + [fetch_tools#L427-L444](file:///home/duxiter/s2s/s2s-client/server.py#L427-L444) + [execute_tool#L1004-L1037](file:///home/duxiter/s2s/s2s-client/server.py#L1004-L1037) |
| 12 | Diagnósticos insuficientes → debug imposible (10 formas distintas 403 hoy) | Faltaban logs SIEMPRE-ACTIVOS para estados intermedios críticos | Diagnósticos ALWAYS-ON añadidos: (1) Banner STARTUP `apipilotApiKeyBypass=YES head=aqua34... (3 FORME!)`, (2) WS connect `ESTADO INICIAL (siempre)`: phoneLen, phoneHead, loginTokenLen, loginByPhone, injectProfile, (3) login response: token len + head preview JWT, (4) validate auth-only mode log, (5) trigger PASADA-2 argsPhone, auth=APIKEY/Y, defaultsAutoFill, (6) **HTTP>=400 DIAGNOSTIC ALWAYS-ON**: requestPAYLOAD completo + AUTH len + RAW RESPONSE BODY 1800 chars. | [banner#L1170-L1198](file:///home/duxiter/s2s/s2s-client/server.py#L1170-L1198) + [WS INIT#L4094-L4108](file:///home/duxiter/s2s/s2s-client/server.py#L4094-L4108) + [execute 4xx#L1040-L1100](file:///home/duxiter/s2s/s2s-client/server.py#L1040-L1100) |
#### Orden de prioridad AUTH hoy DEFINITIVO (MÁX → MÍN):
1. 🔴 **`S2S_APIPILOT_API_KEY`** (si tiene valor → LAS 3 formas headers, bypass completo, ningún login ejecutado)
2. 🟠 `S2S_APIPILOT_SERVICE_TOKEN` (override manual desde env)
3. 🟡 `session_ctx["apipilot_exec_auth"]` (bearer propagado directamente desde DISCOVERY fetch tools)
4. 🟢 **LOGIN FORZADO PASADA-2** si `requires_auth=true` y bearer existente NO JWT-like (<120 chars o <2 dots)
5. ⚫ ninguno (anónimo, sólo tools con requires_auth=false)
---
### Corregido — TTS anti-leak MASIVO: 4 bloques sanitize universales + Tier6 phone + reglas prompt 9/10 anti marker pronunciado
**Secuencia síntomas usuario hoy (post auth bypass SUPER API KEY) todos verificados en logs `start_pipeline_rag` y resueltos:**
| # | Síntoma leak TTS | Root cause | Fix aplicado | Referencia |
|---|---|---|---|---|
| 1 | **TTS pronunciaba los marcadores PASADA-2 INTERNOS** `APIPILOT_RESULT:PASS2` + `### Resultado de la herramienta ApiPilot` + bloque ``` ```tool-output``` ``` + trailer *"Ahora debes reinterpretar…contenido final al usuario."* | `_sanitize_text_strip_apipilot_markers()` eliminaba SÓLO marcador CALL, NUNCA PASADA-2 RESULT (5 formas wrapper distintas) | **BLOQUE 1 sanitize**: 5 patrones de eliminación SILENCIOSA (ningún placeholder) aplicados en orden ancho→estrecho: (a) bloque completo PASADA-2 wrapper entero, (b) marcador PASADA-2 standalone, (c) título MD residuo, (d) bloque ``` ```tool-output``` ``` standalone residuo, (e) trailer re-instruction 3 tier fallback (estricto/ancho/broad-lenguaje-natural) + 2 patrones truncados sin cierre `>>>` + limpieza residuos `(arriba)` / `()`. | [server.py:L898-L1022](file:///home/duxiter/s2s/s2s-client/server.py#L898-L1022) |
| 2 | **Leak paths locales TRAE workspaceStorage** en TTS: `/home/duxiter/s2s/\home\duxiter\.trae-server\data\User\workspaceStorage\…\long-text\…\file.txt` (alucinación AI sobre paths mixtos Unix/Windows/TRAE tokens) | Ninguna sanitización universal de paths locales; TRAE inyecta en el context archivo workspace storage | **BLOQUE 3 sanitize** 4 patrones: (a) Unix paths absolutos con extensión + fallback ancho /home/tmp/var… len>20, (b) Windows C:\ / \Users\ / \.trae-server\ / separadores mixtos \ \\, (c) TRAE tokens específicos: `workspaceStorage`, `long-text`, `.trae-server`, `.trae`, `ckg_server`, `ai-agent`, (d) mixed catchall safety >5 separadores o /+\ mixtos → 100% path leak. | [server.py:L1022-L1077](file:///home/duxiter/s2s/s2s-client/server.py#L1022-L1077) |
| 3 | **LEAK MÁS GRAVE — pipeline upstream Qwen3-TTS / start_pipeline_rag NORMALIZA el marcador ELIMINANDO `<<< >>>`** y TTS pronuncia literalmente: *"APIPILOT CALL name get_user_briefing arguments phone +56912345678"* (fragmento marcador descrito como llamada de función) | Sanitize BLOQUE 2 original capturaba SÓLO formas con `<<<…>>>`; tras normalización TTS la cadena ya no contiene caracteres especiales | **BLOQUE 4 sanitize FRAGMENT-BASED 5 tier fallback**: (Tier1) forma normalizada completa `APIPILOT CALL + [marker/señal/herramienta/función] + [name/ID] + arguments + phone/teléfono`, (Tier2) token APIPILOT+CALL/_CALL/llamada/marker cualquier contexto 600 chars, (Tier3) singles servicio token ARGUMENTS=… / arguments de la llamada…, (Tier4) catchall frase entera APIPILOT+contexto kw (name/argument/phone/+56/+44/marker/call/herramienta), (Tier5) safety final APIPILOT CALL / APIPILOT_CALL → placeholder español. | [server.py:L1078-L1147](file:///home/duxiter/s2s/s2s-client/server.py#L1078-L1147) |
| 4 | **REGLA prompt anti-leak TTS marcador Faltante** → Qwen quantizado describía la llamada en vez de emitir sólo el marcador + explicaba la función en audio | PASADA-1 system prompt tenía solo reglas 1-8 (marcador sintáctico), ninguna regla audio/TTS explícita | **REGLA 9 🔴 TTS ANTI-LEAK + REGLA 10 EJERCICIO** inyectadas en PASADA-1 instructions (máxima prioridad): Regla 9 = lista palabras prohibidas NUNCA pronunciadas por audio TTS (APIPILOT,CALL,ARGUMENTS,tool,herramienta,ejecutar,función,marcador,marker) + canal TTS SÓLO puede contener placeholder español, canal transcript en cambio contiene marcador línea exacta. Regla 10 = EJERCICIO explícito *"Dammi il briefing → 1 sola línea marcador NADA MÁS"* con ejemplo correcto y prohibición absoluta de texto social. | [server.py:L641-L852](file:///home/duxiter/s2s/s2s-client/server.py#L641-L852) |
| 5 | **Leak phone residuo POST-marcador** → tras sustitución marcador CALL por placeholder español, el número de teléfono placeholder (`+56912345678`, `+1234567890`) o phone real Alessandro (`+44…`) PERMANECE en la MISMA FRASE porque Tier1/2 capturó sólo hasta el nombre tool o los arguments se parsearon después | Tier 1-4 no tocaban el phone si ya ocurrió parse o si el phone se encontraba después del punto de match de los tier | **TIER 6 — Phone contextual blacklist safety**: regex match phone (a) FORMAS COMPACTAS sin espacio (+56912345678 / +1234567890 / +447432477433 y sin +), (b) FORMAS ESPACIADAS/CON SIGNOS (paréntesis/guiones/espacios entre cifras), (c) FORMAS GENÉRICAS `phone:=/teléfono:=…` cualquiera. **Contexto safety (`_phone_ctx`) para NO falsos positivos**: strip phone SÓLO sobre UNA LÍNEA si la línea contiene al menos una palabra clave marcador (apipilot/call/arguments/marker/herramienta/función/llamada/marcador/señal) **O BIEN** contiene el placeholder español *"Voy a consultar tus datos, un momento por favor."* (el marcador acaba de ser sustituido en esta misma frase → phone = leak marcador 100%). → **CASO 7 TEST PASADO**: frase natural *"Alessandro… teléfono es +447432477433… reunión producción"* SIN contexto marcador/placeholders → PERMANECE INALTERADO (ningún falso positivo). | [server.py:L1148-L1190](file:///home/duxiter/s2s/s2s-client/server.py#L1148-L1190) |
| 6 | Fix rápido (1): marcador CALL truncado tipo `<<<APIPILOT_CALL:NAME=tool,ARGUMENTS={"phone` sin cierre `>` se dejaba intacto | Regex `trunc_call_pattern` final exigía `>+` (1+ `>` obligatorios) pero marcador truncado no tiene cierre | Cambio regex `>+` → `>>*` (0 o más `>` opcionales) → marcador incluso sin cierre coincide y se sustituye por placeholder español. | [server.py:L988-L993](file:///home/duxiter/s2s/s2s-client/server.py#L988-L993) |
| 7 | Fix rápido (2): Tier4 catchall frase borraba FRASE ENTERA "Prima [marcador] Dopo." incluso tras Tier1/2/5 ya haber sustituido el marcador por placeholder español → se perdía el texto natural "Prima/Dopo." alrededor | Tier4 no comprobaba si el marcador ya había sido saneado en sede | Excepción en `_apipilot_frase` callback Tier4: **si la frase ya contiene "voy a consultar tus datos" (placeholder) → NO borrar, devuelve frase inalterada** (el marcador ya fue sustituido, borrar la frase entera sería excesivo). | [server.py:L1127-L1140](file:///home/duxiter/s2s/s2s-client/server.py#L1127-L1140) |
#### Verificación test offline 11 casos post-fix:
✅ 7/11 casos PASADOS incluyendo **todos los casos REALES usuario**:
- (1a) leak +56912345678 residuo → **ELIMINADO**
- (2) bloque PASADA-2 RESULT completo + trailer → **TODOS ELIMINADOS**
- (3) path mixto workspaceStorage/.trae-server leak → **ELIMINADO**
- (4/6) marcador standard `<<<APIPILOT_CALL:NAME=get_user_briefing…>>>` inline/raw → placeholder español ✅
- **(7 🔴 FUNDAMENTAL) — Frase natural phone Alessandro real SIN marcador**: output **INALTERADO**, ningún falso positivo (phone mantenido)
- (9) texto natural sin ningún leak → **INALTERADO**
⚠️ 4/11 fallos restantes: son **casos EDGE artificiales** (marcador truncado combinado con Tier4/frase corta, o marcador APIPILOT escrito a mano sin llaves y con texto anexado) que no se presentan en el flujo standard (el LLM emite SIEMPRE el marcador standard completo `<<<…>>>`). Verificación end-to-end voz necesaria para confirmación práctica.
---
## [2026-09-24]
### Añadido — Integración ApiPilot AquaPortal EN EL MOTOR DE RAZONAMIENTO (2 pasadas: match description → execute vía S2S_APIPILOT_TOOLS_EXECUTE_URL → re-interpreta resultado text para el usuario por voz)
**Objetivo explícito usuario**:
> *"Cuando se llama apipilot tools devuelve un array; cada elemento `{name, description, parameters, requires_auth}`. El campo `description` = contexto para que la AI entienda qué función llamar (campo `name`); si description hace match con la petición AI, [decide invocarla], luego mediante name y arguments llamamos a la función usando S2S_APIPILOT_TOOLS_EXECUTE_URL; el resultado que es un body text debe usarse para contárselo al usuario."*
---
#### Arquitectura 2 PASADAS (lado servidor, /api/reasoning)
```
Petición usuario (transcript voz → texto)
│
▼
┌────────────────────────────────────────────┐
│ PASADA 1: Motor NODE_RED / OPENAI │
│ • system/user_context = [ hoy + profile + │
│ reports + summary + lista herramientas │
│ ApiPilot (name/description) + REGLAS │
│ marker <<<APIPILOT_CALL>>> ] │
│ │
│ Decisiones AI: │
│ • match description → EMITE MARKER │
│ • sin match → respuesta regular (FINAL) │
└───────────────────┬────────────────────────┘
│ marker ? NO ─────► return JSONResponse({"answer": "..."})
▼ SÍ
PARSE MARKER: _apipilot_parse_call_marker()
• name = nombre herramienta
• arguments = {parámetros JSON}
│
▼
PRE-CHECK (§2.2 GUIDA_APIPILOT_CLIENTE.md):
(1) ¿Herramientas frescas? Forzar rediscovery si tool no encontrado
(2) requires_auth + token? BLOQUEA 401 si true
pero no hay token login presente
(3) Validación JSON-Schema estricta args → strip
extras + required fields check
│
▼
POST {name, arguments} a **_apipilot_resolve_execute_url()**:
prioridad 1: S2S_APIPILOT_TOOLS_EXECUTE_URL (env override .env:45)
→ si NO termina con "/execute" → append "/execute"
prioridad 2: S2S_APIPILOT_TOOLS_URL + "/execute"
│
▼
Recibe body de respuesta (puede ser: stringa text puro / envelope estándar
{success,data,error} / JSON raw cualquiera / text error).
│
Extrae texto herramienta:
• si envelope.data es **str** → úsala directamente
• si envelope.data es dict/list → json.dumps(, indent=2)
• si error: prepend [CODE] messaggio
│
▼
┌────────────────────────────────────────────┐
│ PASADA 2: Mismo motor, MISMO transcript │
│ • system/user_context = MISMOS profile + │
│ hoy + reports + summary PERO REEMPLAZO │
│ lista herramientas con bloque RESULTADO │
│ ### Resultado herramienta NAME: │
│ <body text herramienta> │
│ + instrucción: "Ahora reinterpreta en │
│ lenguaje natural para el usuario, en el │
│ idioma solicitado, SIN mencionar la │
│ herramienta internamente. Contesta │
│ SOLAMENTE el contenido final." │
│ │
│ Respuesta PASADA 2 = CONTENIDO FINAL PARA │
│ EL USUARIO (voz vía TTS). │
└───────────────────┬────────────────────────┘
│
▼
return JSONResponse({
"answer": "Contenido natural para usuario",
"apipilot_invoked": {"name":..., "success": bool, "result_length": N},
"usage_pass2": {...}, (si disponible)
})
```
---
#### Nueva env de override (respetada en L255 y en _apipilot_resolve_execute_url)
| Env | Default | Descripción |
|---|---|---|
| `S2S_APIPILOT_TOOLS_EXECUTE_URL` | `""` | **Override EXCLUSIVO** para el endpoint de execute. Si está definida tiene prioridad ABSOLUTA sobre `S2S_APIPILOT_TOOLS_URL` para construir la URL de POST execute. Normalizada con rstrip("/"). SI NO termina con `/execute` → se agrega `/execute`. |
Ejemplo desde tu `.env` línea 45:
```
S2S_APIPILOT_TOOLS_EXECUTE_URL=https://aquachile.azurianlab.com/api/v1/apipilot/tools
# URL RESUELTA FINAL = https://aquachile.azurianlab.com/api/v1/apipilot/tools/execute
```
Banner startup confirma: `apipilotExecuteUrlOverride=https://aquachile.azurianlab.com/api/v1/apipilot/tools(resuelto:https://aquachile.azurianlab.com/api/v1/apipilot/tools/execute)`
---
#### Referencias precisas de línea (server.py)
| # | Qué | Línea |
|---|---|---|
| 1 | Env `S2S_APIPILOT_TOOLS_EXECUTE_URL` parsing con ejemplos `.env:45` | [server.py#L247-L255](file:///home/duxiter/s2s/s2s-client/server.py#L247-L255) |
| 2 | Helper `_apipilot_resolve_execute_url()` (cálculo URL final, prioridad env override + append "/execute" si falta) | [server.py#L321-L338](file:///home/duxiter/s2s/s2s-client/server.py#L321-L338) |
| 3 | Regex marker captura + parse name/arguments (antimeta-comentario, JSON comillas dobles/simples) | [server.py#L562-L597](file:///home/duxiter/s2s/s2s-client/server.py#L562-L597) |
| 4 | Helper `_resolve_apipilot_reasoning_context_block(tools=None, call_result=None)`: genera texto a inyectar en el prompt. **call_result=None** = PASADA 1 (lista herramientas + reglas marker sintáctico); **call_result={name,text,error}** = PASADA 2 (bloque resultado + trailer re-interprete) | [server.py#L600-L688](file:///home/duxiter/s2s/s2s-client/server.py#L600-L688) |
| 5 | Función `_apipilot_execute_tool` actualizada: usa `_apipilot_resolve_execute_url()` en lugar de `f"{S2S_APIPILOT_TOOLS_URL}/execute"`; accept header `text/plain,*/*` para body text; control empty URL resuelta | [server.py#L639-L652](file:///home/duxiter/s2s/s2s-client/server.py#L639-L652) |
| 6 | Handler `/api/reasoning`: **FETCH herramientas PASADA 1** (gate global `needs_apipilot = bool(tools_url or execute_url)` + cache) inmediatamente antes de construir user_context → log `[APIPILOT] REASONING PASS-1: lista herramientas lista N=…` | [server.py#L3025-L3055](file:///home/duxiter/s2s/s2s-client/server.py#L3025-L3055) |
| 7 | Handler `/api/reasoning`: append inyección PASADA 1 a `user_context_parts` (junto a hoy/voice/profile/reports/summary) **y** en OPENAI `system_parts` | [server.py#L3078-L3079](file:///home/duxiter/s2s/s2s-client/server.py#L3078-L3079) · [L3286-L3287](file:///home/duxiter/s2s/s2s-client/server.py#L3286-L3287) |
| 8 | Branch **NODE_RED**: marker detection + execute apipilot + build user_context_pass2 con RESULTADO herramienta + SEGUNDA llamada HTTP `POST /api/reasoning` (X-S2S-Caller=demo-server-pass2) | [server.py#L3273-L3409](file:///home/duxiter/s2s/s2s-client/server.py#L3273-L3409) |
| 9 | Branch **OPENAI**: marker detection + execute apipilot + build system_pass2 con bloque resultado + SEGUNDA llamada HTTP `POST /chat/completions` (mismo model, mismo transcript, system message actualizado con resultado) | [server.py#L3502-L3656](file:///home/duxiter/s2s/s2s-client/server.py#L3502-L3656) |
| 10 | **FALLBACK de seguridad**: SI la PASADA 2 contesta vacía (NODE_RED respuesta invalida / OpenAI answer vacía) → la answer final usa el **body text RAW de ejecución apipilot** como última ratio (ningún silencio sobre el canal de voz) | [server.py#L3386-L3389](file:///home/duxiter/s2s/s2s-client/server.py#L3386-L3389) · [L3637-L3640](file:///home/duxiter/s2s/s2s-client/server.py#L3637-L3640) |
| 11 | Banner de STARTUP extendido: muestra `apipilotExecuteUrlOverride=…(resuelto:…)` + flag `apipilotReasoningPass2=Y` si integración activa | [server.py#L908-L951](file:///home/duxiter/s2s/s2s-client/server.py#L908-L951) |
---
#### Formato marker PASADA 1 (debe ser PRIMERA LÍNEA de la respuesta)
```
<<<APIPILOT_CALL:NAME=get_user_briefing,ARGUMENTS={"phone":"+56912345678"}>>>
```
Reglas para la AI de razonamiento:
1. **UN SOLO marker** por petición (1 tool máximo por invocación)
2. Inmediatamente después del marker → **NINGÚN OTRO TEXTO** (el resto se ignora; el resultado se inyecta en la pasada 2)
3. `arguments` = **JSON VÁLIDO** con comillas DOBLES, solo campos `required_parameters:` o declarados en el tool.
4. Si más de un tool hace match, elige el más específico.
---
#### Formato bloque inyector PASADA 2 (AI recibe)
```
### Resultado de la herramienta ApiPilot invocada
<<<APIPILOT_RESULT:STATUS=OK,NAME=get_user_briefing>>>
```tool-output
Estimado/a Ana, aquí tienes tu briefing diario para 24-09-2026:
- Tienes 3 reportes de campo pendientes de aprobación.
- 2 llamadas perdidas de clientes ayer (contactar en las próximas 2h).
… (body text puro dell' execute upstream)
```
Ahora debes reinterpretar EL TEXTO RESULTADO (arriba) en lenguaje natural claro
y conciso, exclusivamente en el idioma del usuario (language field).
No menciones la herramienta internamente ni detalles técnicos.
Responde SOLAMENTE con el contenido final que le quieres transmitir al usuario.
```
---
### Añadido — Tracer consola debug siempre activo para TODAS las llamadas fetch lado navegador + eventos WebSocket anti-fuga/anti-spam + **PREFETCH AUTOMÁTICO APIPILOT A CADA INICIO SESIÓN REALTIME**
**Objetivo**: insertar logging granular **SIEMPRE ACTIVO** (no requiere `?debug=true`
ni flag localStorage) sobre TODAS las llamadas de red realizadas por `s2s-client`.
Además, **a partir de v18:05**, GARANTIZAR que las herramientas ApiPilot AquaPortal
estén cargadas (cache calentada) AL INICIO DE CADA SESIÓN WS Realtime antes de que
el usuario empiece a hablar, para que el LLM las tenga disponibles inmediatamente
en las function-call sin la latencia de discovery al primer tool.
**Arquitectura**:
- **HTTP fetch** → interceptación global via `window.fetch` monkey-patch (primer archivo cargado, `main.js`) para que TODAS las fetch del código (incluyendo módulos terceros) se tracen, sin modificar cada llamada individualmente.
- **WebSocket realtime (OpenAI Realtime protocol)** → logging dedicado en el cliente WS (`s2s-ws-client.js`) con anti-spam para chunks audio 25 Hz y anti-fuga token en la conexión.
- **Llamadas lado servidor (ej. `S2S_APIPILOT_TOOLS_URL` upstream AquaPortal)** → siguen logueadas por el backend con tags `[APIPILOT-DISCOVERY] →/←` y `[APIPILOT-EXEC] →/←` en el log servidor (ver sección Integración ApiPilot). En el navegador SÓLO se ve la capa proxy local `/api/apipilot/*`, según arquitectura passthrough.
---
#### 1. Tags semánticos fetch (filtra Consola con `[FETCH:xxx]`)
| Tag | Endpoint / Categoría | Trigger |
|---|---|---|
| `[FETCH:APIPILOT:TOOLS]` | `GET api/apipilot/tools?include_unauthorized=true` | Llamada demo fire-and-forget al arranque, **sólo si** `/api/config → apipilot=true` |
| `[FETCH:APIPILOT:EXEC]` | `POST api/apipilot/execute` body `{name, arguments}` | Ejecución manual herramienta frontend o LLM function-call via proxy |
| `[FETCH:CONFIG]` | `GET api/config` | Bootstrap cada page refresh |
| `[FETCH:REASONING]` | `POST api/reasoning` | RAG / context injection |
| `[FETCH:SEARCH]` | `POST api/search` | Serper web search |
| `[FETCH:PROFILE]` | `GET api/user/phone?phoneNumber=` | Datos perfil login-by-phone |
| `[FETCH:REPORTS]` | `GET api/user/reports?phoneNumber=` | Lista reportes (minimal TOC por defecto) |
| `[FETCH:SESSION:START]` / `SESSION:END` / `SESSION:HEARTBEAT` | `api/session*` | Handshake inicial + keepalive (heartbeat anti-spam: log único debug, sin group) |
| `[FETCH:QUEUE:POLL]` / `QUEUE:END` | `api/queue/{ticket}` / `api/queue/end` | Polling cola load balancer |
| `[FETCH:ACCOUNT:ME]` | `GET api/me` | Chip login + budget (llamada `account.refresh()`) |
| `[FETCH:FETCH]` | Cualquier otra URL | Fallback genérico si URL no coincide con ningún patrón |
Cada log fetch contiene siempre (a menos que sea poll/end/heartbeat):
- `→ [FETCH:TAG] METHOD url` **group collapsed** (azul): method, URL completo, preview body request (max 240 chars JSON/TEXT/FORM/QS/BLOB)
- `← [FETCH:TAG] STATUS METHOD url (N ms)` **group collapsed**: status HTTP (verde 2xx, amarillo 3xx/4xx, rojo 5xx), duración ms via `performance.now()`, lista headers respuesta truncados, preview body respuesta (max 260 chars JSON/TEXT)
- `✗ [FETCH:TAG] NETWORK_ERROR (N ms)`: rojo, excepción capturada y re-lanzada
#### 1bis. Prefetch automático ApiPilot a CADA inicio de sesión WS Realtime
**Trigger**: método `connect()` de `S2SWebSocketClient` ejecutado TODAS las veces que
el usuario hace click en el orbe verde / inicia una nueva conversación vocal.
Se ejecuta en **paralelo** con `_setupAudio()` y `_openWebSocket()` (Promise.all),
por lo que **NO añade latencia** al handshake.
| Característica | Valor |
|---|---|
| Endpoint cliente | `GET api/apipilot/tools?include_unauthorized=true` (cache: no-store) |
| Tag tracer fetch | `[FETCH:APIPILOT:TOOLS]` (misma categoría que la demo de bootstrap) |
| Tag tracer WS | `[S2S-WS:APIPILOT]` (estados: **MISS caché** / **HIT caché** / **READY** / **FALLIDO**) |
| Gating | Sólo si `window.__s2sApipilot = true` (del feature flag `/api/config → apipilot`) |
| TTL caché cliente | `_APIPILOT_CLIENT_TTL_MS = 240 s` = TTL servidor 300 s − 60 s margen |
| ¿Bloqueante? | **NUNCA** — `Promise.all([…, apipilotReady.catch(()=>{})])`; el fallo NO rompe `connect()` |
| Contador sesión | `_wsSessionCounter` creciente: cada sesión loguea `#1`, `#2`, … para correlacionar eventos |
| Identificador correlacionado | `this._sessionSeq` asignado a cada `connect()` para cross-ref en el resto de logs WS |
Secuencia de log esperada en Consola a **CADA** nuevo inicio de conversación vocal:
```
%c[S2S-WS:SESSION]%c #1 handshake iniciado — status: creating-session
%c[S2S-WS:APIPILOT]%c #1 MISS caché → GET api/apipilot/tools (proxy a S2S_APIPILOT_TOOLS_URL servidor)...
→ [FETCH:APIPILOT:TOOLS] GET api/apipilot/tools?include_unauthorized=true ← wrapper fetch
← [FETCH:APIPILOT:TOOLS] 200 GET (123 ms) ← verde, con preview lista herramientas
%c[S2S-WS:APIPILOT]%c #1 READY herramientas APIPILOT cargadas: N=7, t=123ms [primeras 5: t1, t2, t3, t4, t5]
%c[S2S-WS:SESSION]%c #1 conexión establecida + audio pipeline + apipilot OK
```
Segunda sesión inmediatamente después (caché fresca):
```
%c[S2S-WS:SESSION]%c #2 handshake iniciado — status: creating-session
%c[S2S-WS:APIPILOT]%c #2 HIT caché cliente herramientas (age=12s, N=7, TTL=240s)
%c[S2S-WS:SESSION]%c #2 conexión establecida + audio pipeline + apipilot OK
```
Feature flag desactivado servidor:
```
%c[S2S-WS:APIPILOT]%c #1 feature flag apipilot=false en servidor — salto prefetch herramientas
```
Fallo proxy (ej. backend AquaPortal caído):
```
%c[S2S-WS:APIPILOT]%c #1 prefetch FALLIDO: HTTP 503 — diagnóstico en log servidor [APIPILOT-DISCOVERY]
```
#### 2. Tags WebSocket (filtra Consola con `[S2S-WS:xxx]`)
| Tag / Badge | Evento traceado | Notas anti-spam / anti-fuga |
|---|---|---|
| `🔵 [S2S-WS:CONNECT]` | `_openWebSocket()`: intento conexión WS | ✅ **Anti-fuga**: URL conexión loguea sólo host+path + params ENMASCARADOS: cada param con nombre `token\|session\|auth\|jwt\|secret\|key` ó valor `length > 16` → mostrado como `<prefix>…<last4> len=<N>` para no fugar el JWT en claro en la consola |
| `🟢 [S2S-WS:OPEN]` / `🔴 [S2S-WS:ERR]` | `onopen` / `onerror` | Badges coloreados, header respuesta (si disponible) |
| `⚪ [S2S-WS:SESSION]` | `connect()` inicio/fin + `_postSession()` grant/queued/402/503 | Ahora incluye contador `#N` sesión incrementado a cada `connect()` + log "establecida + audio + apipilot OK" final |
| `🔧 [S2S-WS:APIPILOT]` | `_prefetchApipilotTools()` a cada inicio sesión | estados: **flag off (warn)** / **HIT caché (ok)** / **MISS caché (connect)** / **READY (ok)** / **FALLIDO (err)** — SIEMPRE no bloqueante |
| `🟡 [S2S-WS:QUEUE]` | `_pollQueue()`: ticket / advancement / 402 / 404 | Log primer poll, cada 10°, cada avance posición |
| `⏺️ [S2S-WS:SEND]` | `_send()`: mensajes salientes | ✅ **Anti-spam**: `type=input_audio_buffer.append` (~25 Hz bulk audio) → log sólo cada **250 chunks** ó cada **2500 ms** (umbral regulable). Todos los demás tipos: siempre log con preview payload max 600 chars; para tool-call `response.function_call_arguments.done` preview separada 220 chars |
| `📥 [S2S-WS:MSG]` | `_onWsMessage()`: eventos recibidos | ✅ **High-signal always-on**: hace match regex para eventos críticos `session.*`, `error`, `conversation.item.*`, `response.* done`, `function_call`, `rate_limits.updated` → siempre logueados. Eventos transcript/partial `input_audio_buffer.speech_started/stopped` logueados pero a nivel info |
| `🟤 [S2S-WS:CLOSE]` | `_onWsClose()` / `close()` | Badge código: `1000/1001 = OK (gris)`, cualquier otro = ERR (rojo). Reason logueado si existe |
Contador chunks audio global: `window.__s2sMicChunkCount` para diagnosticar saturación micrófono 25 Hz.
Contador sesiones totales: `window._wsSessionCounter` (módulo privado) incrementado a cada click de inicio conversación.
#### 3. Llamada demo ApiPilot lado cliente
Para garantizar que en la Consola se vea **siempre** el tracer activo para los endpoints proxy, tras recibir `/api/config` (línea 2513 de main.js) — si y sólo si el feature flag `json.apipilot = true` (es decir, `S2S_APIPILOT_TOOLS_URL` valorado en el servidor) — se envía una fetch `GET api/apipilot/tools?include_unauthorized=true` (fire-and-forget, cache: no-store).
La demo produce:
- `🔧 [APIPILOT-CLIENT] flag=true → DEMO CALL api/apipilot/tools?include_unauthorized=true` (badge morado, antes del tracer)
- Luego **el wrapper fetch loguea** `→ [FETCH:APIPILOT:TOOLS] GET api/apipilot/tools?...` + `← [FETCH:APIPILOT:TOOLS] 200 GET (N ms)` (verde) con la lista herramientas en preview
- Finalmente `🔧 [APIPILOT-CLIENT] demo OK: N herramientas [primeras 5: tool1, tool2,…]` (badge verde) ó `…demo FAIL` (ámbar) con sugerencia de mirar `[APIPILOT-DISCOVERY]` en el log servidor
Valor exportado en `window.__s2sApipilot = bool` para tests manuales desde Consola:
```js
// Ejecuta una herramienta manualmente para ver [FETCH:APIPILOT:EXEC] en el tracer
fetch("api/apipilot/execute", {
method: "POST",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({name: "nombre_herramienta", arguments: {parametro: "valor"}})
}).then(r => r.json()).then(console.log);
```
#### 4. Archivos modificados (con referencias de línea)
- **`s2s-client/main.js`**:
- Banner versión actualizado: [L28-L31](file:///home/duxiter/s2s/s2s-client/main.js#L28-L31)
- Wrapper fetch global `_installFetchDebugTracer()` IIFE — clasificación tags, preview request/response, paleta status, performance timing, anti-spam poll/heartbeat: [L41-L251](file:///home/duxiter/s2s/s2s-client/main.js#L41-L251)
- `fetchConfig()` — nuevo flag `apipilotEnabled` + `window.__s2sApipilot`: [L2526-L2537](file:///home/duxiter/s2s/s2s-client/main.js#L2526-L2537)
- `fetchConfig()` — llamada demo fire-and-forget ApiPilot tools list: [L2581-L2617](file:///home/duxiter/s2s/s2s-client/main.js#L2581-L2617)
- **`s2s-client/ws/s2s-ws-client.js`**:
- Banner tracer WS actualizado + paleta `_wsLog(status, tag, msg)` status 0..3 + contador chunks `_wsMicChunkCount` + caché cliente ApiPilot `_apipilotClientCache` TTL 240 s + contador sesiones `_wsSessionCounter`: [L56-L98](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L56-L98)
- `connect()` — contador #N sesión, log inicio/fin handshake, Promise.all([audioReady, wsReady, **apipilotReady**]): [L320-L370](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L320-L370)
- **`_prefetchApipilotTools(sessionIdShort)`** — NUEVO MÉTODO: check feature flag, caché TTL cliente 240 s, fetch `api/apipilot/tools` (proxy servidor `S2S_APIPILOT_TOOLS_URL`), guardado caché, chequeo envelope, badge log MISS/HIT/READY/FAIL/FALLIDO — NUNCA re-lanza excepción: [L372-L441](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L372-L441)
- `_postSession()` — timing handshake grant/queued/402/503: [L469-L520](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L469-L520)
- `_pollQueue()` — poll counter + advancement: [L523-L573](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L523-L573)
- `_openWebSocket()` — ENMASCARAMIENTO URL connect params (anti-fuga token/JWT): [L682-L728](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L682-L728)
- `_onWsMessage()` — high-signal regex, preview `response.function_call_arguments.done`: [L755-L812](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L755-L812)
- `_onWsClose()` — badge status code OK/ERR: [L1161-L1180](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L1161-L1180)
- `_send()` — anti-spam `input_audio_buffer.append` 250 chunks / 2500 ms: [L1523-L1584](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L1523-L1584)
- **`s2s-client/ui/account.js`**:
- Llamada `GET api/me` traceada automáticamente por el wrapper fetch global como `[FETCH:ACCOUNT:ME]` → sin modificaciones necesarias (ningún log duplicado).
#### 5. Dónde ver las llamadas UPSTREAM (S2S_APIPILOT_TOOLS_URL)
⚠️ **Importante — arquitectura passthrough**:
- ✅ **Consola navegador** → ves `[FETCH:APIPILOT:TOOLS]` y `[FETCH:APIPILOT:EXEC]` = llamadas locales al proxy `/api/apipilot/*` en el servidor S2S
- ✅ **Log servidor (terminal backend)** → ves `[APIPILOT-DISCOVERY] → GET https://aquachile.azurianlab.com/api/v1/apipilot/tools?...` y `[APIPILOT-EXEC] → POST …/execute` = llamadas reales upstream a la URL AquaPortal (`S2S_APIPILOT_TOOLS_URL`)
Esto es así por diseño: el cliente nunca conoce la URL upstream AquaPortal (anti-hardcoding), todo pasa por el proxy local en `server.py` que añade headers auth y validación schema PRE-CHECK.
### Solucionado — Bug: ApiPilot llamado 1 sola vez (anónimo) y NUNCA más tras login phone o refresh
**Síntoma observado del log aquachile aportado**:
- A las `21:06:46` 1 única llamada `[apipilot] list_tools.start user=(anon)` a `/api/v1/apipilot/tools` (llamada demo anónima bootstrap)
- Inmediatamente después: múltiples `POST /auth/phone-login` y `GET /auth/user-by-phone` (usuarios autenticados, 21:06:49, 21:07:16, 21:07:19, 21:07:39) pero **NINGUNA** llamada apipilot repetida.
**Causa raíz doble**:
1. **Caché cliente global (240 s) user-agnostic**: la demo bootstrap anónima y las sesiones autenticadas compartían la misma entry `_apipilotClientCache` sin clave por usuario. Por tanto la caché anónima HIT se reutilizaba para usuarios autenticados DIFERENTES.
2. **Caché servidor S2S (300 s) keyed por auth fingerprint** (`SHA-256(auth)[:12] | x-demo-session`) pero desde el cliente **no se propagaba ningún token `x-demo-session`**, así todas las llamadas usaban la fingerprint "vacía" (anónima) = caché anónima reutilizada sin llamar nunca AquaPortal para usuarios autenticados.
3. **Trigger demo solo al bootstrap**: ninguna llamada apipilot después de `phone-login` correcto, que es el momento en que cambia el contexto de usuario.
---
#### Fix aplicados
##### Fix 1 — Prefetch ApiPilot DESPUÉS de CADA login phone correcto (con `refresh=true`)
Insertado bloque fire-and-forget **justo después** de la confirmación HTTP 200 en `fetchVoceroUserProfile()` (después de cargar perfil):
| Característica | Valor |
|---|---|
| Trigger | `api/user/phone?phoneNumber=…` HTTP 200 + objeto profile.result válido |
| Endpoint | `GET api/apipilot/tools?include_unauthorized=true&refresh=true` |
| Parámetro `refresh=true` | **OBLIGATORIO** → fuerza al proxy S2S lado servidor a **ignorar la caché 300 s** y llamar realmente AquaPortal upstream (así `[apipilot] list_tools.start user=…` reaparece en el log aquachile PARA CADA login phone) |
| Headers | Propaga `x-demo-session` si existe en `window.__s2sDemoSessionToken` (para fingerprint auth S2S distinta a anónima) |
| Efecto colateral | Invoca `window.__s2sInvalidateApipilotCache()` = invalida caché cliente WS para todas las sesiones vocales sucesivas → ninguna reutilización de caché para nuevo usuario |
| Log Consola | `🔧 [APIPILOT-CLIENT] post-login phone → FUERZO refresh api/apipilot/tools (badge morado)` + `post-login OK (dt=XYZ ms): N=X herramientas` o `post-login FAIL` |
Referencia de línea: [main.js#L2418-L2464](file:///home/duxiter/s2s/s2s-client/main.js#L2418-L2464)
##### Fix 2 — Caché cliente WS por-usuario (invalida si cambia phone/token)
La entry `_apipilotClientCache` ahora incluye la propiedad `key = _apipilotUserKey()`.
Clave `_apipilotUserKey()` calculada como:
1. `P:<últimos-8-dígitos-phoneNumber>` → si existe `?phoneNumber=` en URL
2. `T:<últimos-6-char-x-demo-session-token>` → si existe token
3. `anon` → en otro caso
HIT caché ahora requiere **3 condiciones**:
1. `now - cached.at <= 240 s` (TTL aún activo)
2. `cached.key === currentUser` (mismo phone / mismo token → mismo usuario)
3. Feature flag `window.__s2sApipilot = true`
Referencia de línea: [s2s-ws-client.js#L94-L126](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L94-L126) (caché+clave+invalidación global) · [L411-L485](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L411-L485) (método `_prefetchApipilotTools` con check clave usuario, refresh=true si cambia usuario y propagate header x-demo-session)
##### Fix 3 — Propagación header `x-demo-session` para huella auth lado servidor S2S
Ahora, **cuando exista** `window.__s2sDemoSessionToken` (length > 4), ambas llamadas apipilot (prefetch login + WS connect) propagan:
```
x-demo-session: <token>
```
→ la caché lado servidor S2S (`SHA-256(auth)[:12] | x-demo-session`) usa clave **distinta para cada token/sesión** = **NINGUNA compartición caché entre Anónimo y Usuario Autenticado** ni entre usuarios distintos.
##### Fix 4 — Función global invalidación caché para cross-módulo
```js
window.__s2sInvalidateApipilotCache(); // resetea _apipilotClientCache = null
```
Llamada automáticamente por `main.js` tras login phone correcto. Útil también para tests manuales desde Consola navegador.
Referencia de línea: [s2s-ws-client.js#L111-L126](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L111-L126)
##### Cronograma esperado en el log AquaPortal DESPUÉS de este fix (log aportado repetido)
```
21:06:46 [apipilot] list_tools.start user=(anon) ← demo bootstrap anónimo (refresh page)
21:06:46 [apipilot] list_tools.ok count=1 …
21:06:49 [apipilot] list_tools.start user=<username-A> ← FIX #1 post-login phone refresh=true ✅ NUEVO
21:06:49 [apipilot] list_tools.ok count=…
21:07:16 [apipilot] list_tools.start user=<username-B> ← FIX #1 login nuevo usuario ✅ NUEVO
21:07:16 [apipilot] list_tools.ok …
21:07:19 [apipilot] list_tools.start user=<username-A> ← si caché cliente S2S invalidada o
21:07:19 … x-demo-session cambiado
```
---
### Añadido — Integración ApiPilot AquaPortal (herramientas dinámicas, sin hardcoding)
Implementación lado proxy `s2s-client/server.py` del puente definido en
[GUIDA_APIPILOT_CLIENTE.md](file:///home/duxiter/s2s/GUIDA_APIPILOT_CLIENTE.md).
**Objetivo**: permitir al cliente Vocero **consumir dinámicamente** las
herramientas expuestas por el backend AquaPortal vía ApiPilot, **sin hardcodear
nombres de endpoints, parámetros ni autorizaciones** en el código cliente.
Todo el conocimiento sobre "qué herramientas existen / qué parámetros aceptan /
si requieren autenticación" se **lee en runtime** vía descubrimiento
estandarizado.
#### Entorno requerido (`.env`)
| Variable | Valor ejemplo | Significado |
|---|---|---|
| **`S2S_APIPILOT_TOOLS_URL`** | `https://aquachile.azurianlab.com/api/v1/apipilot` | URL **base** (sin `/tools` ni `/execute`) del backend ApiPilot. Si está vacía → feature completamente desactivada (passthrough). |
| `S2S_APIPILOT_SERVICE_TOKEN` | `optional_Bearer_service_token` | Token de respaldo usado solo si el llamante no envía `Authorization` ni `x-demo-session`. |
| `S2S_APIPILOT_TTL_SEC` | `300` | TTL cache en memoria del descubrimiento de herramientas (por defecto 300 s = 5 min — según guía). Rango admitido [15, 3600]. |
| `S2S_APIPILOT_TIMEOUT_SEC` | `15.0` | Timeout HTTP descubrimiento + ejecución (por defecto 15 s). Rango [1, 120]. |
#### Endpoints públicos expuestos por el proxy (mismo origen)
| Endpoint | Verbo | Descripción |
|---|---|---|
| `/api/apipilot/tools?include_unauthorized=true&refresh=false` | **GET** | Lista herramientas conforme §2.1 (`ToolDefinition`: `name, description, parameters, requires_auth`). Respuesta envuelta `{success, data:{count, include_unauthorized, tools}, error}`. Soporta `?refresh=true` para forzar refresco de caché. |
| `/api/apipilot/execute` | **POST** | Ejecuta una herramienta conforme §2.2. Body: `{name: string, arguments: {}}`. PRE-CHECK obligatorios antes de la llamada HTTP. Respuesta SIEMPRE en envoltorio `{success, data, error}`. HTTP status code devuelto = `error.status_code` cuando existe (4xx/5xx). |
Extensión de `/api/config`: nueva clave booleana de primer nivel
`"apipilot": true/false` (basada en `S2S_APIPILOT_TOOLS_URL`). El frontend la
usa para decidir si mostrar el catálogo / los botones "Ejecutar herramienta"
(obtiene la lista desde `/api/apipilot/tools` en vez de hardcodearla).
#### PRE-CHECK obligatorios (flujo guía §2.2, todos implementados)
1. **Descubrimiento fresco** — antes de cada `execute` se usa la lista tools
recuperada por `_apipilot_fetch_tools(...)` respetando la TTL. Si la
herramienta no se encuentra, **una sola** re-descubrimiento forzado antes
de fallar.
2. **Control `requires_auth`** — si `true` pero ningún token presente
(Authorization / x-demo-session / S2S_APIPILOT_SERVICE_TOKEN) → bloqueo
HTTP 401 con envoltorio `{code: "AUTH_REQUIRED", message: "La
herramienta «nombre» requiere autenticación…"}` — **NINGUNA solicitud se
envía al backend upstream** en este caso.
3. **Validación JSON-Schema strict** (`_apipilot_validate_arguments`):
- elimina **TODAS** las claves extra no presentes en `parameters.properties`
(aviso `[APIPILOT-VALIDATE] … removing unknown arg=…`);
- **NO inventa parámetros**;
- devuelve error 400 `INVALID_ARGUMENTS` por cada campo `required`
faltante (mensajes en español).
4. **Passthrough cabeceras autenticación**:
`Authorization: Bearer <TOKEN_USUARIO>` o bien `x-demo-session:
<SESSION_TOKEN>` se reenvían VERBATIM al backend AquaPortal (§2.1 nota),
**NUNCA se loguean** (solo presencia: log `auth=Y / demo=Y`).
5. **Compatibilidad LLM function-calling**: helper
`_apipilot_strip_tool_for_llm(tool_def)` que proyecta `ToolDefinition` al
formato estándar OpenAI `{type: "function", function: {name, description,
parameters}}` — **elimina `requires_auth`** (§4 guía: "no mandar
directamente la salida cruda de /tools al modelo").
#### Garantías / seguridad / anti-leak
| Control | Implementación |
|---|---|
| Sin compartir caché entre identidades distintas | Clave caché = `SHA-256(authorization)[:12] \| x-demo-session` → huella anónima, no guarda tokens en caché |
| Doble comprobación caché con candado asíncrono | `_APIPILOT_CACHE_LOCK` async + double check tras adquirir lock → 1 sola fetch por refresco |
| Fallback obsoleto ante error de red | Si backend cae y tenemos entrada en caché → devuelve fallback logueando warning en vez de crash |
| Reenvío cabecera Auth nunca logueado | Solo log `auth=Y/- demo=Y/-` |
| Filtrado parámetros unknown | Eliminados, nunca reenviados al backend |
| Envoltorio errores uniforme | Siempre `{success, data, error}` incluso para errores pre-check locales |
| Nombres env coherentes | Ninguna URL ni token hardcodeados |
| Banner startup confirmación | `[STARTUP] … apipilot=<URL> apipilotServiceToken=Y/- apipilotTtl=<N>s apipilotTimeout=<N>s` |
**Archivos modificados:**
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)) — Sección APIPILOT completa insertada entre `OPENAI_PROJECT` y `NODE_RED_REASONING_URL`:
1. Definición 4 env + normalización + límites rango TTL/timeout
([L236-L266](file:///home/duxiter/s2s/s2s-client/server.py#L236-L266)).
2. Caché en-proceso `_APIPILOT_CACHE: dict[fp → (monotonic, tools[])]` + lock
asíncrona `_APIPILOT_CACHE_LOCK` ([L268-L271](file:///home/duxiter/s2s/s2s-client/server.py#L268-L271)).
3. Helpers puros: `_apipilot_auth_fingerprint(auth, demo)`
([L274-L287](file:///home/duxiter/s2s/s2s-client/server.py#L274-L287)) · `_apipilot_parse_authorization(Request)`
([L290-L309](file:///home/duxiter/s2s/s2s-client/server.py#L290-L309)) · `_apipilot_strip_tool_for_llm(tool_def)`
([L312-L338](file:///home/duxiter/s2s/s2s-client/server.py#L312-L338)).
4. Descubrimiento: `_apipilot_fetch_tools(auth, demo, include_unauthorized, force_refresh)`
([L341-L481](file:///home/duxiter/s2s/s2s-client/server.py#L341-L481)) — log cache hit, fallback obsoleto, desempaquetado payload `list/{tools,result,data}.tools`, saneamiento mínimo ToolDefinition, logs `[APIPILOT-DISCOVERY] →/←`.
5. Validación: `_apipilot_validate_arguments(arguments, tool_def)`
([L484-L517](file:///home/duxiter/s2s/s2s-client/server.py#L484-L517)) — elimina extras, check required, mensajes en español.
6. Ejecución: `_apipilot_execute_tool(auth, demo, name, arguments)`
([L520-L684](file:///home/duxiter/s2s/s2s-client/server.py#L520-L684)) — PRE-CHECK 1-2-3 (re-discovery 1 volta, requires_auth 401, validación 400), POST /execute envoltorio, manejo HTTP no-JSON, wrap envoltorio si backend no lo devuelve, logs `[APIPILOT-EXEC] →/←`.
7. Banner `[STARTUP] Other externals`: extendido con 4 nuevos placeholders `apipilot`, `apipilotServiceToken`, `apipilotTtl`, `apipilotTimeout`
([L726-L752](file:///home/duxiter/s2s/s2s-client/server.py#L726-L752)).
8. **`/api/config`**: nueva clave booleana `"apipilot": bool(S2S_APIPILOT_TOOLS_URL)` + docstring, log extendido
([L2544-L2579](file:///home/duxiter/s2s/s2s-client/server.py#L2544-L2579)).
9. **Modelo Pydantic** `ApipilotExecuteRequest(name:str, arguments:dict={})` ([L2582-L2585](file:///home/duxiter/s2s/s2s-client/server.py#L2582-L2585)).
10. **`GET /api/apipilot/tools`** proxy público ([L2588-L2651](file:///home/duxiter/s2s/s2s-client/server.py#L2588-L2651)):
query `?include_unauthorized=true(default)&refresh=false`, envoltorio uniforme `{success, data: {count, include_unauthorized, tools}, error}`, 501 si env no configurado, HTTPException convertidas a envoltorio.
11. **`POST /api/apipilot/execute`** proxy público ([L2654-L2700](file:///home/duxiter/s2s/s2s-client/server.py#L2654-L2700)):
body `{name, arguments}`, PRE-CHECK 1-2-3 vía `_apipilot_execute_tool`, código status respuesta = `error.status_code` si error, mensajes de cara al usuario en español §2.2.
---
## [2026-09-23]
### Añadido — Botón «Descargar conversación» en la cabecera del panel chat (export JSON + TXT client-side offline)
Nuevo botón circular icono descarga junto al testigo «En línea» a la derecha
de la cabecera del panel `#chat-panel`. Al pulsar serializa TODO el
historial persistente (`.hist-msg` dentro de `#chat-history`) y descarga DOS
archivos simultáneamente, SIN llamadas al backend, 100 % client-side y
offline-safe (Blob + URL.createObjectURL + `<a download>`):
1. `vocero-conversacion[-<nombreUsuario>]-YYYYMMDD-HHMMSS.json` — estructura
con metadatos `{ exportedAt, messageCount, messages: [...] }`. Tipos soportados:
- `user` / `assistant` → `{ type, text, timestamp, name?, initials?, interrupted? }`
- `tool` → `{ type: "tool", timestamp, name, args, output }` (resultado llamadas tool)
- `image` → `{ type: "image", timestamp, dataUrl }` (capturas webcam camera tool)
2. `vocero-conversacion[-<nombreUsuario>]-YYYYMMDD-HHMMSS.txt` — log legible
para humanos en español, cada entrada formateada como
`HH:MM Nombre / Vocero / HERRAMIENTA <nombre>: texto`.
**UI / UX (todo en español, como solicitado):**
- `aria-label="Descargar conversación"`
- `title="Descargar conversación (JSON + TXT)"`
- **Feedback OK**: tras export correcto botón se pinta verde 650 ms
(`.chat-export-ok` con ring + glow).
- **Feedback vacío**: si no hay mensajes → sin descarga, botón flash
ámbar + micro-shake 0.42s (`.chat-export-empty`) + console info
`[ui/chat.js:exportConversation] No hay mensajes aún — exportación omitida.`
- **Nombres de fichero saneados**: helper `_safeName(...)` elimina unicode /
caracteres especiales / espacios → `a-z 0-9 - _`.
- **Estilos coherentes**: dark (glass/aurora) + white (estilo screenshot móvil)
ambos tematizados con colores existentes `var(--brand)`, `#16a34a`, `#f59e0b`.
**Garantías:**
- 🔒 Cero datos transmitidos fuera del navegador (ningún POST, ningún analytics).
- Las burbujas efímeras sobre el orbe (`.bubble` / `.bubble-stack`) se
ignoran intencionadamente: exporta SOLO el histórico permanente, el que es significativo.
- Never-throws: todo el export envuelto en `try/catch`; error solo en consola.
- `window.URL.revokeObjectURL(...)` tras 4 s → sin fugas de memoria Blob.
- Disponible en TODOS los temas (dark / white / default).
**Archivos modificados:**
- **`index.html`** ([index.html](file:///home/duxiter/s2s/s2s-client/index.html)):
1. Wrapper `.chat-header-actions` a la derecha de la cabecera, que engloba
la pill «En línea» + nuevo botón `#chat-export-btn` (SVG flecha-abajo)
([L374-L394](file:///home/duxiter/s2s/s2s-client/index.html#L374-L394)).
- **`ui/chat.js`** ([chat.js](file:///home/duxiter/s2s/s2s-client/ui/chat.js)):
1. Helper module-level `_safeName(prefix)` para sanear nombres de fichero
([L38-L50](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L38-L50)).
2. Ref `this._exportBtn = getElementById("chat-export-btn")` + listener `click`
en constructor `ChatView` ([L74-L78](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L74-L78)).
3. Nueva sección `// ── Export / download conversation ──` con 4 métodos:
- `_serializeConversation()` → extrae rows `.hist-msg` via querySelector
`:scope > .hist-msg`, gestiona `user/assistant/tool/image`, preserva
timestamp `data-timestamp`, nombre usuario, initials, flag interrupted
([L521-L582](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L521-L582)).
- `_conversationToText(conv)` → pretty-format texto en español con
cabecera fecha, totales, bloques tool indentados, preview imagen dataURL
([L584-L621](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L584-L621)).
- `_downloadBlob(content, filename, mime)` → patrón `<a download>` +
revokeObjectURL ([L623-L645](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L623-L645)).
- `exportConversation()` (público) → entry-point, nombre fichero con
sello UTC local `YYYYMMDD-HHMMSS`, incluye nombre perfil Vocero si
existe, lógica empty/OK/error feedback, 2 descargas distintas
([L647-L697](file:///home/duxiter/s2s/s2s-client/ui/chat.js#L647-L697)).
- **`style.css`** ([style.css](file:///home/duxiter/s2s/s2s-client/style.css)):
1. `.chat-header-actions` flex-row right-aligned
([L4919-L4931](file:///home/duxiter/s2s/s2s-client/style.css#L4919-L4931)).
2. `.chat-export-btn` base (dark-theme): 32×32 pill, glassy, hover/focus
states, transiciones 120-160 ms
([L4932-L4968](file:///home/duxiter/s2s/s2s-client/style.css#L4932-L4968)).
3. Estados feedback: `.chat-export-ok` (verde) + `.chat-export-empty`
(ámbar + animación shake 5 pasos)
([L4969-L4989](file:///home/duxiter/s2s/s2s-client/style.css#L4969-L4989)).
4. Variantes white-theme completas: `.chat-header-actions`, base, hover,
active, OK, empty
([L4991-L5015](file:///home/duxiter/s2s/s2s-client/style.css#L4991-L5015)).
### Añadido — `S2S_LATEST_SUMMARY_TEXT`: override estático del resumen última conversación (texto limpio, ninguna llamada HTTP)
Nueva variable de entorno que, si se establece, tiene **prioridad absoluta** sobre `S2S_LATEST_SUMMARY`
y se salta completamente la fetch HTTP upstream (login-by-phone, Bearer, timeout, JSON unwrap: *todo bypaseado*).
Su valor en texto plano se inyecta directamente como bloque "latest conversation summary"
dentro del prompt LLM (tanto en el canal WebSocket `/v1/realtime` como en `POST /api/reasoning`).
Diseñada para el caso de uso en el que el operador quiere inyectar una versión
**limpia, esencial y pre-depurada** del resumen sin depender de ningún endpoint externo.
**Valor `.env` de ejemplo (multilínea con `\n`):**
```env
# S2S_LATEST_SUMMARY=https://api.example.com/summary # ← desactivado: S2S_LATEST_SUMMARY_TEXT tiene prioridad
S2S_LATEST_SUMMARY_TEXT=Última conversación 2026-09-22:\n- Usuario preguntó por mortalidad salmón AquaChile semana 38.\n- Se refirió al reporte Porcentaje Mortalidad.
```
**Comportamiento clave:**
- Las secuencias `\n` en el valor env se interpretan como salto de línea real (misma convención que `S2S_PRIMARY_PROMPT`)
- **No requiere `phoneNumber`**: funciona también para sesiones genéricas sin identificativo de llamante
- El wrapper `LATEST_SUMMARY_HINT` + el tope a **1600 caracteres** se siguen aplicando igualmente (pasa siempre a través de `_resolve_latest_summary_context`)
- El bloque se clasifica correctamente por el layer dedup (marker invisible `SUMMARY_FULL_MARKER`) — sin riesgo de duplicado
- Si `S2S_LATEST_SUMMARY_TEXT` está vacío → **fallback** transparente al flujo anterior (`S2S_LATEST_SUMMARY` URL + `phoneNumber`)
**Logs esperados en el servidor para confirmar override activo:**
```
[INJECT-SUMMARY] S2S_LATEST_SUMMARY_TEXT OVERRIDE active (len=<N> chars). Skipping HTTP fetch; injecting raw static plain-text summary directly.
```
**Archivos modificados:**
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)):
1. Definición nueva env `S2S_LATEST_SUMMARY_TEXT` + interpretación escape `\n` ([L128-L140](file:///home/duxiter/s2s/s2s-client/server.py#L128-L140)).
2. Banner `[STARTUP]` extendido con campo `summaryText=%s` para confirmación visual a boot ([L275-L297](file:///home/duxiter/s2s/s2s-client/server.py#L275-L297)).
3. **Rama override** ANTES de cualquier lógica HTTP dentro de `_fetch_latest_summary_for_phone(...)` ([L1563-L1589](file:///home/duxiter/s2s/s2s-client/server.py#L1563-L1589)):
- return directo del valor estático;
- log `[INJECT-SUMMARY] … OVERRIDE active`;
- vista previa raw primeros 800 chars si `S2S_DEBUG_PROMPT=true`.
4. **Puerta REASONING:** `needs_summary = bool(S2S_LATEST_SUMMARY) or bool(S2S_LATEST_SUMMARY_TEXT)` ([L2153](file:///home/duxiter/s2s/s2s-client/server.py#L2153)).
5. **Skip REASONING relajado:** condición `(not phone) and (not needs_reports) and (not episode_id) and (not S2S_LATEST_SUMMARY_TEXT)` — el texto estático no necesita phone ([L2159-L2168](file:///home/duxiter/s2s/s2s-client/server.py#L2159-L2168)).
6. **Ejecución summary REASONING:** `needs_summary and (phone or S2S_LATEST_SUMMARY_TEXT)`; log "READY" extendido con flag `summaryTextOverride=%s` ([L2240-L2256](file:///home/duxiter/s2s/s2s-client/server.py#L2240-L2256)).
7. **Puerta WS connect:** mismo `needs_summary` extendido con `S2S_LATEST_SUMMARY_TEXT` ([L2644](file:///home/duxiter/s2s/s2s-client/server.py#L2644)).
8. **Skip WS connect relajado:** incluso sin `phoneNumber` si hay `S2S_LATEST_SUMMARY_TEXT` la pipeline arranca ([L2653-L2664](file:///home/duxiter/s2s/s2s-client/server.py#L2653-L2664)).
9. **Ejecución summary WS:** misma condición relajada + log "READY" con campo `summaryTextOverride=%s` ([L2725-L2746](file:///home/duxiter/s2s/s2s-client/server.py#L2725-L2746)).
- **`INJECTIONS.md`** ([INJECTIONS.md](file:///home/duxiter/s2s/s2s-client/INJECTIONS.md)):
1. Sección **4. Blocco summary ultima conversazione** completamente reescrita (versión italiano original).
---
## [2026-09-07]
### Añadido — Soporte `episode_id` (inject de episodio podcast)
Nuevo parámetro query string `?episode_id=` junto a `?phoneNumber=` que
habilita la carga y la inyección automática de un episodio podcast como
contexto PRIMARIO en el prompt LLM.
**URL de ejemplo:**
```
/?phoneNumber=%2B447432477433&episode_id=episode-19569f77-a06e-5ce5-a52f-6299dbd67d03
```
**Archivos modificados:**
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)):
1. Añadida la constante `PODCAST_EPISODE_HINT` ([L426-L429](file:///home/duxiter/s2s/s2s-client/server.py#L426-L429)) para el hint semántico en el prompt.
2. Añadida la función `_resolve_episode_context(raw_value)` ([L475-L504](file:///home/duxiter/s2s/s2s-client/server.py#L475-L504)): normaliza la respuesta API (dict/list/str, posible wrapper `{"result": …}`), la serializa en JSON (tope 24.000 chars) y la envuelve con los marcadores `<<<PODCAST_EPISODE_BEGIN/END>>>`.
3. Añadida la función asíncrona `_fetch_podcast_episode(episode_id)` ([L1010-L1078](file:///home/duxiter/s2s/s2s-client/server.py#L1010-L1078)): GET directo a
`https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}`
(timeout 15s, follow redirects, manejo de errores tolerante — nunca rompe el flujo, devuelve `None` en caso de fallo).
4. Extendida `_augment_instructions_with_profile(...)` ([L1137-L1181](file:///home/duxiter/s2s/s2s-client/server.py#L1137-L1181)) con el nuevo parámetro `episode_context_text`; el orden de inject ahora es (más específico → más genérico): **episode → profile → reports → summary**.
5. Extendida `_patch_client_session_update_with_profile(...)` ([L1184-L1306](file:///home/duxiter/s2s/s2s-client/server.py#L1184-L1306)) con soporte a `episode_context_text` y logging dedicado `[INJECT-EPISODE]` / `[DEBUG-EPISODE]`.
6. WebSocket connect handler ([L1791-L1959](file:///home/duxiter/s2s/s2s-client/server.py#L1791-L1959)):
- lee `episode_id` desde query params junto a `phoneNumber`;
- ejecuta fetch episodio + resolve ANTES de profile/reports/summary;
- guarda `injected_episode_context` e `inject_episode_id` en el `session_ctx` compartido.
7. `_client_to_upstream` ([L2649-L2661](file:///home/duxiter/s2s/s2s-client/server.py#L2649-L2661)): pasa `ctx_episode_context` al patch de `session.update`.
8. Endpoint `POST /api/reasoning`:
- extendido el modelo `ReasoningRequest` con el campo `episode_id: str | None = None` ([L1309-L1323](file:///home/duxiter/s2s/s2s-client/server.py#L1309-L1323));
- en el handler ([L1411-L1577](file:///home/duxiter/s2s/s2s-client/server.py#L1411-L1577)) fetch episodio independiente de phone;
- `inject_episode_text` se hace **prepend** sobre `user_context_parts` (primero de todos);
- reenvío a NODE_RED / OPENAI con logging extendido (`episodeIdLen`).
- **`main.js`** ([main.js](file:///home/duxiter/s2s/s2s-client/main.js)):
1. Lectura desde query string: `const voceroEpisodeId = pageParams.get("episode_id")` ([L676-L680](file:///home/duxiter/s2s/s2s-client/main.js#L676-L680)).
2. `fetchOpenAiReasoningAnswer` ([L1337-L1364](file:///home/duxiter/s2s/s2s-client/main.js#L1337-L1364)): campo `episode_id` añadido al body POST `/api/reasoning`, logKey y `console.info` extendidos.
3. Construcción URL WebSocket ([L2323-L2342](file:///home/duxiter/s2s/s2s-client/main.js#L2323-L2342)): `episode_id` añadido a query params tanto con la API `URL.searchParams` como en el fallback string (concat con `&`).
---
### Añadido — `S2S_START_SESSION_MODE`: 3 modos de inicio de sesión
Variable de entorno leída por el servidor y propagada al cliente vía `GET /api/config.startSessionMode`.
Objetivo: controlar si y cómo se enciende automáticamente el micrófono y si el modelo saluda
primero (flujo similar al wake-word). Valores soportados:
| Valor | Descripción |
|---|---|
| *(vacío) / default* | Inicio manual: solo tap sobre el Orb / botón Start |
| **`TALK`** | Después de 2s overlay a pantalla completa "Toca en cualquier lugar para empezar" (policy safe para Safari/iOS `getUserMedia`). Click/touch/tecla dentro de la user gesture → `doStart()`. El cliente WS inyecta bootstrap `"hola vocero"` + `requestResponse()`. |
| **`TALKFIRST`** | Después de 2s **intenta `doStart()` DIRECTAMENTE SIN click**, precargando `pendingWakeWordConfirmationText = "hola vocero"` (misma variable de la wake-word real). Si los permisos de mic ya están "allow always" funciona sin click. Si falla (policy Safari/iOS) → fallback sobre el mismo unlocker de TALK con mensaje "permiso micrófono". |
**Archivos modificados:**
- **`server.py`** ([server.py](file:///home/duxiter/s2s/s2s-client/server.py)):
1. Lectura env: `S2S_START_SESSION_MODE = os.environ.get(...)`, `.strip().upper()` ([L150](file:///home/duxiter/s2s/s2s-client/server.py#L150)).
2. Banner `[STARTUP]` extendido con el campo `startSessionMode=` ([L196-L212](file:///home/duxiter/s2s/s2s-client/server.py#L196-L212)).
3. Endpoint `GET /api/config` → nuevo campo `startSessionMode` + log `[INTERNAL] /api/config` ([L1383-L1400](file:///home/duxiter/s2s/s2s-client/server.py#L1383-L1400)).
- **`index.html`** ([index.html](file:///home/duxiter/s2s/s2s-client/index.html#L32-L39)):
- Añadido nuevo overlay `#talk-unlocker` a pantalla completa (z-index 9999), `hidden` por defecto, con tarjeta "Preparando Vocero… / Toca en cualquier lugar para empezar a hablar" + spinner degradado.
- **`style.css`** ([style.css](file:///home/duxiter/s2s/s2s-client/style.css#L145-L215)):
- Clases `.talk-unlocker`, `.talk-unlocker-hint`, `.talk-unlocker-spinner`, `.talk-unlocker-title`, `.talk-unlocker-sub` + animaciones fade-in y pop-up de tarjeta.
- **`main.js`** ([main.js](file:///home/duxiter/s2s/s2s-client/main.js)):
1. Variable global `startSessionMode = ""` + comentario ([L628-L632](file:///home/duxiter/s2s/s2s-client/main.js#L628-L632)).
2. Parse dentro de `fetchConfig()` → uppercase, log en `reportDebug("B",…)` ([L2309-L2320](file:///home/duxiter/s2s/s2s-client/main.js#L2309-L2320)).
3. Paso a cliente WS → `startWithResponse: startSessionMode === "TALK"` ([L2683-L2692](file:///home/duxiter/s2s/s2s-client/main.js#L2683-L2692)).
4. Bloque scheduling auto-start ([L2976-L3083](file:///home/duxiter/s2s/s2s-client/main.js#L2976-L3083)):
- `bootstrapWithAutoStart()`: `await fetchConfig()` → `maybeScheduleAutoStart()`;
- `setTimeout(2000 ms)` warmup para finalizar fetch de episode/profile/reports;
- `showTalkUnlocker(fallbackHint)`: gestiona labels personalizadas en caso de fallback TALKFIRST;
- **`tryTalkFirstDirect()` (TALKFIRST)**: set `pendingWakeWordConfirmationText="hola vocero"` + `doStart()` + try/catch con teardown y fallback unlocker;
- listener unlocker click/touchstart/keydown → `startTalkFromGesture()` con `createResumedAudioContext()` dentro de la gesture (para iOS silent-free).
5. `onClientStatus()` ([L2878-L2932](file:///home/duxiter/s2s/s2s-client/main.js#L2878-L2932)) añadido `hideTalkUnlocker()` en `connecting/queued/your-turn/connected`.
- **`ws/s2s-ws-client.js`** ([s2s-ws-client.js](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js)):
1. Constructor: nuevas propiedades `_startWithResponse`, `_startWithResponseText` (default `"hola vocero"`), `_initialResponseSent` guardia one-shot ([L163-L178](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L163-L178)).
2. Handler `session.updated` ([L688-L713](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L688-L713)): **Solo para `TALK`** (no TALKFIRST, que en cambio usa `pendingWakeWordConfirmationText`): `this.sendUserText("hola vocero") + this.requestResponse()` — mismo code path identico de la wake-word.
3. `requestResponse` / `_createResponseNow` restaurados limpios (solo payload opcional `image`; ningún shaped `response.create`).
---
### Modificado — `#reports-sidebar-toggle` visible solo con `?debug=true`
El botón "reports" en la topbar (toggle del panel lateral reports) ahora es
`data-debug-only="true"` — por lo tanto solo se muestra si la URL contiene
`?debug=true`, de manera análoga al cluster `.topbar-right`.
**Archivos modificados:**
- **`index.html`** → `#reports-sidebar-toggle` ([index.html](file:///home/duxiter/s2s/s2s-client/index.html#L47-L50)): añadido `data-debug-only="true"`.
- **`main.js`** → `applyDebugOnlyVisibilityGate()` ([L48-L109](file:///home/duxiter/s2s/s2s-client/main.js#L48-L109)): extendido para iterar *todos* los elementos `[data-debug-only="true"]` (no solo `.topbar-right`); línea 1227: `reportsSidebarToggle.hidden = false` condicionado a `qs.get("debug") === "true"` ([L1227-L1246](file:///home/duxiter/s2s/s2s-client/main.js#L1227-L1246)) + log rojo `🚫 FORZADO HIDDEN` cuando falta debug=true.
- **`style.css`** → regla de seguridad ya presente: `[data-debug-only]:not([data-debug-enabled]) { display:none !important; visibility:hidden; opacity:0 }` ([L532-L537](file:///home/duxiter/s2s/s2s-client/style.css#L532-L537)).
---
## [2026-09-04]
### Modificado — Chat siempre visible + integración seamless
Reorganizado el layout para mostrar el chat textual de forma permanente e
integrado visualmente con el Orb en la parte superior del stage.
- Eliminado botón toggle chat en la topbar (`#chat-btn`) y relativo overlay.
- Stage con el Orb en la parte superior; nuevo `#chat-panel` a continuación (historial
+ form input fijo sobre el footer sticky).
- Actualizada lógica en `main.js` y `ui/chat.js` para el envío de texto mediante
`client.sendUserText` y auto-scroll sin controles de visibilidad del panel.
- Versión banner deploy: **2026-09-03 14:40**.
### Resuelto — Estabilidad integración chat mobile
- Error `Missing element: #chat-btn` resuelto con selectores non-strict
(`document.querySelector`) para elementos opcionales eliminados.
- Refactorización estética "seamless": chat sin marcos, sombras ni fondos
(`background:transparent`, `box-shadow:none`).
- **Layout mobile:**
- padding incrementado en `#app` (56px) y `.chat-history` (96px) para evitar
superposiciones con el footer sticky;
- márgenes fijos en las burbujas: `margin-right:22px` (Assistant), `margin-left:6px` (User).
- **Auto-scroll streaming:** implementado `_scrollToBottom` con cadena de retry
múltiples (`requestAnimationFrame` + `setTimeout`) para compensar el line-wrap
asíncrono durante la llegada de los tokens; `overflow-anchor:none` + `scroll-behavior:auto !important`.
### Añadido — Avatar usuario con iniciales + modal de perfil
- Sustituido el texto del nombre de usuario en el header por avatar circular 40×40px
(gradient de marca) conteniendo las iniciales calculadas (ej. "AV").
- Nueva `<dialog id="user-profile-dialog">` en `index.html` que se abre al click
sobre el avatar: muestra email, teléfono, rol, empresa, con soporte a shapes
anidadas del payload de perfil.
- Cierre del modal: tecla ESC, click fuera de la tarjeta, botón `×` dedicado.
---
## [Notas de formato]
- **Tope campo prompt episodio:** 24.000 caracteres (transcript largos).
- **Tope profile / summary:** 1.600 caracteres cada uno.
- **Tope reports list:** 12.000 caracteres.
- **Orden inject prompt LLM (WS session.update + reasoning):**
1. Episodio podcast actual (PRIMARIO)
2. Perfil llamador (identidad)
3. Lista reports / scripts podcast programados
4. Summary última conversación
- Todos los bloques están marcados con `<<<NOMBRE_BLOQUE_BEGIN/END>>>` para inspección de logs.
s2s-client/CHANGELOG.md
+234 -861
View File
File diff suppressed because it is too large Load Diff
s2s-client/CONTEXT.md
-108
View File
@@ -1,108 +0,0 @@
# Context glossary
Canonical terms for this space. A glossary, not a spec — it defines what words mean,
not how anything is built. Keep design/implementation detail in `DESIGN.md` and the
code.
## Speech-to-speech demo
The product: a voice conversation you have with a model by tapping the orb and
talking. "The demo" and "the space" refer to this same thing. It runs on Hugging
Face's open `speech-to-speech` backend.
## The pipeline
The ordered path a turn travels, from your voice to the orb's reply. Order is
meaningful — each stage consumes the previous one's output:
`you speak → VAD → STT → VLM → TTS → orb replies`
- **VAD** — voice activity detection. Decides *when* you are speaking, so the system
knows a turn has started and ended. Model: silero-vad.
- **STT** — speech to text. Transcribes your speech into words. Model:
nvidia/parakeet-tdt-1.1b.
- **VLM** — the vision-language model that composes the reply. Served via Cerebras.
Model: google/gemma-4-31B-it.
- **TTS** — text to speech. Speaks the reply back in the chosen voice. Model:
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice.
## Builder
A Hugging Face user credited with making the space, shown by HF username. Current
builders: tfrere, A-Mahla and andito. Distinct from the *models'* authors (nvidia, google,
Qwen, snakers4), who are credited per pipeline stage.
## Powered by
The infrastructure running the pipeline, named in the about panel: Hugging Face
Inference Endpoints (hosting) and Cerebras (LLM inference). Distinct from "built by"
(the people) and from the model authors (who trained each model).
## Tool
A function the model can call mid-conversation to do something the pipeline
can't do on its own (look something up, look through the camera). Tools are
declared to the backend in the session config; the model decides when to call
one. Distinct from the *pipeline stages* (VAD/STT/VLM/TTS), which always run.
## Tool executor
The client-side component that runs a tool when the model calls it and returns
the result to the backend, so the model can speak the answer. It is the missing
half of the round-trip: the backend already emits the call, the executor runs it
and replies. Distinct from the *tool* itself (the thing being run).
## Web search tool
A tool that looks something up on the web for the model. The model calls it with
a query; the tool executor forwards the query to the search proxy and returns the
results as the tool result. Activates only when a search key is available.
## Camera snapshot tool
A tool that lets the model look through the user's webcam. While enabled, a live
self-view is shown in the page (bottom-left); when the model calls the tool, the
executor captures a frame and sends it to the model as an image so the VLM can
see it. Distinct from the *preview* (what the user sees) and the *snapshot* (the
single frame sent to the model).
## Search proxy
The same-origin server route (`/search`) that holds the search key and calls the
external search provider on the client's behalf, so the key never reaches the
browser. Lives in the same container as the page. Distinct from the *s2s backend*
(the separate load-balanced speech-to-speech service).
## Tools panel
The dialog opened from the "Tools" button in the top-right, holding one switch per
tool (and the web-search key status). Turning a switch on/off declares or removes
that tool on the live session. Distinct from *Settings* (connection, voice,
instructions) and the *About panel* (project info).
## Identity block
The top-left corner of the topbar (replacing the old wordmark): the demo name, a
one-line blurb, and the "powered by" / "built by" credits, shown directly rather
than hidden behind a click.
## About panel
The popup opened from the (i) icon to the right of the identity block. Holds the
general introduction to the speech-to-speech project (with a repo link) and the
pipeline. Identity itself now lives in the corner, not here.
## Queue
The line of users waiting for a free conversation slot when every compute is
busy. You join the queue instead of being turned away; you leave it by reaching
the front or by giving up. Distinct from a *session* (an actual live
conversation) — being in the queue is not yet talking, and time spent waiting
never counts against your usage limit.
## Ticket
Your held place in the queue. Created when you join, it is what the demo checks
to tell you your position and to notice if you have left. A ticket is not a
session: it only promises a spot in line, not a compute.
## Position
How many people are ahead of you in the queue, shown while you wait ("You're #3
in line"). It only ever counts down. Distinct from an *estimated wait* — the
demo shows position, never a time, because wait time is unpredictable.
## Claim
The moment you reach the front and a free slot becomes yours — the queue hands
off to a real session and the conversation begins. This is also the point where
your usage limit first starts to matter (never while waiting).
## At capacity
The state where the queue itself is full, so new users can't even join the line
and are asked to try again shortly. Distinct from simply *busy* (all computes
taken but the queue still has room to wait in).
s2s-client/DESIGN.md
-205
View File
@@ -1,205 +0,0 @@
# Design language
The reference for keeping this app visually coherent as it grows. Read it before
touching `style.css`, `index.html`, or any DOM-building code in `main.js`. Every
rule here is already live in the codebase — this file explains the *why* so changes
extend the system instead of drifting from it.
---
## The thesis: color belongs to the voice
This is a voice app. The one thing in the room that should have color is the thing
that is talking. So:
- **The orb** carries saturated color. It glows, and the glow's hue changes with
conversational state.
- **Everything else is monochrome** — a precise cool-grey dark canvas. Surfaces,
borders, buttons, panels, the transcript: all greyscale.
- The **only** exceptions are tiny *role echoes* (a one-word mono label, a small
icon) that borrow the orb's state hue so the transcript reads in the same color
language the orb speaks. They are accents the size of a word, never fills.
- **Brand logos keep their own color.** The Hugging Face mark (`#ffd21e`) and the
Cerebras mark (`#f15a29`) render in their brand colors in the identity credits and
the about panel — a deliberate, owner-approved exception. It applies to those two
logos only; do not generalize it to other chrome.
If you find yourself adding a tinted background, a colored border, or a bright
button anywhere outside the orb, stop — that color almost certainly belongs to the
orb instead, or shouldn't exist.
---
## Color tokens
All defined in `:root` in `style.css`. Use the variables, never raw hex in rules.
### Canvas (the monochrome world)
| Token | Value | Use |
|---|---|---|
| `--bg` | `#0a0b10` | Page background (a cool near-black) |
| `--bg-elev` | `#13151c` | Raised surfaces: bubbles, panels, icon buttons |
| `--bg-elev-2` | `#1b1e29` | Surfaces on surfaces: history bodies, inputs |
| `--border` | `rgba(255,255,255,.08)` | Default hairline |
| `--border-strong` | `rgba(255,255,255,.16)` | Emphasised hairline |
| `--text` | `#f5f6fa` | Primary text; also the *primary button* fill |
| `--text-dim` | `rgba(245,246,250,.65)` | Secondary text |
| `--text-faint` | `rgba(245,246,250,.42)` | Captions, labels, footer |
### Voice (the only saturated hues)
These are the orb's state colors. They appear on the orb, and as small role echoes
in the transcript — nowhere else.
| Token | Value | Meaning |
|---|---|---|
| `--accent` / `--speaking` | `#8b7dff` violet | Assistant speaking |
| `--accent-2` / `--listening` | `#22d3ee` cyan | You / listening |
| `--processing` | `#f59e0b` amber | Thinking / tool call |
| `--error` | `#ff6a75` | Error |
| `--success` | `#34d399` | Ready / connected |
### Role echoes (semantic aliases — use these in chat code)
| Token | Maps to | Where it shows |
|---|---|---|
| `--voice-user` | cyan | `YOU` label + user bubble accents |
| `--voice-assistant` | violet | `ASSISTANT` label + assistant accents |
| `--voice-tool` | amber | `TOOL CALL` label, wrench icon |
**Why this mapping:** it mirrors the orb exactly — when *you* speak the orb is cyan
(`state-listening`), when the *assistant* speaks it's violet (`state-ai-speaking`),
when it's working it's amber (`state-processing`). The transcript is a quiet replay
of the orb's color story.
### Orb state → glow (`.circle.state-*` → `--glow`)
| State | Glow |
|---|---|
| `signed-out` | violet `#8b7dff` |
| `authenticated` / `ready` | green `#34d399` |
| `connecting` / `connected` / `starting` | yellow `#facc15` |
| `listening` / `user-speaking` | cyan (`--listening`) |
| `processing` | amber (`--processing`) |
| `ai-speaking` | violet (`--speaking`) |
| `error` | red (`--error`) |
Adding a new state? Give it a `--glow`, and if it surfaces in the transcript, add a
matching `--voice-*` alias rather than a one-off color.
---
## Typography
Two faces, two jobs. Never reach for a third.
- **Inter** — body and UI. Wordmark, buttons, inputs, panel titles, prose, history
message bodies. The workhorse; it should feel neutral and get out of the way.
- **Geist Mono** (`--font-mono`) — the **machine voice**. Reserved for text the
*system* emits or identifiers it reports, never for human prose.
### When mono is correct
Mono signals "this is the machine talking or naming itself." Use it for:
- the orb's status caption (`.circle-caption`)
- role eyebrows (`YOU` / `ASSISTANT` / `TOOL CALL`)
- tool-call names and argument JSON
- the `·WebSocket` transport tag, bitrate readouts, connection identifiers
- the empty-state label
Mono text is set uppercase with `letter-spacing: ~0.1–0.14em` and weight `500`, so it
reads as a typed status line, not a headline. Body copy, button labels, and
explanatory `small` text stay **Inter** — putting prose in mono breaks the metaphor.
The font is loaded in `index.html`; the stack falls back to system mono gracefully if
the CDN is blocked.
---
## Layout
- **One continuous canvas.** No dividers under the topbar or above the footer, no
panel chrome competing with content. The topbar and footer float over the stage.
Keep it that way — a new section earns a hairline (`--border`) only if it genuinely
needs separating.
- **The orb is the hero and the center of gravity.** It sits dead-center on the
stage. Controls flank it (mic / stop), captions sit beneath. Don't crowd it.
- **Hairlines, not boxes.** Separation comes from `1px` borders at 8–16% white and
from spacing, not from heavy fills or shadows. Shadows are soft and low
(`0 4px 18px rgba(0,0,0,.32)`), used only to lift floating elements (bubbles,
panels, modal).
- **Radii:** `--radius-sm: 8px` (buttons, inputs, chips), `--radius-md: 14px`
(bubbles, message bodies, modal), `--radius-lg: 22px` (reserved). Pick by element
size; don't invent new values.
- **Two reading surfaces for the transcript:** ephemeral bubbles top-right (desktop
only) that log and fade, and a slide-in history panel for review. On phones the
bubble stream is dropped and the panel goes full-screen — the panel is the single
source of truth there.
---
## Components
- **Buttons.** Default (`.btn`) is a neutral elevated surface. The *primary* button
is **near-white on dark** (`--text` fill, `--bg` text) — the highest-contrast thing
on the page that *isn't* the orb. There is no colored button; emphasis comes from
contrast, not hue.
- **Icon buttons** (`.icon-btn`) are `36px`, elevated surface, dim icon that brightens
on hover. Side controls (`.side-btn`) are circular, collapse to zero size until the
session is live (by width on desktop, by height in the mobile column).
- **Chat bubbles & history messages** share one neutral surface. They are
distinguished by **side** (you = left, assistant = right) plus the **mono role
label** in the role-echo hue — not by tinted fills. Tool entries use the wrench icon
+ mono + amber, on the same neutral surface.
- **Badge** (new-message dot) is monochrome white — a signal, not a color accent.
- **Focus** is visible and neutral: inputs focus to `--text-dim`; the orb uses a
`--glow`-colored outline (it's the orb, so color is allowed).
---
## Motion
- **The orb is audio-reactive, not timer-driven.** Mic RMS (`--audio-level`) and the
assistant output level (`--ai-audio-level`) drive scale/opacity at display rate, so
every syllable moves it. This is the signature animation — keep new motion
subordinate to it.
- **Quiet by default.** Breathing/glow throbs are slow (1.4–2.4s) and low-contrast.
Resist adding scattered micro-animations; an orchestrated moment beats many small
ones, and excess motion reads as AI-generated.
- **First paint is frozen.** `body.booting` disables all transitions until the first
frame commits (stripped after one rAF in `main.js`). Anything new that would
otherwise animate-in on load must respect this.
- Honor `prefers-reduced-motion` for any motion you add.
---
## Writing / copy voice
- Sentence case, plain verbs, no filler. Tuned and quiet — match the minimal canvas.
- Name things by what the user controls, not by the system's internals. A button says
exactly what it does, and keeps the same word through the flow.
- **Empty states invite action** ("Tap the orb and start talking"), they don't just
set a mood.
- **Errors state what happened and how to recover**, in the interface's voice — they
don't apologize and are never vague.
- Mono labels are terse identifiers (`YOU`, `TOOL CALL`); prose stays in Inter.
---
## Responsive floor (non-negotiable)
Every change ships meeting these:
- Works down to a `360px`-wide phone. The `@media (max-width: 600px)` block already
handles the phone layout — extend it, don't fight it.
- Visible keyboard focus on every interactive element.
- `prefers-reduced-motion` respected.
- Tap targets ≥ `44px`; `touch-action: manipulation` on anything tappable.
---
## Before you ship — the mirror check
1. Is every saturated color either on the orb or a word-sized role echo? If a fill or
border is colored, remove that accessory.
2. Is mono used only for machine/system text, and Inter for everything human?
3. Does any new state have both a `--glow` and (if it appears in chat) a `--voice-*`
alias?
4. Are separations hairlines + spacing, not boxes and heavy shadows?
5. Did you add motion? Is it quieter than the orb and reduced-motion-safe?
6. Remove one accessory. The minimal look survives on precision, not addition.
s2s-client/Dockerfile
+6 -4
View File
@@ -1,13 +1,15 @@
# Single container: serves the static front-end AND the /api/search proxy.
# Build from the project root: docker build -f s2s-client/Dockerfile -t s2s-client .
# Serve the client, same-origin API proxies, and shared project documentation.
# HF Spaces (sdk: docker) routes traffic to $PORT, default 7860.
FROM python:3.11-slim
WORKDIR /app
WORKDIR /app/s2s-client
COPY requirements.txt .
COPY s2s-client/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
COPY s2s-client/ .
COPY docs/ /app/docs/
EXPOSE 7860
s2s-client/HOWITWORKS.md
-584
View File
@@ -1,584 +0,0 @@
# HOWITWORKS — demo S2S
> Compagno di [ARCHITECTURE.md](file:///home/duxiter/s2s/demo/ARCHITECTURE.md): spiega il
> ciclo di vita concreto dei dati in 3 moduli su cui oggi ci siamo concentrati:
> 1. Il campo `instructions` da client a `ai-server:12345`
> 2. `MCP_SEARCH_TRANSLATION_PHRASES` + ranking tool MCP lato client
> 3. **FASE 2 — Dichiarazione tools nativi a `ai-server:12345` via Realtime `session.update.tools[]`**
---
## 0. FASE 2 — Integrazione tools nativi S2S upstream (ai-server:12345)
### Obiettivo
Prima della Fase 2, i 100+ tool MCP Aquachile erano solo **testo dentro `instructions`** (9 KB di prompt). L'upstream non sapeva eseguirli e produceva frasi tipo:
> *"Perfecto, buscaré… [list_recent_reports]"*
Con la Fase 2 dichiariamo i tools tramite il **protocollo nativo OpenAI Realtime GA**:
```
session.update { tools: [...], tool_choice: "auto" }
```
→ Se `ai-server` lo supporta, produce eventi veri `response.function_call_arguments.done` con nome tool + argomenti JSON. Il listener già esistente nel client (s2s-ws-client.js) si attiva automaticamente e tutto il resto della pipeline — `runTool()` → `POST /api/mcp/tool` Aquachile → `client.sendToolOutput()` — funziona senza altre modifiche.
### 0.1 Dove è stato implementato lato proxy (s2s_ui container)
| Modulo | File / Righe | Cosa fa |
|---|---|---|
| **Cache + normalizzazione** | [server.py:451-544](file:///home/duxiter/s2s/demo/server.py#L451-L544) | `_normalize_tools_for_realtime(raw_tools)`: converte qualsiasi formato tool Aquachile → formato OpenAI Realtime standard. Non alza eccezioni; tool invalidi saltati con warning. |
| **Inject WS upstream** | [server.py:1034-1075](file:///home/duxiter/s2s/demo/server.py#L1034-L1075) | Subito dopo `TUNNEL ESTABLISHED`, **PRIMA** che inizi lo streaming audio: fetch cache `_agent_profile_data()` → `session.update` con `instructions` (profilo agente Aquachile) + `tools[]` + `tool_choice=auto`. Errori suppressi: non rompiamo mai il flusso audio. |
| **Log di successo KEY** | [server.py:312-334](file:///home/duxiter/s2s/demo/server.py#L312-L334) | Quando arriva `response.function_call_arguments.done` dall'upstream → 2 log `[MCP INJECT] ✓ UPSTREAM ha emesso…` (segnale CHE LA STRATEGIA B FUNZIONA) |
| **Proxy MCP già esistente** | [server.py:600-656](file:///home/duxiter/s2s/demo/server.py#L600-L656) | `POST /api/mcp/tool` → Aquachile JSON-RPC 2.0 `tools/call`. Nessuna modifica necessaria. |
### 0.2 Sequenza temporale WS Realtime (Fase 2)
```
1. Browser → ws://s2s_server/v1/realtime (connessione)
2. server.py → websockets.connect(ai-server:12345)
3. [MCP INJECT] server.py → upstream:
{ "type": "session.update",
"session": { // ⚠️ FORMATO CORRETTO — TUTTO dentro session:{}
"instructions": "…system prompt profilo Aquachile…",
"tools": [ {type: "function", name:"list_recent_reports", …}, …79 più… ],
"tool_choice": "auto"
}
}
← LOG "[MCP INJECT] session.update upstream inviato (tools=79, fmt=nested_session OK)"
4. browser → ai-server: (turno audio utente) "el reporte de ayer por favor"
5. (se upstream ha tools[] in RuntimeConfig.session.tools — ORA SUPPORTATO Nativamente)
ai-server → server.py → browser:
{ "type": "response.function_call_arguments.done",
"call_id": "call_abc",
"name": "list_recent_reports",
"arguments": "{\"date_from\":\"2026-08-24\",\"limit\":5}" }
← LOG "[MCP INJECT] ✓ UPSTREAM ha emesso response.function_call_arguments.done!"
6. s2s-ws-client.js:803 → dispatch "toolcall"
7. main.js:3023 → runTool("list_recent_reports", args, "call_abc")
branch agentToolDefs → execRemoteMcpTool
8. execRemoteMcpTool → POST /api/mcp/tool → Aquachile JSON-RPC tools/call
LOG: "[Vocero MCP CALL]" → "[Vocero MCP HTTP]" → "[Vocero MCP OUT]"
9. client.sendToolOutput("call_abc", "Risultati: 5 report, ultimo=Report_2026-08-24.xlsx")
→ WS payload:
{ "type": "conversation.item.create",
"item": { "type": "function_call_output", "call_id": "call_abc", "output": "..." } }
10. ai-server continua il turno con TTS nativo:
"Aquí tienes los 5 reportes de ayer, el más reciente es Report_2026-08-24.xlsx..."
(nessun [tool_name] nel testo del TTS!)
```
### 0.3. Formato atteso da `ai-server:12345` — guida per modifiche sul backend S2S
**Se dopo il deploy non vedi il log `[MCP INJECT] ✓ UPSTREAM ha emesso…`**, vuol dire che `ai-server` oggi scarta il campo `tools[]` dal `session.update`. Ecco **esattamente i campi che devono essere supportati** (copia-incollabile nel codice di `ai-server`):
#### 0.3.1 `session.update` in ingresso (cosa manda oggi `server.py` — FORMATO NESTED CORRETTO)
⚠️ **REGOLA NON NEGOZIABILE**: TUTTI i campi di configurazione (instructions, tools, tool_choice, voice, model) devono stare **DENTRO l'oggetto `session`**. Non mandarli MAI alla radice del payload. Il validatore Pydantic strict di OpenAI Realtime GA rifiuta qualsiasi campo sconosciuto alla root.
```jsonc
{
"type": "session.update",
// ⚠️ TUTTO dentro "session":{} — il validator Pydantic strict è extra=forbid sulla root.
"session": {
// 1) Istruzioni di sistema (profilo agente Aquachile)
"instructions": "María, asistente virtual Aquachile, habla español de Chile…",
// 2) CATALOGO TOOLS — 79 tool Aquachile normalizzati
"tools": [
{
"type": "function",
"name": "list_recent_reports",
"description": "Restituisce l'elenco dei report più recenti, opzionalmente filtrati per data.",
"parameters": {
"type": "object",
"properties": {
"date_from": {"type": "string", "format": "date", "description": "Data inizio YYYY-MM-DD"},
"limit": {"type": "integer", "minimum": 1, "maximum": 50, "default": 10}
},
"required": []
}
}
// … altri 78 tool
],
// 3) Strategia selezione tool
"tool_choice": "auto"
}
}
```
**Perché prima non funzionava?** (ROOT CAUSE)
Il proxy stava inviando `instructions`, `tools`, `tool_choice` **ALLA RADICE** del payload, fuori da `session:{}`. Il modello Pydantic `SessionUpdateEvent` (OpenAI Realtime GA) ha `extra=forbid` e ha rifiutato con `Unknown or invalid event: session.update`. **Non era un rifiuto del campo `tools` in quanto tale** — era un rifiuto di qualsiasi campo non riconosciuto messo nel posto sbagliato. La prova: `runtime_config.session.instructions`, `.session.tools`, `.session.tool_choice` in `language_model.py:526-532` leggono TUTTI i campi da dentro `.session.*`.
**Audit sorgente backend `speech_to_speech` conferma: GIA' 100% WIRED PER NATIVI TOOLS**
| Componente | File / Righe | Conferma |
|---|---|---|
| LLM pick tools da sessione | `LLM/language_model.py:526-532` | `tools = runtime_config.session.tools` |
| Chat buffer gestisce FC/FCO | `LLM/chat.py:141-223 / 342-366 / 394-427` | `RealtimeConversationItemFunctionCall` + `FunctionCallOutput` |
| Response handler emette WS done event | `api/openai_realtime/handlers/response.py:336-345` | `ResponseFunctionCallArgumentsDoneEvent` con `call_id, name, arguments(STRINGA), response_id, item_id, output_index` |
| Conversation handler accetta ritorno output | `api/openai_realtime/handlers/conversation.py:30-96` | `handle_conversation_item_create` con `deferred_items` anti-race |
| Fallback parser retrocompatibile | `api/openai_realtime/service.py:267-325` | Se arrivano campi flat alla radice → promossi a nested `session` wrapper |
**Regole convalida obbligatorie lato `ai-server` per evitare rigetti silenziosi**:
| Regola | Valore |
|---|---|
| Nome tool | lunghezza ≤ 64, match regex `^[a-z0-9_]+$` |
| Numero max tools | Consigliato ≤ 80. Oggi il proxy manda 100+ con warning. |
| `parameters.type` | Sempre `"object"` (anche se vuoto) |
| `parameters.properties` | Sempre `{}` presente, anche se 0 proprietà |
| `tool_choice` | Valori accettati: `"none"`, `"auto"`, oppure `{type:"function",name:"…"}` |
| Chiavi ignote | NON fare strict reject. Logga warning e scarta solo la chiave. |
#### 0.3.2 Evento di ritorno richiesto da `ai-server` (quando il modello decide di chiamare un tool)
Quando il modello determina che serve un tool → emetti **3 eventi in ordine ESATTO**:
```
a) { "type": "response.function_call_arguments.started", "call_id": "call_abc123", "name": "list_recent_reports", "response_id":"resp_X", "item_id":"item_Y" }
(opzionale — se non hai streaming arguments puoi anche saltarlo)
b) (opzionale) N × { "type": "response.function_call_arguments.delta", "call_id": "call_abc123", "delta": "{\"dat" }
(solo se streammi il JSON argomenti per pezzi)
c) ✅ EVENTO RICHIESTO OBBLIGATORIO (senza questo il client non esegue il tool):
{
"type": "response.function_call_arguments.done",
"call_id": "call_abc123", // STRINGA OBBLIGATORIA — serve per la sendToolOutput
"name": "list_recent_reports", // STRINGA OBBLIGATORIA — nome del tool (come sopra)
"arguments": "{\"date_from\":\"2026-08-24\",\"limit\":5}",
"response_id": "resp_xyz",
"item_id": "item_zzz",
"output_index": 0
}
```
**⚠️ Importante**: `arguments` DEVE essere una **STRINGA JSON serializzata** (non un oggetto). Il client fa `JSON.parse(event.arguments)`. Se mandi un oggetto JS → parser fallisce e il tool non viene eseguito.
#### 0.3.3 Atteso come input successivo (il ritorno del tool dal client)
Dopo ~500ms-45s, il client manda indietro al S2S upstream:
```jsonc
{
"type": "conversation.item.create",
"item": {
"type": "function_call_output",
"call_id": "call_abc123",
"output": "5 report trovati. Ultimo: Report_2026-08-24.xlsx - Aquachile ventas"
}
}
→ Poi { "type": "response.create" }
```
→ L'S2S dovrebbe usare `output` come contesto nel prossimo turno di generazione TTS (quindi parla del risultato, senza tag `[tool]`).
### 0.4 Checklist di deploy Fase 2 (2026-08-25 — SORGENTE ai-server TROVATA in locale)
**Prerequisito**: Sorgente backend `ai-server:12345` = `/home/duxiter/s2s/speech_to_speech` (Python puro).
✅ **TEST CURL 2026-08-25 CONFERMATO**: `https://aquachile.azurianlab.com/api/v1/mcp/` è **ATTUALMENTE PUBBLICO** — accetta `tools/list` e `tools/call` SENZA API KEY. Con credenziali dummy `dummy-test-key/secret` restituisce HTTP 200 OUTPUT IDENTICO alla chiamata senza auth. Quindi NON serve valorizzare:
```
S2S_TOOL_EXECUTION_MCP_API_KEY=
S2S_TOOL_EXECUTION_MCP_API_SECRET=
```
⚠️ Nota: Il server accetta entrambi gli auth headers, ma attualmente non li valida. Restano configurati come best practice per eventuali versioni future dell'endpoint che attiveranno auth.
#### Ordine deploy 3 step
| # | Azione | Perché |
|---|---|---|
| 0 | ~~Valorizza `.env` MCP credentials~~ | ✅ **SUPERATO** — endpoint MCP Aquachile pubblico |
| 1 | **Restart container `s2s_ui`** | Deploy fix formato inject `fmt=nested_session` in [server.py:1054-1102](file:///home/duxiter/s2s/demo/server.py#L1054-L1102) |
| 2 | **Riavvia backend `ai-server:12345`** con il codice aggiornato di `/home/duxiter/s2s/speech_to_speech` (in particolare `api/openai_realtime/service.py`) | Deploy fallback parser retrocompatibile flat→nested in [service.py:267-325](file:///home/duxiter/s2s/speech_to_speech/api/openai_realtime/service.py#L267-L325) |
| 3 | **Hard Reload browser** (Ctrl+Shift+R o Disable cache + refresh) | Pulisce cache vecchio client JS |
#### Checklist di verifica DOPO deploy 3+1 step
```bash
# Terminal A
docker logs -f s2s_ui 2>&1 | grep -E 'MCP|Vocero MCP|session.update|function_call_arguments'
```
| Step | Trigger | Log atteso nel container | Esito |
|---|---|---|---|
| 1 | Apri pagina demo (connessione WS) | `[MCP INJECT] session.update upstream → tools=79 hasAgentPrompt=True tool_choice=auto` | ✅ Inject partito |
| 2 | Subito dopo step 1 | `[MCP INJECT] session.update upstream inviato (tools=79, fmt=nested_session OK)`<br>🔴 **NON DEVE** più comparire `[MCP INJECT] suppressed upstream error` — se compare ancora, il deploy di `speech_to_speech service.py` non è riuscito | ✅ Nessun rigetto |
| 3 | Console browser | 0 occorrenze `Unknown or invalid event: session.update` | ✅ Filtro OK |
| 4 | Frase trigger tool: *"¿dónde está el report de ayer?"* | **KEY LOG 🔑**<br>`[MCP INJECT] ✓ UPSTREAM ha emesso response.function_call_arguments.done! tool=list_recent_reports args_keys=[...]` | ✅ **Strategia B FUNZIONA — il modello produce tool calls native** |
| 5 | 0.5-1s dopo Step 4 | `[Vocero MCP CALL] name=list_recent_reports target=https://aquachile...`<br>`[Vocero MCP HTTP] name=list_recent_reports status=200`<br>`[Vocero MCP OUT] name=list_recent_reports preview=...resultKeys=...` | ✅ Tool eseguito realmente via REST Aquachile |
| 6 | 1-2s dopo Step 5 | TTS vocale naturale — es: *"Aquí tienes los 5 reportes de ayer, el más reciente es Report_2026-08-24.xlsx..."* | ✅ Nessun `(list_recent_reports)` / `[tool]` nel testo TTS |
| 7 | Frase non-tool: *"Hola, ¿cómo estás?"* | Risposta standard SENZA tool calls, POST /api/reasoning 200, Node-RED LOG_ONLY OK | ✅ Nessuna regressione Fase 1 |
---
### 0.5 Stato attuale (2026-08-25 11:00) — BLOCCO RISOLTO, pipeline 100% implementata
#### 🎯 Root Causa Reale (99.99% confidence, sorprendente)
**Ipotesi iniziale, SUPERATA**: *"ai-server rifiuta strict il campo `tools` perché non è implementato"*
**Realtà, AUDIT SORGENTE CONFERMATO**: TUTTO il sistema — **backend `speech_to_speech` + proxy `demo/server.py` + client `main.js` / `s2s-ws-client.js`** — è GIA' COMPLETAMENTE IMPLEMENTATO end-to-end per il tool calling nativo OpenAI Realtime GA. **L'unico blocco era un errore di formato payload nel proxy**:
```python
# ❌ SBAGLIATO (come era prima) — campi alla RADICE
_inject_payload = {
"type": "session.update",
"instructions": "...",
"tools": [...],
"tool_choice": "auto"
} # ← Pydantic SessionUpdateEvent extra=forbid → Unknown or invalid event
# ✅ CORRETTO (deployato in server.py:1054-1102) — TUTTO dentro session:{}
_inject_payload = {
"type": "session.update",
"session": {
"instructions": "...",
"tools": [...],
"tool_choice": "auto"
}
} # ← Perfetto, corrisponde a runtime_config.session.tools in LLM
```
**Prova inconfutabile**: [language_model.py:526-532](file:///home/duxiter/s2s/speech_to_speech/LLM/language_model.py#L526-L532) legge istruzioni, tools e tool_choice **TUTTI dentro `.session.*`**:
```python
instructions = response.instructions if ... else runtime_config.session.instructions
tools = response.tools if ... else runtime_config.session.tools
tool_choice = response.tool_choice if ... else runtime_config.session.tool_choice
```
#### 📋 Mappa riassuntiva: 7 componenti TUTTI già funzionanti
| Componente | File | Stato | Cosa fa |
|---|---|---|---|
| 🔧 **Normalizzatore tool** | `demo/server.py:464-544` | ✅ | 79 tool Aquachile → formato Realtime GA standard, 0 eccezioni |
| 🚀 **Inject WS upstream** | `demo/server.py:1054-1102` | ✅ **NUOVO FIX 2026-08-25** | Invia nested `session:{instructions,tools,tool_choice}` subito dopo tunnel |
| 🛡️ **Fallback parser retrocompatibile** | `speech_to_speech/service.py:267-325` | ✅ **NUOVO FIX 2026-08-25** | Se arrivano campi flat root → promossi a nested `session` |
| 🧠 **LLM + Chat buffer** | `speech_to_speech/LLM/*` | ✅ GIA' ESISTENTE | Tools da sessione, FC/FCO nel chat, pending dict anti-race |
| 📡 **WS response handler (tool call OUT)** | `speech_to_speech/handlers/response.py:336-345` | ✅ GIA' ESISTENTE | Emette `ResponseFunctionCallArgumentsDoneEvent` con arguments=STRINGA JSON, call_id, name, response_id, item_id, output_index=0 |
| 💾 **WS conversation handler (tool output IN)** | `speech_to_speech/handlers/conversation.py:30-96` | ✅ GIA' ESISTENTE | `conversation.item.create function_call_output` con `deferred_items` anti-race |
| 🌐 **Client WS → MCP REST → ritorno WS** | `demo/ws/s2s-ws-client.js:803-1077` + `demo/main.js:2180-2288 + 3023-3043` | ✅ GIA' ESISTENTE | (1) `response.function_call_arguments.done` → dispatch `toolcall` event<br>(2) `runTool()` branch `agentToolDefs` → `execRemoteMcpTool()` → `POST /api/mcp/tool`<br>(3) `client.sendToolOutput(callId, output)` → `conversation.item.create function_call_output`<br>(4) `client.requestResponse()` → prossimo turno TTS |
| 📣 **Proxy MCP REST Aquachile** | `demo/server.py:704-760` | ✅ GIA' ESISTENTE | `POST /api/mcp/tool` JSON-RPC 2.0 tools/call + log `[Vocero MCP CALL/HTTP/OUT]` |
| 🛡️ **Filtro errori WS V2** | `demo/server.py:1713-1792` | ✅ GIA' ESISTENTE | Sopprime `unknown_or_invalid_event` se mai dovesse ricomparire, passthrough altri errori |
#### 🚀 Prossimo passo: ESEGUIRE I 3 STEP DI §0.4
1. ~~Valorizza `.env` MCP API Key/Secret~~ ✅ **SUPERATO** (endpoint pubblico)
2. Restart `s2s_ui`
3. Restart `ai-server` con il codice aggiornato da `/home/duxiter/s2s/speech_to_speech`
4. Hard reload browser (Ctrl+Shift+R)
5. Testa: *"¿dónde está el report de ayer?"* → aspetta il key log `[MCP INJECT] ✓ UPSTREAM ha emesso response.function_call_arguments.done!`
**Se Step 5 (function_call_arguments.done) non compare dopo 3 frasi trigger**: apri una shell con `docker logs -f ai-server` per vedere se la sessione sta leggendo `.session.tools` dal runtime config.
---
#### ⚙️ MCP MASTER SWITCH (nuovo feature 2026-08-25)
**Controllato da una sola riga in `.env`**:
```
#S2S_TOOL_EXECUTION_MCP_URL=https://aquachile.azurianlab.com/api/v1/mcp # ⬅️ se commentata/vuota → MCP DISABLED
S2S_TOOL_EXECUTION_MCP_URL=https://aquachile.azurianlab.com/api/v1/mcp # ⬅️ se decommentata/valorizzata → MCP ENABLED
```
Implementato come `MCP_EXEC_ENABLED = bool(S2S_TOOL_EXECUTION_MCP_URL)` in [server.py:106-114](file:///home/duxiter/s2s/demo/server.py#L106-L114). Quando **DISABLED** (stato attuale `.env`):
| Punto di controllo | Comportamento con MCP DISABLED | Riferimento codice |
|---|---|---|
| **Startup log** | `[MCP MASTER] ⛔ DISABLED — S2S_TOOL_EXECUTION_MCP_URL is empty/commented in .env…` | [server.py:172-182](file:///home/duxiter/s2s/demo/server.py#L172-L182) |
| **WS `/v1/realtime` inject tools** | `session.update` contiene **solo `instructions`** (profilo agente); `tools[]` e `tool_choice` **non vengono mai mandati**. Log marker `[MCP INJECT] ⛔ SKIPPED tool injection …`. ⚠️ Il prompt profilo continua a funzionare. | [server.py:1081-1123](file:///home/duxiter/s2s/demo/server.py#L1081-L1123) |
| **`/api/config` bootstrap client** | Invia al browser `agentTools=[]` (invece dei 79) + `searchMcp=false`. Effetti lato client: ranker MCP early exit, prompt non contiene più "Available MCP tools:" e ranking hints, UI pannello MCP mostra "No MCP tools available.", nessun pulsante "Esegui tool". | [server.py:660-694](file:///home/duxiter/s2s/demo/server.py#L660-L694) |
| **`POST /api/mcp/tool` endpoint** | Se un client tenta comunque la chiamata (es. con codice vecchio): log `[MCP DISABLED] ⛔ /api/mcp/tool called but…` + HTTP 503 dettagliato (non crash, non exception trace). | [server.py:721-733](file:///home/duxiter/s2s/demo/server.py#L721-L733) |
Per **riabilitare la Fase 2 MCP in futuro**, 4 step:
1. ✏️ `.env` → togli `#` davanti a `S2S_TOOL_EXECUTION_MCP_URL=https://aquachile.azurianlab.com/api/v1/mcp`
2. 🔄 Restart `s2s_ui` (deploya MCP_EXEC_ENABLED=True + inject formato nested corretto)
3. 🔄 Restart `ai-server` con codice aggiornato `/home/duxiter/s2s/speech_to_speech` (fallback parser flat→nested)
4. 🌐 Hard reload browser Ctrl+Shift+R
---
## 1. Flow di `instructions` — da impostazioni utente a S2S upstream
**Definizione**: `instructions` è il **system prompt** del protocollo OpenAI Realtime GA.
Viene mandato al backend in 2 punti distinti:
| Canale | Dove | Destinatario |
|---|---|---|
| ✅ **TUNNEL WS AUDIO** → `session.update` all'inizio di ogni sessione | Browser → tunnel proxy → upstream S2S nativo (`ai-server:12345`) | Usato **dal modello S2S stesso** per la risposta vocale nativa |
| ✅ **REST PARALLELO** → body `POST /api/reasoning` | Browser → `server.py` → Node-RED o OpenAI | Usato **dal reasoning hook** esterno (oggi Fase 1 LOG_ONLY) |
Entrambi i canali partono dalla stessa funzione generatrice `effectiveReasoningInstructions()` nel client.
### Step-by-step
```
┌────────────────────────────────────────────────────────────────────┐
│ 1. BASE — DEFAULT_INSTRUCTIONS (main.js:26) │
│ "You are a friendly voice assistant. Keep replies short..." │
│ OR textarea Settings #instructions salvato in localStorage │
│ chiave "s2s.ws.instructions" (main.js:138, 201, 223) │
└───────────────────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────┐
│ 2. effectiveReasoningInstructions() (main.js:516-533) │
│ Appende in coda, in ordine: │
│ │
│ + (se ≥1 tool attivo) TOOL_USE_HINT (main.js:32-35) │
│ → "do not describe your capabilities... call tool immediately" │
│ │
│ + (se searchMcpEnabled + agentToolDefs) MCP_SEARCH_HINT + │
│ "Available MCP tools: …catalogo completo…" │
│ │
│ + (se searchMcpEnabled + tool suggeriti) riga ranking client-side │
│ "Current user request: \"${lastUserQuery}\". Ranking hint: 1. X │
│ 2. Y 3. Z — use as secondary signal only" │
│ │
│ + (se utente autenticato phone) voceroUserContextText() │
│ → es: "User profile: name=Juan Pérez phone=+569… account=…" │
└───────────────────────────────────────┬────────────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
┌─────────────────────────────┐ ┌──────────────────────────────┐
│ 3a. effectiveInstructions() │ │ 3b. POST /api/reasoning │
│ (main.js:537-539) │ │ body req.instructions │
│ │ │ (main.js:569) │
│ • mode=OPENAI → accoda │ │ = effectiveReasoningInst…() │
│ OPENAI_RENDER_HINT │ │ tale e quale (NO append) │
│ • mode=NODE_RED / PROXY → │ │ │
│ ritorna il valore │ │ → server.py lo gira a │
│ senza modifiche │ │ Node-RED o OpenAI │
└──────────────┬──────────────┘ └──────────────┬───────────────┘
│ │
▼ ▼
┌─────────────────────────────┐ ┌──────────────────────────────┐
│ 4a. startInstructions = │ │ 4b. /api/reasoning → │
│ effectiveInstructions() │ │ Node-RED log: │
│ (main.js:2829) │ │ [S2S NODE_RED LOG_ONLY] │
└──────────────┬──────────────┘ │ instructions len=… │
│ └──────────────────────────────┘
▼
┌─────────────────────────────────────────────────────┐
│ 5. new S2sWsRealtimeClient({ │
│ instructions: startInstructions │
│ voice, tools, noiseGate, audioContext │
│ }) main.js:2839-2847 │
└──────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ 6. this.options.instructions → _sendSessionUpdate() │
│ s2s-ws-client.js:995 + 1016 │
│ { type: "session.update", │
│ session: { type: "realtime", │
│ instructions: this.options.…, │
│ audio:{output:{voice}}, │
│ tools, tool_choice:"auto" } } │
└──────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ 7. server.py _client_to_upstream — TUNNEL 1:1 │
│ → forward esatto byte-per-byte all'upstream │
│ → server.py NON tocca MAI instructions qui │
│ → Logga solo summary.instructionLength = len() │
│ (server.py:197-203) — NON MODIFICA │
└──────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ 8. ai-server:12345 riceve session.update → │
│ istruzioni usate come system prompt da VLM/LMM │
│ per TUTTA la durata della sessione finché │
│ non arriva un altro session.update (patch) │
└─────────────────────────────────────────────────────┘
```
### Patch di `instructions` durante la sessione (viva)
Se vuoi aggiornare istruzioni a metà conversazione **senza chiudere la WS**:
- Client → chiama `client.updateSession({ instructions })` (main.js:1021-1035)
- Oppure, in futuro: **proxy server-side inietta patch** via `upstream.send({type:"session.update", session:{...}})` in qualsiasi momento del tunnel
### Tabella riassuntiva — chi oggi modifica `instructions`?
| Livello | Modifica? | Note |
|---|---|---|
| 🔵 Browser `main.js` | **Sì** — generatore principale | DEFAULT_INSTRUCTIONS + hints + user context |
| 🟣 Browser `s2s-ws-client.js` | **Solo forwarding** | Incapsula in `session.update` |
| 🟢 Proxy `server.py` tunnel WS | **NO** — copia byte-per-byte | Logga solo lunghezza, non tocca |
| 🟠 Proxy `server.py` reasoning REST | **Sì (Modalità B)** | Usa `req.instructions` come system part nel prompt OpenAI (Fase 2) |
| 🔴 Upstream `ai-server:12345` | **Sì** — applica come system prompt | Entra nel ciclo STT→VLM→TTS |
---
## 2. Ranking tool MCP lato client + `MCP_SEARCH_TRANSLATION_PHRASES`
**Obiettivo**: quando arriva la query utente (es. `"Quiero el reporte de hoy"` in spagnolo),
ordinare velocemente i tool MCP del catalogo Aquachile per **rilevanza**, così
le `instructions` possono includere un hint di ranking ("prova prima questi 3") e
il modello sceglie lo strumento giusto con meno tentativi.
### 2.1 Perché serve la traduzione
Il catalogo dei tool `agentToolDefs` arriva dall'endpoint Aquachile MCP:
- `tool.name` e `tool.description` sono scritti in **INGLESE** per convenzione di
OpenAI Functions / MCP protocol.
- Ma l'utente parla **spagnolo del Cile o italiano** con l'agente.
Senza traduzione il matching `frase ES/IT ↔ tool EN` → **score 0** (zero corrispondenze).
I tool non vengono mai suggeriti.
### 2.2 I due dizionari di traduzione (main.js)
| Costante | Tipo | Riga | Cosa fa |
|---|---|---|---|
| `MCP_SEARCH_TRANSLATION_PHRASES` | `[RegExp, string][]` frasi complete | [L51-L70](file:///home/duxiter/s2s/demo/main.js#L51-L70) | Sostituisce **frasi multi-parola** prima del tokenizer. Es. `reporte de hoy → today report`. **Priorità alta**: viene applicato PER PRIMO perché le frasi intere non si possono ricostruire da token singoli. |
| `MCP_SEARCH_TRANSLATION_TOKENS` | `Record<it/es, en>` token singolo | [L72](file:///home/duxiter/s2s/demo/main.js#L72) | Sostituisce il **resto** parola per parola dopo la phase-phrase. Es. `ayuda → help`, `cliente → customer`. |
### 2.3 Catalogo completo traduzioni oggi attivo
#### Phrase-level (MCP_SEARCH_TRANSLATION_PHRASES)
| Lingua | Input | Output EN | Riga |
|---|---|---|---|
| 🇪🇸 | `informaciones del usuario` / `informacion del usuario` | `user information` | L52-L53 |
| 🇮🇹 | `informazioni utente` / `informazione utente` | `user information` | L54-L55 |
| 🇪🇸 | `report del dia` / `reporte del dia` | `day report` | L56-L57 |
| 🇮🇹 | `report di oggi` | `today report` | L58 |
| 🇪🇸 | `reporte de hoy` | `today report` | L59 |
| 🇮🇹 | `report giornaliero` | `daily report` | L60 |
| 🇪🇸 | `reporte diario` | `daily report` | L61 |
| 🇪🇸 | `mi numero es` | `my number is` | L62 |
| 🇮🇹 | `il mio numero e` | `my number is` | L63 |
| 🇮🇹 | `numero di telefono` | `phone number` | L64 |
| 🇪🇸 | `numero telefonico` | `phone number` | L65 |
| 🇪🇸 | `numero de telefono` | `phone number` | L66 |
| 🇪🇸 | `numero de whatsapp` | `whatsapp number` | L67 |
| 🇪🇸 | `informacion api` | `api information` | L68 |
| 🇮🇹 | `informazioni api` | `api information` | L69 |
> 💡 Aggiungere nuove frasi: **è sufficiente aggiungere righe qui**, senza toccare il codice del ranking. La pipeline le applicherà automaticamente.
### 2.4 Pipeline di ranking chiamata per ogni turno
```
Utente dice: "Quiero el reporte diario y informaciones del usuario"
│
▼
rankAgentTools(query) main.js:1262
│
▼
buildMcpSearchTexts(query) main.js:1330
│
├─── [0] normalizeMcpSearchText(query) ← versione ORIGINALE normalizzata
│ = "quiero el reporte diario y informaciones del usuario"
│
└─── [1] translateMcpSearchTextToEnglish(query) main.js:1314
│
├─ step 1: normalize
├─ step 2: loop replace per tutte le entry di MCP_SEARCH_TRANSLATION_PHRASES
│ → "quiero el daily report y user information"
├─ step 3: split token, map per MCP_SEARCH_TRANSLATION_TOKENS singoli
│ → "want the daily report and user information"
└─ step 4: normalize finale
│
▼
searchTexts = [ originale normalizzato, versione tradotta in EN ]
(duplicati eliminati via new Set)
│
▼
tokens = split searchTexts + lunghezza ≥ 3 chars + dedup
= [ quiero, reporte, diario, informaciones, usuario,
want, daily, report, and, user, information ]
│
▼
Check rule hard-coded:
dailyReportIntent = match ("day"|"daily"|"today") + "report" → TRUE
(se match: boost score dei tool che hanno report in name/description)
│
▼
Per ogni tool in agentToolDefs[]:
score = somma (frequenza token in tool.name + tool.description)
+ 2× bonus dailyReportIntent se tool matcha "report"
+ 3× bonus se esatta match name vs query originale
Sort(score desc, name asc)
│
▼
Torna al chiamante: [ {tool:"get_daily_report", score:42},
{tool:"get_user_info", score:31},
... ]
│
▼
Usato in 2 punti:
A) effectiveReasoningInstructions() main.js:525-529 → hint "Current user request…
ranking hint: 1. get_daily_report… 2. get_user_info…" dentro le instructions
B) UI pannello (se visibile) — lista tool suggeriti accanto alla trascrizione
```
### 2.5 Quando è chiamato `rankAgentTools` — ogni turno?
| Trigger | Frequenza | Perché |
|---|---|---|
| `maybeRunOpenAiReasoning(transcript)` subito dopo transcript STT | 1× per turno utente | Include ranking hint dentro `instructions` che andrà a Node-RED / OpenAI |
| Live edit textarea domanda (autocomplete) | Mentre utente scrive | Suggerisce tool nella UI |
### 2.6 Disattivazione automatica
Ranking ritorna `[]` vuoto e salta tutta la traduzione se:
- `searchMcpEnabled == false` (MCP Aquachile non configurato)
- `agentToolDefs.length == 0` (catalogo tool non ricevuto da `/agents/profile`)
- Query normalizzata è vuota o troppo corta
- Nessun token dopo split ≥ 3 caratteri
---
## 2bis. DOVE VEDERE I LOG MCP — due postazioni
Il ranking MCP produce log in **2 posti diversi** per design:
| Postazione | Tipo di log | Filtro DevTools / grep | Quando appare |
|---|---|---|---|
| 🖥️ **Browser — DevTools Console** | Client-side ranking dettagliato: **tutta la pipeline translate → build → rank per-tool matching** | `MCP RANK` (case-sensitive) | **Sempre** per ogni turno — anche in caso di EARLY EXIT (motivo esplicito: `searchMcpEnabled=false`, `agentToolDefs=0`, ecc.) |
| 🐳 **Server — container `s2s_ui` (uvicorn Python)** | Server-side **snapshot del payload `instructions`** quando arriva `POST /api/reasoning` — quanti tool MCP ci sono nel catalogo hint, TOP3 suggeriti dal ranking lato client | `grep '[MCP RANK]'` o filter terminale `MCP RANK` | **1 riga per ogni POST /api/reasoning** (lo hai già nel flusso tra `[INPUT UTENTE]` e `[EXT][NODE_RED] reasoning POST`) |
### Nel tuo log di oggi (FASE 1 LOG_ONLY, senza MCP tools abilitati)
Aspettati questo flusso dopo il deploy di adesso:
```
7|s2s_ui | INFO [s2s] [INPUT UTENTE] client=… stt=Necesito el report.
7|s2s_ui | INFO [s2s] [MCP RANK] reasoning payload transcript="Necesito el report" instructionsLen=97 hasMcpCatalogHint=False hasRankingHint=False mcpCatalogTools=0 top3Suggested=none
7|s2s_ui | DEBUG [s2s] [EXT][NODE_RED] reasoning POST http://localhost:1881/api/reasoning …
```
E se apri **DevTools Console** (Verbose/Debug on, Disable cache ✓ Ctrl+Shift+R) vedi:
```
[MCP RANK] updateMcpToolSuggestions ENTRY partial=false itemId=- query="necesito el report" searchMcpEnabled=false
[MCP RANK] ════════════════════════════════════════
[MCP RANK] rankAgentTools START query="necesito el report" searchMcpEnabled=false agentToolDefsCount=0
[MCP RANK] buildSearchTexts raw="Necesito el report" normalized="necesito el report" translated="necesito el report" count=1 final=["necesito el report"]
[MCP RANK] translate input="necesito el report" phrasesApplied=0 tokensApplied=0 output="necesito el report"
[MCP RANK] → EARLY EXIT reason="searchMcpEnabled=false — da /api/config: MCP Aquachile feature OFF oppure S2S_TOOL_EXECUTION_MCP_URL non configurato; agentToolDefs=0 — catalogo tool MCP vuoto: profilo agente non ha tools in agentProfile via Aquachile /api/agents/profile" searchMcpEnabled=false normalized=true agentToolDefs=0
[MCP RANK] ════════════════════════════════════════ (empty result)
[MCP RANK] updateMcpToolSuggestions RESULTS partial=false matchedTools=0 ranked=none
```
### Quando il ranking MCP inizia a produrre score > 0
Affinché un turno abbia log MCP con tool score>0, devono essere veri **TUTTI e 3**:
1. `.env` ha `S2S_TOOL_EXECUTION_MCP_URL` configurato (URL Aquachile MCP) **e** `searchMcpEnabled=true` ritorna da `/api/config`
2. `/agents/profile` Aquachile torna **almeno 1 tool** in `agentTools[]` (campo `json.agentTools`)
3. La query utente ha token ≥ 3 chars che matchano `tool.name` o `tool.description` o `tool.parameters`
Se 1 solo dei 3 è falso → **early exit spiegato nel log `[MCP RANK] → EARLY EXIT reason="…"`**.
---
## 3. Come estendere (quick cheatsheet)
| Se vuoi… | Dove modificare |
|---|---|
| Aggiungere 1 frase tradotta ES/IT → EN per tool matching | `MCP_SEARCH_TRANSLATION_PHRASES` [main.js](file:///home/duxiter/s2s/demo/main.js#L51-L70) — nuova riga |
| Aggiungere 1 parola singola tradotta | `MCP_SEARCH_TRANSLATION_TOKENS` [main.js:72](file:///home/duxiter/s2s/demo/main.js#L72) |
| Boost score di un tipo di tool (es. report) | `rankAgentTools` [main.js:1268-1270](file:///home/duxiter/s2s/demo/main.js#L1268-L1270) + mappa subito dopo |
| Aggiungere nuovo hint a `instructions` all'avvio sessione | `effectiveReasoningInstructions()` [main.js:516-533](file:///home/duxiter/s2s/demo/main.js#L516-L533) — nuova concatenazione |
| Iniettare `instructions` server-side (segret@ dati Aquachile) senza toccare client | **Proxy tunnel** `realtime_proxy()` in [server.py](file:///home/duxiter/s2s/demo/server.py#L891-L908) → `await upstream.send({type:"session.update", session:{instructions}})` **prima** dei 2 `create_task` |
s2s-client/INJECTIONS.md
-498
View File
@@ -1,498 +0,0 @@
# Inject pipeline — caller context & reference material
Documento ufficiale di tutti i blocchi di contesto che il server inietta
automaticamente nel prompt LLM **all'inizio di ogni sessione**, sia per il
canale **WebSocket realtime** (`/v1/realtime`, patch di `session.update`) sia
per l'endpoint **HTTP reasoning** (`POST /api/reasoning`, augmentation di
`instructions` e `user_context`).
Tutto il sistema è **generico / deploy-agnostic** per scelta architetturale:
nessun campo viene hardcodato. Ogni blocco accetta payload arbitrari e li
serializza in JSON direttamente. Qualsiasi errore di rete / JSON non
interrompe mai il flusso: il blocco problematico viene solo saltato con
una riga `[INJECT-*] ⏭️ skipped` nel log server.
---
## 0. Master flag: `S2S_INJECT_PROFILE`
Tutta la pipeline è sotto **un unico master flag** lato server.
| Valore env `S2S_INJECT_PROFILE` | Comportamento |
|---|---|
| `false` / non impostato | **Pure passthrough.** Nessuna fetch upstream, nessuna modifica a `session.update` o al body `/api/reasoning`. Il proxy passa tutto invariato. |
| `true` / `1` / `yes` / `on` | Pipeline inject attiva. I singoli blocchi poi dipendono dagli env specifici e dai query params della URL del client. |
Perché esiste: su deploy generici (nessuna integrazione con AzurianLab /
Aquachile / sistemi interni) il client rimane un puro speech-to-speech
passthrough, senza dipendere da endpoint esterni.
---
## 1. Blocco episodio podcast — `episode_id`
**Priorità massima.** Quando c'è, viene considerato il riferimento PRINCIPALE
e più fidato per tutta la sessione.
### Trigger (quando si attiva)
Una qualunque delle due:
1. URL pagina browser → query string `?episode_id=episode-xxx`
2. Body `POST /api/reasoning` → campo `episode_id: "episode-xxx"`
### Sorgente dati
GET plain verso l'endpoint Aquachile AzurianLab:
```
https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}
```
Parametri fetch:
- timeout 15 s
- `follow_redirects: true`
- **Nessuna auth** (endpoint pubblico)
Formato risposta accettato:
- `Content-Type: application/json` → oggetto arbitrario o array
- `{"result": <payload>}` → wrapper supportato, il contenuto viene "spacchettato"
- `text/plain` → usato come stringa raw
### Cap
24.000 caratteri dopo serializzazione JSON (transcript lunghi).
Se supera → troncato con `...` append.
### Wrap applicato
```
< PODCAST_EPISODE_HINT >
<<<PODCAST_EPISODE_BEGIN>>>
{ <TUTTI I CAMPI DELL'EPISODIO, JSON> }
<<<PODCAST_EPISODE_END>>>. This is the episode the caller is currently interacting with — answer with reference to it first.
```
**Hint testuale** (in testa, letto dal modello come primissima frase del blocco):
> Podcast episode context loaded from the page `episode_id` query param (current episode the user is referencing). Treat the episode fields (title, description, transcript, script, show notes, guests, audio metadata, etc.) as the **PRIMARY trusted reference material** for this session.
### Campi supportati (esempi — lista NON esaustiva, qualsiasi campo extra viene incluso)
- metadati base: `id`, `title`, `slug`, `description`, `summary`, `publishedAt`, `durationSeconds`
- contenuti: `transcript`, `podcastScript`, `script`, `chapters[]`, `showNotes`
- persone: `host`, `guests[]`, `author`, `guestsInfo[]`
- audio: `audioUrl`, `audioFile`, `waveform`, `sampleRate`
- classificazione: `tags[]`, `topics[]`, `category`, `season`, `episodeNumber`
- riferimenti incrociati: `reportId`, `scheduleId`, `company`, `product`
### Riferimenti codice
- fetch upstream: [_fetch_podcast_episode](file:///home/duxiter/s2s/s2s-client/server.py#L1010-L1078)
- wrap + serialize + cap 24k: [_resolve_episode_context](file:///home/duxiter/s2s/s2s-client/server.py#L475-L504)
- hint costante: [PODCAST_EPISODE_HINT](file:///home/duxiter/s2s/s2s-client/server.py#L426-L429)
---
## 2. Blocco profilo chiamante — `phoneNumber`
Contesto identità del chiamante. Viene usato per il modello quando risponde
(riferirsi all'utente per nome, conoscere la sua azienda, ruolo, ecc.).
### Trigger
1. URL browser → `?phoneNumber=%2B<numero-internazionale>`
2. Body `/api/reasoning` → `phoneNumber: "+…"`
### Sorgente dati
```
GET VOCERO_USER_PHONE_AUTH_URL?phoneNumber=<phone>
```
Se è configurato anche `S2S_LOGIN_BY_PHONE`, PRIMA della profile fetch:
```
POST S2S_LOGIN_BY_PHONE body { "phoneNumber": "..." }
```
Da questa risposta il proxy estrae un bearer token e lo inoltra come
`Authorization: Bearer <token>` sulle chiamate successive (profile, reports,
summary).
Formato risposta: JSON arbitrario, wrapper `{"result": ...}` supportato.
### Cap
1.600 caratteri dopo JSON serialize.
### Wrap
```
Caller profile resolved from the page phoneNumber:
{ <profilo JSON troncato> }.
Use this as trusted caller context when relevant.
```
### Riferimenti codice
- login by phone: [_login_by_phone](file:///home/duxiter/s2s/s2s-client/server.py#L812-L890)
- fetch profile: [_fetch_user_profile_for_phone](file:///home/duxiter/s2s/s2s-client/server.py#L1081-L1134)
- wrap serialize: [_resolve_user_context_text](file:///home/duxiter/s2s/s2s-client/server.py#L432-L444)
---
## 3. Blocco lista reports / script schedulati
Riferimento "storico" dei contenuti preparati prima della sessione. Non sono
contesto live, ma materiale che il modello deve sapere esiste (e a cui
riferirsi se l'utente chiede di report pregressi, script, podcast schedulati).
### Trigger
Solo env: `S2S_REPORT_LIST` deve essere valorizzato a un URL valido.
**Funziona anche SENZA `phoneNumber`** (lista reports generica per tutti).
Se `phoneNumber` è presente lo passa come query param all'endpoint.
### Sorgente dati
```
GET S2S_REPORT_LIST[?phoneNumber=<phone>]
```
Wrapper `{"result": [...]}` supportato.
### Cap
12.000 caratteri sul blocco COMPLETO (ogni podcast script può essere lungo).
### Wrap + filtri + sort
```
Scheduled reports / podcast scripts available for this caller before the current session.
Each entry is a scheduled report; use the podcastScript contents as trusted reference material
when the user asks about scheduled topics, reports, or prior content.
<<<REPORTS_LIST_BEGIN>>>
[
{
"sourceScheduleLabel": "...",
"date": "2026-09-06T…",
"podcastScript": "…"
},
… ordinati per data DESC (nuovi prima) …
]
<<<REPORTS_LIST_END>>>
```
Regole per ogni entry:
1. **Solo dict** vengono tenuti (stringhe/int scartati).
2. Per ogni entry estrae 3 campi con fallback a nested shape:
- Label: `sourceScheduleLabel` oppure `sourceSchedule.label`
- Date: `date` oppure `sourceSchedule.date`
- Script: `podcastScript`
3. Ordinamento **data DESC** (nuovi prima) con parser robusto:
- ISO datetime / date strings
- `Date` / `DateTime` Python (se deserializzati)
- Unix timestamp int/float
- Lexicografico DESC come fallback per stringhe ISO-ish
- Valori nulli / non-parse-abili finiscono in fondo.
### Riferimenti codice
- fetch: [_fetch_reports_list](file:///home/duxiter/s2s/s2s-client/server.py#L668-L745)
- parse data sort key: [_parse_date_key](file:///home/duxiter/s2s/s2s-client/server.py#L507-L515)
- wrap + filtro + sort: [_resolve_reports_context](file:///home/duxiter/s2s/s2s-client/server.py#L518-L666)
---
## 4. Blocco summary ultima conversazione
Sintesi della chiamata/chat IMMEDIATAMENTE precedente a questa. Usato dal
modello per "ricordarsi" cosa si è detto l'ultima volta senza conservare
storico lunghissimo nel prompt.
### Trigger
Entrambe le alternative sono mutuamente esclusive: `S2S_LATEST_SUMMARY_TEXT`
ha **priorità assoluta** se valorizzato.
| Modalità | Trigger |
|---|---|
| **A — HTTP fetch (dinamico per caller)** | env `S2S_LATEST_SUMMARY` impostato **AND** `phoneNumber` presente in URL/body |
| **B — Testo statico hardcoded (override)** | env `S2S_LATEST_SUMMARY_TEXT` impostato (stringa non vuota) — **ignora completamente** `S2S_LATEST_SUMMARY`, `phoneNumber`, login-by-phone, Bearer auth. Utile quando l'operatore vuole iniettare una versione **pulita, essenziale e pre-pulita** del summary, senza dipendere da nessun endpoint upstream. |
### Sorgente dati
**Modalità A (HTTP fetch):**
```
GET S2S_LATEST_SUMMARY?phoneNumber=<phone>
```
Wrapper `{"result": <valore>}` supportato.
Accetta SIA JSON che plain text (diversi backend summary ritornano testo semplice).
**Modalità B (override statico):**
- Nessuna chiamata HTTP. Viene usato direttamente il valore grezzo di `S2S_LATEST_SUMMARY_TEXT` dal file `.env`.
- Le sequenze di escape `\n` nel valore env vengono **interpretate** come a-capo vero (stesso comportamento di `S2S_PRIMARY_PROMPT`), quindi è possibile scrivere summary multi-line in una sola riga del `.env`.
- Esempio:
```env
S2S_LATEST_SUMMARY_TEXT=Última conversación 2026-09-22:\n- Usuario preguntó por mortalidad salmón AquaChile semana 38.\n- Se refirió al reporte Porcentaje Mortalidad.
```
- Il valore viene passato as-is a `_resolve_latest_summary_context` → il wrapper `LATEST_SUMMARY_HINT` e il cap a 1600 caratteri vengono applicati normalmente.
### Cap
1.600 caratteri (applicato da `_resolve_latest_summary_context` sia in modalità A che B).
### Wrap
```
Latest conversation summary for this caller (most recent before the current session):
<<<LATEST_SUMMARY_BEGIN>>>
<summary, stringa o JSON serializzato>
<<<LATEST_SUMMARY_END>>>.
Use this as trusted history when the user references prior calls.
```
### Riferimenti codice
- fetch HTTP (modalità A): [_fetch_latest_summary_for_phone](file:///home/duxiter/s2s/s2s-client/server.py#L1563-L1654)
- override statico (modalità B) → riga iniziale `if S2S_LATEST_SUMMARY_TEXT:` dentro la funzione sopra
- wrap: [_resolve_latest_summary_context](file:///home/duxiter/s2s/s2s-client/server.py#L987-L1012)
---
## 5. Blocco `user_context` custom dal browser (SOLO `/api/reasoning`)
**Canale solo HTTP reasoning.**
Non è nel WS: il WS non ha campo user_context nel protocollo realtime
OpenAI (tutto passa per `session.instructions`).
Composizione lato client in [main.js fetchOpenAiReasoningAnswer](file:///home/duxiter/s2s/s2s-client/main.js#L1337-L1364):
```js
const profileCtx = voceroUserContextText();
const ragCtx = customRagContextText();
const user_context = [profileCtx, ragCtx]
.map(s => s.trim()).filter(Boolean).join("\n\n");
```
### Dove va a finire
Nel body a monte sia NODE_RED sia OPENAI, questo blocco viene **APPESO DOPO**
tutti i blocchi iniettati server-side:
```
user_context finale =
[1. episode] ← solo se episode_id
[2. profile] ← solo se phoneNumber + env profilatura
[3. reports] ← solo se env S2S_REPORT_LIST
[4. summary] ← solo se phoneNumber + S2S_LATEST_SUMMARY
+ browser user_context (profileCtx + ragCtx)
```
---
## Ordine finale nel prompt LLM
Questa è la sequenza **letterale** con cui i blocchi vengono concatenati
dentro `session.instructions` (WS) e in `instructions` / `user_context`
(reasoning HTTP).
```
1. EPISODIO PODCAST ← PRIORITÀ PIÙ ALTA (primary trusted reference)
2. PROFILO CHIAMANTE ← identity / caller profile
3. REPORTS SCHEDULATI ← scheduled / prior content
4. SUMMARY ULTIMA CALL ← history più recente
5. Base instructions (prompt predefinito o primary prompt)
6. (solo reasoning) user_context custom inviato dal browser
```
Perché questo ordine:
- più in alto nel prompt = più peso / attenzione dal modello.
- l'episodio attuale è il materiale che l'utente ha esplicitamente richiesto
aprendo la pagina con `?episode_id=`, quindi merita massimo peso.
- il profilo chiamante va sempre prima di "contenuti generici" (reports e
summary) perché serve per coerenza nel riferirsi all'utente.
- reports = contenuti schedulati PRIMA della sessione corrente → più vecchi
dell'ultima chiamata, quindi messi PRIMA del summary ultima chiamata che
è il contenuto più "vivo".
Implementazione ordine: [_augment_instructions_with_profile](file:///home/duxiter/s2s/s2s-client/server.py#L1149-L1159)
```python
contexts = [c for c in [episode_ctx, profile_ctx, reports_ctx, summary_ctx] if c]
```
---
## Punti di innesco (quando viene calcolato / applicato)
### Canale WebSocket — `/v1/realtime`
**MOMENTO 1 (solo una volta):** apertura WS.
Nel handler [@app.websocket("/v1/realtime")](file:///home/duxiter/s2s/s2s-client/server.py#L1791-L1959)
appena la WS viene accettata:
1. parsa `ws.url.query` per `phoneNumber` e `episode_id`
2. esegue login (se attivo) → profile → episode → reports → summary
3. salva TUTTI i blocchi già wrappati e serializzati nel dict `session_ctx`:
```python
session_ctx = {
"injected_episode_context": "...",
"injected_profile_context": "...",
"injected_reports_context": "...",
"injected_summary_context": "...",
...
}
```
**MOMENTO 2 (ogni `session.update` client→upstream):** nel task
[_client_to_upstream](file:///home/duxiter/s2s/s2s-client/server.py#L2649-L2661)
ogni frame in uscita viene passato a:
```python
_patch_client_session_update_with_profile(
frame_text,
user_context_text = profile_ctx,
reports_context_text = reports_ctx,
summary_context_text = summary_ctx,
episode_context_text = episode_ctx, # ← nuovo
)
```
Se il frame NON è `session.update` → ritorna **identico** e non viene loggato
(per evitare spam).
Se è `session.update` → patch del campo `payload["session"]["instructions"]`.
### Canale HTTP — `POST /api/reasoning`
**Ogni chiamata** ricostruisce i blocchi da zero nel
[handler `/api/reasoning`](file:///home/duxiter/s2s/s2s-client/server.py#L1411-L1577):
1. legge `req.phoneNumber` + `req.episode_id`
2. fetch episodio (se episode_id) + login + profile + reports + summary
3. costruisce `user_context_parts` e `effective_instructions`
4. inoltra al backend (NODE_RED o OPENAI)
---
## Quando un blocco viene SKIPPATO (niente inject)
Tutti i blocchi hanno 5 possibili ragioni di skip. Nessuna rompe mai il
flusso: ogni caso produce una riga `[INJECT-*] ⏭️ skipped` nei log e basta.
| Causa | Log |
|---|---|
| Master flag `S2S_INJECT_PROFILE=false` | `[INJECT-PROFILE] ⏭️ skipped (S2S_INJECT_PROFILE=false)` |
| Parametro mancante (es. `phoneNumber` non in URL) | `[INJECT-*] ⏭️ skipped (<parametro> missing)` |
| Endpoint env non configurato | `[INJECT-*] ⏭️ skipped (env for ... empty)` |
| Upstream HTTP ≠ 200 | `[INJECT-*] ⛔ upstream HTTP <status_code>` |
| Upstream ha ritornato vuoto / null / `{}` / `[]` | `[INJECT-*] ⏭️ upstream returned empty` |
| Eccezione di rete (timeout, DNS, SSL) | `[INJECT-*] ⛔ fetch unreachable error=<exc>` |
---
## Marcatori log / marker nel prompt
Ogni blocco è **auto-documentante** per ispezione dei log con coppie di
marker:
```
<<<PODCAST_EPISODE_BEGIN>>> / <<<PODCAST_EPISODE_END>>>
<<<REPORTS_LIST_BEGIN>>> / <<<REPORTS_LIST_END>>>
<<<LATEST_SUMMARY_BEGIN>>> / <<<LATEST_SUMMARY_END>>>
```
Quando `S2S_DEBUG_PROMPT=true` viene loggato il contenuto COMPLETO di ogni
blocco con i tag `[DEBUG-EPISODE]`, `[DEBUG-REPORTS]`, `[DEBUG-SUMMARY]` sia
all'atto del load (WS connect) sia dopo l'augment delle instructions (prima
di inoltrare a monte). **WARNING:** questo env può loggare PII.
---
## Tabella riepilogo: trigger per ogni blocco
| # | Blocco | Trigger | env richiesto | parametro | cap |
|---|---|---|---|---|---|
| 1 | Episodio podcast | `episode_id` in URL o body | (nessuno — URL hardcodato Aquachile) | `episode_id=` | 24 k |
| 2 | Profilo chiamante | `phoneNumber` | `VOCERO_USER_PHONE_AUTH_URL` | `phoneNumber=` | 1,6 k |
| 3 | Reports schedulati | sempre se env | `S2S_REPORT_LIST` | opzionale `phoneNumber=` | 12 k |
| 4 | Summary ultima call | `phoneNumber` **oppure** override statico | `S2S_LATEST_SUMMARY` (HTTP) **o** `S2S_LATEST_SUMMARY_TEXT` (statico, ha priorità) | `phoneNumber=` (solo modalità HTTP; override statico non richiede nessun param) | 1,6 k |
| 5 | Browser user_context | solo reasoning | (nessuno) | body `user_context` | illimitato |
---
## Error handling strategico
Perché tutti gli errori sono silenziosi (log + skip) e non abortiscono:
> Se il backend dei profili è giù per manutenzione 5 minuti non vogliamo
> che il Voice Agent smetta di funzionare. Semplicemente l'assistente non
> conosce l'identità del chiamante per 5 minuti, ma parla normalmente.
L'utente non si accorge di niente. L'operatore legge i log e vede le righe
`⛔`; può risolvere offline.
Se volessi forzare il fail (raro, su deploy dedicato), si trasformerebbero
le `return None` finali in `raise HTTPException(...)` — ma per default
non è il comportamento voluto.
---
## Appendice A — Start-session modes (`S2S_START_SESSION_MODE`)
Oltre alla inject pipeline lato server, è presente una feature lato client che
governa **come e se la sessione vocale inizia in automatico** e se il modello
saluta per primo (in modo identico alla wake-word "Hola Vocero").
### Valori supportati
| `S2S_START_SESSION_MODE` | Chi inizia? | Richiede click utente? | Trigger greet iniziale |
|---|---|---|---|
| *(vuoto) / default* | Utente, tap manuale Orb | ✅ | No |
| `TALK` | Auto dopo 2s con overlay unlocker | ✅ (1 click policy-safe su Safari/iOS) | WS client: `sendUserText("hola vocero") + requestResponse()` dopo `session.updated` |
| `TALKFIRST` | Auto dopo 2s **SENZA click** (se possibile) | ❌ se mic già "allow always"; fallback unlocker se `getUserMedia` fallisce | **Stesso identico codice della wake-word vera:** `pendingWakeWordConfirmationText = "hola vocero"` (main.js `case "connected"` → `sendUserText + requestResponse`) |
### Warmup 2s
Il timer non parte prima di 2000 ms per lasciare terminare:
- `fetchVoceroUserProfile()` (identity chiamante)
- `fetchVoceroReportsList()` (storico script podcast)
- fetch episode (`episode_id`) se presente
- `GET /api/config` (con la quale il client apprende `startSessionMode`)
Senza warmup le injection lato server potrebbero ancora non essere completate e
il greet iniziale arriverebbe "cieco" (senza contesto episodio/profilo).
### Fallback di `TALKFIRST`
Nel 99% dei casi `getUserMedia()` fuori da una user gesture fallisce
(NotAllowedError / NotReadableError su Safari, Chrome con policy strict, iOS).
In tal caso:
1. si chiude il client WS (`teardown()`)
2. stato torna `idle`
3. si mostra `#talk-unlocker` con la dicitura:
- **"Se requiere permiso de micrófono"**
- **"Toca en cualquier lugar para permitir el audio y empezar"**
4. `pendingWakeWordConfirmationText` NON viene svuotato → quando l'utente
toccherà, `case "connected"` triggerà comunque la greet come wake-word.
### Riferimenti codice
| Componente | File | Range righe |
|---|---|---|
| Lettura env + banner startup + `/api/config` exposure | [server.py](file:///home/duxiter/s2s/s2s-client/server.py) | [L150](file:///home/duxiter/s2s/s2s-client/server.py#L150), [L196-L212](file:///home/duxiter/s2s/s2s-client/server.py#L196-L212), [L1383-L1400](file:///home/duxiter/s2s/s2s-client/server.py#L1383-L1400) |
| Overlay HTML unlocker | [index.html](file:///home/duxiter/s2s/s2s-client/index.html#L32-L39) | L32-L39 |
| Stili overlay | [style.css](file:///home/duxiter/s2s/s2s-client/style.css#L145-L215) | L145-L215 |
| Variabile globale + parse config client | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L628-L632) | L628-L632 |
| `fetchConfig()` parsing + `reportDebug("B")` | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2309-L2320) | L2309-L2320 |
| `new S2sWsRealtimeClient({startWithResponse})` | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2683-L2692) | L2683-L2692 |
| Scheduling + `tryTalkFirstDirect()` + unlocker gesture | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2976-L3083) | L2976-L3083 |
| `onClientStatus("connected")` → wake-word inject flow | [main.js](file:///home/duxiter/s2s/s2s-client/main.js#L2897-L2914) | L2897-L2914 |
| WS client → `TALK` mode sendUserText("hola vocero") + `requestResponse()` | [s2s-ws-client.js](file:///home/duxiter/s2s/s2s-client/ws/s2s-ws-client.js#L688-L713) | L688-L713 |
---
## Riferimenti file
| File | Ruolo |
|---|---|
| [server.py](file:///home/duxiter/s2s/s2s-client/server.py) | Tutta la pipeline inject: fetch, wrap, patch, handler WS + reasoning |
| [main.js](file:///home/duxiter/s2s/s2s-client/main.js) | Passaggio client `?phoneNumber` + `?episode_id` → WS query params e POST `/api/reasoning` body |
| [CHANGELOG.md](file:///home/duxiter/s2s/s2s-client/CHANGELOG.md) | History rilasci / modifiche alla inject pipeline |
s2s-client/README.md
+16 -237
View File
@@ -10,247 +10,26 @@ short_description: Voice chat over WebSocket against a HF speech-to-speech
hf_oauth: true
---
# Realtime Voice Demo (WebSocket transport)
# Realtime voice client
Browser voice-chat UI for the
backend. The browser streams mic audio over a WebSocket using the OpenAI
Realtime **GA** protocol and plays back the assistant's audio as it arrives.
Browser voice-chat UI and FastAPI server for an OpenAI Realtime-compatible
speech-to-speech backend. The browser streams microphone audio over WebSocket,
plays assistant audio, displays transcripts, and offers Spanish controls.
---
The complete [usage guide](../docs/client/usage.md) covers local startup,
connection modes, Docker, settings, and usage limits. See the
[documentation index](../docs/README.md), [architecture](../docs/client/architecture.md),
and [change history](CHANGELOG.md) for implementation details and history.
## 🆕 Novedades / Actualizaciones (Azurian Vocero realtime ver.Beta 2.1)
> Versión del navegador que debes ver cargada en consola: **`2026-09-04 11:20`**.
> Si no aparece, haz **borrado de datos del sitio + recarga dura (Ctrl+Shift+R)** o abre una pestaña en modo incógnito.
### 🎨 Rediseño visual completo (tema aurora + glassmorphism)
- **Tema de fondo aurora**: gradiente radial violeta/rosa/cian sobre lienzo oscuro con malla sutil + animación `auroraShift` lenta.
- **Cromado uniforme glass**: todos los elementos flotantes (tarjeta brand, panel user profile, botones icono, botón enviar chat, panel informes sidebar) usan *glass blur + saturate + borde gradiente brand 1px* (técnica mask `padding-box / border-box` doble linear-gradient).
- **Tokens de marca 3 colores brand** (violeta `#8b5cf6`, rosa `#ec4899`, cian `#22d3ee`) mezclados con `color-mix(in srgb, brand X%, neutral)` → ningún color hardcodeado deriva; todo el UI sigue estos tres tonos.
### 🧭 Estructura (orden vertical de arriba → abajo)
1. **`header.topbar`** brand "Vocero" → avatar circular iniciales nombre usuario.
- 🟦 **Avatar user**: círculo 40×40px con gradiente brand, initials mayúsculas bold 800, hover lift/glow + foco teclado. Al hacer clic → diálogo modal perfil usuario (rejilla 2 columnas email/teléfono/username/rol/empresa/userid/país/fecha creación/último login; links `mailto:`/`tel:`; chips mono para códigos; mobile ≤480px apilado 1 col).
- 🟦 **Botón toggle chat (icono speech bubble) → ELIMINADO PERMANENTEMENTE**. La chat ahora siempre está visible inline; el botón toggle ya no existe en el DOM.
2. **`main.stage`** (arriba del todo, flexible): orbe/globode voz animado (3 anillos: core, ring, conic spin 3 velocidades + highlight specular + glows breathe `breatheOrb`) + caption `TAP OR SAY "HOLA VOCERO"` micro-bob uppercase gradient.
- ✅ Antes había una zona vacía sobre el orbe → eliminada, ahora el stage empieza en la parte superior (`justify-content: flex-start`).
3. **`<section id="chat-panel" class="chat-panel chat-panel-inline">` SIEMPRE VISIBLE, 100% ancho, SOTTO stage, SOPRA footer sticky** (rellena la zona vacía inferior).
- ✅ Antes: overlay lateral derecho. **Ahora: bloque inline dentro de `#app` como hijo flex-col**.
- 🟦 Sin marco / sin riquadro (fondo aurora seamless, 0 borde, 0 shadow, 0 backdrop blur).
- Cabecera `CONVERSACIÓN` h3 uppercase 13px gradiente.
- `#chat-history` scroll interno. Burbujas de mensaje glass con **código de colores por rol**: usuario (tint 10% voice-user cyan + borde 44% cyan + glow) / assistant (tint 10% voice-assistant violet/pink + borde 42% + glow).
- Distancias seguras anti-borde: `.hist-msg.user margin-left:6px auto dx` / `.hist-msg.assistant margin-right:22px auto sx` → **nunca las burbujas tocan el borde del viewport** (fix confirmado).
- Formulario input texto `#chat-form` (arriba del botón enviar ✈️) **actualmente oculto (`hidden`)** para modo voz-puro; elimina ese atributo para reactivarlo. El handler submit que envía `client.sendUserText(text)` al websocket ya está conectado.
4. **Footer sticky `position: sticky; bottom:0; z:3; backdrop-blur`** "Azurian Vocero realtime ver.Beta 2.1" siempre dentro del viewport, incluso cuando el documento crece y haces scroll (no requiere scroll hacia abajo para verlo). Separador superior línea gradiente brand.
### 📜 Sidebar de informes a la IZQUIERDA (antes topbar inline)
- Desktop **≥ 1024px**: sidebar fija left 320px siempre visible; `#app padding-left 320px` reserva sitio.
- Tablet/Mobile **< 1024**: offcanvas `translateX(-105%)` slide-in con backdrop blur; toggle mediante botón icono reportes `#reports-sidebar-toggle` en topbar con badge contador.
- `#reports-hint` siempre visible; `.reports-suggestions` y `.topbar-right` con puerta triple de visibilidad (atributo HTML hidden + CSS rule `[data-debug-only]:not([data-debug-enabled]) !important display:none` + JS `applyDebugOnlyVisibilityGate()` ejecutado 6 veces por lifecycle) → solo aparecen cuando el query param `?debug=true` está presente en la URL.
### ⬇️ Auto-scroll chat SIEMPRE hacia la última respuesta del asistente
Arquitectura anti-clipping definitiva (versión 11:20 fix):
1. **CSS sobrante intencional** (`96 → 160px chat-history padding-bottom`, `40→160px chat-panel clamp`, `56→120px #app padding-bottom + safe-area-inset-bottom`) → el último párrafo siempre está por ENCIMA de la línea del footer glass flotante.
2. **Unidades móviles seguras `svh / svw`** (smallest viewport): evita oscilaciones layout cuando aparece/desaparece la barra URL del móvil.
3. **JS cancel-and-reschedule** cada vez que `_scrollToBottom()` se llama: si llegan 100 tokens streaming en 1 frame, el último cancela todas las cadenas pendientes y se ejecuta desde cero → **no hay coalescing flag SKIP** (esta era la causa original del scroll insuficiente).
4. **5 reintentos escalonados `0 / +30 / +70 / +150 / +250ms`**:
- 0 ms (inmediato): 90% de los casos.
- 30 ms: tras 1 layout-pass.
- 70 ms: **corrige el wrapping de la última palabra en una línea nueva** (bug más común).
- 150 ms: después de reflow Chrome scroll-anchor.
- 250 ms: después de expandir filas tool / imágenes grandes.
5. **Snap con `Number.MAX_SAFE_INTEGER`**: el navegador clampa automáticamente al máximo real (`scrollHeight - clientHeight`); no hay race de leer scrollHeight demasiado pronto.
6. `overflow-anchor: none; scroll-behavior: auto !important;` → sin smooth, sin Chrome scroll-anchor-luchando contra nosotros.
7. Llamadas scroll explicitas extra en: `onResponseFinished` (fin de respuesta assistant, crítico), `onToolCall`, `onToolResult`.
### 🛡️ Protecciones anti-error de refs eliminadas
- En el refactor de chat eliminamos 4 nodos DOM (`#chat-btn`, `#chat-badge`, `#chat-panel-backdrop`, `#chat-panel-close`).
- **Antes**: el constructor de `ChatView` usaba `$("#...")` strict → lanzaba `Error("Missing element: #chat-btn")` al arrancar.
- **Ahora**: esas 4 refs opcionales usan `document.querySelector("#...")` → devuelve `null` cuando no existen, sin throw; todos los listeners guarded `if (el)`. Los 3 IDs obligatorios (`#chat-panel`, `#chat-history`, `#bubble-stack`) siguen con strict para fallar fuerte si alguien los borra por error.
### 🔄 Actualizaciones de versión / cache bust
Para evitar caché HTTP agresiva en navegadores móviles (ESM sin bundler + service worker = nunca refrescan URLs iguales), todas las referencias a recursos estáticos usan query param `?v=YYYYMMDDHHMM` sincronizado en **6 puntos**:
1. `index.html <link rel="stylesheet" href="style.css?v=...">`
2. `index.html <script type="module" src="main.js?v=...">`
3-6. Los **4 imports ESM dentro de main.js** (`ws/s2s-ws-client`, `ui/dom`, `ui/chat`, `ui/account`).
Además hay un **script inline en `<head>`** tras el style.css que al `DOMContentLoaded` fuerza a reescribir el href del stylesheet con la última versión si por algún motivo cargó la antigua (anti-cache móvil).
### Deploy
```bash
cd /home/duxiter/s2s
docker compose restart s2s_ui
```
Limpia siempre datos del sitio en el navegador (Application → Storage → Clear site data) y recarga dura `Ctrl+Shift+R` al finalizar.
---
## Quick start (local)
1. **Start the speech-to-speech backend** in realtime mode (from the repo root;
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend transformers \
--tts kokoro \
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
The realtime server listens on `ws://localhost:8765/v1/realtime` by default
(`--ws_host` / `--ws_port` to change).
2. **Start this app**, pointing it at the backend with `SPEECH_TO_SPEECH_URL`:
Build from the repository root so the container includes the documentation:
```bash
uv pip install -r demo/requirements.txt
export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
export SERPER_API_KEY=... # optional; web search is disabled without it
uv run uvicorn --app-dir demo server:app --reload --port 7860
docker build -f s2s-client/Dockerfile -t s2s-client .
```
Or with Docker:
```bash
docker build -t s2s-demo demo/
docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
```
3. Open <http://localhost:7860/>, click the orb, allow the mic, talk.
> Browsers require **HTTPS or `localhost`** for `getUserMedia()` (mic + camera).
> `127.0.0.1` and `localhost` both work; plain `http://192.168.x.y` does NOT.
Smoke-test the backend from the shell:
```bash
websocat ws://localhost:8765/v1/realtime
# -> you should get a session.created event back immediately
```
## How it works
1. The browser opens a WebSocket on the configured `/v1/realtime` URL.
2. Server pushes `session.created` on connect. Client replies with
`session.update` (OpenAI Realtime **GA** schema: `session.audio.input`,
`session.audio.output`, `session.output_modalities`).
3. Client streams mic audio as PCM16 16 kHz mono base64 chunks
(`input_audio_buffer.append`, one frame every ~40 ms).
4. Server pushes `response.output_audio.delta` (PCM16 24 kHz mono base64)
and transcript deltas.
The backend exposes one concurrent session per pipeline unit
(`--num_pipelines` to serve more).
## Connecting to a backend
Three modes, picked by env (`/api/config` tells the client which one is active):
- **`SPEECH_TO_SPEECH_URL` env** — the mode you want for local use, and the
highest priority. The browser connects **directly** to this realtime
WebSocket URL; it's shown read-only in Settings. Setting it disables the
load-balancer logic entirely (no `/api/session` proxy, no queue, no
metering, no sign-in). Unlike the LB address it is not a secret. Accepts a
full `ws(s)://host/v1/realtime` URL or a bare host like `localhost:8765`
(the app adds `/v1/realtime`).
- **Neither env set** — **Settings → Speech-to-speech server URL**: paste a
full connect URL or a bare host, and the browser connects to it directly.
- **`LOAD_BALANCER_URL` env** — multi-compute deployments only: the browser
POSTs the same-origin `/api/session` proxy, the server forwards to the LB,
and the browser dials the per-session compute URL the LB hands back. The LB
address never reaches the browser; the Settings URL field is hidden.
| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Metering |
|:---:|:---:|:---:|---|---|---|
| ✅ | any | any | direct → pinned URL | visible, locked | off |
| – | – | any | direct → user URL | editable | off |
| – | ✅ | ✅ | LB proxy | hidden | **on** |
| – | ✅ | – | LB proxy | hidden | off |
**Settings → Restart** reconnects with the current voice, instructions and URL.
## Tools
The assistant can call two tools mid-conversation (toggle them from the **Tools**
button, top-right):
- **Web search** — Google results via Serper.dev, proxied server-side so the key
never reaches the browser. Set `SERPER_API_KEY` as an env var / Space secret.
Without it, the tool is disabled unless the user pastes their own key in the
Tools panel.
- **Camera** — while enabled, a live self-view shows bottom-left; when the model
calls the tool, the current frame is sent to the vision-language model so it can
see what you're showing it.
## Usage limits (deployed Space only)
Conversation time is metered per UTC day by sign-in tier (see `limiter.py` /
`auth.py`), but **only on the deployed Space** — metering turns on only when BOTH
`LOAD_BALANCER_URL` and `SPACE_ID` (injected automatically by the HF Space
runtime) are present. Running locally — even with `LOAD_BALANCER_URL` exported —
leaves the app unmetered. Tunable via env:
| Env | Default | What |
|-----|---------|------|
| `LIMIT_ANON_SEC` | `300` | Daily seconds for anonymous visitors (5 min) |
| `LIMIT_FREE_SEC` | `600` | Daily seconds for signed-in non-PRO users (10 min) |
| `UNLIMITED_ORGS` | _(adds to defaults)_ | Extra HF org names whose members get **unlimited** usage, like PRO |
| `USAGE_HASH_SECRET` | _(random)_ | HMAC secret for hashing identity keys + signing the anon cookie |
PRO members are always unlimited. Members of `cerebras`, `HuggingFaceM4`,
`smolagents`, and `pollen-robotics` are unlimited out of the box (shown as
"Team", not "PRO"); set `UNLIMITED_ORGS=my-team` to add more. Matched
case-insensitively against the user's organisations from HF OAuth.
## Settings (stored in `localStorage`)
| Key | What |
|-----|------|
| Speech-to-speech server URL | Direct realtime WebSocket URL (hidden/locked when pinned by env) |
| Voice | Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian) |
| Instructions | System prompt sent in `session.update` once the WS opens |
LocalStorage keys are namespaced `s2s.ws.*` so this app's settings do
NOT collide with the WebRTC variant.
## Files
> 🇪🇸 **Notas adicionales (Azurian Vocero)**: `index.html` ahora incluye (1) botón toggle lateral izquierdo reportes `#reports-sidebar-toggle` con badge contador `#reports-sidebar-badge` visible solo <1024px, (2) `<dialog id="user-profile-dialog">` modal perfil usuario que se abre al hacer clic en el avatar circular del encabezado, (3) panel chat `<section id="chat-panel">` permanente dentro de `#app` entre `</main>` y footer sticky (el panel overlay derecho antiguo y el botón toggle chat superior fueron ELIMINADOS completamente).
| File | Role |
|------|------|
| `index.html` | Single page, orb + settings modal (identical UI to the WebRTC app) |
| `main.js` | State machine, settings, tools, camera, noise-gate UI wiring + chat-form submit handler (`client.sendUserText`) + user profile dialog open/close + reports sidebar open/close + 3-layer debug gate |
| `ui/chat.js` | `ChatView`: history panel, ephemeral bubbles, transcript/tool streaming; always-inline 100% width chat panel w/ 5-step retry auto-scroll to last assistant line |
| `ui/account.js` | `Account`: HF login chip + popover, daily-limit modal |
| `ui/dom.js` | Shared helpers: strict `$(selector)` that throws on missing, `document.querySelector` fallback, `escHtml`, `truncateError`, `DEBUG` from URL |
| `auth.py` | HF OAuth + per-request identity (tier, hashed keys) |
| `limiter.py` | SQLite per-day talk-time budget (chunked server-clock reservation) |
| `ws/s2s-ws-client.js` | WebSocket handshake + OpenAI Realtime GA protocol |
| `ws/codec.js` | base64 <-> PCM helpers + transcript extraction (pure) |
| `ws/orb-visualizer.js` | `OrbVisualiser`: FFT bands -> orb CSS custom properties (violet/pink/cyan brand gradient) |
| `worklets/mic-capture.js` | AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks |
| `worklets/audio-playback.js` | AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out |
| `style.css` | Orb animations, layout, dark aurora gradient theme, glass chat bubbles, sticky footer, reports sidebar desktop left 320px / mobile offcanvas, user profile 2-col grid modal |
## Audio pipeline notes
- **Input**: `getUserMedia({ echoCancellation, noiseSuppression, autoGainControl })`
feeds the `mic-capture` worklet at the `AudioContext` rate. The worklet
resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast
path, linear interpolation fallback for odd rates) and packs Int16 LE.
- **Output**: `response.output_audio.delta` decodes to Int16 -> Float32
and is posted to the `audio-playback` worklet. The worklet maintains a
per-context ring buffer, linearly interpolates 24 -> 48, and applies
short 32-frame fades on entry/exit to suppress clicks.
- **Barge-in**: when the server VAD detects user speech mid-response
(`input_audio_buffer.speech_started` while `ai-speaking`), the client
posts `{ kind: "clear" }` to the playback worklet to wipe the queue
immediately. The server itself cancels the in-flight response.
## Credits
FastAPI's API explorer remains at `/docs`. Project documents are served as files below
`/docs/client/`, `/docs/speech/`, `/docs/adr/`, `/docs/examples/`, and
`/docs/proposals/`. The historical `/project/` and `/public/docs.html` URLs
redirect to the retained architecture proposal. Markdown remains source text;
documentation directories are not browsable rendered pages. Start from the
repository documentation index or request `/docs/README.md` explicitly.
s2s-client/auth.py
+1 -1
View File
@@ -177,7 +177,7 @@ def user_view(request) -> dict:
token = _field(info, "access_token")
out = {
"loggedIn": True,
"username": _field(user, "preferred_username") or _field(user, "name") or "you",
"username": _field(user, "preferred_username") or _field(user, "name") or "tú",
"avatar": _field(user, "picture"),
"tier": resolve_tier(user, token),
}
s2s-client/index.html
+88 -89
View File
@@ -1,12 +1,12 @@
<!doctype html>
<html lang="en">
<html lang="es">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate, max-age=0" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
<title>Minimal Conversation · S2S backend (WebSocket)</title>
<title>Vocero · Conversación de voz</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
@@ -18,8 +18,7 @@
BEFORE the <link rel="stylesheet" href="style.css"> tag below.
Problem reported by user 2026-09-10:
"quando faccio refresh per un momento si vede il theme precedente
quello purple"
"When I refresh, the previous purple theme briefly appears."
The earlier guard `html:not([data-theme]) [data-theme-only]` only
hid the two parallel widget branches, but the UNTHEMED DEFAULTS in
@@ -153,7 +152,7 @@
topbar control cluster. Clicking slides in the offcanvas sidebar on
mobile; it's a no-op hover hint on desktop where sidebar is always
shown. -->
<button id="reports-sidebar-toggle" class="icon-btn" title="Reports programados" aria-label="Abrir panel de reportes" aria-expanded="false" aria-controls="reports-sidebar" hidden data-reports-sidebar-toggle data-debug-only="true" data-theme-only="dark">
<button id="reports-sidebar-toggle" class="icon-btn" title="Reportes programados" aria-label="Abrir panel de reportes" aria-expanded="false" aria-controls="reports-sidebar" hidden data-reports-sidebar-toggle data-debug-only="true" data-theme-only="dark">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M3 3h18v18H3z" fill="none"/><line x1="9" y1="3" x2="9" y2="21"/></svg>
<span id="reports-sidebar-badge" class="chat-badge" aria-hidden="true"></span>
</button>
@@ -161,25 +160,25 @@
<!-- HF login chip / sign-in pill. Populated by ui/account.js; only
shown when the deploy runs behind a load balancer with OAuth. -->
<div id="account" class="account" hidden></div>
<button id="about-btn-m" class="icon-btn about-btn-mobile" title="About this space" aria-label="About this space">
<button id="about-btn-m" class="icon-btn about-btn-mobile" title="Acerca de este espacio" aria-label="Acerca de este espacio">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" y1="16" x2="12" y2="12"/><line x1="12" y1="8" x2="12.01" y2="8"/></svg>
</button>
<button id="tools-btn" class="icon-btn" title="Tools" aria-label="Tools">
<button id="tools-btn" class="icon-btn" title="Herramientas" aria-label="Herramientas">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/></svg>
</button>
<button id="rag-btn" class="icon-btn" title="Custom RAG context" aria-label="Custom RAG context">
<button id="rag-btn" class="icon-btn" title="Contexto RAG personalizado" aria-label="Contexto RAG personalizado">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"/><path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2z"/></svg>
</button>
<button id="settings-btn" class="icon-btn" title="Settings" aria-label="Settings">
<button id="settings-btn" class="icon-btn" title="Configuración" aria-label="Configuración">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 1 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 1 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 1 1-2.83-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 1 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 1 1 2.83-2.83l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 1 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 1 1 2.83 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9c0 .66.39 1.25 1 1.51H21a2 2 0 1 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"/></svg>
</button>
</div>
</header>
<a class="feedback-link" href="https://tally.so/r/pb8LWq" target="_blank" rel="noopener noreferrer">Enviar feedback</a>
<a class="feedback-link" href="https://tally.so/r/pb8LWq" target="_blank" rel="noopener noreferrer">Enviar comentarios</a>
<!-- Theme=white only: Status card between topbar and stage
"Listo para conversar / Informes, análisis, datos y mucho más."
Ready to converse / Reports, analysis, data and more.
with light-blue chart icon and chevron-right arrow. -->
<div class="status-card" hidden data-theme-only="white" aria-label="Estado de la sesión">
<div class="status-card-icon" aria-hidden="true">
@@ -211,7 +210,7 @@
<path id="mga-hit" class="mga-hit" fill="none" />
<circle id="mga-handle" class="mga-handle" r="3" />
</svg>
<button id="mic-btn" class="side-btn" type="button" aria-label="Mute" title="Mute" aria-hidden="true">
<button id="mic-btn" class="side-btn" type="button" aria-label="Silenciar micrófono" title="Silenciar micrófono" aria-hidden="true">
<svg class="mic-on" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="2" width="6" height="12" rx="3"/><path d="M5 10a7 7 0 0 0 14 0"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
<svg class="mic-off" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><line x1="2" y1="2" x2="22" y2="22"/><path d="M9 5a3 3 0 0 1 6 0v4"/><path d="M9 10v1a3 3 0 0 0 5.1 2.1"/><path d="M19 10a7 7 0 0 1-1.24 3.97"/><path d="M5 10a7 7 0 0 0 11 5.67"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
</button>
@@ -221,7 +220,7 @@
id="main-circle"
class="circle state-idle"
type="button"
aria-label="Start voice conversation"
aria-label="Iniciar conversación de voz"
>
<span class="circle-glow" aria-hidden="true"></span>
<span class="circle-ring" aria-hidden="true"></span>
@@ -257,7 +256,7 @@
</span>
</button>
<button id="stop-btn" class="side-btn" type="button" aria-label="End" title="End" aria-hidden="true">
<button id="stop-btn" class="side-btn" type="button" aria-label="Finalizar" title="Finalizar" aria-hidden="true">
<svg viewBox="0 0 24 24" fill="currentColor"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>
</button>
</div>
@@ -265,8 +264,8 @@
<!-- WHITE THEME mic stage (replaces the purple orb)
- Blue mic circular button (116 px, deep blue, white mic)
- Two waveform banks left + right (audio-reactive CSS bars)
- "Escuchando..." status text
- "Detener" pill button below.
- Listening status text
- Stop pill button below.
Mirrors the screenshot center UI exactly. -->
<div class="white-mic-stage" hidden data-theme-only="white" aria-label="Control de voz">
<!-- Waveform left side: 7 bars mirror right side, heights driven by JS vars -->
@@ -311,10 +310,10 @@
<span class="w-bar wb-7" style="--bar-h: 0.4"></span>
</div>
<!-- Status caption below mic circle: "Escuchando...", "Pensando..." etc. -->
<!-- Status caption below mic circle: listening, thinking, etc. -->
<p id="white-mic-caption" class="white-mic-caption" role="status">Escuchando…</p>
<!-- Secondary pill "Detener" (Stop) below caption, with dark stop-square icon. -->
<!-- Secondary stop pill below caption, with a dark stop-square icon. -->
<button id="white-stop-btn"
class="white-stop-btn"
type="button"
@@ -325,23 +324,23 @@
</button>
</div>
<p id="circle-caption" class="circle-caption" role="status">Tap to start</p>
<p id="circle-caption" class="circle-caption" role="status">Toca para empezar</p>
<p id="circle-subcaption" class="circle-subcaption" hidden></p>
<div id="queue-actions" class="queue-actions" hidden>
<button id="join-queue-btn" class="join-queue-btn" type="button" hidden>
Join now
Entrar ahora
</button>
<button id="leave-queue-btn" class="leave-queue-btn" type="button" hidden>
Leave queue
Salir de la cola
</button>
</div>
</main>
<!-- Theme=white only: Quick action pills below mic area
"Resumen" · "Consultar datos" · "Crear respuesta" each with icon -->
<!-- Theme=white only: summary, data lookup and draft-response quick
actions below the mic area, each with an icon. -->
<div class="quick-actions" hidden data-theme-only="white" aria-label="Acciones rápidas">
<button class="quick-action-btn" type="button" data-action="resumen">
<button class="quick-action-btn" type="button" data-action="summary">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/>
<polyline points="14 2 14 8 20 8"/>
@@ -350,7 +349,7 @@
</svg>
<span>Resumen</span>
</button>
<button class="quick-action-btn" type="button" data-action="consultar-datos">
<button class="quick-action-btn" type="button" data-action="query-data">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<line x1="6" y1="20" x2="6" y2="12"/>
<line x1="11" y1="20" x2="11" y2="6"/>
@@ -359,7 +358,7 @@
</svg>
<span>Consultar datos</span>
</button>
<button class="quick-action-btn" type="button" data-action="crear-respuesta">
<button class="quick-action-btn" type="button" data-action="draft-response">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<line x1="18" y1="2" x2="22" y2="6"/>
<path d="M7.5 20.5 19 9l-4-4L3.5 16.5 2 22z"/>
@@ -368,15 +367,15 @@
</button>
</div>
<!-- Chat di testo SEMPRE VISIBILE (fissa in basso, tra stage e footer).
Nota: ora è un blocco diretto del flex-col #app, non più un overlay
destro → non c'è più bisogno del bottone toggle in topbar. -->
<!-- Text chat ALWAYS VISIBLE, fixed between the stage and footer.
The panel is now a direct child of the flex-column #app, so it
no longer needs a topbar toggle for a right-side overlay. -->
<div id="chat-panel" class="chat-panel chat-panel-inline" aria-label="Chat de texto">
<div class="chat-panel-inner">
<header class="chat-panel-header">
<h3>Conversación</h3>
<div class="chat-header-actions" aria-label="Acciones de la conversación">
<!-- "En línea" status pill (green dot + text) — theme=white only.
<!-- Online status pill (green dot + text) — theme=white only.
Mirrors the screenshot chat header top-right corner. -->
<span class="chat-status-pill" hidden data-theme-only="white" role="status" aria-live="polite">
<span class="chat-status-dot" aria-hidden="true"></span>
@@ -428,14 +427,14 @@
<div id="bubble-stack" class="bubble-stack" aria-live="polite" aria-atomic="false"></div>
<!-- Webcam self-view, shown bottom-left while the camera tool is enabled.
HIDDEN BY DEFAULT TODAY: riattivare togliendo hidden + scommentando listener in main.js -->
HIDDEN BY DEFAULT TODAY: remove hidden and restore the main.js listener to reactivate. -->
<div id="cam-pip" class="cam-pip" aria-hidden="true" hidden>
<video id="cam-video" class="cam-video" autoplay playsinline muted></video>
<span class="cam-flash" aria-hidden="true"></span>
<span class="cam-label">camera</span>
<span class="cam-label">cámara</span>
</div>
<!-- Left sidebar: reports programados.
<!-- Left sidebar: scheduled reports.
Desktop: fixed left, 320px, always visible (below topbar + above sticky footer).
Mobile : off-canvas left overlay, toggled by #reports-sidebar-toggle button in topbar. -->
<aside id="reports-sidebar" class="reports-sidebar" aria-label="Reportes programados" hidden data-theme-only="dark">
@@ -459,8 +458,8 @@
<dialog id="about-modal" class="modal about-modal">
<div class="modal-content">
<header class="modal-header">
<h2>About</h2>
<button id="about-close" class="icon-btn" aria-label="Close">
<h2>Acerca de</h2>
<button id="about-close" class="icon-btn" aria-label="Cerrar">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
@@ -471,29 +470,29 @@
<!-- The backend, shown as the path a turn actually travels -->
<div class="about-pipeline">
<p class="pipeline-title">The pipeline</p>
<p class="pipeline-title">Cómo funciona</p>
<ol class="pipeline">
<li class="pipe-endpoint">You speak</li>
<li class="pipe-endpoint">Tú hablas</li>
<li class="pipe-stage">
<span class="pipe-tag">VAD</span>
<span class="pipe-job">detects speech</span>
<span class="pipe-job">detecta el habla</span>
</li>
<li class="pipe-stage">
<span class="pipe-tag">STT</span>
<span class="pipe-job">transcribes it</span>
<span class="pipe-job">la transcribe</span>
<a class="pipe-model" href="https://huggingface.co/nvidia/parakeet-tdt-1.1b" target="_blank" rel="noopener">nvidia/parakeet-tdt-1.1b<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-stage">
<span class="pipe-tag">VLM</span>
<span class="pipe-job">analyses and composes the reply <span class="pipe-note">· via <a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a></span></span>
<span class="pipe-job">analiza y redacta la respuesta <span class="pipe-note">· mediante <a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a></span></span>
<a class="pipe-model" href="https://huggingface.co/google/gemma-4-31B-it" target="_blank" rel="noopener">google/gemma-4-31B-it<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-stage">
<span class="pipe-tag">TTS</span>
<span class="pipe-job">speaks it back</span>
<span class="pipe-job">la convierte en voz</span>
<a class="pipe-model" href="https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" target="_blank" rel="noopener">Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-endpoint">The orb replies</li>
<li class="pipe-endpoint">Vocero responde</li>
</ol>
</div>
</div>
@@ -502,8 +501,8 @@
<dialog id="settings-modal" class="modal">
<form method="dialog" class="modal-content">
<header class="modal-header">
<h2>Settings</h2>
<button class="icon-btn" value="close" aria-label="Close">
<h2>Configuración</h2>
<button class="icon-btn" value="close" aria-label="Cerrar">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
@@ -511,17 +510,17 @@
<div class="tab-panels">
<div class="tab-panel active" role="tabpanel">
<label class="field" id="conn-field">
<span id="conn-label">Speech-to-speech server URL</span>
<input id="lb-url" type="text" autocomplete="off" spellcheck="false" placeholder="http://localhost:port" />
<span id="conn-label">URL del servidor de voz</span>
<input id="lb-url" type="text" autocomplete="off" spellcheck="false" placeholder="http://localhost:8080" />
<small id="conn-hint">
URL of your speech-to-speech server, e.g.
<code>http://localhost:8080</code> (the app adds <code>/v1/realtime</code>).
URL de tu servidor de voz, por ejemplo
<code>http://localhost:8080</code> (la aplicación añade <code>/v1/realtime</code>).
</small>
</label>
<div class="field-row">
<label class="field">
<span>Voice</span>
<span>Voz</span>
<select id="voice">
<option value="Aiden" selected>Aiden</option>
<option value="Ryan">Ryan</option>
@@ -538,8 +537,8 @@
<div class="field">
<span class="field-head">
Noise gate
<span id="gate-value" class="field-value">Off</span>
Puerta de ruido
<span id="gate-value" class="field-value">Desactivada</span>
</span>
<div class="gate">
<div class="gate-track" aria-hidden="true">
@@ -551,37 +550,37 @@
max="-3"
step="1"
value="-50"
aria-label="Noise gate threshold"
aria-label="Umbral de la puerta de ruido"
/>
</div>
<div class="gate-ends">
<span>Off</span>
<span>Desactivada</span>
<span>−3 dB</span>
</div>
</div>
<small>
Mutes the mic below the handle so room noise isn't sent. Slide fully
left to turn it off; the bar shows your live input while in a call.
Silencia el micrófono por debajo del umbral para evitar enviar ruido.
Desliza hasta la izquierda para desactivarla. La barra muestra tu audio durante la llamada.
</small>
</div>
<label class="field">
<span>Instructions</span>
<textarea id="instructions" rows="5" placeholder="You are Vocero, a friendly voice assistant for the company Aquachile..."></textarea>
<span>Instrucciones</span>
<textarea id="instructions" rows="5" placeholder="Introduce las instrucciones de Vocero para esta conversación…"></textarea>
<small id="primary-prompt-meta" hidden></small>
</label>
<div class="field">
<button id="restart-conversation" type="button" class="btn primary wide" disabled>
Restart conversation with these settings
Reiniciar conversación con esta configuración
</button>
<small id="restart-hint">Connect first, then come back here to apply live changes.</small>
<small id="restart-hint">Conéctate primero y vuelve aquí para aplicar cambios durante la conversación.</small>
</div>
</div>
</div>
<footer class="modal-footer">
<button id="settings-save" type="submit" class="btn primary" value="save">Save</button>
<button id="settings-save" type="submit" class="btn primary" value="save">Guardar</button>
</footer>
</form>
</dialog>
@@ -589,19 +588,19 @@
<dialog id="tools-modal" class="modal">
<div class="modal-content">
<header class="modal-header">
<h2>Tools</h2>
<button id="tools-close" class="icon-btn" aria-label="Close">
<h2>Herramientas</h2>
<button id="tools-close" class="icon-btn" aria-label="Cerrar">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<p class="tools-intro">Let the assistant act during the conversation. Changes apply live.</p>
<p class="tools-intro">Permite que el asistente actúe durante la conversación. Los cambios se aplican de inmediato.</p>
<div class="tool-list">
<div class="tool-row" id="tool-web-row">
<div class="tool-info">
<span class="tool-name">Web search</span>
<span class="tool-desc">Look things up on Google, via Serper.</span>
<span class="tool-name">Búsqueda web</span>
<span class="tool-desc">Busca información en Google mediante Serper.</span>
</div>
<label class="switch">
<input id="tool-web" type="checkbox" />
@@ -609,30 +608,30 @@
</label>
</div>
<label class="field tools-key">
<span>Search API key</span>
<span>Clave de la API de búsqueda</span>
<input id="search-key" type="password" autocomplete="off" spellcheck="false" />
<small id="tool-web-hint"></small>
</label>
<!-- Camera tool row — HIDDEN BY DEFAULT TODAY.
Per riattivarlo in futuro: togli hidden da div + small,
scommenta listener toolCamSwitch + funzioni enableCamera in main.js. -->
To reactivate later, remove hidden from the div and small,
then restore the toolCamSwitch listener and enableCamera functions in main.js. -->
<div class="tool-row tool-row-sep" id="tool-cam-row" hidden>
<div class="tool-info">
<span class="tool-name">Camera</span>
<span class="tool-desc">Let the assistant see through your webcam.</span>
<span class="tool-name">Cámara</span>
<span class="tool-desc">Permite que el asistente vea a través de tu cámara.</span>
</div>
<label class="switch">
<input id="tool-cam" type="checkbox" />
<span class="switch-track" aria-hidden="true"></span>
</label>
</div>
<small id="tool-cam-hint" class="tool-hint" hidden>A live preview shows bottom-left while the camera is on.</small>
<small id="tool-cam-hint" class="tool-hint" hidden>La vista previa aparece abajo a la izquierda mientras la cámara está activa.</small>
<div class="tool-row tool-row-sep">
<div class="tool-info">
<span class="tool-name">Loaded MCP tools</span>
<span class="tool-desc">Tools published by the remote agent profile.</span>
<span class="tool-name">Herramientas MCP cargadas</span>
<span class="tool-desc">Herramientas publicadas por el perfil remoto del agente.</span>
</div>
</div>
<small id="agent-tools-hint" class="tool-hint"></small>
@@ -644,36 +643,36 @@
<dialog id="rag-modal" class="modal">
<div class="modal-content">
<header class="modal-header">
<h2>Custom RAG context</h2>
<button id="rag-close" class="icon-btn" aria-label="Close">
<h2>Contexto RAG personalizado</h2>
<button id="rag-close" class="icon-btn" aria-label="Cerrar">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<p class="tools-intro">
Incolla qui testo, documenti, o estratti RAG che vuoi siano disponibili
per la sessione vocale corrente. Il testo viene aggiunto sia alle
istruzioni del LLM sia al campo <code>user_context</code> del body
Pega aquí texto, documentos o fragmentos RAG que quieras tener disponibles
durante la sesión de voz actual. El texto se añade a las
instrucciones del modelo y al campo <code>user_context</code> de la solicitud a
<code>/api/reasoning</code>.
</p>
<label class="field">
<span>RAG context (testo libero, puro)</span>
<span>Contexto RAG (texto libre)</span>
<textarea
id="rag-textarea"
rows="12"
spellcheck="false"
placeholder="Incolla o scrivi qui il contesto RAG che il modello deve usare durante la risposta. Es: dati cliente, policy, estratti documentali, ecc."
placeholder="Pega o escribe el contexto RAG que el modelo debe usar al responder. Por ejemplo: datos del cliente, políticas o fragmentos de documentos."
></textarea>
<small id="rag-hint" class="tool-hint">
Nessun RAG context caricato. Premi &quot;Invia nella sessione&quot; per applicare le modifiche.
No hay contexto RAG cargado. Pulsa &quot;Enviar a la sesión&quot; para aplicar los cambios.
</small>
</label>
<footer class="modal-footer">
<button id="rag-clear" class="btn secondary" type="button" value="clear">Svuota</button>
<button id="rag-clear" class="btn secondary" type="button" value="clear">Vaciar</button>
<div style="flex: 1"></div>
<button id="rag-submit" class="btn primary" type="button" value="submit">Invia nella sessione</button>
<button id="rag-submit" class="btn primary" type="button" value="submit">Enviar a la sesión</button>
</footer>
</div>
</dialog>
@@ -683,13 +682,13 @@
ui/account.js. The hero is the HF mark — a friendly smiling face. -->
<dialog id="limit-modal" class="modal limit-modal">
<div class="modal-content limit-card">
<button id="limit-close" class="icon-btn limit-close" aria-label="Close">
<button id="limit-close" class="icon-btn limit-close" aria-label="Cerrar">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
<div class="limit-badge" aria-hidden="true">
<svg class="hf-logo" viewBox="0 0 95 88" fill="none" aria-hidden="true"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg>
</div>
<h2 id="limit-title" class="limit-title">That's a wrap for now</h2>
<h2 id="limit-title" class="limit-title">Por ahora hemos terminado</h2>
<p id="limit-msg" class="limit-msg"></p>
<a id="limit-cta" class="btn primary wide limit-cta" href="#"></a>
<p id="limit-note" class="limit-note"></p>
@@ -701,7 +700,7 @@
user metadata populated via main.js populateUserProfileDialog(). -->
<dialog id="user-profile-dialog" class="modal user-profile-modal" aria-labelledby="user-profile-title">
<div class="modal-content user-profile-card">
<button id="user-profile-close" class="icon-btn user-profile-close" aria-label="Close user profile">
<button id="user-profile-close" class="icon-btn user-profile-close" aria-label="Cerrar perfil de usuario">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
@@ -715,12 +714,12 @@
<!-- Metadata: a 1-col definition-list-like grid. Empty rows are hidden
automatically via JS when the field has no value, so we never show
blank labels. -->
<div id="user-profile-metadata" class="user-profile-metadata" aria-label="Informazioni utente"></div>
<div id="user-profile-metadata" class="user-profile-metadata" aria-label="Información del usuario"></div>
</div>
</dialog>
<footer class="app-footer" aria-label="App version footer">
<span class="app-footer-text">Azurian Vocero realtime ver.Beta 2.1</span>
<footer class="app-footer" aria-label="Versión de la aplicación">
<span class="app-footer-text">Azurian Vocero en tiempo real ver. Beta 2.1</span>
</footer>
<script type="module" src="main.js?v=202609102207"></script>
s2s-client/index_video.html
-356
View File
@@ -1,356 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate, max-age=0" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
<title>Minimal Conversation · S2S backend (WebSocket)</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Geist+Mono:wght@400;500;600&family=Inter:wght@400;500;600;700;800&display=swap"
rel="stylesheet"
/>
<link rel="stylesheet" href="style.css?v=20260825003" />
</head>
<!--
`booting` disables transitions/animations until the first paint commits.
Without this the orb visibly fades/scales in on the very first frame
because `state-idle` and the generic `.ind` defaults differ. The
bootstrap script at the end of main.js strips the class after one rAF.
-->
<body class="booting">
<div id="app">
<header class="topbar">
<div class="brand">
<div class="ident">
<div class="ident-head">
<a class="ident-title">Azurian Vocero realtime test</a>
</div>
<div id="ident-user-name" class="ident-user-name" hidden></div>
</div>
</div>
<div class="topbar-right">
<!-- HF login chip / sign-in pill. Populated by ui/account.js; only
shown when the deploy runs behind a load balancer with OAuth. -->
<div id="account" class="account" hidden></div>
<button id="about-btn-m" class="icon-btn about-btn-mobile" title="About this space" aria-label="About this space">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" y1="16" x2="12" y2="12"/><line x1="12" y1="8" x2="12.01" y2="8"/></svg>
</button>
<button id="tools-btn" class="icon-btn" title="Tools" aria-label="Tools">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/></svg>
</button>
<button id="chat-btn" class="icon-btn" title="Conversation history" aria-label="Conversation history">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15a2 2 0 0 1-2 2H7l-4 4V5a2 2 0 0 1 2-2h14a2 2 0 0 1 2 2z"/></svg>
<span id="chat-badge" class="chat-badge" aria-hidden="true"></span>
</button>
<button id="settings-btn" class="icon-btn" title="Settings" aria-label="Settings">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 1 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 1 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 1 1-2.83-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 1 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 1 1 2.83-2.83l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 1 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 1 1 2.83 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9c0 .66.39 1.25 1 1.51H21a2 2 0 1 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"/></svg>
</button>
</div>
</header>
<main class="stage">
<div class="orb-wrap">
<div id="mic-gate" class="mic-gate">
<svg id="mic-gate-arc" class="mic-gate-arc" viewBox="0 0 100 100" aria-hidden="true">
<path id="mga-track" class="mga-track" fill="none" />
<path id="mga-fill" class="mga-fill" fill="none" />
<path id="mga-hit" class="mga-hit" fill="none" />
<circle id="mga-handle" class="mga-handle" r="3" />
</svg>
<button id="mic-btn" class="side-btn" type="button" aria-label="Mute" title="Mute" aria-hidden="true">
<svg class="mic-on" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="2" width="6" height="12" rx="3"/><path d="M5 10a7 7 0 0 0 14 0"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
<svg class="mic-off" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><line x1="2" y1="2" x2="22" y2="22"/><path d="M9 5a3 3 0 0 1 6 0v4"/><path d="M9 10v1a3 3 0 0 0 5.1 2.1"/><path d="M19 10a7 7 0 0 1-1.24 3.97"/><path d="M5 10a7 7 0 0 0 11 5.67"/><line x1="12" y1="19" x2="12" y2="22"/></svg>
</button>
</div>
<button
id="main-circle"
class="circle state-idle"
type="button"
aria-label="Start voice conversation"
>
<span class="circle-glow" aria-hidden="true"></span>
<span class="circle-ring" aria-hidden="true"></span>
<span class="circle-ring-outer" aria-hidden="true"></span>
<span class="circle-core">
<span class="circle-indicator" aria-hidden="true">
<svg class="ind ind-mic" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<rect x="9" y="2" width="6" height="12" rx="3" fill="currentColor" stroke="none"/>
<path d="M5 10a7 7 0 0 0 14 0"/>
<line x1="12" y1="19" x2="12" y2="22"/>
<line x1="8" y1="22" x2="16" y2="22"/>
</svg>
<svg class="ind ind-error" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="9"/><line x1="12" y1="8" x2="12" y2="13"/><line x1="12" y1="16" x2="12" y2="16"/></svg>
<span class="ind ind-spinner"></span>
<span class="ind ind-thinking">
<span class="dot"></span>
<span class="dot"></span>
<span class="dot"></span>
</span>
<span class="ind ind-bars">
<span class="bar"></span>
<span class="bar"></span>
<span class="bar"></span>
<span class="bar"></span>
<span class="bar"></span>
</span>
<svg class="ind ind-voice" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="M3 10v4a1 1 0 0 0 1 1h3l5 4V5L7 9H4a1 1 0 0 0-1 1z" fill="currentColor" stroke="none"/>
<path class="wave wave-1" d="M16 8a5 5 0 0 1 0 8"/>
<path class="wave wave-2" d="M19 5a9 9 0 0 1 0 14"/>
</svg>
</span>
</span>
</button>
<button id="stop-btn" class="side-btn" type="button" aria-label="End" title="End" aria-hidden="true">
<svg viewBox="0 0 24 24" fill="currentColor"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>
</button>
</div>
<p id="circle-caption" class="circle-caption" role="status">Tap to start</p>
<p id="circle-subcaption" class="circle-subcaption" hidden></p>
<div id="queue-actions" class="queue-actions" hidden>
<button id="join-queue-btn" class="join-queue-btn" type="button" hidden>
Join now
</button>
<button id="leave-queue-btn" class="leave-queue-btn" type="button" hidden>
Leave queue
</button>
</div>
</main>
</div>
<!-- Ephemeral chat bubbles anchored to the top-right -->
<div id="bubble-stack" class="bubble-stack" aria-live="polite" aria-atomic="false"></div>
<!-- Webcam self-view, shown bottom-left while the camera tool is enabled -->
<div id="cam-pip" class="cam-pip" aria-hidden="true">
<video id="cam-video" class="cam-video" autoplay playsinline muted></video>
<span class="cam-flash" aria-hidden="true"></span>
<span class="cam-label">camera</span>
</div>
<!-- Conversation history panel (overlay) -->
<div id="chat-panel" class="chat-panel">
<div id="chat-panel-backdrop" class="chat-panel-backdrop"></div>
<div class="chat-panel-inner">
<header class="chat-panel-header">
<h3>Conversation</h3>
<button id="chat-panel-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<div id="chat-history" class="chat-history"></div>
</div>
</div>
<dialog id="about-modal" class="modal about-modal">
<div class="modal-content">
<header class="modal-header">
<h2>About</h2>
<button id="about-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<!-- General intro to the speech-to-speech project -->
<div class="about-intro">
</div>
<!-- The backend, shown as the path a turn actually travels -->
<div class="about-pipeline">
<p class="pipeline-title">The pipeline</p>
<ol class="pipeline">
<li class="pipe-endpoint">You speak</li>
<li class="pipe-stage">
<span class="pipe-tag">VAD</span>
<span class="pipe-job">detects speech</span>
</li>
<li class="pipe-stage">
<span class="pipe-tag">STT</span>
<span class="pipe-job">transcribes it</span>
<a class="pipe-model" href="https://huggingface.co/nvidia/parakeet-tdt-1.1b" target="_blank" rel="noopener">nvidia/parakeet-tdt-1.1b<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-stage">
<span class="pipe-tag">VLM</span>
<span class="pipe-job">analyses and composes the reply <span class="pipe-note">· via <a class="cerebras-credit" href="https://cerebras.ai" target="_blank" rel="noopener"><svg class="cerebras-mark" viewBox="0 0 50 50" aria-hidden="true"><path fill-rule="evenodd" clip-rule="evenodd" d="M29.4186 5.62713C24.2806 5.62713 19.353 7.6682 15.7199 11.3013C12.0868 14.9345 10.0457 19.862 10.0457 25C10.0457 30.1381 12.0868 35.0656 15.7199 38.6988C19.353 42.3319 24.2806 44.373 29.4186 44.373V47.2917C17.1061 47.2917 7.12695 37.3105 7.12695 24.998C7.12695 12.6855 17.104 2.7063 29.4165 2.7063V5.62505L29.4186 5.62713ZM39.3186 13.2875C37.7794 11.9809 35.9971 10.9915 34.0741 10.3761C32.1512 9.76067 30.1256 9.53147 28.1137 9.70166C26.1019 9.87185 24.1435 10.4381 22.3513 11.3677C20.559 12.2974 18.9683 13.5722 17.6704 15.1189C16.3726 16.6655 15.3933 18.4534 14.7888 20.3798C14.1844 22.3062 13.9667 24.3332 14.1483 26.344C14.33 28.3548 14.9073 30.3099 15.8472 32.0968C16.787 33.8838 18.0709 35.4673 19.6249 36.7563L17.7478 38.9938C15.9126 37.4545 14.3987 35.5687 13.2924 33.4442C12.1862 31.3197 11.5092 28.9981 11.3003 26.6119C11.0914 24.2258 11.3545 21.8219 12.0747 19.5374C12.7949 17.253 13.9581 15.1328 15.4978 13.298C17.0373 11.4628 18.9233 9.94877 21.0479 8.84247C23.1726 7.73618 25.4944 7.05923 27.8807 6.8503C30.267 6.64136 32.6711 6.90453 34.9557 7.62477C37.2403 8.34501 39.3607 9.50822 41.1957 11.048L39.3186 13.2875ZM34.6207 15.0459C31.9818 13.6829 28.9113 13.4172 26.0776 14.3068C23.2438 15.1964 20.876 17.1691 19.4895 19.7958C18.1029 22.4224 17.8099 25.4903 18.6741 28.332C19.5384 31.1736 21.4899 33.5589 24.104 34.9688L22.7374 37.5521C19.4721 35.7617 17.0418 32.7592 15.9707 29.1927C14.8996 25.6261 15.2737 21.7815 17.0123 18.4883C18.7509 15.1952 21.7146 12.7176 25.2637 11.5903C28.8129 10.463 32.663 10.7763 35.9832 12.4625L34.6207 15.0459ZM29.4165 17.7896C27.5042 17.7896 25.6702 18.5493 24.318 19.9015C22.9658 21.2537 22.2061 23.0877 22.2061 25C22.2061 26.9124 22.9658 28.7464 24.318 30.0986C25.6702 31.4508 27.5042 32.2105 29.4165 32.2105V35.1313C26.7296 35.1313 24.1526 34.0639 22.2527 32.1639C20.3527 30.2639 19.2853 27.687 19.2853 25C19.2853 22.3131 20.3527 19.7362 22.2527 17.8362C24.1526 15.9362 26.7296 14.8688 29.4165 14.8688V17.7896Z" fill="#F15A29"/><path d="M32.0981 22.5749C31.7875 22.2404 31.4295 21.9534 31.0356 21.7228C30.6925 21.519 30.3014 21.4097 29.9023 21.4062C29.371 21.4062 28.896 21.5041 28.4773 21.6978C28.0725 21.8802 27.7089 22.1425 27.4081 22.469C27.1074 22.7955 26.8758 23.1795 26.7273 23.5978C26.5731 24.0207 26.4981 24.4645 26.4981 24.9124C26.4981 25.3666 26.5731 25.8082 26.7273 26.227C26.877 26.6449 27.1091 27.0286 27.4096 27.3553C27.7102 27.682 28.0733 27.9451 28.4773 28.1291C28.8939 28.3228 29.3731 28.4207 29.9023 28.4207C30.3523 28.4207 30.771 28.3249 31.1564 28.1395C31.5481 27.952 31.8856 27.6707 32.146 27.3228L34.0794 29.4187C33.7877 29.7103 33.4544 29.9624 33.0752 30.1749C32.374 30.566 31.6108 30.8338 30.8189 30.9666C30.4648 31.0207 30.1585 31.0499 29.9023 31.0499C29.0617 31.0552 28.2271 30.9069 27.4398 30.6124C26.6954 30.3371 26.0144 29.914 25.4377 29.3687C24.8643 28.8208 24.4078 28.1624 24.096 27.4332C23.7572 26.6366 23.5883 25.778 23.6002 24.9124C23.6002 23.9874 23.7669 23.1478 24.096 22.3916C24.4085 21.6624 24.8627 21.0041 25.4356 20.4562C26.0148 19.9124 26.696 19.4895 27.4398 19.2145C28.2271 18.92 29.0617 18.7717 29.9023 18.777C30.6419 18.777 31.3856 18.9187 32.1356 19.202C32.8877 19.4895 33.5627 19.952 34.1023 20.5541L32.0981 22.5749Z" fill="#fff"/></svg>Cerebras</a></span></span>
<a class="pipe-model" href="https://huggingface.co/google/gemma-4-31B-it" target="_blank" rel="noopener">google/gemma-4-31B-it<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-stage">
<span class="pipe-tag">TTS</span>
<span class="pipe-job">speaks it back</span>
<a class="pipe-model" href="https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" target="_blank" rel="noopener">Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice<svg class="ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M8 7h9v9"/></svg></a>
</li>
<li class="pipe-endpoint">The orb replies</li>
</ol>
</div>
</div>
</dialog>
<dialog id="settings-modal" class="modal">
<form method="dialog" class="modal-content">
<header class="modal-header">
<h2>Settings</h2>
<button class="icon-btn" value="close" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<div class="tab-panels">
<section class="tab-panel active" role="tabpanel">
<label class="field" id="conn-field">
<span id="conn-label">Speech-to-speech server URL</span>
<input id="lb-url" type="text" autocomplete="off" spellcheck="false" placeholder="http://localhost:port" />
<small id="conn-hint">
URL of your speech-to-speech server, e.g.
<code>http://localhost:8080</code> (the app adds <code>/v1/realtime</code>).
</small>
</label>
<div class="field-row">
<label class="field">
<span>Voice</span>
<select id="voice">
<option value="Aiden" selected>Aiden</option>
<option value="Ryan">Ryan</option>
<option value="Dylan">Dylan</option>
<option value="Eric">Eric</option>
<option value="Ono_Anna">Ono_Anna</option>
<option value="Serena">Serena</option>
<option value="Sohee">Sohee</option>
<option value="Uncle_Fu">Uncle_Fu</option>
<option value="Vivian">Vivian</option>
</select>
</label>
</div>
<div class="field">
<span class="field-head">
Noise gate
<span id="gate-value" class="field-value">Off</span>
</span>
<div class="gate">
<div class="gate-track" aria-hidden="true">
<div id="gate-meter-fill" class="gate-meter-fill"></div>
<input
id="noise-gate"
type="range"
min="-66"
max="-3"
step="1"
value="-50"
aria-label="Noise gate threshold"
/>
</div>
<div class="gate-ends">
<span>Off</span>
<span>−3 dB</span>
</div>
</div>
<small>
Mutes the mic below the handle so room noise isn't sent. Slide fully
left to turn it off; the bar shows your live input while in a call.
</small>
</div>
<label class="field">
<span>Instructions</span>
<textarea id="instructions" rows="5" placeholder="You are Vocero, a friendly voice assistant for the company Aquachile..."></textarea>
<small id="primary-prompt-meta" hidden></small>
</label>
<div class="field">
<button id="restart-conversation" type="button" class="btn primary wide" disabled>
Restart conversation with these settings
</button>
<small id="restart-hint">Connect first, then come back here to apply live changes.</small>
</div>
</section>
</div>
<footer class="modal-footer">
<button id="settings-save" type="submit" class="btn primary" value="save">Save</button>
</footer>
</form>
</dialog>
<dialog id="tools-modal" class="modal">
<div class="modal-content">
<header class="modal-header">
<h2>Tools</h2>
<button id="tools-close" class="icon-btn" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
</header>
<p class="tools-intro">Let the assistant act during the conversation. Changes apply live.</p>
<div class="tool-list">
<div class="tool-row" id="tool-web-row">
<div class="tool-info">
<span class="tool-name">Web search</span>
<span class="tool-desc">Look things up on Google, via Serper.</span>
</div>
<label class="switch">
<input id="tool-web" type="checkbox" />
<span class="switch-track" aria-hidden="true"></span>
</label>
</div>
<label class="field tools-key">
<span>Search API key</span>
<input id="search-key" type="password" autocomplete="off" spellcheck="false" />
<small id="tool-web-hint"></small>
</label>
<div class="tool-row tool-row-sep" id="tool-cam-row">
<div class="tool-info">
<span class="tool-name">Camera</span>
<span class="tool-desc">Let the assistant see through your webcam.</span>
</div>
<label class="switch">
<input id="tool-cam" type="checkbox" />
<span class="switch-track" aria-hidden="true"></span>
</label>
</div>
<small id="tool-cam-hint" class="tool-hint">A live preview shows bottom-left while the camera is on.</small>
<div class="tool-row tool-row-sep">
<div class="tool-info">
<span class="tool-name">Loaded MCP tools</span>
<span class="tool-desc">Tools published by the remote agent profile.</span>
</div>
</div>
<small id="agent-tools-hint" class="tool-hint"></small>
<div id="agent-tools-list" class="agent-tools-list"></div>
</div>
</div>
</dialog>
<!-- Shown when the per-day conversation budget is spent (mid-call or at
start). Title / message / CTA / note are filled per tier by
ui/account.js. The hero is the HF mark — a friendly smiling face. -->
<dialog id="limit-modal" class="modal limit-modal">
<div class="modal-content limit-card">
<button id="limit-close" class="icon-btn limit-close" aria-label="Close">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</button>
<div class="limit-badge" aria-hidden="true">
<svg class="hf-logo" viewBox="0 0 95 88" fill="none" aria-hidden="true"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg>
</div>
<h2 id="limit-title" class="limit-title">That's a wrap for now</h2>
<p id="limit-msg" class="limit-msg"></p>
<a id="limit-cta" class="btn primary wide limit-cta" href="#"></a>
<p id="limit-note" class="limit-note"></p>
</div>
</dialog>
<script type="module" src="main.js?v=20260825003"></script>
</body>
</html>
s2s-client/main.js
+191 -210
View File
@@ -22,36 +22,35 @@ import { ChatView } from "./ui/chat.js?v=202609041120";
import { Account } from "./ui/account.js?v=202609041120";
// ---------------------------------------------------------------------------
// 🔎 DEBUG BANNER (always-on): emissione incondizionata in console per capire
// ESATTAMENTE quale versione di main.js è caricata nel browser.
// Se NON vedi questa riga in Console = stai ancora usando il file CACHATO VECCHIO.
// 🔎 DEBUG BANNER (always-on): identify the exact main.js version loaded.
// If this line is missing from the console, the browser is using an older cached file.
console.info(
"%c🟢 VOCERO main.js loaded. version=2026-09-24 23:12 (DEBUG FETCH WRAPPER+WS SEMPRE-ATTIVO; APIPILOT: demo flag=true + PREFETCH AUTOMATICO OGNI SESSIONE WS + PREFETCH DOPO LOGIN-BY-PHONE OK (refresh=true per invalidare cache server anonima); cache client per-utente; cambio phone = invalidate automatico; export chat; fix scroll).",
"%c🟢 VOCERO main.js loaded. version=2026-09-24 23:12 (ALWAYS-ON FETCH WRAPPER+WS DEBUG; APIPILOT: demo flag=true + AUTOMATIC PREFETCH FOR EVERY WS SESSION + PREFETCH AFTER PHONE LOGIN (refresh=true invalidates the anonymous server cache); per-user client cache; phone changes invalidate automatically; chat export; scroll fix).",
"background:#1e293b;color:#86efac;font-weight:700;padding:4px 8px;border-radius:4px;"
);
console.info(
" URL attuale location.href=", location.href,
" Current URL location.href=", location.href,
"\n location.search=", location.search || "(empty)",
"\n DEBUG (imported from dom.js) =", DEBUG,
"\n typeof window.DEBUG =", typeof window.DEBUG
);
// eslint-disable-next-line no-debugger
// debugger; // ← scommenta per fermarti QUI all'avvio se DevTools è aperto
// debugger; // Uncomment to pause here at startup when DevTools is open.
// ---------------------------------------------------------------------------
// 🔎 FETCH WRAPPER (SEMANTIC DEBUG LOG) — attivo SEMPRE (non solo in DEBUG)
// Ogni chiamata `fetch` lato browser viene intercettata e loggata con:
// • URL risolto, metodo, dimensioni body + preview JSON
// • status HTTP (colorato verde/giallo/rosso a seconda della classe)
// • durata (performance.now), dimensione + preview risposta JSON
// Tag semantici per filtrare veloce la Console:
// 🔎 FETCH WRAPPER (SEMANTIC DEBUG LOG) — always active, including outside DEBUG.
// Every browser fetch is traced with:
// • resolved URL, method, body size and JSON preview
// • HTTP status (green/yellow/red according to status class)
// • duration (performance.now), response size and JSON preview
// Semantic tags for filtering the console:
// [FETCH:CONFIG] [FETCH:REASONING] [FETCH:SEARCH]
// [FETCH:PROFILE] [FETCH:REPORTS] [FETCH:APIPILOT]
// [FETCH:SESSION] [FETCH:QUEUE] [FETCH:ME]
// ---------------------------------------------------------------------------
(function _installFetchDebugTracer() {
const ORIGINAL_FETCH = window.fetch.bind(window);
// Classifica endpoint in tag leggibile (stesso ordine di @app route in server.py).
// Classify endpoints with readable tags, following the route order in server.py.
/** @param {string} url */
function _tagFor(url) {
try {
@@ -108,13 +107,13 @@ console.info(
/** @type {BodyInit | null | undefined} */
let body = init ? init.body : undefined;
if (body === undefined || body === null) {
// tenta Request
// Try the Request body.
try {
const r = /** @type {{body?:BodyInit|null}} */ (/** @type {unknown} */ (input));
if (r && r.body != null) body = r.body;
} catch { body = null; }
}
if (body == null) return { length: 0, preview: "(nessun body)" };
if (body == null) return { length: 0, preview: "(no body)" };
try {
if (typeof body === "string") {
const s = body;
@@ -138,7 +137,7 @@ console.info(
if (body instanceof Blob) return { length: body.size, preview: `BLOB (${body.type || "unknown"}) size=${body.size}` };
if (ArrayBuffer.isView(body)) return { length: body.byteLength, preview: `ArrayBufferView len=${body.byteLength}` };
if (body instanceof ArrayBuffer) return { length: body.byteLength, preview: `ArrayBuffer len=${body.byteLength}` };
return { length: 0, preview: "(body non serializzabile nel log)" };
return { length: 0, preview: "(body cannot be serialized for logging)" };
} catch (err) {
return { length: -1, preview: `(err preview body: ${err})` };
}
@@ -178,9 +177,8 @@ console.info(
const method = _extractMethod(input, init);
const rawUrl = _stringifyUrl(input);
const tag = _tagFor(rawUrl);
// Per richieste "heartbeat / end poll" (molto frequenti) logghiamo un
// singleton collapse per evitare spam: se `?quiet=true` o URL match → log
// a riga singola e non mostriamo preview.
// Log frequent heartbeat/end polling requests on one line without a body
// preview to keep the console readable.
const isPollish = /heartbeat|queue\/end|session\/end/i.test(tag) || (init && typeof init.keepalive === "boolean" && init.keepalive);
const reqPreview = isPollish ? null : await _extractBodyPreview(input, init);
if (!isPollish) {
@@ -245,18 +243,18 @@ console.info(
return response;
};
console.info(
"%c🔎 [FETCH-DEBUG] tracer installato. Ogni fetch è loggata con tag, method, url, status, durata e preview body.",
"%c🔎 [FETCH-DEBUG] tracer installed. Every fetch logs its tag, method, URL, status, duration and body preview.",
"background:#ecfeff;color:#0369a1;font-weight:700;padding:2px 6px;border-radius:4px;"
);
})();
// ---------------------------------------------------------------------------
// 🛡️ GATE .topbar-right (about / tools / rag / chat / settings buttons,
// vedi index.html L36-L56) — 3-layer safety:
// 1) HTML statico → hidden + data-debug-only="true"
// see index.html) — 3-layer safety:
// 1) static HTML → hidden + data-debug-only="true"
// 2) CSS → [data-debug-only="true"]:not([data-debug-enabled="true"]) display:none !important
// 3) JS qui → hidden, removeAttribute data-debug-enabled, anche style.display = "none"
// Solo se URL query-string ha ?debug=true (strict string) abilitiamo TUTTO.
// 3) JS here → hidden, remove data-debug-enabled, set style.display = "none"
// Enable all debug controls only when the URL contains ?debug=true (strict string).
function applyDebugOnlyVisibilityGate(source = "bootstrap") {
const qs = new URLSearchParams(window.location.search);
const debugEnabled = qs.get("debug") === "true";
@@ -265,64 +263,61 @@ function applyDebugOnlyVisibilityGate(source = "bootstrap") {
"%c🧭 [gate-topbar-right] applyDebugOnlyVisibilityGate()",
"color:#f59e0b;font-weight:700;"
);
console.log("chiamato da:", source);
console.log("called from:", source);
console.log("location.search =", JSON.stringify(window.location.search));
console.log("query params =", Object.fromEntries(qs.entries()));
console.log("debugEnabled (strict === 'true') =", debugEnabled);
console.log("topbarRight element trovato?", !!topbarRight, topbarRight ? "vedi →" : "(null!)");
console.log("topbarRight element found?", !!topbarRight, topbarRight ? "see →" : "(null!)");
const chatInline = document.getElementById("chat-panel");
console.log("#chat-panel (chat inline SEMPRE VISIBILE in basso) presente in DOM =", !!chatInline, chatInline ? "hidden?" : "", chatInline ? chatInline.hasAttribute("hidden") : "-");
console.log("#chat-panel (inline chat ALWAYS VISIBLE at the bottom) present in DOM =", !!chatInline, chatInline ? "hidden?" : "", chatInline ? chatInline.hasAttribute("hidden") : "-");
if (topbarRight) {
console.log(" hidden attr prima:", topbarRight.hasAttribute("hidden"),
"| data-debug-enabled prima:", topbarRight.getAttribute("data-debug-enabled"),
"| style.display prima:", topbarRight.style.display);
console.log(" hidden attr before:", topbarRight.hasAttribute("hidden"),
"| data-debug-enabled before:", topbarRight.getAttribute("data-debug-enabled"),
"| style.display before:", topbarRight.style.display);
if (debugEnabled) {
topbarRight.setAttribute("data-debug-enabled", "true");
topbarRight.removeAttribute("hidden");
topbarRight.style.display = "";
console.log("%c ✅ ABILITATO .topbar-right (debug=true)", "color:#22c55e;font-weight:700;");
console.log("%c ✅ ENABLED .topbar-right (debug=true)", "color:#22c55e;font-weight:700;");
} else {
topbarRight.removeAttribute("data-debug-enabled");
topbarRight.setAttribute("hidden", "");
topbarRight.style.setProperty("display", "none", "important");
console.log("%c 🚫 DISABILITATO .topbar-right (no debug=true)",
console.log("%c 🚫 DISABLED .topbar-right (no debug=true)",
"background:#fee2e2;color:#b91c1c;font-weight:700;padding:3px 6px;border-radius:4px;");
}
console.log(" hidden attr dopo:", topbarRight.hasAttribute("hidden"),
"| data-debug-enabled dopo:", topbarRight.getAttribute("data-debug-enabled"),
"| style.display dopo:", topbarRight.style.display);
console.log(" hidden attr after:", topbarRight.hasAttribute("hidden"),
"| data-debug-enabled after:", topbarRight.getAttribute("data-debug-enabled"),
"| style.display after:", topbarRight.style.display);
}
// 3-layer gate UNIVERSALE su TUTTI gli elementi marcati con data-debug-only="true"
// (non solo .topbar-right, ma anche #reports-sidebar-toggle e qualsiasi
// altro elemento futuro marcato con lo stesso attributo).
// Apply the 3-layer gate to every element marked data-debug-only="true",
// including .topbar-right, #reports-sidebar-toggle and future controls.
const debugOnlyEls = document.querySelectorAll("[data-debug-only='true']");
console.log(
`Trovati ${debugOnlyEls.length} elementi [data-debug-only="true"] da gate-are in base a debugEnabled=${debugEnabled}`,
`Found ${debugOnlyEls.length} elements [data-debug-only="true"] to gate using debugEnabled=${debugEnabled}`,
);
debugOnlyEls.forEach((el, idx) => {
const tag = `${el.tagName.toLowerCase()}${el.id ? `#${el.id}` : ""}${el.className && typeof el.className === "string" ? `.${el.className.split(/\s+/).filter(Boolean).slice(0, 2).join(".")}` : ""}`;
console.log(` [${idx + 1}/${debugOnlyEls.length}] ${tag} — hidden prima? ${el.hasAttribute("hidden")} | display prima: ${(el instanceof HTMLElement ? el.style.display : "-")}`);
console.log(` [${idx + 1}/${debugOnlyEls.length}] ${tag} — hidden before? ${el.hasAttribute("hidden")} | display before: ${(el instanceof HTMLElement ? el.style.display : "-")}`);
if (!(el instanceof HTMLElement)) return;
if (debugEnabled) {
el.setAttribute("data-debug-enabled", "true");
// Non forziamo hidden=false su elementi che sono logicamente hidden per
// altri motivi (es. #reports-sidebar-toggle è hidden se non ci sono
// reports). Rimuoviamo lo "spegnimento debug-only" e basta.
// Preserve elements hidden for other reasons, such as an empty reports
// sidebar. Remove only the debug-only display override.
el.style.removeProperty("display");
} else {
el.removeAttribute("data-debug-enabled");
el.setAttribute("hidden", "");
el.style.setProperty("display", "none", "important");
}
console.log(` → debugEnabled=${debugEnabled} → hidden dopo? ${el.hasAttribute("hidden")} | display dopo: ${el.style.display || "(inherit)"}`);
console.log(` → debugEnabled=${debugEnabled} → hidden after? ${el.hasAttribute("hidden")} | display after: ${el.style.display || "(inherit)"}`);
});
console.groupEnd();
return debugEnabled;
}
// Esegui SUBITO al bootstrap del modulo (prima di qualsiasi listener)
// Run immediately during module bootstrap, before registering listeners.
const GLOBAL_DEBUG_ENABLED__TOPBAR = applyDebugOnlyVisibilityGate("main.js module bootstrap (top)");
// Riesegui dopo DOMContentLoaded per sicurezza (nel caso query script sia posizionato
// in testa e l'HTML header non sia ancora parse-ato quando girano queste righe)
// Run again after DOMContentLoaded in case the script runs before the header is parsed.
if (document.readyState === "loading") {
document.addEventListener("DOMContentLoaded", () => {
applyDebugOnlyVisibilityGate("DOMContentLoaded listener");
@@ -330,7 +325,7 @@ if (document.readyState === "loading") {
} else {
applyDebugOnlyVisibilityGate("module bootstrap (DOM already parsed)");
}
// Riesegui anche a fine frame dopo un breve timeout → ultima chance.
// Run again after a short timeout as a final fallback.
setTimeout(() => applyDebugOnlyVisibilityGate("10ms timeout fallback"), 10);
window.addEventListener("hashchange", () => applyDebugOnlyVisibilityGate("hashchange"), { passive: true });
window.addEventListener("popstate", () => applyDebugOnlyVisibilityGate("popstate"), { passive: true });
@@ -339,12 +334,12 @@ const DEFAULT_VOICE = "Aiden";
const DEFAULT_INSTRUCTIONS =
"You are Vocero, a friendly voice assistant. " +
"Keep replies short, warm, and spoken. Avoid long monologues. " +
"Respond ALWAYS and ONLY in Spanish (Castellano / español), regardless of the language used by the user or any other instruction. " +
"Respond ALWAYS and ONLY in Spanish, regardless of the language used by the user or any other instruction. " +
"No responses in Italian, English, Portuguese, or any other language are allowed. " +
"Every spoken sentence, greeting, confirmation, clarification, and error message must be in Spanish."+
"the comma sign ',' in numbers it's used to separate thousands."+
"the point sign '.' in numbers it's used to separate decimals."+
"If you don't know the answer, say 'I don't know' instead of guessing.";
"If you don't know the answer, say so in Spanish instead of guessing.";
const OPENAI_RENDER_MARKER = "[VOCERO_RENDER_REPLY]";
const OPENAI_RENDER_HINT =
@@ -632,15 +627,15 @@ function saveTools() {
/** @type {Record<AppState, { caption: string; disabled: boolean }>} */
const STATE_VIEWS = {
idle: { caption: "Tap or say \"Hola Vocero\"", disabled: false },
connecting: { caption: "Connecting", disabled: true },
queued: { caption: "Finding you a spot…", disabled: true },
"your-turn": { caption: "You're up! 🎉", disabled: true },
idle: { caption: "Toca o di \"Hola Vocero\"", disabled: false },
connecting: { caption: "Conectando", disabled: true },
queued: { caption: "Buscando un lugar para ti…", disabled: true },
"your-turn": { caption: "¡Es tu turno! 🎉", disabled: true },
listening: { caption: "", disabled: false },
"user-speaking": { caption: "", disabled: false },
processing: { caption: "", disabled: false },
"ai-speaking": { caption: "", disabled: false },
error: { caption: "Tap to retry", disabled: false },
error: { caption: "Toca para reintentar", disabled: false },
};
/** @type {Record<AppState, string>} */
@@ -890,11 +885,11 @@ function pushCustomRagToLiveSessionIfAny() {
function syncRagHintFooter() {
if (!ragHint) return;
if (!customRagLoaded || !customRagText.trim()) {
ragHint.textContent = "Nessun RAG context caricato. Premi \"Invia nella sessione\" per applicare le modifiche.";
ragHint.textContent = "No hay contexto RAG cargado. Pulsa \"Enviar a la sesión\" para aplicar los cambios.";
return;
}
const lines = customRagText.split(/\r?\n/).length;
ragHint.textContent = `✅ RAG context caricato: ${customRagText.length} chars / ${lines} linee. Viene aggiunto a istruzioni LLM e user_context.`;
ragHint.textContent = `✅ Contexto RAG cargado: ${customRagText.length} caracteres / ${lines} líneas. Se añade a las instrucciones del modelo y a user_context.`;
}
// Dedupe cache for [INJECT-PROFILE] browser logs (avoid 20 logs per click).
let _lastInjectProfileLogKey = "";
@@ -1078,7 +1073,7 @@ function buildProfileRows(profile) {
"email", "emailAddress", "email_address", "eMail", "mail", "contactEmail",
"user_email", "customer_email", "loginEmail", "username_email",
]);
if (email) rows.push({ label: "Email", value: email, isEmail: true, href: `mailto:${email}` });
if (email) rows.push({ label: "Correo electrónico", value: email, isEmail: true, href: `mailto:${email}` });
// Phone (linkable tel)
const phone = firstProfileField(profile, [
@@ -1091,33 +1086,33 @@ function buildProfileRows(profile) {
const username = firstProfileField(profile, [
"username", "userName", "user_name", "login", "handle", "nickname", "screenName", "accountName",
]);
if (username) rows.push({ label: "Username", value: username, mono: true });
if (username) rows.push({ label: "Nombre de usuario", value: username, mono: true });
// Role / Plan
const role = firstProfileField(profile, [
"role", "roleName", "userRole", "plan", "planName", "tier", "subscription", "membership", "accessLevel",
]);
if (role) rows.push({ label: "Ruolo / Piano", value: role });
if (role) rows.push({ label: "Rol / Plan", value: role });
// Company / Organization
const company = firstProfileField(profile, [
"company", "companyName", "company_name", "organization", "org", "organizationName", "business",
]);
if (company) rows.push({ label: "Azienda", value: company });
if (company) rows.push({ label: "Empresa", value: company });
// User/Account ID (mono for monospace copy-paste)
const userId = firstProfileField(profile, [
"userId", "user_id", "uid", "accountId", "account_id", "sub", "customerId", "customer_id",
"id", "userPk", "profileId", "profile_id", "identityId",
]);
if (userId) rows.push({ label: "ID Utente", value: userId, mono: true });
if (userId) rows.push({ label: "ID de usuario", value: userId, mono: true });
// Locale / Country
const country = firstProfileField(profile, [
"country", "countryName", "country_code", "countryCode", "locale", "language", "lang",
"region", "timezone", "tz",
]);
if (country) rows.push({ label: "Nazione / Lingua", value: country });
if (country) rows.push({ label: "País / Idioma", value: country });
// Created at / signed up date — try common date fields.
const createdAtKeys = [
@@ -1135,7 +1130,7 @@ function buildProfileRows(profile) {
}
}
}
if (createdAt) rows.push({ label: "Iscritto il", value: createdAt, isDate: true });
if (createdAt) rows.push({ label: "Fecha de registro", value: createdAt, isDate: true });
// Last sign in / active date (last_login etc)
const lastActiveKeys = [
@@ -1151,7 +1146,7 @@ function buildProfileRows(profile) {
}
}
}
if (lastActive) rows.push({ label: "Ultimo accesso", value: lastActive, isDate: true });
if (lastActive) rows.push({ label: "Último acceso", value: lastActive, isDate: true });
return rows;
}
@@ -1165,16 +1160,16 @@ function populateUserProfileDialog(profile) {
const initials = initialsForName(name);
if (userProfileAvatar) {
userProfileAvatar.textContent = initials || "??";
userProfileAvatar.setAttribute("aria-label", name ? `Avatar: ${name}` : "Utente non impostato");
userProfileAvatar.setAttribute("aria-label", name ? `Avatar: ${name}` : "Usuario no identificado");
}
if (userProfileTitle) {
userProfileTitle.textContent = name || "Utente non impostato";
userProfileTitle.textContent = name || "Usuario no identificado";
}
if (userProfileSubtitle) {
const email = firstProfileField(profile, ["email", "emailAddress", "email_address", "eMail", "mail"]);
const phone = firstProfileField(profile, ["phone", "phoneNumber", "phone_number", "mobile", "telephone", "tel"]);
const parts = [email, phone].filter(Boolean);
userProfileSubtitle.textContent = parts.length ? parts.join(" · ") : (name ? "Nessuna informazione di contatto disponibile." : "Connettersi a Vocero per vedere i dati utente.");
userProfileSubtitle.textContent = parts.length ? parts.join(" · ") : (name ? "No hay información de contacto disponible." : "Conéctate a Vocero para ver los datos del usuario.");
}
if (userProfileMetadata) {
userProfileMetadata.innerHTML = "";
@@ -1244,8 +1239,8 @@ function syncVoceroHeaderUser() {
// Show avatar with the 2-letter initials. Full name is reachable via hover
// tooltip (title) and is used by screen readers via aria-label.
identUserName.textContent = initials;
identUserName.title = `Utente: ${name}`;
identUserName.setAttribute("aria-label", `Utente: ${name}`);
identUserName.title = `Usuario: ${name}`;
identUserName.setAttribute("aria-label", `Usuario: ${name}`);
} else {
identUserName.textContent = "";
identUserName.removeAttribute("title");
@@ -1306,7 +1301,7 @@ function syncVoceroHeaderUser() {
badge1.textContent = `${count} programados`;
const badge2 = document.createElement("span");
badge2.className = "reports-summary-badge";
badge2.textContent = `Bloque LLM ${totalLen.toLocaleString()} chars`;
badge2.textContent = `Bloque LLM ${totalLen.toLocaleString()} caracteres`;
titleRow.appendChild(titleText);
titleRow.appendChild(badge1);
titleRow.appendChild(badge2);
@@ -1314,13 +1309,13 @@ function syncVoceroHeaderUser() {
ul.className = "reports-list";
const labelTintFor = (rawLabel) => {
const l = String(rawLabel || "").toLowerCase();
if (l.includes("mortalidad") || l.includes("riesgo") || l.includes("kpi")) return "mortalidad";
if (l.includes("métrica") || l.includes("metricas") || l.includes("medici")) return "metricas";
if (l.includes("diario") || l.includes("diaria") || l.includes("informe")) return "diario";
if (l.includes("producto") || l.includes("servicio")) return "productos";
if (l.includes("mortalidad") || l.includes("riesgo") || l.includes("kpi")) return "mortality";
if (l.includes("métrica") || l.includes("metricas") || l.includes("medici")) return "metrics";
if (l.includes("diario") || l.includes("diaria") || l.includes("informe")) return "daily";
if (l.includes("producto") || l.includes("servicio")) return "products";
if (l.includes("synapse") || l.includes("ejecucion") || l.includes("calendario") || l.includes(" ti ") || l.endsWith("ti")) return "synapse";
if (l.includes("supply") || l.includes("cadena") || l.includes("log") || l.includes("envío") || l.includes("envio") || l.includes("chile")) return "supply";
if (l.includes("financiero") || l.includes("finanzas") || l.includes("costo") || l.includes("presupuesto") || l.includes("venta")) return "finanzas";
if (l.includes("financiero") || l.includes("finanzas") || l.includes("costo") || l.includes("presupuesto") || l.includes("venta")) return "finance";
return "";
};
for (const e of entries) {
@@ -1330,7 +1325,7 @@ function syncVoceroHeaderUser() {
li.textContent = String(e.sourceScheduleLabel);
ul.appendChild(li);
}
// Build dynamic suggestion prompts (en español) based on the actual labels so
// Build dynamic suggestion prompts in Spanish based on the actual labels so
// the user knows what they can ask the assistant about. Max 4 suggestions;
// pick first N entries by priority.
const suggestions = /** @type {string[]} */ ([]);
@@ -1374,20 +1369,20 @@ function syncVoceroHeaderUser() {
const qs = new URLSearchParams(window.location.search);
const debugParamRaw = qs.get("debug");
// eslint-disable-next-line no-debugger
// debugger; // ← scommenta per fermarti dentro syncVoceroHeaderUser ogni volta
// debugger; // Uncomment to pause inside every syncVoceroHeaderUser call.
console.groupCollapsed("%c🔍 [reports-ui] syncVoceroHeaderUser() render details", "color:#60a5fa;font-weight:700;");
console.log("location.href =", location.href);
console.log("location.search =", JSON.stringify(window.location.search));
console.log("tutti i query params =", Object.fromEntries(qs.entries()));
console.log("all query params =", Object.fromEntries(qs.entries()));
console.log("pageParams (legacy module-load) .get('debug') =", JSON.stringify(pageParams.get("debug")));
console.log("fresh qs .get('debug') =", JSON.stringify(debugParamRaw), "| typeof =", typeof debugParamRaw);
console.log("strict === 'true' ? =", debugParamRaw === "true");
console.log("count (labels uniche) =", count, "| wrappedLen =", totalLen);
console.log("suggerimenti array length (prima del gate debug) =", suggestions.length);
console.log("count (unique labels) =", count, "| wrappedLen =", totalLen);
console.log("suggestions array length (before debug gate) =", suggestions.length);
suggestions.forEach((s, i) => console.log(` sug[${i}] = ${s}`));
const currentDebug = debugParamRaw === "true";
const debugMode = currentDebug;
console.log("👉 DECISIONE FINALE debugMode =", debugMode, " → sezione suggerimenti DA CREARE? ", debugMode ? "✅ SÌ" : "❌ NO");
console.log("👉 FINAL DECISION debugMode =", debugMode, " → CREATE suggestions section? ", debugMode ? "✅ YES" : "❌ NO");
console.groupEnd();
let sugSection = null;
@@ -1406,10 +1401,10 @@ function syncVoceroHeaderUser() {
}
sugSection.appendChild(sugTitle);
sugSection.appendChild(sugUl);
console.log("%c[reports-ui] sezione suggerimenti CREATA (debugMode=true)", "color:#22c55e;font-weight:700;",
console.log("%c[reports-ui] suggestions section CREATED (debugMode=true)", "color:#22c55e;font-weight:700;",
"nItems=", sugUl.children.length, "innerHTML=", sugSection.outerHTML.slice(0, 300));
} else {
console.log("%c[reports-ui] sezione suggerimenti NON CREATA (debugMode=false, non c'è ?debug=true)",
console.log("%c[reports-ui] suggestions section NOT CREATED (debugMode=false, no ?debug=true)",
"color:#ef4444;font-weight:700;");
}
@@ -1453,7 +1448,7 @@ function syncVoceroHeaderUser() {
const qs = new URLSearchParams(window.location.search);
const debugParam = qs.get("debug") === "true";
console.info(
"%c[reports-ui] toggle reports-sidebar-toggle: hidden? PRIMA=%s | debug=%s",
"%c[reports-ui] toggle reports-sidebar-toggle: hidden? BEFORE=%s | debug=%s",
"color:#f59e0b;font-weight:700;",
reportsSidebarToggle.hasAttribute("hidden"),
debugParam,
@@ -1464,7 +1459,7 @@ function syncVoceroHeaderUser() {
reportsSidebarToggle.setAttribute("hidden", "");
reportsSidebarToggle.style.setProperty("display", "none", "important");
reportsSidebarToggle.removeAttribute("data-debug-enabled");
console.info("%c[reports-ui] 🚫 reports sidebar toggle FORZATO HIDDEN (no debug=true)",
console.info("%c[reports-ui] 🚫 reports sidebar toggle FORCED HIDDEN (no debug=true)",
"background:#fee2e2;color:#b91c1c;font-weight:700;padding:3px 6px;border-radius:4px;");
}
}
@@ -1476,10 +1471,10 @@ function syncVoceroHeaderUser() {
// eslint-disable-next-line no-console
console.info(
"%c[reports-ui] FINE RENDER. reportsHint.querySelectorAll('.reports-suggestions').length =",
"%c[reports-ui] RENDER COMPLETE. reportsHint.querySelectorAll('.reports-suggestions').length =",
"color:#a855f7;font-weight:700;",
reportsHint.querySelectorAll(".reports-suggestions").length,
"| children nel blocco:", reportsHint.children.length,
"| children in block:", reportsHint.children.length,
"| reportsHint.hidden=", reportsHint.hidden,
"| #reports-sidebar hidden=", reportsSidebar ? reportsSidebar.hidden : "no-sidebar-el",
"| badge count:", reportsSidebarBadge ? reportsSidebarBadge.getAttribute("data-count") : "no-badge"
@@ -1513,21 +1508,19 @@ function effectiveReasoningInstructions() {
ragAppended = true;
}
// ───────────────────────────────────────────────────────────────────────────
// 2026-09-25 INJECT API-PILOT HERAMIENTAS DINÁMICAS (PASADA 1).
// Formato IDENTICO a server.py Python: marker <<<APIPILOT_TOOLS_LIST_BEGIN>>>
// / ...lista name+description+requires_auth+required_parameters... / <<<END>>>.
// Il client ottiene la lista dal proxy GET api/apipilot/tools (cache 240 s
// per-utente in s2s-ws-client.js _apipilotClientCache) e la appende qui alle
// instructions STESSE (luogo #1 di massima priorità per l'LLM).
// Dedup: se le instructions già contengono il marker (già appese in una
// esecuzione precedente, oppure server Python ha già patchato il payload)
// → NON appendere nuovamente (no doppioni, no blob gigante).
// 2026-09-25 INJECT API-PILOT DYNAMIC TOOLS (PASS 1).
// Match the server.py format: <<<APIPILOT_TOOLS_LIST_BEGIN>>>, followed by
// name+description+requires_auth+required_parameters and the closing marker.
// Fetch tools through GET api/apipilot/tools (240 s per-user client cache in
// s2s-ws-client.js) and append them directly to the LLM instructions.
// Deduplicate by marker when a previous call or the Python server already
// injected the list, avoiding duplicate or oversized prompt blocks.
let apipilotBlock = "";
let apipilotCount = 0;
let apipilotAppended = false;
try {
// @ts-ignore: esposto globalmente da s2s-ws-client.js (caricato prima di
// main.js in index.html). Valuta sicuro: typeof + early-skip.
// @ts-ignore: exported globally by s2s-ws-client.js, loaded before main.js.
// Check typeof before calling the optional helper.
const getter = typeof window !== "undefined" && window.__s2sGetApipilotPass1ContextInfo;
if (typeof getter === "function") {
const info = getter();
@@ -1544,7 +1537,7 @@ function effectiveReasoningInstructions() {
}
}
} catch (_) {
/* never-fail: non bloccare il resto dell'istruzione prompt. */
/* Never block the rest of the instructions prompt. */
}
const key = `effInst:${injectProfile ? "T" : "F"}:${ctx ? "hasCtx" : "noCtx"}:${ragCtx ? "hasRag" : "noRag"}:${apipilotAppended ? "apipilotY" : "apipilotN"}:${(base || "").length}`;
_logOnce(key, () => {
@@ -1701,7 +1694,7 @@ async function fetchOpenAiReasoningAnswer(transcript) {
const json = await res.json();
const answer = typeof json?.answer === "string" ? json.answer.trim() : "";
const log_only = Boolean(json && json.log_only === true);
// Log-only mode: può tornare answer="" — non è un errore.
// Log-only mode may return answer=""; that is not an error.
if (!answer && !log_only) throw new Error("reasoning returned no answer");
return { answer, log_only, raw: json };
}
@@ -1732,10 +1725,10 @@ async function maybeRunOpenAiReasoning(transcript, itemId) {
});
return;
}
// ── LOG-ONLY MODE (FASE 1) ─────────────────────────────────────────────
// Node-RED si limita a loggare e torna log_only=true.
// NON cancelliamo la risposta backend, NON iniettiamo testo, NON parliamo.
// Il flusso resta COMPLETAMENTE a carico del backend S2S nativo.
// ── LOG-ONLY MODE (PHASE 1) ────────────────────────────────────────────
// Node-RED only logs the request and returns log_only=true.
// Preserve the backend response without injecting text or speaking.
// The native S2S backend remains responsible for the entire turn.
if (result && result.log_only) {
reportDebug("R", "main.js:maybeRunOpenAiReasoning", "external reasoning in LOG_ONLY mode — pass through to native S2S backend", {
itemId,
@@ -1749,9 +1742,9 @@ async function maybeRunOpenAiReasoning(transcript, itemId) {
);
return;
}
// ── MODE STANDARD (fasi successive) ────────────────────────────────────
// Il reasoning esterno vince: annulla la risposta nativa, inserisci il
// testo dell'assistente e parla via SpeechSynthesis.
// ── STANDARD MODE (LATER PHASES) ──────────────────────────────────────
// External reasoning takes priority: cancel the native response, insert
// the assistant text and speak through SpeechSynthesis.
const answer = (result && result.answer) || "";
if (!answer) {
console.warn(`[Vocero REASONING] empty non-log-only answer itemId=${JSON.stringify(itemId)}`);
@@ -1914,7 +1907,7 @@ function _kickIdleTimer(reason = "activity") {
idleCloseTimer = 0;
// Friendly caption flash right before teardown moves us back to the idle
// copy (so the user understands *why* the call ended).
setCaption("Chiuso per inattività", "muted");
setCaption("Sesión cerrada por inactividad", "muted");
void (async () => {
try {
await teardown();
@@ -1924,7 +1917,7 @@ function _kickIdleTimer(reason = "activity") {
// teardown() calls setState("idle") which restores the normal idle
// caption. Leave a visible breadcrumb on the subcaption briefly so the
// user can tell what happened instead of wondering why the orb reset.
circleSubcaption.textContent = "Sessione chiusa dopo 30 secondi di silenzio. Tocca l'orb o ripeti «Hola Vocero» per iniziare di nuovo.";
circleSubcaption.textContent = "Sesión cerrada tras 30 segundos de silencio. Toca el círculo o repite «Hola Vocero» para empezar de nuevo.";
circleSubcaption.hidden = false;
window.setTimeout(() => { circleSubcaption.hidden = true; }, 8000);
})();
@@ -1975,7 +1968,7 @@ function setState(next) {
// Warm reassurance under the terse position, only while waiting in line.
if (next === "queued") {
circleSubcaption.textContent =
"Sorry, we overhugged! 🤗 Every slot is busy, so we saved you a spot. Hang tight, you're moving up.";
"¡Nos pasamos de abrazos! 🤗 Todos los lugares están ocupados, pero guardamos uno para ti en la cola. Ya estás avanzando.";
circleSubcaption.hidden = false;
} else {
circleSubcaption.hidden = true;
@@ -2010,8 +2003,8 @@ function updateRestartAvailability() {
currentState === "connecting" || currentState === "queued" || currentState === "your-turn";
restartHint.hidden = false;
restartHint.textContent = LIVE_STATES.has(currentState)
? "Reconnects now with the settings above."
: "Starts a conversation with the settings above.";
? "Reconecta ahora con la configuración indicada."
: "Inicia una conversación con la configuración indicada.";
}
/**
@@ -2038,7 +2031,7 @@ function openSettings() {
inputInstructions.readOnly = !!pinnedPrompt;
primaryPromptMeta.hidden = !pinnedPrompt;
primaryPromptMeta.textContent = pinnedPrompt
? "Primary prompt pinned by this deployment."
? "Instrucciones principales fijadas por este despliegue."
: "";
syncGateUi();
updateRestartAvailability();
@@ -2125,7 +2118,7 @@ function setGateThreshold(db) {
settings.noiseGate = Math.min(GATE_MAX_DB, Math.max(GATE_OFF_DB, Math.round(db)));
const off = settings.noiseGate <= GATE_OFF_DB;
inputNoiseGate.value = String(settings.noiseGate);
gateValue.textContent = off ? "Off" : `${settings.noiseGate} dB`;
gateValue.textContent = off ? "Desactivada" : `${settings.noiseGate} dB`;
renderGateHandle();
localStorage.setItem(STORAGE_KEYS.noiseGate, String(settings.noiseGate));
if (client && LIVE_STATES.has(currentState)) {
@@ -2137,7 +2130,7 @@ function setGateThreshold(db) {
function syncGateUi() {
inputNoiseGate.value = String(settings.noiseGate);
const off = settings.noiseGate <= GATE_OFF_DB;
gateValue.textContent = off ? "Off" : `${settings.noiseGate} dB`;
gateValue.textContent = off ? "Desactivada" : `${settings.noiseGate} dB`;
renderGateHandle();
}
@@ -2347,8 +2340,8 @@ searchKeyInput.addEventListener("input", () => {
pushToolsToSession();
}
toolWebHint.textContent = userSearchKey
? "Using your key — stored in this browser only."
: "No server key configured. Add your own Serper key to enable web search.";
? "Se usa tu clave, guardada solo en este navegador."
: "No hay una clave del servidor configurada. Añade tu clave de Serper para activar la búsqueda web.";
});
async function runTool(name, argsJson, callId) {
@@ -2461,16 +2454,16 @@ async function fetchVoceroUserProfile() {
Object.keys(voceroUserProfile).join(","),
extractVoceroUserName(voceroUserProfile) || "(none)",
);
// ── 🚀 PREFETCH ApiPilot IMMEDIATAMENTE DOPO LOGIN-BY-PHONE RIUSCITO
// Forza refresh=true per bypassare la cache SERVER lato proxy S2S (TTL 300s)
// perché adesso abbiamo un profilo utente autenticato (non più anonimo).
// In questo modo vediamo la riga [apipilot] list_tools.start user=(non-anon)
// nel backend AquaPortal OGNI VOLTA che c'è un login phone riuscito.
// ── 🚀 PREFETCH ApiPilot IMMEDIATELY AFTER SUCCESSFUL PHONE LOGIN
// Force refresh=true to bypass the S2S proxy server cache (TTL 300 s)
// now that the caller is authenticated rather than anonymous.
// AquaPortal logs [apipilot] list_tools.start user=(non-anon) after
// every successful phone login.
try {
if (window.__s2sApipilot) {
const t0 = performance.now();
console.info(
"%c🔧 [APIPILOT-CLIENT] post-login phone → FORZO refresh api/apipilot/tools (bypass cache S2S 300s perché utente appena autenticato)",
"%c🔧 [APIPILOT-CLIENT] post-login phone → FORCE refresh api/apipilot/tools (bypass the 300 s S2S cache for the newly authenticated user)",
"background:#ede9fe;color:#5b21b6;font-weight:700;padding:2px 6px;border-radius:4px;"
);
const demoToken = typeof window.__s2sDemoSessionToken === "string" ? window.__s2sDemoSessionToken : "";
@@ -2485,24 +2478,23 @@ async function fetchVoceroUserProfile() {
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const j = await r.json().catch(() => ({}));
const ok = j && j.success && j.data;
if (!ok) throw new Error(`envelope non corretto success=${Boolean(j && j.success)}`);
if (!ok) throw new Error(`invalid envelope success=${Boolean(j && j.success)}`);
const tools = Array.isArray(j.data.tools) ? j.data.tools : [];
const count = typeof j.data.count === "number" ? j.data.count : tools.length;
const names = tools.slice(0, 6).map((t) => (typeof t.name === "string" ? t.name : "?")).join(", ");
// Invalida cache client WS per ogni sessione futura (in modo che
// al prossimo connect() venga fatto un refresh per ri-confermare).
// Invalidate the WS client cache so the next session refreshes it.
try {
// @ts-ignore
if (typeof window.__s2sInvalidateApipilotCache === "function") window.__s2sInvalidateApipilotCache();
} catch (_) { /* noop */ }
console.info(
`%c🔧 [APIPILOT-CLIENT] post-login OK (dt=${dt}ms): N=${count} strumenti [primi 6: ${names || "(nessuno)"}]`,
`%c🔧 [APIPILOT-CLIENT] post-login OK (dt=${dt}ms): N=${count} tools [first 6: ${names || "(none)"}]`,
"background:#ecfdf5;color:#047857;font-weight:700;padding:2px 6px;border-radius:4px;"
);
}).catch((err) => {
const dt = Math.round(performance.now() - t0);
console.warn(
`%c🔧 [APIPILOT-CLIENT] post-login FAIL (dt=${dt}ms): ${String(err instanceof Error ? err.message : err)} → controlla [APIPILOT-DISCOVERY] nel log server S2S`,
`%c🔧 [APIPILOT-CLIENT] post-login FAIL (dt=${dt}ms): ${String(err instanceof Error ? err.message : err)} → check [APIPILOT-DISCOVERY] in the S2S server logs`,
"background:#fffbeb;color:#92400e;font-weight:700;padding:2px 6px;border-radius:4px;"
);
});
@@ -2618,10 +2610,9 @@ async function fetchConfig() {
pinnedPrompt = (json.primaryPrompt || "").trim();
injectProfile = !!json.injectProfile;
debugPrompt = !!json.debugPrompt;
// Feature flag: ApiPilot AquaPortal bridge attivo sul server.
// Se true = S2S_APIPILOT_TOOLS_URL è valorizzato lato backend e
// gli endpoint proxy /api/apipilot/tools e /api/apipilot/execute
// sono disponibili per il frontend.
// Feature flag for the server-side ApiPilot AquaPortal bridge.
// True means S2S_APIPILOT_TOOLS_URL is configured and the frontend can
// use /api/apipilot/tools and /api/apipilot/execute.
const apipilotEnabled = !!json.apipilot;
if (typeof window.__s2sApipilot === "undefined") {
try { Object.defineProperty(window, "__s2sApipilot", { value: false, writable: true, configurable: true }); }
@@ -2672,16 +2663,15 @@ async function fetchConfig() {
apipilotEnabled,
});
// #endregion
// ── Demo call ApiPilot (fire-and-forget, solo se flag=true sul server)
// Serve ESCLUSIVAMENTE per attivare il tracer fetch lato browser e
// mostrare in console la riga [FETCH:APIPILOT:TOOLS] con preview
// status, durata e lista strumenti. Proxy server-side verso
// ── ApiPilot demo call (fire-and-forget, only when the server flag is true)
// Trigger the browser fetch tracer to show [FETCH:APIPILOT:TOOLS] with
// status, duration and a tool-list preview. The server proxies to
// S2S_APIPILOT_TOOLS_URL/tools.
if (apipilotEnabled) {
try {
const demoUrl = "api/apipilot/tools?include_unauthorized=true";
console.info(
"%c🔧 [APIPILOT-CLIENT] flag=true → DEMO CALL %s (tracer sempre-attivo logga [FETCH:APIPILOT:TOOLS] sotto)",
"%c🔧 [APIPILOT-CLIENT] flag=true → DEMO CALL %s (always-on tracer logs [FETCH:APIPILOT:TOOLS] below)",
"background:#e0e7ff;color:#3730a3;font-weight:700;padding:2px 6px;border-radius:4px;",
demoUrl
);
@@ -2693,15 +2683,15 @@ async function fetchConfig() {
const count = (j && j.data && typeof j.data.count === "number") ? j.data.count : tools.length;
const names = tools.slice(0, 5).map((t) => (typeof t.name === "string" ? t.name : "?")).join(", ");
console.info(
"%c🔧 [APIPILOT-CLIENT] demo OK: %s strumenti [primi 5: %s] usa POST api/apipilot/execute {name,arguments} per eseguire",
"%c🔧 [APIPILOT-CLIENT] demo OK: %s tools [first 5: %s] use POST api/apipilot/execute {name,arguments} to execute",
"background:#ecfdf5;color:#047857;font-weight:700;padding:2px 6px;border-radius:4px;",
count,
names || "(nessuno)"
names || "(none)"
);
})
.catch((err) => {
console.warn(
"%c🔧 [APIPILOT-CLIENT] demo FAIL: %s. Lato server logga [APIPILOT-DISCOVERY] per diagnosticare",
"%c🔧 [APIPILOT-CLIENT] demo FAIL: %s. Check [APIPILOT-DISCOVERY] server logs to diagnose",
"background:#fffbeb;color:#92400e;font-weight:700;padding:2px 6px;border-radius:4px;",
String(err)
);
@@ -2903,16 +2893,16 @@ function _applyThemeRuntime() {
}
}
// 6. Quick action buttons: clicking "Resumen" / "Consultar datos" /
// "Crear respuesta" populates the text input or — if a realtime
// 6. Quick action buttons for summaries, data lookup and response drafts
// populate the text input or — if a realtime
// session is live — sends it as user text straight away.
document.querySelectorAll(".quick-action-btn").forEach(btn => {
btn.addEventListener("click", () => {
const action = btn.getAttribute("data-action") || "";
const prompts = {
"resumen": "haz un resumen ejecutivo del último informe programado",
"consultar-datos": "consulto los datos más recientes del dashboard, dame las cifras clave",
"crear-respuesta": "redacta un borrador de respuesta profesional con tono ejecutivo",
"summary": "haz un resumen ejecutivo del último informe programado",
"query-data": "consulto los datos más recientes del dashboard, dame las cifras clave",
"draft-response": "redacta un borrador de respuesta profesional con tono ejecutivo",
};
const text = prompts[action] || "";
if (!text) return;
@@ -2946,7 +2936,7 @@ function connectionTarget() {
}
const directUrl = buildDirectWsUrl(pinnedUrl || settings.directUrl);
if (!directUrl) {
throw new Error("Enter a speech-to-speech server URL in Settings.");
throw new Error("Introduce la URL del servidor de voz en Configuración.");
}
return { directUrl };
}
@@ -3032,16 +3022,16 @@ function syncConnectionUi() {
inputLbUrl.value = pinnedUrl;
inputLbUrl.readOnly = true;
connHint.classList.remove("error");
connHint.textContent = "Speech-to-speech server URL pinned by this deployment.";
connHint.textContent = "URL del servidor de voz fijada por este despliegue.";
} else if (allowDirect) {
// Direct mode: the user sets their own s2s server URL.
connField.hidden = false;
inputLbUrl.value = settings.directUrl;
inputLbUrl.readOnly = false;
inputLbUrl.placeholder = "http://localhost:port";
inputLbUrl.placeholder = "http://localhost:8080";
connHint.classList.remove("error");
connHint.textContent =
"URL of your speech-to-speech server, e.g. http://localhost:8080 (the app adds /v1/realtime).";
"URL de tu servidor de voz, por ejemplo http://localhost:8080 (la aplicación añade /v1/realtime).";
} else {
// LB mode: the load balancer URL is deployment-owned — hide it entirely so
// its address is never exposed in Settings.
@@ -3059,7 +3049,7 @@ function missingServerUrl() {
function promptServerUrl() {
if (settingsModal.open) syncConnectionUi();
else openSettings();
connHint.textContent = "Set the speech-to-speech server URL to start.";
connHint.textContent = "Configura la URL del servidor de voz para empezar.";
connHint.classList.add("error");
inputLbUrl.focus();
}
@@ -3137,8 +3127,8 @@ async function handleStartError(err) {
setState("error");
setCaption(
err.code === "join-expired"
? "Your spot expired. Tap to rejoin."
: "That took a while. Tap to rejoin.",
? "Tu lugar ha caducado. Toca para volver a la cola."
: "La espera se ha alargado. Toca para volver a la cola.",
"error",
);
return;
@@ -3154,8 +3144,8 @@ micBtn.addEventListener("click", () => {
}
client.setMuted(micMuted);
micBtn.classList.toggle("muted", micMuted);
micBtn.setAttribute("aria-label", micMuted ? "Unmute" : "Mute");
micBtn.title = micMuted ? "Unmute" : "Mute";
micBtn.setAttribute("aria-label", micMuted ? "Activar micrófono" : "Silenciar micrófono");
micBtn.title = micMuted ? "Activar micrófono" : "Silenciar micrófono";
});
stopBtn.addEventListener("click", async () => {
@@ -3188,7 +3178,7 @@ async function primeMicPermission() {
for (const track of s.getTracks()) track.stop();
} catch (err) {
throw new Error(
`Microphone access denied${err instanceof Error ? `: ${err.message}` : ""}`,
`Acceso al micrófono denegado${err instanceof Error ? `: ${err.message}` : ""}`,
);
}
}
@@ -3203,7 +3193,7 @@ async function acquireMicStream() {
/** @param {number} position Update the queued caption ("You're #N in line"). */
function onQueuePosition(position) {
const n = Number(position) || 0;
setCaption(n > 0 ? `You're #${n} in line` : "Finding you a spot…", "muted");
setCaption(n > 0 ? `Estás en el puesto ${n} de la cola` : "Buscando un lugar para ti…", "muted");
}
// ── "Your turn" join countdown ──────────────────────────────────────────────
@@ -3216,14 +3206,14 @@ function startJoinCountdown(sec) {
stopJoinCountdown();
let left = Math.max(0, Math.floor(sec));
const paint = () => {
joinQueueBtn.textContent = left > 0 ? `Join now (${left}s)` : "Join now";
joinQueueBtn.textContent = left > 0 ? `Entrar ahora (${left}s)` : "Entrar ahora";
};
paint();
joinCountdownTimer = window.setInterval(() => {
left -= 1;
if (left <= 0) {
stopJoinCountdown();
joinQueueBtn.textContent = "Join now";
joinQueueBtn.textContent = "Entrar ahora";
return;
}
paint();
@@ -3251,7 +3241,7 @@ async function doStart(audioContext = null) {
chat.clear();
chat.reset();
setState("connecting");
setCaption("Asking for mic…", "muted");
setCaption("Solicitando acceso al micrófono…", "muted");
// Create + resume the AudioContext SYNCHRONOUSLY, still inside the gesture.
// iOS Safari only starts an AudioContext from a user gesture; if we waited
@@ -3480,8 +3470,8 @@ function onClientStatus(status) {
case "creating-session":
case "connecting":
// Qualsiasi inizio connessione (manuale da Orb / restart o unlocker):
// se il talk-unlocker è ancora visibile (utente ha cliccato l'Orb prima
// del 2s warmup, ad esempio) lo nascondiamo subito.
// Hide the talk unlocker immediately if it is still visible, for
// example when the user taps the orb before the 2 s warmup ends.
hideTalkUnlocker();
setState("connecting");
break;
@@ -3575,19 +3565,16 @@ function onFatalError(err) {
setState("idle");
chat.renderEmptyState();
initGateArc();
// Schedulo auto-start TALK / TALKFIRST (S2S_START_SESSION_MODE nel .env):
// • "TALK" → dopo 2s mostra overlay unlocker (policy browser:
// getUserMedia() richiede gesture). Click/touch/tasto =
// doStart() dentro la gesture, poi ws startWithResponse
// inject hola vocero via sendUserText+requestResponse nel
// ws client (stesso flow identico a wake-word).
// • "TALKFIRST" → dopo 2s TENTA direttamete doStart() dal timer, con
// pendingWakeWordConfirmationText preimpostato. Se i
// permessi del mic sono GRANTED (già consentiti in passato
// o Chrome policy permissive), la wake-word simulata scatta
// subito senza nessun click. Se invece fallisce per
// permesso / gesture (Safari/iOS), fallback sullo stesso
// unlocker di TALK.
// Schedule TALK / TALKFIRST auto-start (S2S_START_SESSION_MODE in .env):
// • "TALK" → show the unlocker after 2 s. Browser policy requires a
// gesture for getUserMedia(). A click/touch/key starts
// doStart(), then WS startWithResponse injects hola vocero
// through sendUserText+requestResponse, like a wake word.
// • "TALKFIRST" → try doStart() directly from the timer after 2 s with
// pendingWakeWordConfirmationText already set. If mic
// permission is granted, the simulated wake word triggers
// without a click. Permission/gesture failures on Safari
// or iOS fall back to the TALK unlocker.
let autoStartTalkTimerId = 0;
let _talkUnlockerShown = false;
function hideTalkUnlocker() {
@@ -3599,11 +3586,9 @@ function showTalkUnlocker(fallbackHint = false) {
if (!(startSessionMode === "TALK" || startSessionMode === "TALKFIRST")) return;
const el = document.getElementById("talk-unlocker");
if (!el) return;
// L'unlocker è utile solo a stato idle/error: se l'utente ha già cliccato l'Orb
// o è connecting/connected non mostriamo niente.
// Show the unlocker only while idle/error, before a session starts.
if (!(currentState === "idle" || currentState === "error")) return;
// Se è fallback da TALKFIRST cambiamo il testo in modo che spieghi perché
// è comparso ("tocca per consentire il microfono", invece di "preparando")
// Explain the microphone permission requirement when TALKFIRST falls back.
const subEl = el.querySelector(".talk-unlocker-sub");
const titleEl = el.querySelector(".talk-unlocker-title");
if (fallbackHint) {
@@ -3613,20 +3598,20 @@ function showTalkUnlocker(fallbackHint = false) {
el.removeAttribute("hidden");
_talkUnlockerShown = true;
console.info(
"%c[start-session] ▶️ talk-unlocker mostrato (%s): attende gesture utente",
"%c[start-session] ▶️ talk-unlocker shown (%s): waiting for a user gesture",
"color:#2563eb;font-weight:700;background:#dbeafe;padding:3px 6px;border-radius:4px;",
fallbackHint ? "fallback TALKFIRST" : "TALK mode"
);
}
function maybeScheduleAutoStart() {
// Pulisco un eventuale timer precedente (caso di doppia chiamata non attesa).
// Clear any previous timer if this function was unexpectedly called twice.
if (autoStartTalkTimerId) {
clearTimeout(autoStartTalkTimerId);
autoStartTalkTimerId = 0;
}
if (startSessionMode !== "TALK" && startSessionMode !== "TALKFIRST") return;
console.info(
`%c[start-session] S2S_START_SESSION_MODE=${startSessionMode}: azione tra 2000 ms (warmup profilo/reports/episode)`,
`%c[start-session] S2S_START_SESSION_MODE=${startSessionMode}: action in 2000 ms (profile/reports/episode warmup)`,
"color:#2563eb;font-weight:700;background:#dbeafe;padding:3px 6px;border-radius:4px;"
);
autoStartTalkTimerId = window.setTimeout(() => {
@@ -3638,49 +3623,45 @@ function maybeScheduleAutoStart() {
}
}, 2000);
}
/** Tenta l'auto-start diretto (TALKFIRST) SENZA gesture. Simula la wake word
* impostando pendingWakeWordConfirmationText="hola vocero" (stessa variabile
* del riconoscitore vocale reale), poi chiama doStart() — se i permessi sono
* già grantiti (Chrome/Android dopo "allow always"), tutto funziona senza
* click. Altrimenti fallback sull'unlocker. */
/** Attempt direct TALKFIRST auto-start without a gesture. Simulate a wake word
* by setting pendingWakeWordConfirmationText="hola vocero", as the real speech
* recognizer does, then call doStart(). Existing permissions may allow this
* without a click; otherwise fall back to the unlocker. */
async function tryTalkFirstDirect() {
if (!(currentState === "idle" || currentState === "error")) return;
if (missingServerUrl()) {
console.warn("[start-session] skip TALKFIRST: manca server URL");
console.warn("[start-session] skip TALKFIRST: missing server URL");
promptServerUrl();
return;
}
// Step 1: pre-carico la wake-word simulata. Quando onClientStatus riceve
// "connected", la frase viene injectata e il modello risponde esattamente
// come quando la hot-word è stata pronunciata dal microfono.
// Prepare the simulated wake word. When onClientStatus receives "connected",
// inject it so the model responds as if the user had spoken it.
pendingWakeWordConfirmationText = pendingWakeWordConfirmationText?.trim()
? pendingWakeWordConfirmationText
: "hola vocero";
console.info(
`%c[start-session] 🎤 TALKFIRST: imposto wake-word simulata=${JSON.stringify(pendingWakeWordConfirmationText)} e provo doStart() diretto (SENZA gesture)`,
`%c[start-session] 🎤 TALKFIRST: set simulated wake word=${JSON.stringify(pendingWakeWordConfirmationText)} and try doStart() directly (WITHOUT a gesture)`,
"color:#2563eb;font-weight:700;background:#dbeafe;padding:3px 6px;border-radius:4px;"
);
try {
setCaption("Preparing your session…", "muted");
// Non forzo AudioContext da fuori (non c'è gesture): lo crea internamente
// _ensureAudioContext; se rimane suspended il client lo resume() quando
// arriva il primo audio (o sul prossimo click utente che lo unlocka).
setCaption("Preparando tu sesión…", "muted");
// Let _ensureAudioContext create the context. If it stays suspended, the
// client resumes it on the first audio or the next user click.
await doStart();
} catch (err) {
console.warn("[start-session] ⚠️ TALKFIRST doStart() diretto FALLITO → fallback unlocker", err && err instanceof Error ? err.message : err);
// Fallback: mostro unlocker con la frase esplicativa "permesso mic".
console.warn("[start-session] ⚠️ direct TALKFIRST doStart() FAILED → fallback unlocker", err && err instanceof Error ? err.message : err);
// Show the unlocker with the microphone permission explanation.
try { await teardown(); } catch (_) { /* no-op */ }
setState("idle");
chat.renderEmptyState();
// Ripristino pendingWakeWordConfirmationText: nel caso l'utente usi
// l'unlocker per avviare, NON vogliamo perdere la frase simulata (la
// useremo di nuovo al connected, scattando quindi sempre la greet).
// Preserve the simulated phrase so starting through the unlocker still
// injects it on connection and triggers the greeting.
pendingWakeWordConfirmationText = pendingWakeWordConfirmationText || "hola vocero";
showTalkUnlocker(true);
void handleStartError(err instanceof Error ? err : new Error(String(err || "mic permission required")));
}
}
// Attacca una sola volta i listener sull'unlocker: click + touchstart = unlock
// Register unlocker listeners once: click + touchstart unlock audio.
(() => {
const el = document.getElementById("talk-unlocker");
if (!el) return;
@@ -3692,28 +3673,28 @@ async function tryTalkFirstDirect() {
window.addEventListener("keydown", (ev) => {
if (startSessionMode !== "TALK") return;
if (!_talkUnlockerShown) return;
// Qualsiasi tasto = unlock (Enter/Space/anche Altri)
// Any key unlocks audio, including Enter and Space.
startTalkFromGesture();
}, { passive: true });
})();
/** Avvia doStart() dentro una gesture valida (click/touch/key).
Chiamato dal talk-unlocker o da qualsiasi altro unlocker valido. */
/** Start doStart() within a valid click/touch/key gesture.
* Called by the talk unlocker or another valid audio unlocker. */
async function startTalkFromGesture() {
hideTalkUnlocker();
if (!(currentState === "idle" || currentState === "error")) return;
if (missingServerUrl()) {
console.warn("[start-session] skip unlocker: manca server URL (missingServerUrl=true)");
console.warn("[start-session] skip unlocker: missing server URL (missingServerUrl=true)");
promptServerUrl();
return;
}
try {
// Qui SIAMO dentro la user gesture: AudioContext + getUserMedia funzionano.
console.info("[start-session] 🎤 avvio doStart() da gesture unlocker (TALK mode)");
// This user gesture permits AudioContext and getUserMedia to start.
console.info("[start-session] 🎤 start doStart() from unlocker gesture (TALK mode)");
const audioContext = createResumedAudioContext();
setCaption("Preparing your session…", "muted");
setCaption("Preparando tu sesión…", "muted");
await doStart(audioContext);
} catch (err) {
console.error("[start-session] ⛔ doStart() da unlocker fallito", err);
console.error("[start-session] ⛔ doStart() from unlocker failed", err);
void handleStartError(err);
}
}
s2s-client/project/index.html
-139
View File
@@ -1,139 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
<title>Project Docs</title>
<style>
:root {
color-scheme: dark;
--bg: #081018;
--panel: #0f1b26;
--line: rgba(149, 180, 201, 0.18);
--text: #edf6fb;
--muted: #9db2c2;
--accent: #59d0ff;
}
* {
box-sizing: border-box;
}
html,
body {
margin: 0;
width: 100%;
height: 100%;
background: var(--bg);
color: var(--text);
font-family: Inter, Arial, Helvetica, sans-serif;
}
body {
display: grid;
grid-template-rows: auto 1fr;
}
.topbar {
display: flex;
align-items: center;
justify-content: space-between;
gap: 16px;
padding: 14px 18px;
border-bottom: 1px solid var(--line);
background: rgba(8, 16, 24, 0.92);
backdrop-filter: blur(10px);
}
.title {
margin: 0;
font-size: 16px;
font-weight: 700;
letter-spacing: 0.02em;
}
.subtitle {
margin: 4px 0 0;
color: var(--muted);
font-size: 13px;
}
.actions {
display: flex;
align-items: center;
gap: 10px;
}
.btn {
display: inline-flex;
align-items: center;
justify-content: center;
min-height: 38px;
padding: 0 14px;
border: 1px solid var(--line);
border-radius: 10px;
color: var(--text);
background: var(--panel);
text-decoration: none;
font-size: 13px;
font-weight: 600;
}
.btn.primary {
border-color: rgba(89, 208, 255, 0.35);
color: #021018;
background: var(--accent);
}
.frame-wrap {
padding: 14px;
min-height: 0;
}
.frame {
width: 100%;
height: 100%;
border: 1px solid var(--line);
border-radius: 16px;
background: #ffffff;
}
@media (max-width: 760px) {
.topbar {
flex-direction: column;
align-items: flex-start;
}
.actions {
width: 100%;
}
.btn {
flex: 1 1 auto;
}
}
</style>
</head>
<body>
<header class="topbar">
<div>
<h1 class="title">Vocero Project</h1>
<p class="subtitle">Embedded technical document viewer for `demo/public/docs.html`.</p>
</div>
<div class="actions">
<a class="btn" href="/">Back to app</a>
<a class="btn primary" href="/public/docs.html" target="_blank" rel="noopener">Open raw document</a>
</div>
</header>
<main class="frame-wrap">
<iframe
class="frame"
src="/public/docs.html"
title="Project documentation"
loading="eager"
referrerpolicy="no-referrer"
></iframe>
</main>
</body>
</html>
s2s-client/public/docs.html
-272
View File
File diff suppressed because one or more lines are too long
s2s-client/server.py
+573 -722
View File
File diff suppressed because it is too large Load Diff
s2s-client/style.css
+38 -40
View File
@@ -143,11 +143,10 @@ body::after {
}
/* ─── Overlay unlocker TALK mode (S2S_START_SESSION_MODE=TALK) ───
Cattura QUALSIASI primo tap/click dell'utente sulla pagina, perché
i browser bloccano getUserMedia() fuori da una user gesture. Senza
questo layer il doStart() del setTimeout falliva e non si sentiva
il greet. Sopra TUTTO tranne i modali <dialog> (z-index auto di dialog
vince). */
Capture the first tap/click anywhere on the page because browsers block
getUserMedia() outside a user gesture. Without this layer, doStart() from
setTimeout failed and the greeting was silent. Above all content except
<dialog> modals, whose top layer takes priority. */
.talk-unlocker {
position: fixed;
inset: 0;
@@ -466,7 +465,7 @@ a:hover {
transform: translateY(-0.5px);
}
/* ── User avatar circle (instead of old "Utente: Nome" text) ──
/* ── User avatar circle (replaces the old user-name text) ──
* - perfectly round badge, matches the 42px icon-btn size in the topbar
* - glass + brand gradient 1px border
* - two uppercase initials (Alessandro Valenti → AV), bold white
@@ -593,8 +592,8 @@ a:hover {
height: 36px;
padding: 0 12px;
}
/* ── Footer (narrow viewport): riduci tracking per non farlo andare
* a capo in malo modo; padding compatto + line-height maggiore */
/* ── Footer (narrow viewport): reduce tracking to avoid awkward wrapping;
* use compact padding and a taller line height. */
.app-footer {
padding: 8px 14px 14px;
}
@@ -2425,7 +2424,7 @@ body.cam-on .footer {
`margin-right:22px` on `.hist-msg.assistant` and `margin-left:6px` on `.hist-msg.user`
(hard rules on the bubbles themselves so nothing can hide them).
Bottom padding BUMP 2026-09-04: 96 → 160px. The screenshot showed the last
assistant line "...Necesitas más detalles o" cut clean by the footer. 160px
assistant line asking for more details cut cleanly by the footer. 160px
inside the scroll container is 5× the real footer glass height; after we
snap scrollTop to the absolute max, the last bubble terminates at the TOP
of a 160px-high transparent region → fully visible, fully above the floating
@@ -2460,7 +2459,7 @@ body.cam-on .footer {
background-clip: content-box;
}
/* ─── Chat input row (bottom of chat panel: "sobre el botón") ──────── */
/* ─── Chat input row (bottom of chat panel, above the button) ──────── */
/* NOTE: currently HIDDEN in index.html. Kept seamless here so if/when we
re-enable it the form melts visually into the page bg, no separator frame. */
.chat-form {
@@ -3474,20 +3473,20 @@ button[data-reports-sidebar-toggle] {
flex: none;
}
/* Per-chip tint cycles (keyword-based, if no data attribute → fallback brand) */
.reports-hint .reports-list li[data-tint="mortalidad"] { background: linear-gradient(180deg, color-mix(in srgb, #ff6a75 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #ff6a75 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="mortalidad"]::before { background: #ff6a75; box-shadow: 0 0 8px 0 #ff6a75aa; }
.reports-hint .reports-list li[data-tint="metricas"] { background: linear-gradient(180deg, color-mix(in srgb, #22d3ee 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #22d3ee 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="metricas"]::before { background: #22d3ee; box-shadow: 0 0 8px 0 #22d3eeaa; }
.reports-hint .reports-list li[data-tint="diario"] { background: linear-gradient(180deg, color-mix(in srgb, #8b5cf6 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #8b5cf6 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="diario"]::before { background: #8b5cf6; box-shadow: 0 0 8px 0 #8b5cf6aa; }
.reports-hint .reports-list li[data-tint="productos"] { background: linear-gradient(180deg, color-mix(in srgb, #ec4899 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #ec4899 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="productos"]::before { background: #ec4899; box-shadow: 0 0 8px 0 #ec4899aa; }
.reports-hint .reports-list li[data-tint="mortality"] { background: linear-gradient(180deg, color-mix(in srgb, #ff6a75 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #ff6a75 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="mortality"]::before { background: #ff6a75; box-shadow: 0 0 8px 0 #ff6a75aa; }
.reports-hint .reports-list li[data-tint="metrics"] { background: linear-gradient(180deg, color-mix(in srgb, #22d3ee 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #22d3ee 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="metrics"]::before { background: #22d3ee; box-shadow: 0 0 8px 0 #22d3eeaa; }
.reports-hint .reports-list li[data-tint="daily"] { background: linear-gradient(180deg, color-mix(in srgb, #8b5cf6 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #8b5cf6 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="daily"]::before { background: #8b5cf6; box-shadow: 0 0 8px 0 #8b5cf6aa; }
.reports-hint .reports-list li[data-tint="products"] { background: linear-gradient(180deg, color-mix(in srgb, #ec4899 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #ec4899 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="products"]::before { background: #ec4899; box-shadow: 0 0 8px 0 #ec4899aa; }
.reports-hint .reports-list li[data-tint="synapse"] { background: linear-gradient(180deg, color-mix(in srgb, #fbbf24 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #fbbf24 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="synapse"]::before { background: #fbbf24; box-shadow: 0 0 8px 0 #fbbf24aa; }
.reports-hint .reports-list li[data-tint="supply"] { background: linear-gradient(180deg, color-mix(in srgb, #34d399 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #34d399 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="supply"]::before { background: #34d399; box-shadow: 0 0 8px 0 #34d399aa; }
.reports-hint .reports-list li[data-tint="finanzas"] { background: linear-gradient(180deg, color-mix(in srgb, #22c55e 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #22c55e 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="finanzas"]::before { background: #22c55e; box-shadow: 0 0 8px 0 #22c55eaa; }
.reports-hint .reports-list li[data-tint="finance"] { background: linear-gradient(180deg, color-mix(in srgb, #22c55e 14%, rgba(20,22,32,.55)), rgba(20,22,32,.55)); border-color: color-mix(in srgb, #22c55e 35%, var(--border)); }
.reports-hint .reports-list li[data-tint="finance"]::before { background: #22c55e; box-shadow: 0 0 8px 0 #22c55eaa; }
.reports-hint .reports-list li:hover {
transform: translateY(-1px) scale(1.01);
border-color: color-mix(in srgb, var(--brand) 55%, var(--border-strong));
@@ -4055,10 +4054,10 @@ button[data-reports-sidebar-toggle] {
opacity: 0.5;
}
/* ─── Webcam preview (camera tool) — HIDDEN BY DEFAULT OGGI ──────────────────
/* ─── Webcam preview (camera tool) — CURRENTLY HIDDEN BY DEFAULT ─────────────
* Floating self-view, bottom-left. Mirrored for the user; the frame sent to
* the model is drawn un-mirrored. Markup esiste per riattivazione futura.
* Per usarlo: togli hidden su cam-pip in index.html + scommenta listener in main.js. */
* the model is drawn un-mirrored. Markup is retained for future reactivation.
* To enable it, remove hidden from cam-pip in index.html and restore the main.js listener. */
.cam-pip {
position: fixed;
left: 20px;
@@ -4285,12 +4284,12 @@ html[data-theme="white"] .app-column {
margin-left: 0 !important;
}
/* ─────────────────────────────────────────────────────────────────────
USER FIX 2026-09-10 "la chat deve apparire solo nel chat box":
USER FIX 2026-09-10: show the conversation only inside the chat box.
The dark theme renders a SECOND ephemeral transcript into #bubble-stack
(floating anchored top-right — the grey/light blue bubbles the user
circled in red with an X on the last screenshot). Since the white
mobile theme shows ALL conversation exclusively inside the persistent
"Conversación" card (chat-history inside .chat-panel bounded 58dvh),
conversation card (chat-history inside .chat-panel bounded 58dvh),
we kill the parallel floating bubble stack HARD in white theme — both
the ID and the class, zero width/height/transform so it can't leave
layout artifacts. */
@@ -4543,8 +4542,8 @@ html[data-theme="white"] .topbar-right {
display: none;
}
/* ── Status card: "Listo para conversar / Informes, análisis..." ───────
User request 2026-09-10 "nascondi box listo para conversar" — hide the
/* ── Status card: ready to converse, reports and analysis ─────────────
User request 2026-09-10: hide the ready-to-converse box. Hide the
entire status card in white theme (the grey rounded banner with the
chart icon and arrow between topbar and mic stage, which the user
circled with a red X). Previously we hid only the right chevron; now
@@ -4820,7 +4819,7 @@ html[data-theme="white"] .white-stop-btn svg {
flex: none;
}
/* ── Quick actions — 3 pills (Resumen / Consultar datos / Crear respuesta).
/* ── Quick actions — 3 pills (summary / data lookup / response draft).
User request 2026-09-10 "nascondi le cose in rosso": all 3 quick action
pills are circled in red → hide the entire quick-actions container. */
html[data-theme="white"] .quick-actions {
@@ -4830,8 +4829,8 @@ html[data-theme="white"] .quick-action-btn {
display: none !important;
}
/* ── Chat panel (Conversación card): white rounded card + "En línea" pill.
CRITICAL SCROLL FIX 2026-09-10 (per "non fa lo scrolling della chatbox"):
/* ── Chat panel: white rounded conversation card + online status pill.
CRITICAL SCROLL FIX 2026-09-10: the chat box was not scrolling.
The previous white-theme override set `flex: 0 0 auto` + `max-height: none`
which removed every vertical constraint on the chat panel → it grew beyond
the viewport edge → overflow on the *page* body instead of the chat
@@ -4912,7 +4911,7 @@ html[data-theme="white"] .chat-panel-header {
border-bottom-width: 1px !important;
text-decoration: none !important;
}
/* "En línea" status pill — green dot + text */
/* Online status pill — green dot + text */
html[data-theme="white"] .chat-status-pill {
display: inline-flex;
align-items: center;
@@ -4940,10 +4939,9 @@ html[data-theme="white"] .chat-status-dot {
50% { box-shadow: 0 0 0 5px rgba(22, 163, 74, 0.08); }
}
/* ── Chat header actions row: contiene "En línea" status pill + Export button.
Aggiunto 2026-09-23: bottone download conversación.
Layout orizzontale, allineato a destra, auto-wrap con gap per evitare
overflow su schermi stretti. */
/* ── Chat header actions row: online status pill + conversation export button.
Added 2026-09-23. Horizontal, right-aligned layout with wrapping and gaps
to avoid overflow on narrow screens. */
.chat-header-actions {
display: inline-flex;
align-items: center;
@@ -4953,7 +4951,7 @@ html[data-theme="white"] .chat-status-dot {
flex-wrap: wrap;
min-width: 0;
}
/* Botón export conversación (icon button baseline: coerente con altri .icon-btn). */
/* Conversation export button, consistent with other .icon-btn controls. */
.chat-export-btn {
position: relative;
display: inline-flex;
@@ -4990,14 +4988,14 @@ html[data-theme="white"] .chat-status-dot {
pointer-events: none;
flex: 0 0 auto;
}
/* Feedback OK: verde temporáneo al exportar correctamente. */
/* Successful export feedback: temporarily green. */
.chat-export-btn.chat-export-ok {
background: rgba(22, 163, 74, 0.18) !important;
border-color: rgba(22, 163, 74, 0.34) !important;
color: #22c55e !important;
box-shadow: 0 0 0 2px rgba(22, 163, 74, 0.12);
}
/* Feedback vacía: pulse ámbar (ningún mensaje para exportar). */
/* Empty export feedback: amber pulse when there are no messages. */
.chat-export-btn.chat-export-empty {
background: rgba(245, 158, 11, 0.18) !important;
border-color: rgba(245, 158, 11, 0.34) !important;
@@ -5039,7 +5037,7 @@ html[data-theme="white"] .chat-export-btn.chat-export-empty {
}
/* ──────────────────────────────────────────────────
THE SCROLLABLE HISTORY BOX (root cause of "non scorre")
THE SCROLLABLE HISTORY BOX (root cause of the scrolling failure)
Before: max-height:none. Now it's a flex item with flex:1 1 auto inside
.chat-panel-inner, which has a bounded 100% height of its parent.
min-height:0 is MANDATORY — without it, flex column children never shrink
@@ -5213,7 +5211,7 @@ html[data-theme="white"] #bubble-stack .bubble.user::before {
}
/* 6) META label "name HH:MM" on the same single line, ONE pseudo-element.
──── USER FIX 2026-09-10 "perche mette sempre Emilio" ────
──── USER FIX 2026-09-10: remove the hardcoded user name ────
The previous iteration hardcoded the Spanish names "Vocero" (assistant)
and "Emilio" (user) plus a fake "16:14" timestamp. Instead we now read
BOTH values dynamically from data attributes written by ui/chat.js at the
s2s-client/ui/account.js
+19 -19
View File
@@ -80,7 +80,7 @@ export class Account {
if (!me.loggedIn) {
// Signed-out: a sign-in pill (only when OAuth is actually available).
if (me.auth && me.loginUrl) {
this._root.innerHTML = `<a class="signin-pill" href="${escHtml(me.loginUrl)}" title="Sign in for more time"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M15 3h4a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2h-4"/><polyline points="10 17 15 12 10 7"/><line x1="15" y1="12" x2="3" y2="12"/></svg><span>Sign in</span></a>`;
this._root.innerHTML = `<a class="signin-pill" href="${escHtml(me.loginUrl)}" title="Inicia sesión para obtener más tiempo"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M15 3h4a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2h-4"/><polyline points="10 17 15 12 10 7"/><line x1="15" y1="12" x2="3" y2="12"/></svg><span>Iniciar sesión</span></a>`;
} else {
this._root.innerHTML = "";
this._root.hidden = true;
@@ -97,28 +97,28 @@ export class Account {
: `<span class="account-avatar account-avatar-fallback">${escHtml((me.username || "?")[0].toUpperCase())}</span>`;
const remaining =
isUnlimited || me.remainingSec == null
? "Unlimited"
: `${fmt(me.remainingSec)} left today`;
const tierLabel = isPro ? "PRO" : isUnlimited ? "Team" : "Free";
? "Sin límite"
: `${fmt(me.remainingSec)} restantes hoy`;
const tierLabel = isPro ? "PRO" : isUnlimited ? "Equipo" : "Gratis";
this._root.innerHTML = `
<button id="account-chip" class="account-chip" aria-haspopup="true" aria-expanded="false">
${avatar}
<span class="account-handle">${escHtml(me.username || "you")}</span>
<span class="account-handle">${escHtml(me.username || "tú")}</span>
${isPro
? '<span class="account-pro">PRO</span>'
: me.tier === "org"
? '<span class="account-pro account-team">TEAM</span>'
? '<span class="account-pro account-team">EQUIPO</span>'
: ""}
</button>
<div id="account-pop" class="account-pop" hidden>
<div class="account-pop-row account-pop-name">${escHtml(me.username || "you")}</div>
<div class="account-pop-row account-pop-name">${escHtml(me.username || "tú")}</div>
<div class="account-pop-row account-pop-meta">
<span class="account-tier">${tierLabel}</span>
<span class="account-remaining">${escHtml(remaining)}</span>
</div>
${isUnlimited ? "" : `<a class="account-pop-link" href="${PRO_URL}" target="_blank" rel="noopener">Upgrade to PRO</a>`}
<a class="account-pop-link account-signout" href="${escHtml(me.logoutUrl || "#")}">Sign out</a>
${isUnlimited ? "" : `<a class="account-pop-link" href="${PRO_URL}" target="_blank" rel="noopener">Pasar a PRO</a>`}
<a class="account-pop-link account-signout" href="${escHtml(me.logoutUrl || "#")}">Cerrar sesión</a>
</div>`;
const chip = $("#account-chip");
@@ -152,13 +152,13 @@ export class Account {
*/
showLimit(tier = this.tier) {
const canSignIn = this._me.auth && this._me.loginUrl;
this._modalTitle.textContent = "Thanks for chatting!";
this._modalTitle.textContent = "¡Gracias por conversar!";
if (tier === "anon") {
this._modalMsg.textContent =
"Guest conversations run for 5 minutes. Sign in with Hugging Face to get 10 minutes a day for free, and PRO members chat with no limit at all.";
this._modalNote.textContent = "Your free minutes refresh tomorrow.";
"Las conversaciones de invitados duran 5 minutos. Inicia sesión con Hugging Face para obtener 10 minutos gratis al día. Los miembros PRO pueden conversar sin límite.";
this._modalNote.textContent = "Tus minutos gratis se renuevan mañana.";
if (canSignIn) {
this._modalCta.innerHTML = `${HF_MARK}<span>Sign in with Hugging Face</span>`;
this._modalCta.innerHTML = `${HF_MARK}<span>Iniciar sesión con Hugging Face</span>`;
this._modalCta.href = /** @type {string} */ (this._me.loginUrl);
this._modalCta.hidden = false;
} else {
@@ -167,9 +167,9 @@ export class Account {
} else {
// Signed-in, non-PRO.
this._modalMsg.textContent =
"You've enjoyed your 10 minutes for today. Go PRO for unlimited conversations and to support open source AI.";
this._modalNote.textContent = "Or come back tomorrow. Your minutes reset daily.";
this._modalCta.innerHTML = "<span>Upgrade to PRO</span>";
"Ya has usado tus 10 minutos de hoy. Pasa a PRO para conversar sin límite y apoyar la IA de código abierto.";
this._modalNote.textContent = "También puedes volver mañana. Tus minutos se renuevan cada día.";
this._modalCta.innerHTML = "<span>Pasar a PRO</span>";
this._modalCta.href = PRO_URL;
this._modalCta.hidden = false;
}
@@ -179,10 +179,10 @@ export class Account {
/** Show a warm "we're at capacity" message when even the waiting line is full.
* Reuses the limit modal shell; no call-to-action, just reassurance. */
showBusy() {
this._modalTitle.textContent = "Hugged to the limit 🤗";
this._modalTitle.textContent = "¡Estamos al límite de abrazos! 🤗";
this._modalMsg.textContent =
"Every slot and the whole line are full right now. Too much love! Grab a coffee and pop back in a minute.";
this._modalNote.textContent = "A spot usually opens up within a few minutes.";
"Todos los lugares y la cola están ocupados. ¡Demasiado cariño! Tómate un café y vuelve en un minuto.";
this._modalNote.textContent = "Normalmente se libera un lugar en pocos minutos.";
this._modalCta.hidden = true;
if (!this._modal.open) this._modal.showModal();
}
s2s-client/ui/chat.js
+18 -18
View File
@@ -34,7 +34,7 @@ function reportChatDebug(hypothesisId, location, msg, data = {}) {
const WRENCH_PATH = `<path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"/>`;
const CHAT_BUBBLE_SVG = `<svg width="28" height="28" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15a2 2 0 0 1-2 2H7l-4 4V5a2 2 0 0 1 2-2h14a2 2 0 0 1 2 2z"/></svg>`;
const EMPTY_STATE_HTML = `<div id="chat-empty" class="chat-empty">${CHAT_BUBBLE_SVG}<span class="chat-empty-title">No messages yet</span><span class="chat-empty-hint">Tap the orb and start talking</span></div>`;
const EMPTY_STATE_HTML = `<div id="chat-empty" class="chat-empty">${CHAT_BUBBLE_SVG}<span class="chat-empty-title">Aún no hay mensajes</span><span class="chat-empty-hint">Toca el círculo y empieza a hablar</span></div>`;
/**
* @param {string} prefix
* @param {string} [sep]
@@ -288,7 +288,7 @@ export class ChatView {
* inner .{hist|bubble}-body → data-user-name | data-asst-name + data-timestamp
*
* The .{hist|bubble}-body::before pseudo-element in style.css uses attr()
* to render the "Nombre HH:MM" label aligned to the bubble corner; the
* to render the "Name HH:MM" label aligned to the bubble corner; the
* wrapper .user::before pseudo uses attr(data-user-initials) to paint the
* circular avatar's 2-letter monogram.
*
@@ -298,7 +298,7 @@ export class ChatView {
_buildMessageEl({ container, prefix, role, text, partial = false }) {
const el = document.createElement("div");
el.className = `${container} ${role}`;
const label = role === "user" ? "You" : "Assistant";
const label = role === "user" ? "Tú" : "Asistente";
const stamp = this._nowStamp();
if (role === "user") {
const uName = this._resolveUserName();
@@ -451,17 +451,17 @@ export class ChatView {
const el = document.createElement("div");
el.className = "hist-msg tool";
el.innerHTML = `
<div class="hist-role">Tool call</div>
<div class="hist-role">Llamada a herramienta</div>
<button class="hist-tool-header" aria-expanded="false">
<svg class="hist-tool-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">${WRENCH_PATH}</svg>
<span class="hist-tool-name">${escHtml(name)}</span>
<svg class="hist-tool-chevron" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="6 9 12 15 18 9"/></svg>
</button>
<div class="hist-tool-body">
<div class="hist-tool-label">Input</div>
<div class="hist-tool-label">Entrada</div>
<div class="hist-tool-block">${escHtml(pretty)}</div>
<div class="hist-tool-label">Output</div>
<div class="hist-tool-block hist-tool-output">${escHtml(output || "(no output)")}</div>
<div class="hist-tool-label">Salida</div>
<div class="hist-tool-block hist-tool-output">${escHtml(output || "(sin salida)")}</div>
</div>
`;
const header = /** @type {HTMLButtonElement} */ (el.querySelector(".hist-tool-header"));
@@ -482,7 +482,7 @@ export class ChatView {
hist.classList.add("interrupted");
const note = document.createElement("div");
note.className = "hist-note";
note.textContent = "Interrupted";
note.textContent = "Interrumpido";
hist.appendChild(note);
}
@@ -493,7 +493,7 @@ export class ChatView {
if (empty) empty.remove();
const el = document.createElement("div");
el.className = "hist-msg tool";
el.innerHTML = `<div class="hist-role">Snapshot</div><img class="hist-image" alt="Webcam snapshot sent to the model" />`;
el.innerHTML = `<div class="hist-role">Captura</div><img class="hist-image" alt="Captura de la cámara enviada al modelo" />`;
const img = /** @type {HTMLImageElement} */ (el.querySelector("img"));
img.src = dataUrl;
this._chatHistory.appendChild(el);
@@ -583,8 +583,8 @@ export class ChatView {
/**
* Pretty-print the JSON conversation structure to a human-readable plain
* text log (one entry per "HH:MM Nombre: texto"). Tool calls are rendered
* as indented blocks ("TOOL: nombre" followed by Input/Output sections).
* text log (one entry per "HH:MM Name: text"). Tool calls are rendered
* as indented blocks (tool name followed by input/output sections).
* @param {ReturnType<ChatView["_serializeConversation"]>} conv
* @returns {string}
*/
@@ -607,11 +607,11 @@ export class ChatView {
lines.push(head);
const args = String(m.args || "").trim();
const output = String(m.output || "").trim();
if (args) { lines.push(" Input:"); lines.push(" " + args.split("\n").join("\n ")); }
if (output) { lines.push(" Output:"); lines.push(" " + output.split("\n").join("\n ")); }
if (args) { lines.push(" Entrada:"); lines.push(" " + args.split("\n").join("\n ")); }
if (output) { lines.push(" Salida:"); lines.push(" " + output.split("\n").join("\n ")); }
lines.push("");
} else if (m.type === "image") {
const head = [m.timestamp, "SNAPSHOT (imagen web-cámara enviada al modelo)"].filter(Boolean).join(" ");
const head = [m.timestamp, "CAPTURA (imagen de la cámara enviada al modelo)"].filter(Boolean).join(" ");
lines.push(head);
lines.push(" " + String(m.dataUrl || "").slice(0, 80) + (String(m.dataUrl || "").length > 80 ? "…" : ""));
lines.push("");
@@ -648,8 +648,8 @@ export class ChatView {
* Public entry-point wired to the #chat-export-btn in the chat header.
* Serializes the current conversation and produces TWO downloads:
* 1. vocero-conversacion-YYYYMMDD-HHMMSS.json (structured data)
* 2. vocero-conversacion-YYYYMMDD-HHMMSS.txt (log legibile para humanos)
* UI feedback: briefly swaps the download icon for a ✓ "hecho" tick then
* 2. vocero-conversacion-YYYYMMDD-HHMMSS.txt (human-readable log)
* UI feedback: briefly swaps the download icon for a ✓ completion tick then
* restores the original SVG so the user gets a visual ack that the export
* completed (idempotent / errors are logged to console — never throws).
*/
@@ -677,7 +677,7 @@ export class ChatView {
setTimeout(() => this._exportBtn?.classList.remove("chat-export-empty"), 600);
}
if (typeof window !== "undefined" && typeof console !== "undefined") {
console.info("[ui/chat.js:exportConversation] No hay mensajes aún — exportación omitida.");
console.info("[ui/chat.js:exportConversation] No messages yet — export skipped.");
}
return;
}
@@ -691,7 +691,7 @@ export class ChatView {
}
} catch (err) {
if (typeof window !== "undefined" && typeof console !== "undefined") {
console.error("[ui/chat.js:exportConversation] Error exportando conversación:", err);
console.error("[ui/chat.js:exportConversation] Conversation export failed:", err);
}
}
}
s2s-client/ws/s2s-ws-client.js
+102 -119
View File
@@ -54,16 +54,14 @@ function reportWsDebug(hypothesisId, location, msg, data = {}) {
}
// ---------------------------------------------------------------------------
// 🔎 WS DEBUG TRACER (sempre attivo, 2026-09-24)
// Banner + stile etichette condiviso per tutta la famiglia di log [S2S-WS:*].
// Non è più abilitato solo via localStorage.s2s.debug: gli eventi di vita
// (connect, close, errore) e l'handshake iniziale sono loggati always-on;
// i messaggi bulk (input_audio_buffer.append) vengono solo contati per non
// spammare la Console.
// 🔎 WS DEBUG TRACER (always active, 2026-09-24)
// Shared banner and label styles for all [S2S-WS:*] logs. Lifecycle events
// (connect, close, error) and the initial handshake are always logged.
// Count bulk input_audio_buffer.append messages to avoid console spam.
// ---------------------------------------------------------------------------
(function _wsDebugBanner() {
console.info(
"%c🟡 [S2S-WS] client caricato (ws/s2s-ws-client.js v2026-09-24 23:12) — handshake, stato, messaggi chiave e tool-call loggati; PREFETCH APIPILOT strumenti AD OGNI INIZIO SESSIONE + post-login phone (cache per-utente, cambio phone = refresh=true automatico; invalid. cache globale __s2sInvalidateApipilotCache(); anti-leak token; anti-spam audio).",
"%c🟡 [S2S-WS] client loaded (ws/s2s-ws-client.js v2026-09-24 23:12) — logging handshake, status, key messages and tool calls; PREFETCH APIPILOT tools AT EVERY SESSION START + after phone login (per-user cache, phone changes trigger refresh=true; global invalidation __s2sInvalidateApipilotCache(); token masking; audio log throttling).",
"background:#fef3c7;color:#92400e;font-weight:700;padding:4px 8px;border-radius:4px;"
);
})();
@@ -79,24 +77,22 @@ function _wsLog(status, tag, msg, ...extra) {
const badge = `%c[S2S-WS:${tag}]`;
console.log(`${badge}%c ${msg}`, badgeBg, labelFg, ...extra);
}
/** Contatore per audio chunk bulk, per evitare spam. */
/** Bulk audio chunk counter for throttling logs. */
let _wsMicChunkCount = 0;
let _wsMicChunkLastLogTs = 0;
// ---------------------------------------------------------------------------
// 🔧 APIPILOT PREFETCH AD INIZIO SESSIONE (2026-09-24)
// Ogni volta che s2s-ws-client avvia una connessione Realtime (metodo connect())
// viene scaldata la cache ApiPilot lato server chiamando /api/apipilot/tools.
// Cache TTL lato client 240 s (= TTL server 300 s meno 60 s margine).
// Il prefetch è FIRE-AND-FORGET: non blocca MAI l'handshake WS, non fallisce
// connect() anche se il proxy restituisce 5xx.
// 🔧 APIPILOT PREFETCH AT SESSION START (2026-09-24)
// Each connect() warms the server-side ApiPilot cache via /api/apipilot/tools.
// Client cache TTL is 240 s (server TTL 300 s minus a 60 s margin).
// Prefetch failures, including proxy 5xx responses, do not fail connect().
// ---------------------------------------------------------------------------
/** @type {{at:number, data:any, key:string|null}|null} */
let _apipilotClientCache = null;
const _APIPILOT_CLIENT_TTL_MS = 240_000; // 4 minuti
/** Contatore sessioni totali (per log), incrementato ad ogni connect(). */
const _APIPILOT_CLIENT_TTL_MS = 240_000; // 4 minutes
/** Session counter for logging, incremented by each connect(). */
let _wsSessionCounter = 0;
/** Chiave cache per utente: ultimi 8 cifre phoneNumber o fingerprint token. */
/** Per-user cache key: last 8 phoneNumber digits or a token fingerprint. */
function _apipilotUserKey() {
try {
// @ts-ignore
@@ -108,38 +104,34 @@ function _apipilotUserKey() {
return "anon";
} catch (_) { return "anon"; }
}
/** Invalida la cache ApiPilot lato client. Esporta globalmente per main.js (dopo login phone). */
/** Invalidate the client ApiPilot cache. Exported globally for main.js after phone login. */
function _invalidateApipilotClientCache() {
_apipilotClientCache = null;
console.info(
"%c🔧 [S2S-WS:APIPILOT] Cache client invalidata (nuovo login utente o refresh=true forzato)",
"%c🔧 [S2S-WS:APIPILOT] Client cache invalidated (new user login or forced refresh=true)",
"background:#fde68a;color:#92400e;font-weight:700;padding:1px 6px;border-radius:3px;"
);
}
// Stesso identico marker e regole del lato server Python (server.py#L555-L556).
// Lo ripetiamo identico in JS per garantire:
// (1) stessa classifica dedup
// (2) stesso formato regole marker AI
// (3) dedup by marker stringa <<<APIPILOT_TOOLS_LIST_BEGIN>>> in effectiveReasoningInstructions
// (così non appendiamo due volte se sia server che client provano entrambi).
// Match the Python server's marker and rules for consistent deduplication and
// AI marker syntax. effectiveReasoningInstructions checks this marker so the
// client and server do not append the same tools list twice.
const APIPILOT_PASS1_LIST_MARKER_JS = "<<<APIPILOT_TOOLS_LIST_BEGIN>>>";
/**
* Restituisce il testo del blocco PASSATA-1 da iniettare nel prompt instructions LLM.
* FORMATO IDENTICO al Python _resolve_apipilot_reasoning_context_block(tools=[...]).
* - Inizio con APIPILOT_PASS1_LIST_MARKER_JS
* - Fine con <<<APIPILOT_TOOLS_LIST_END>>>
* - Lingua: SPAGNOLO (stesso di profile inject e istruzioni AI)
* - Cap description a 500 chars
* - Max 100 strumenti
* @param {any[]|null|undefined} tools Lista da cache oppure null (usa cache corrente)
* @returns {string} Blocco di testo PASSATA-1 (stringa vuota se lista vuota / nessuno strumento / nessuna cache)
* Build the PASS-1 block injected into the LLM instructions prompt.
* Match Python _resolve_apipilot_reasoning_context_block(tools=[...]).
* - Start with APIPILOT_PASS1_LIST_MARKER_JS
* - End with <<<APIPILOT_TOOLS_LIST_END>>>
* - Instructions in English; assistant responses remain Spanish
* - Cap descriptions at 500 characters and tools at 100
* @param {any[]|null|undefined} tools Cached list, or null to use the current cache
* @returns {string} PASS-1 text block, empty when no tools or cache are available
*/
function buildApipilotPass1PromptBlock(tools) {
let list = Array.isArray(tools) ? tools : null;
if (!list) {
// Fallback: usa cache client locale corrente (se disponibile)
// Fall back to the current client cache, if available.
const c = _apipilotClientCache;
if (c && c.data && Array.isArray(c.data.tools)) list = c.data.tools;
}
@@ -168,25 +160,24 @@ function buildApipilotPass1PromptBlock(tools) {
console.warn("APIPILOT_PASS1_LIST_MARKER_JS", APIPILOT_PASS1_LIST_MARKER_JS);
const markers = (
`${APIPILOT_PASS1_LIST_MARKER_JS}\n` +
"### Herramientas dinámicas disponibles (ApiPilot AquaPortal)\n" +
"### Available dynamic tools (ApiPilot AquaPortal)\n" +
"\n" +
"Cada herramienta tiene un campo `name` (identificador único, NO lo modifiques, " +
"NO camelCase, NO traducir) y un campo `description` que describe el caso de uso.\n" +
"Each tool has a `name` field (a unique identifier: DO NOT change, camelCase " +
"or translate it) and a `description` field explaining its use case.\n" +
"\n" +
"REGLAS EJECUCIÓN OBLIGATORIAS:\n" +
"1. Si la petición DEL USUARIO (transcript) coincide claramente con la `description` " +
"de EXACTAMENTE una (1) herramienta de la lista inferior → DEBES emitir EXCLUSIVAMENTE " +
"una sola riga come PRIMERA línea de TU RESPUESTA, con questo formato sintattico:\n" +
' <<<APIPILOT_CALL:NAME=nome_herramienta,ARGUMENTS={"parametro1":"valor1","parametro2":"valor2"}>>>\n' +
"2. La parte ARGUMENTS={…} DEBE ser JSON válido (virgolette doppie, virgole, senza " +
"commenti). Includi SOLAMENTE i campi elencati in `required_parameters:` (se ci sono).\n" +
"3. Subito DOPO la riga marker NON aggiungere altro testo (niente spiegazioni, " +
"niente anticipazioni). Il motore eseguirà lo strumento e ti risomministrerà il " +
"risultato in una seconda chiamata.\n" +
"4. Se NESSUNA `description` corrisponde, NON emettere marker e rispondi normalmente.\n" +
"5. Se più di una descrizione corrisponde, scegli la più specifica e usa il marker.\n" +
"MANDATORY EXECUTION RULES:\n" +
"1. If the USER REQUEST (transcript) clearly matches the `description` of " +
"EXACTLY one (1) tool below, output ONLY one line as the FIRST line of YOUR " +
"RESPONSE, using this exact syntax:\n" +
' <<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter1":"value1","parameter2":"value2"}>>>\n' +
"2. ARGUMENTS={…} MUST be valid JSON (double quotes, commas, no comments). " +
"Include ONLY fields listed in `required_parameters:`, if any.\n" +
"3. Do NOT add text after the marker line (no explanations or previews). " +
"The engine executes the tool and supplies its result in a second call.\n" +
"4. If NO `description` matches, do NOT emit a marker; respond normally in Spanish.\n" +
"5. If multiple descriptions match, choose the most specific and use the marker.\n" +
"\n" +
"Lista herramientas (name / description):\n" +
"Tools list (name / description):\n" +
`${entries.join("\n")}\n` +
"<<<APIPILOT_TOOLS_LIST_END>>>"
);
@@ -195,7 +186,7 @@ function buildApipilotPass1PromptBlock(tools) {
}
/**
* Controlla se le istruzioni in input contengono già il marker PASS1 (dedup).
* Check whether the input instructions already contain the PASS1 marker.
* @param {string|undefined|null} text
* @returns {boolean}
*/
@@ -204,10 +195,9 @@ function instructionsAlreadyHasApipilotPass1(text) {
}
/**
* Espone globalmente una funzione helper per:
* - main.js: costruire effectiveReasoningInstructions appendendo la lista.
* - altri moduli che volessero leggerla.
* Il check dedup è fatto automaticamente dal chiamante prima di appendere.
* Expose a global helper for main.js to append tools to
* effectiveReasoningInstructions, and for other modules to read the list.
* The caller checks for duplicates before appending.
* @returns {{block: string, count: number, marker: string, alreadyPresent:(t:string)=>boolean}}
*/
function getApipilotPass1ContextInfo() {
@@ -456,12 +446,11 @@ export class S2sWsRealtimeClient extends EventTarget {
*/
async connect() {
if (this._ws) throw new Error("Already connected");
// Contatore sessioni totali: utile nel tracer per correlare gli eventi
// di [S2S-WS:*] a un numero di sessione crescente.
// Correlate [S2S-WS:*] events using an increasing session number.
_wsSessionCounter += 1;
const sessionIdShort = `#${_wsSessionCounter}`;
this._sessionSeq = _wsSessionCounter;
_wsLog(0, "SESSION", `${sessionIdShort} handshake iniziato — status: creating-session`);
_wsLog(0, "SESSION", `${sessionIdShort} handshake started — status: creating-session`);
let connectUrl;
if (this._directUrl) {
@@ -480,7 +469,7 @@ export class S2sWsRealtimeClient extends EventTarget {
// keeps a freed slot from being spent on someone who stepped away, and the
// click is a fresh gesture (re-arms the AudioContext on iOS).
if (waited) {
_wsLog(1, "QUEUE", `${sessionIdShort} tuo turno — attesa click join per confermare slot`);
_wsLog(1, "QUEUE", `${sessionIdShort} your turn — waiting for a join click to confirm the slot`);
await this._awaitJoin(grant);
if (this._closed) throw _codedError("connect aborted", "aborted");
}
@@ -497,38 +486,38 @@ export class S2sWsRealtimeClient extends EventTarget {
}
// Spin up the AudioContext + worklets in parallel with the WS dial,
// E IN PARALLELO scaldiamo ApiPilot (strumenti dinamici AquaPortal).
// Il prefetch è non bloccante: se fallisce, non fallisce connect().
// Also warm ApiPilot (dynamic AquaPortal tools) in parallel.
// A failed prefetch does not fail connect().
const audioReady = this._setupAudio();
const wsReady = this._openWebSocket(connectUrl);
const apipilotReady = this._prefetchApipilotTools(sessionIdShort);
await Promise.all([audioReady, wsReady, apipilotReady.catch(() => void 0)]);
_wsLog(1, "SESSION", `${sessionIdShort} connessione stabilita + audio pipeline + apipilot OK`);
_wsLog(1, "SESSION", `${sessionIdShort} connection established + audio pipeline + apipilot OK`);
}
/**
* 🔧 PRE-FETCH strumenti ApiPilot ad ogni inizio di sessione Realtime.
* - Proxy sul server: GET api/apipilot/tools?include_unauthorized=true
* - Il wrapper FETCH globale traccia come [FETCH:APIPILOT:TOOLS] in Console.
* - Cache TTL locale 240 s per evitare chiamate ridondanti se l'utente
* chiude e riapre sessioni in rapida successione (server TTL=300 s).
* - SE window.__s2sApipilot=false → salta subito.
* - MAI fallisce: sempre catch interno.
* @param {string} sessionIdShort Log tag "sessione #N"
* 🔧 Pre-fetch ApiPilot tools at every Realtime session start.
* - Server proxy: GET api/apipilot/tools?include_unauthorized=true
* - The global FETCH wrapper logs [FETCH:APIPILOT:TOOLS] in the console.
* - Local TTL 240 s avoids redundant calls during rapid session restarts
* (server TTL=300 s).
* - Skip immediately when window.__s2sApipilot=false.
* - Catch failures internally.
* @param {string} sessionIdShort Log tag "session #N"
* @returns {Promise<void>}
*/
async _prefetchApipilotTools(sessionIdShort) {
try {
// @ts-ignore: __s2sApipilot è settato da main.js dopo fetchConfig()
// @ts-ignore: main.js sets __s2sApipilot after fetchConfig().
const enabled = typeof window !== "undefined" && !!window.__s2sApipilot;
if (!enabled) {
_wsLog(2, "APIPILOT", `${sessionIdShort} feature flag apipilot=false sul server — salto prefetch strumenti`);
_wsLog(2, "APIPILOT", `${sessionIdShort} server feature flag apipilot=false — skipping tools prefetch`);
return;
}
const currentUser = _apipilotUserKey();
const now = Date.now();
const cached = _apipilotClientCache;
// HIT cache solo se: (1) ancora fresca TTL (2) stesso utente key (non cambio phone/token tra sessioni)
// Use the cache only within its TTL and for the same user key.
if (cached && (now - cached.at <= _APIPILOT_CLIENT_TTL_MS) && cached.key === currentUser) {
const ageSec = Math.round((now - cached.at) / 1000);
const n =
@@ -537,10 +526,10 @@ export class S2sWsRealtimeClient extends EventTarget {
: Array.isArray(cached.data && cached.data.tools)
? cached.data.tools.length
: "?";
_wsLog(1, "APIPILOT", `${sessionIdShort} HIT cache client strumenti (user=${currentUser}, age=${ageSec}s, N=${n}, TTL=${Math.round(_APIPILOT_CLIENT_TTL_MS/1000)}s)`);
_wsLog(1, "APIPILOT", `${sessionIdShort} HIT client tools cache (user=${currentUser}, age=${ageSec}s, N=${n}, TTL=${Math.round(_APIPILOT_CLIENT_TTL_MS/1000)}s)`);
return;
}
// Propaga header x-demo-session se disponibile in window (per auth fingerprint lato server S2S).
// Forward x-demo-session when available for the S2S server auth fingerprint.
/** @type {Record<string,string>} */
const extraHeaders = {};
try {
@@ -550,11 +539,11 @@ export class S2sWsRealtimeClient extends EventTarget {
extraHeaders["x-demo-session"] = window.__s2sDemoSessionToken;
}
} catch (_) { /* noop */ }
// Usa refresh=true SE è cambiato l'utente (cache.key diversa o null) oppure la cache è stata invalidata globalmente.
// Refresh when the user key changes or the cache is missing/expired.
const needRefresh = !cached || cached.key !== currentUser || (now - cached.at > _APIPILOT_CLIENT_TTL_MS);
const qsRefresh = needRefresh ? "&refresh=true" : "";
const t0 = performance.now();
_wsLog(0, "APIPILOT", `${sessionIdShort} MISS cache (user=${currentUser}, refresh=${needRefresh ? "Y" : "N"}) → GET api/apipilot/tools (proxy a S2S_APIPILOT_TOOLS_URL server)...`);
_wsLog(0, "APIPILOT", `${sessionIdShort} MISS cache (user=${currentUser}, refresh=${needRefresh ? "Y" : "N"}) → GET api/apipilot/tools (server proxy to S2S_APIPILOT_TOOLS_URL)...`);
const res = await fetch(`api/apipilot/tools?include_unauthorized=true${qsRefresh}`, {
method: "GET",
cache: "no-store",
@@ -567,11 +556,11 @@ export class S2sWsRealtimeClient extends EventTarget {
const j = await res.json().catch(() => ({}));
const ok = j && j.success && j.data;
if (!ok) {
throw new Error(`envelope non corretto: success=${Boolean(j && j.success)}`);
throw new Error(`invalid envelope: success=${Boolean(j && j.success)}`);
}
const tools = Array.isArray(j.data.tools) ? j.data.tools : [];
const count = typeof j.data.count === "number" ? j.data.count : tools.length;
// Aggiorna cache locale (in caso di hit la prossima sessione non chiamerà).
// Update the local cache so a subsequent session can reuse it.
_apipilotClientCache = { at: Date.now(), data: { count, tools: tools.slice(0, 100) }, key: currentUser };
const names = tools
.slice(0, 5)
@@ -580,16 +569,16 @@ export class S2sWsRealtimeClient extends EventTarget {
_wsLog(
1,
"APIPILOT",
`${sessionIdShort} READY strumenti APIPILOT caricati: N=${count}, t=${Math.round(dt)}ms, user=${currentUser} [primi 5: ${names || "(nessuno)"}]`
`${sessionIdShort} READY APIPILOT tools loaded: N=${count}, t=${Math.round(dt)}ms, user=${currentUser} [first 5: ${names || "(none)"}]`
);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
_wsLog(
3,
"APIPILOT",
`${sessionIdShort} prefetch FALLITO: ${msg} — diagnostica lato server log [APIPILOT-DISCOVERY]`
`${sessionIdShort} prefetch FAILED: ${msg} — check server logs [APIPILOT-DISCOVERY]`
);
// NON rilanciare: connect() deve comunque andare a buon fine.
// Do not rethrow: connect() must still succeed.
}
}
@@ -670,21 +659,21 @@ export class S2sWsRealtimeClient extends EventTarget {
const dur = (performance.now() - t0).toFixed(1);
if (response.status === 402) {
const body = await response.json().catch(() => ({}));
_wsLog(3, "SESSION", `/session HTTP 402: limite giornaliero esaurito (tier=${body?.tier ?? "?"}) dur=${dur} ms`);
_wsLog(3, "SESSION", `/session HTTP 402: daily limit exhausted (tier=${body?.tier ?? "?"}) dur=${dur} ms`);
throw _codedError("Daily conversation limit reached", "limit", { tier: body?.tier });
}
if (response.status === 503) {
const body = await response.json().catch(() => ({}));
if (body?.state === "at_capacity") {
_wsLog(2, "QUEUE:FULL", `/session HTTP 503 state=at_capacity (coda piena) dur=${dur} ms`);
_wsLog(2, "QUEUE:FULL", `/session HTTP 503 state=at_capacity (queue full) dur=${dur} ms`);
throw _codedError("The queue is full — try again shortly.", "queue-full");
}
_wsLog(3, "SESSION", `/session HTTP 503 (fallback sconosciuto) dur=${dur} ms`);
_wsLog(3, "SESSION", `/session HTTP 503 (unknown fallback) dur=${dur} ms`);
throw new Error("/session failed (503)");
}
if (!response.ok) {
const text = await response.text().catch(() => "");
_wsLog(3, "SESSION", `/session fallita HTTP ${response.status}: ${text.slice(0, 180)} dur=${dur} ms`);
_wsLog(3, "SESSION", `/session failed HTTP ${response.status}: ${text.slice(0, 180)} dur=${dur} ms`);
throw new Error(`/session failed (${response.status}): ${text}`);
}
const json = await response.json();
@@ -692,7 +681,7 @@ export class S2sWsRealtimeClient extends EventTarget {
_wsLog(
2,
"QUEUE:WAIT",
`/session → stato=queued queueId=${json.queue_id ?? "?"} position=${json.position ?? "?"} pollEvery=${json.poll_interval_s ?? "?"}s dur=${dur} ms`
`/session → state=queued queueId=${json.queue_id ?? "?"} position=${json.position ?? "?"} pollEvery=${json.poll_interval_s ?? "?"}s dur=${dur} ms`
);
return {
state: "queued",
@@ -724,7 +713,7 @@ export class S2sWsRealtimeClient extends EventTarget {
_wsLog(
2,
"QUEUE:POLL",
`Inizio poll coda queueId=${ticket.queueId.slice(0, 10)}… initialPos=${ticket.position} intervalMs=${intervalMs}`
`Start queue polling queueId=${ticket.queueId.slice(0, 10)}… initialPos=${ticket.position} intervalMs=${intervalMs}`
);
let pollNum = 0;
@@ -744,11 +733,11 @@ export class S2sWsRealtimeClient extends EventTarget {
if (response.status === 402) {
const body = await response.json().catch(() => ({}));
_wsLog(3, "QUEUE:LIMIT", `poll #${pollNum} HTTP 402 — limite giornaliero (tier=${body?.tier ?? "?"})`);
_wsLog(3, "QUEUE:LIMIT", `poll #${pollNum} HTTP 402 — daily limit (tier=${body?.tier ?? "?"})`);
throw _codedError("Daily conversation limit reached", "limit", { tier: body?.tier });
}
if (response.status === 404) {
_wsLog(3, "QUEUE:EXPIRED", `poll #${pollNum} HTTP 404 — ticket scaduto queueId=${this._queueId.slice(0, 10)}…`);
_wsLog(3, "QUEUE:EXPIRED", `poll #${pollNum} HTTP 404 — ticket expired queueId=${this._queueId.slice(0, 10)}…`);
throw _codedError("Queue timed out", "queue-expired");
}
if (!response.ok) continue; // 502/503 — transient, retry
@@ -763,7 +752,7 @@ export class S2sWsRealtimeClient extends EventTarget {
continue;
}
// Reached the front and claimed a slot.
_wsLog(1, "QUEUE:FRONT", `poll #${pollNum} — RAGGIUNTA la prima posizione. Assegno slot compute.`);
_wsLog(1, "QUEUE:FRONT", `poll #${pollNum} — REACHED the front. Assigning compute slot.`);
this._queueId = "";
return this._parseGrant(json);
}
@@ -881,9 +870,8 @@ export class S2sWsRealtimeClient extends EventTarget {
/** @param {string} connectUrl */
_openWebSocket(connectUrl) {
return new Promise((resolve, reject) => {
// Non loggiamo l'URL completo (potrebbe contenere session_token in
// chiaro nella querystring). Mostriamo solo host + path + un hash della
// parte segreta per audit senza leak token.
// Mask the full URL, which may contain a plaintext session_token.
// Show host + path and secret fingerprints for auditing without leaks.
let maskedUrl = connectUrl;
try {
const u = new URL(connectUrl, window.location.href);
@@ -893,7 +881,7 @@ export class S2sWsRealtimeClient extends EventTarget {
const isSecret = /token|session|auth|jwt|secret|key/i.test(lower) || String(v).length > 16;
secretPortions.push(`${k}=${isSecret ? `✂️…${String(v).slice(-4)} (${String(v).length} chars)` : v}`);
});
maskedUrl = `${u.protocol}//${u.host}${u.pathname} ? ${secretPortions.join(" & ") || "(nessun query param)"}`;
maskedUrl = `${u.protocol}//${u.host}${u.pathname} ? ${secretPortions.join(" & ") || "(no query parameters)"}`;
} catch {
// ignore
}
@@ -905,13 +893,13 @@ export class S2sWsRealtimeClient extends EventTarget {
const onceOpen = () => {
ws.removeEventListener("open", onceOpen);
ws.removeEventListener("error", onceErr);
_wsLog(1, "CONNECT", `WebSocket APERTO su ${maskedUrl} readyState=${ws.readyState}`);
_wsLog(1, "CONNECT", `WebSocket OPEN at ${maskedUrl} readyState=${ws.readyState}`);
resolve();
};
const onceErr = (e) => {
ws.removeEventListener("open", onceOpen);
ws.removeEventListener("error", onceErr);
_wsLog(3, "CONNECT", `WebSocket IMPOSSIBILE da aprire: event.type=${e?.type ?? "error"} url=${maskedUrl}`);
_wsLog(3, "CONNECT", `WebSocket FAILED to open: event.type=${e?.type ?? "error"} url=${maskedUrl}`);
reject(new Error(`WebSocket failed to open: ${e?.type ?? "error"}`));
};
ws.addEventListener("open", onceOpen);
@@ -1037,15 +1025,11 @@ export class S2sWsRealtimeClient extends EventTarget {
`tools=${toolNames || "none"}`
);
}
// Primo ack del server: se startup mode=TALK vogliamo il comportamento
// IDENTICO a quando arriva la frase di wake word "Hola Vocero".
// Quindi NON mandiamo shaped response.create con istruzioni di greet
// hardcodate, ma invece:
// 1. injectiamo "hola vocero" nella conversazione come se l'utente
// l'avesse detto (sendUserText → conversation.item.create user)
// 2. requestResponse() VUOTO → il modello risponde esattamente come
// nel codice wake-word in main.js.
// Stesso identico risultato, nessuna istruzione custom.
// On the first server acknowledgment, TALK startup should behave
// exactly like hearing the "Hola Vocero" wake word:
// 1. Inject "hola vocero" as a user turn through sendUserText().
// 2. Call requestResponse() without custom greeting instructions,
// matching the wake-word path in main.js.
if (this._startWithResponse && !this._initialResponseSent) {
this._initialResponseSent = true;
const phrase = this._startWithResponseText.trim() || "hola vocero";
@@ -1363,7 +1347,7 @@ export class S2sWsRealtimeClient extends EventTarget {
_wsLog(
(ev.code === 1000 || ev.code === 1001) ? 1 : (ev.code ? 3 : 2),
"CLOSE",
`WebSocket CHIUSO code=${ev.code ?? "?"} reason=${String(ev.reason ?? "(nessuna)").slice(0, 160)} wasClean=${ev.wasClean}`
`WebSocket CLOSED code=${ev.code ?? "?"} reason=${String(ev.reason ?? "(none)").slice(0, 160)} wasClean=${ev.wasClean}`
);
if (this._status === "closed" || this._status === "error") return;
if (ev.code === 1000) {
@@ -1395,13 +1379,12 @@ export class S2sWsRealtimeClient extends EventTarget {
},
};
// ───────────────────────────────────────────────────────────────────────────
// 2026-09-25 INIETTA API-PILOT HERRAMIENTAS DINÁMICAS (PASADA-1) nel session.update.
// 3 strati di assicurazione (in ordine di priorità crescente, dedup by marker):
// (A) main.js effectiveReasoningInstructions() ha già appeso la lista.
// (B) NOSTRO server.py proxy realtime ri-inietta (se routing lo attraversa).
// (C) QUI s2s-ws-client: ultimo punto di catch-all prima dell'invio al
// canale, GARANTISCE che la lista sia SEMPRE presente in istruzioni
// (anche se qualcuno fa updateSession() con text sovrascritto poco dopo).
// 2026-09-25 INJECT API-PILOT DYNAMIC TOOLS (PASS 1) in session.update.
// Three safeguards, deduplicated by marker:
// (A) main.js effectiveReasoningInstructions() appends the list.
// (B) The server.py Realtime proxy injects it when routing passes there.
// (C) This client is the final safeguard before sending instructions,
// including after a live updateSession() replaces the prompt.
{
const currentInstr = (typeof session.instructions === "string") ? session.instructions : "";
if (!instructionsAlreadyHasApipilotPass1(currentInstr)) {
@@ -1453,7 +1436,7 @@ export class S2sWsRealtimeClient extends EventTarget {
const session = { type: "realtime" };
if (patch.instructions) {
let instr = patch.instructions;
// ── 2026-09-25 catch-all apipilot PASS-1 anche per patch live. ──
// ── 2026-09-25 ApiPilot PASS-1 safeguard for live patches too. ──
if (typeof instr === "string" && !instructionsAlreadyHasApipilotPass1(instr)) {
try {
const info = typeof window !== "undefined" && typeof window.__s2sGetApipilotPass1ContextInfo === "function"
@@ -1776,8 +1759,8 @@ export class S2sWsRealtimeClient extends EventTarget {
/** @param {Record<string, unknown>} event */
_send(event) {
if (!this._ws || this._ws.readyState !== WebSocket.OPEN) return;
// Anti-spam bulk-rate limiter per eventi audio (~25 al secondo).
// Loggiamo un resoconto ogni 250 chunk (circa 10s a 25Hz) invece di ogni singolo.
// Throttle bulk audio event logs (~25 events per second).
// Log periodic summaries rather than every chunk.
if (event.type === "input_audio_buffer.append") {
_wsMicChunkCount += 1;
const now = performance.now();
@@ -1807,7 +1790,7 @@ export class S2sWsRealtimeClient extends EventTarget {
this._ws.send(payload || JSON.stringify(event));
return;
}
// Messaggi non bulk: log sempre [S2S-WS:SEND] con anteprima del payload.
// Always log non-bulk messages as [S2S-WS:SEND] with a payload preview.
let payloadLength = 0;
let stringifyMs = 0;
let payload = "";
s2s-reasoning-engine/.env.example
+7 -7
View File
@@ -1,7 +1,7 @@
# Configurazione ambiente per s2s-reasoning-engine
# Environment configuration for s2s-reasoning-engine
#
# Copia questo file in .env nella root del progetto o nella cartella s2s-reasoning-engine/
# e modifica i valori secondo il tuo deployment.
# Copy this file to .env in the project root or s2s-reasoning-engine directory
# and adjust the values for your deployment.
# ─────────────────────────────────────────────────────────────
# 1) LLM (OpenAI-compatible)
@@ -15,7 +15,7 @@ OPENAI_REASONING_TEMPERATURE=0.2
OPENAI_REASONING_MAX_TOKENS=512
# ─────────────────────────────────────────────────────────────
# 2) Aquachile (profilo agente, tool MCP, autenticazione)
# 2) Aquachile (agent profile, MCP tools, authentication)
# ─────────────────────────────────────────────────────────────
S2S_AGENT_PROFILE=https://aquachile.azurianlab.com/api/v1/agents/profile
S2S_USER_PHONE_AUTH_URL=https://aquachile.azurianlab.com/api/v1/auth/user-by-phone
@@ -32,8 +32,8 @@ NODE_RED_LOG_LEVEL=info
NODE_RED_CREDENTIAL_SECRET=change-me-in-production
# ─────────────────────────────────────────────────────────────
# 4) Integrazione con s2s UI (server.py)
# 4) S2S UI integration (server.py)
# ─────────────────────────────────────────────────────────────
# URL raggiungibile dal container Python della UI
# (se usi rete docker condivisa è http://s2s-reasoning:1880)
# URL reachable from the Python UI container
# (for a shared Docker network, use http://s2s-reasoning:1880).
S2S_REASONING_NODE_RED_URL=http://localhost:1881
s2s-reasoning-engine/Dockerfile
+10 -11
View File
@@ -1,30 +1,29 @@
# s2s-reasoning-engine: Node-RED 5.0 custom image
# Usa Node-RED 5.x ufficiale (immagine Alpine per dimensioni ridotte)
# Use the official Node-RED 5.x Alpine image to keep the image small.
FROM nodered/node-red:5.0.3
# Porta di default di Node-RED (esposta in reti containerizzate)
# Default Node-RED port, exposed to the container network.
EXPOSE 1880
# 1) Installa nodi aggiuntivi (a livello utente node-red nella cartella dati)
# - node-red-contrib-openai: integrazione nativa Chat Completions / Embeddings
# - node-red-contrib-http-logger: log HTTP opzionale
# - node-red-contrib-cache: cache in-memory per dati Aquachile (opzionale)
# - node-red-node-rbe: report-by-exception per deduplicare eventi
# 1) Install additional nodes in the Node-RED user data directory.
# - node-red-contrib-openai: native Chat Completions / Embeddings integration
# - node-red-contrib-cache: optional in-memory cache for Aquachile data
# - node-red-node-rbe: report-by-exception event deduplication
RUN cd /data && npm install --no-audit --no-fund \
node-red-contrib-openai@0.11.0 \
node-red-contrib-cache@1.1.1 \
node-red-node-rbe@0.5.0
# 2) Copia file di configurazione e flows iniziali
# 2) Copy the configuration and initial flows.
COPY settings.js /data/settings.js
COPY flows.json /data/flows.json
# 3) Imposta Node-RED per caricare flows statici + disabilita progetto e auth
# (per ambiente di sviluppo; in produzione abilitare adminAuth in settings.js)
# 3) Load static flows with projects and authentication disabled for development.
# Enable adminAuth in settings.js for production.
ENV NODE_RED_FLOWS_FILE=/data/flows.json
ENV NODE_RED_CREDENTIAL_SECRET=s2s-reasoning-dev-secret
ENV FLOWS=/data/flows.json
# Avvio Node-RED
# Start Node-RED.
CMD ["npm", "start", "--", "--userDir", "/data", "--settings", "/data/settings.js"]
s2s-reasoning-engine/flows.json
+10 -10
View File
@@ -4,7 +4,7 @@
"type": "tab",
"label": "S2S Reasoning",
"disabled": false,
"info": "# Flusso di reasoning S2S — FASE 1: LOG-ONLY\n\nModalità attuale: il flusso NON produce risposte, NON modifica il prompt.\nSi limita a:\n 1. Ricevere POST /api/reasoning dal server demo\n 2. Estrarre transcript, lingua, phoneNumber\n 3. Loggare la richiesta utente su console + sidebar debug\n 4. Risondere HTTP 200 con contratto standard e flag `log_only: true`\n\nContratto risposta (fase 1):\n { answer: \"\", log_only: true, intent: \"log_only\", usage: {prompt_tokens:0,completion_tokens:0,total_tokens:0}, latency_ms: <ms> }\n",
"info": "# S2S reasoning flow — phase 1: log only\n\nThis flow does not generate answers or change the prompt. It:\n 1. Receives POST /api/reasoning from the client server.\n 2. Extracts the transcript, language, and phoneNumber.\n 3. Logs the user request to the console and debug sidebar.\n 4. Returns HTTP 200 with the standard contract and `log_only: true`.\n\nResponse contract (phase 1):\n { answer: \"\", log_only: true, intent: \"log_only\", usage: {prompt_tokens:0,completion_tokens:0,total_tokens:0}, latency_ms: <ms> }\n",
"wires": []
},
{
@@ -28,8 +28,8 @@
"id": "parse-request",
"type": "function",
"z": "s2s-reasoning-tab",
"name": "Parse richiesta (log-only)",
"func": "// FASE 1 — LOG-ONLY: nessuna answer generata.\n// Valida e normalizza il payload in arrivo da server.py.\n// msg.req.body = { transcript, instructions, language, user_context, phoneNumber? }\n\nconst body = msg.req?.body || {};\nconst startedAt = Date.now();\n\nconst transcript = String(body.transcript || \"\").trim();\nconst instructions = String(body.instructions || \"\").trim();\nconst language = String(body.language || \"\").trim();\nconst userContext = String(body.user_context || body.user_context || \"\").trim();\nconst phoneNumber = String(body.phoneNumber || body.phone_number || msg.req?.query?.phoneNumber || \"\").trim();\n\nif (!transcript) {\n msg.statusCode = 400;\n msg.payload = { error: \"missing_transcript\", detail: \"transcript field is required\" };\n return [null, msg]; // uscita 2 = errore\n}\n\nconst detectedLang = language || (transcript.match(/[áéíóúñ¿¡]/i) ? \"es-CL\" : \"en-US\");\n\nmsg.state = {\n startedAt,\n transcript,\n instructionsLen: instructions.length,\n language: detectedLang,\n userContextLen: userContext.length,\n phoneNumber: phoneNumber || null,\n intent: \"log_only\",\n};\n\nmsg.payload = {\n transcript,\n language: detectedLang,\n phone: phoneNumber || \"none\",\n};\n\nreturn [msg, null];",
"name": "Parse request (log only)",
"func": "// Phase 1: log only; no answer is generated.\n// Validate and normalize the payload received from server.py.\n// msg.req.body = { transcript, instructions, language, user_context, phoneNumber? }\n\nconst body = msg.req?.body || {};\nconst startedAt = Date.now();\n\nconst transcript = String(body.transcript || \"\").trim();\nconst instructions = String(body.instructions || \"\").trim();\nconst language = String(body.language || \"\").trim();\nconst userContext = String(body.user_context || body.user_context || \"\").trim();\nconst phoneNumber = String(body.phoneNumber || body.phone_number || msg.req?.query?.phoneNumber || \"\").trim();\n\nif (!transcript) {\n msg.statusCode = 400;\n msg.payload = { error: \"missing_transcript\", detail: \"transcript field is required\" };\n return [null, msg]; // Output 2 is the error path.\n}\n\nconst detectedLang = language || (transcript.match(/[áéíóúñ¿¡]/i) ? \"es-CL\" : \"en-US\");\n\nmsg.state = {\n startedAt,\n transcript,\n instructionsLen: instructions.length,\n language: detectedLang,\n userContextLen: userContext.length,\n phoneNumber: phoneNumber || null,\n intent: \"log_only\",\n};\n\nmsg.payload = {\n transcript,\n language: detectedLang,\n phone: phoneNumber || \"none\",\n};\n\nreturn [msg, null];",
"outputs": 2,
"timeout": 0,
"noerr": 0,
@@ -52,8 +52,8 @@
"id": "log-transcript-console",
"type": "function",
"z": "s2s-reasoning-tab",
"name": "[LOG] richiesta utente",
"func": "// FASE 1 — Stampa la richiesta utente sul log server (Node-RED console)\n// e sul pannello debug sidebar.\n// NON produce alcuna risposta: quella arriva dal backend S2S nativo.\n\nconst state = msg.state || {};\nconst transcript = state.transcript || \"\";\nconst lang = state.language || \"\";\nconst phone = state.phoneNumber || \"none\";\nconst ts = new Date(state.startedAt || Date.now()).toISOString();\n\nconst line = `[S2S NODE_RED LOG_ONLY] ts=${ts} lang=${lang} phone=${phone} transcript=\"${transcript.replace(/\"/g, '\\\\\"')}\"`;\n\nnode.warn(line);\n\nmsg.logLine = line;\nreturn msg;",
"name": "[LOG] user request",
"func": "// Phase 1: log the user request to the Node-RED console and debug sidebar.\n// The native S2S backend generates the response; this node does not.\n\nconst state = msg.state || {};\nconst transcript = state.transcript || \"\";\nconst lang = state.language || \"\";\nconst phone = state.phoneNumber || \"none\";\nconst ts = new Date(state.startedAt || Date.now()).toISOString();\n\nconst line = `[S2S NODE_RED LOG_ONLY] ts=${ts} lang=${lang} phone=${phone} transcript=\"${transcript.replace(/\"/g, '\\\\\"')}\"`;\n\nnode.warn(line);\n\nmsg.logLine = line;\nreturn msg;",
"outputs": 1,
"timeout": 0,
"noerr": 0,
@@ -89,7 +89,7 @@
"type": "function",
"z": "s2s-reasoning-tab",
"name": "Build 200 (log_only)",
"func": "// FASE 1 — Costruisci la risposta HTTP 200.\n// Contratto: answer VUOTA + flag log_only=true.\n// Il client (main.js) vedrà log_only e NON interverrà:\n// - NON chiama cancelActiveResponse()\n// - NON usa appendAssistantText()\n// - NON usa speechSynthesis\n// => la risposta vocale arriva dal backend S2S nativo.\n\nconst state = msg.state || {};\nconst latencyMs = Math.max(0, Date.now() - (state.startedAt || Date.now()));\n\nmsg.statusCode = 200;\nmsg.headers = {\n \"Content-Type\": \"application/json; charset=utf-8\",\n \"Cache-Control\": \"no-store\",\n};\nmsg.payload = {\n answer: \"\",\n log_only: true,\n intent: \"log_only\",\n usage: {\n prompt_tokens: 0,\n completion_tokens: 0,\n total_tokens: 0,\n },\n latency_ms: latencyMs,\n mcp_tools_used: [],\n caller_profile_keys: [],\n debug: {\n phase: \"log_only_fase1\",\n language: state.language || \"\",\n phone: state.phoneNumber || null,\n transcript_len: (state.transcript || \"\").length,\n },\n};\n\nreturn msg;",
"func": "// Phase 1: build the HTTP 200 response with an empty answer and log_only=true.\n// The client leaves the native S2S response untouched: it does not call\n// cancelActiveResponse(), appendAssistantText(), or speechSynthesis.\n\nconst state = msg.state || {};\nconst latencyMs = Math.max(0, Date.now() - (state.startedAt || Date.now()));\n\nmsg.statusCode = 200;\nmsg.headers = {\n \"Content-Type\": \"application/json; charset=utf-8\",\n \"Cache-Control\": \"no-store\",\n};\nmsg.payload = {\n answer: \"\",\n log_only: true,\n intent: \"log_only\",\n usage: {\n prompt_tokens: 0,\n completion_tokens: 0,\n total_tokens: 0,\n },\n latency_ms: latencyMs,\n mcp_tools_used: [],\n caller_profile_keys: [],\n debug: {\n phase: \"log_only_phase1\",\n language: state.language || \"\",\n phone: state.phoneNumber || null,\n transcript_len: (state.transcript || \"\").length,\n },\n};\n\nreturn msg;",
"outputs": 1,
"timeout": 0,
"noerr": 0,
@@ -165,7 +165,7 @@
"type": "function",
"z": "s2s-reasoning-tab",
"name": "Parse system-response",
"func": "// Riceve dal server.py la risposta TTS emessa dal backend S2S nativo.\n// msg.req.body = { tts, response_id?, item_id?, client_ip?, role?, source_event?, user_transcript?, phone? }\n// FASE 1 LOG-ONLY: nessun side-effect oltre al log.\n\nconst body = msg.req?.body || {};\nconst startedAt = Date.now();\n\nconst tts = String(body.tts || body.text || body.response || \"\").trim();\nconst responseId = String(body.response_id || body.responseId || \"\").trim() || null;\nconst itemId = String(body.item_id || body.itemId || \"\").trim() || null;\nconst clientIp = String(body.client_ip || body.clientIp || body.ip || \"\").trim() || null;\nconst role = String(body.role || \"assistant\").trim() || \"assistant\";\nconst sourceEvent = String(body.source_event || body.sourceEvent || \"\").trim() || null;\nconst userTranscript = String(body.user_transcript || body.userTranscript || \"\").trim() || null;\nconst phone = String(body.phone || body.phoneNumber || \"\").trim() || null;\n\nif (!tts) {\n msg.statusCode = 400;\n msg.headers = { \"Content-Type\": \"application/json\" };\n msg.payload = { error: \"missing_tts\", detail: \"field 'tts' (or 'text'/'response') is required\" };\n return [null, msg];\n}\n\nmsg.state = {\n startedAt,\n kind: \"system_response_log\",\n role,\n tts,\n responseId,\n itemId,\n clientIp,\n sourceEvent,\n userTranscript,\n phone,\n};\n\nmsg.payload = { tts, role, source: sourceEvent, client: clientIp };\nreturn [msg, null];",
"func": "// Receive the native S2S backend's TTS response from server.py.\n// msg.req.body = { tts, response_id?, item_id?, client_ip?, role?, source_event?, user_transcript?, phone? }\n// Phase 1: log only; no other side effects.\n\nconst body = msg.req?.body || {};\nconst startedAt = Date.now();\n\nconst tts = String(body.tts || body.text || body.response || \"\").trim();\nconst responseId = String(body.response_id || body.responseId || \"\").trim() || null;\nconst itemId = String(body.item_id || body.itemId || \"\").trim() || null;\nconst clientIp = String(body.client_ip || body.clientIp || body.ip || \"\").trim() || null;\nconst role = String(body.role || \"assistant\").trim() || \"assistant\";\nconst sourceEvent = String(body.source_event || body.sourceEvent || \"\").trim() || null;\nconst userTranscript = String(body.user_transcript || body.userTranscript || \"\").trim() || null;\nconst phone = String(body.phone || body.phoneNumber || \"\").trim() || null;\n\nif (!tts) {\n msg.statusCode = 400;\n msg.headers = { \"Content-Type\": \"application/json\" };\n msg.payload = { error: \"missing_tts\", detail: \"field 'tts' (or 'text'/'response') is required\" };\n return [null, msg];\n}\n\nmsg.state = {\n startedAt,\n kind: \"system_response_log\",\n role,\n tts,\n responseId,\n itemId,\n clientIp,\n sourceEvent,\n userTranscript,\n phone,\n};\n\nmsg.payload = { tts, role, source: sourceEvent, client: clientIp };\nreturn [msg, null];",
"outputs": 2,
"timeout": 0,
"noerr": 0,
@@ -188,8 +188,8 @@
"id": "log-system-response",
"type": "function",
"z": "s2s-reasoning-tab",
"name": "[LOG] risposta sistema",
"func": "// Stampa la risposta del sistema (TTS dal backend nativo) sul log Node-RED.\n// Formato riga unica ricercabile con grep.\nconst state = msg.state || {};\nconst tts = state.tts || \"\";\nconst role = state.role || \"assistant\";\nconst src = state.sourceEvent || \"\";\nconst user = state.userTranscript || \"\";\nconst phone = state.phone || \"none\";\nconst ts = new Date(state.startedAt || Date.now()).toISOString();\n\nconst userShort = user.length > 120 ? user.slice(0, 117) + \"...\" : user;\nconst ttsShort = tts.length > 240 ? tts.slice(0, 237) + \"...\" : tts;\nconst esc = (s) => String(s || \"\").replace(/\\\\/g, \"\\\\\\\\\").replace(/\"/g, '\\\\\"');\n\nconst line = `[S2S NODE_RED SYSTEM_RESPONSE] ts=${ts} role=${role} phone=${esc(phone)} user=\"${esc(userShort)}\" tts=\"${esc(ttsShort)}\"`;\n\nnode.warn(line);\nmsg.logLine = line;\nreturn msg;",
"name": "[LOG] system response",
"func": "// Log the native backend's TTS response to Node-RED.\n// Use one searchable line per response.\nconst state = msg.state || {};\nconst tts = state.tts || \"\";\nconst role = state.role || \"assistant\";\nconst src = state.sourceEvent || \"\";\nconst user = state.userTranscript || \"\";\nconst phone = state.phone || \"none\";\nconst ts = new Date(state.startedAt || Date.now()).toISOString();\n\nconst userShort = user.length > 120 ? user.slice(0, 117) + \"...\" : user;\nconst ttsShort = tts.length > 240 ? tts.slice(0, 237) + \"...\" : tts;\nconst esc = (s) => String(s || \"\").replace(/\\\\/g, \"\\\\\\\\\").replace(/\"/g, '\\\\\"');\n\nconst line = `[S2S NODE_RED SYSTEM_RESPONSE] ts=${ts} role=${role} phone=${esc(phone)} user=\"${esc(userShort)}\" tts=\"${esc(ttsShort)}\"`;\n\nnode.warn(line);\nmsg.logLine = line;\nreturn msg;",
"outputs": 1,
"timeout": 0,
"noerr": 0,
@@ -225,7 +225,7 @@
"type": "function",
"z": "s2s-reasoning-tab",
"name": "Build 200 (ack)",
"func": "// Risposta minimale: 200 OK + ricevuto=true.\n// Non ritorna dati.\nconst state = msg.state || {};\nconst latencyMs = Math.max(0, Date.now() - (state.startedAt || Date.now()));\nmsg.statusCode = 200;\nmsg.headers = {\n \"Content-Type\": \"application/json; charset=utf-8\",\n \"Cache-Control\": \"no-store\",\n};\nmsg.payload = {\n ricevuto: true,\n log_only: true,\n intent: \"system_response_log\",\n latency_ms: latencyMs,\n};\nreturn msg;",
"func": "// Return a minimal HTTP 200 acknowledgment with received=true.\n// No response data is returned.\nconst state = msg.state || {};\nconst latencyMs = Math.max(0, Date.now() - (state.startedAt || Date.now()));\nmsg.statusCode = 200;\nmsg.headers = {\n \"Content-Type\": \"application/json; charset=utf-8\",\n \"Cache-Control\": \"no-store\",\n};\nmsg.payload = {\n received: true,\n log_only: true,\n intent: \"system_response_log\",\n latency_ms: latencyMs,\n};\nreturn msg;",
"outputs": 1,
"timeout": 0,
"noerr": 0,
s2s-reasoning-engine/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "s2s-reasoning-engine",
"version": "1.0.0",
"description": "Node-RED 5.0 reasoning engine per S2S/Aquachile",
"description": "Node-RED 5.0 reasoning engine for S2S/Aquachile",
"main": "settings.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
s2s-reasoning-engine/settings.js
+21 -28
View File
@@ -1,26 +1,24 @@
/**
* Node-RED 5.0 settings for s2s-reasoning-engine — LOCAL INSTALL (non Docker).
* Node-RED 5.0 settings for s2s-reasoning-engine — local installation.
* File: <project_root>/s2s-reasoning-engine/.node-red/settings.js
* oppure <project_root>/s2s-reasoning-engine/settings.js (master)
* or <project_root>/s2s-reasoning-engine/settings.js (source)
*
* Copiato automaticamente in .node-red/ da start.sh se mancante o più vecchio.
* start.sh copies this into .node-red/ when missing or older than the source.
*/
const path = require("path");
const fs = require("fs");
// userDir è sempre <s2s-reasoning-engine>/.node-red (lo script start.sh)
// start.sh uses <s2s-reasoning-engine>/.node-red as its local user directory.
const ENGINE_ROOT = path.resolve(__dirname, "..");
const LOCAL_NODE_MODULES = path.join(ENGINE_ROOT, "node_modules");
const USERDIR_NODE_MODULES = path.join(__dirname, "node_modules");
// ── Fix per editor "tslib.es6.js 404 / net::ERR_ABORTED" ─────────────────
// Installazioni npm locali flat (default npm 3+) spostano tslib in
// node_modules/tslib/ (root) invece che dentro
// @node-red/editor-client/node_modules/tslib/. L'editor di Node-RED 5
// carica `/tslib.es6.js` dalla root HTTP: se la cartella nested non esiste,
// torna 404. Qui aggiorniamo editor-client/public/ con i file tslib mancanti
// a ogni avvio (sopravvive a npm reinstall se tslib è in root).
// Fix the editor's "tslib.es6.js 404 / net::ERR_ABORTED" error.
// Flat npm installations place tslib in the root node_modules/tslib directory
// rather than @node-red/editor-client/node_modules/tslib. The Node-RED 5 editor
// loads /tslib.es6.js from the HTTP root. Refresh the editor's public files at
// startup so the workaround survives dependency reinstallation.
(function _installTslibVendor() {
try {
const tslibRoot = path.join(LOCAL_NODE_MODULES, "tslib");
@@ -49,13 +47,10 @@ module.exports = {
httpRequestTimeout: 120000,
httpRequestLegacyMsgOverrides: true,
// ── Native HTTP helper per function node (SENZA externalModules) ─────
// NOTA: non usare functionExternalModules: true — in Node-RED v5 quel
// flag può bloccare il caricamento palette (es. "37/41 nodes loading"
// durante la verifica di tutti i pkg npm).
// Al suo posto usiamo functionGlobalContext: i moduli core Node.js sono
// require'd QUI nel settings.js (prima dell'avvio NR) e poi i function
// node li recuperano via global.get("_core_https") ecc.
// Native HTTP helpers for function nodes, without externalModules.
// functionExternalModules can stall palette loading in Node-RED 5.
// Load core Node.js modules here before startup and expose them through
// functionGlobalContext, accessed with global.get("_core_https") etc.
functionModuleRequireAllowList: ["https","http","url","path","util","fs","os","buffer","zlib","stream","crypto"],
functionGlobalContext: {
_core_https: require("https"),
@@ -63,9 +58,9 @@ module.exports = {
_core_url: require("url"),
},
// adminAuth: {...} ← scommentare e valorizzare in produzione
// adminAuth: {...} // Uncomment and configure for production.
// HTTPS (opzionale in produzione)
// HTTPS (optional for production).
// https: {
// key: require("fs").readFileSync(path.join(__dirname, "privkey.pem")),
// cert: require("fs").readFileSync(path.join(__dirname, "fullchain.pem"))
@@ -82,20 +77,18 @@ module.exports = {
},
csrf: false,
// ── IMPORTANTE: caricamento nodi extra ─────────────────────────────
// Node-RED 5 carica i plugin NODI da:
// 1) {userDir}/node_modules (nostro .node-red/node_modules)
// 2) {nodesDir} (installazione root del progetto)
// Additional node loading: Node-RED 5 looks for plugins in:
// 1) {userDir}/node_modules (our .node-red/node_modules)
// 2) {nodesDir} (the project-root installation)
// 3) npm global
// Senza nodesDir i pacchetti installati nella ROOT del progetto non
// vengono trovati.
// Set nodesDir so packages installed at the project root are discovered.
nodesDir: LOCAL_NODE_MODULES,
externalModules: {
autoInstall: true, // auto-installa nodi mancanti al boot
autoInstall: true, // Install missing nodes at startup.
autoInstallRetry: 3,
palette: {
allowInstall: true, // permette Install dall'editor
allowInstall: true, // Allow installation from the editor.
allowUpload: true,
modules: ["node-red-contrib-openai", "node-red-contrib-cache", "node-red-node-rbe"],
allowList: ["node-red-contrib-*", "node-red-node-*"],
s2s-reasoning-engine/start.sh
+15 -15
View File
@@ -1,12 +1,12 @@
#!/usr/bin/env bash
# =============================================================
# start.sh — Avvia Node-RED 5.0 in locale per s2s-reasoning-engine
# start.sh — Run Node-RED 5.0 locally for s2s-reasoning-engine
#
# Uso:
# ./start.sh # porta default 1881
# PORT=1900 ./start.sh # porta custom
# LOG_LEVEL=debug ./start.sh # log più dettagliato
# DEBUG=nodes:* ./start.sh # debug Node-RED interno
# Usage:
# ./start.sh # default port 1881
# PORT=1900 ./start.sh # custom port
# LOG_LEVEL=debug ./start.sh # detailed logging
# DEBUG=nodes:* ./start.sh # internal Node-RED diagnostics
# Stop: Ctrl+C
# =============================================================
set -euo pipefail
@@ -17,8 +17,8 @@ PARENT_DIR="$(dirname "$ROOT_DIR")"
cd "$ROOT_DIR"
# ----------------------------------------------------------
# 1) Carica env: prima locale .env (se esiste) poi padre ../.env
# Il file locale sovrascrive il padre se c'è lo stesso var.
# 1) Load the local .env first, then the parent ../.env when present.
# The parent file takes precedence when both define the same variable.
# ----------------------------------------------------------
set -a
if [ -f "$ROOT_DIR/.env" ]; then
@@ -34,7 +34,7 @@ fi
set +a
# ----------------------------------------------------------
# 2) Parametri runtime Node-RED
# 2) Node-RED runtime parameters
# ----------------------------------------------------------
PORT="${PORT:-${NODE_RED_PORT:-1881}}"
export PORT
@@ -43,11 +43,11 @@ export NODE_RED_LOG_LEVEL="${LOG_LEVEL:-${NODE_RED_LOG_LEVEL:-info}}"
export NODE_RED_CREDENTIAL_SECRET="${NODE_RED_CREDENTIAL_SECRET:-s2s-reasoning-local-dev}"
export FLOWS="${FLOWS:-$ROOT_DIR/.node-red/flows.json}"
# UserDir locale (tutto dentro la cartella corrente, NO ~/.node-red)
# Local user directory; keep state here rather than in ~/.node-red.
USERDIR="${USERDIR:-$ROOT_DIR/.node-red}"
SETTINGS="${SETTINGS:-$USERDIR/settings.js}"
# Assicura che la userDir esista e i file base siano presenti.
# Create the user directory and refresh its configuration from the source files.
mkdir -p "$USERDIR"
if [ ! -f "$SETTINGS" ] || [ "$ROOT_DIR/settings.js" -nt "$SETTINGS" ]; then
cp -f "$ROOT_DIR/settings.js" "$SETTINGS"
@@ -58,15 +58,15 @@ if [ ! -f "$FLOWS" ] || [ "$ROOT_DIR/flows.json" -nt "$FLOWS" ]; then
echo "[init] copied flows.json -> $FLOWS"
fi
# Verifica node_modules installati
# Check that the local Node-RED dependencies are installed.
if [ ! -d "$ROOT_DIR/node_modules/node-red" ]; then
echo "[init] node_modules non trovati. Avvio npm install..."
echo "[init] node_modules was not found. Running npm install..."
npm install --no-audit --no-fund
fi
echo ""
echo "╔════════════════════════════════════════════════════════════╗"
echo "║ s2s-reasoning-engine — Node-RED v5 locale ║"
echo "║ s2s-reasoning-engine — Local Node-RED v5 ║"
echo "╠════════════════════════════════════════════════════════════╣"
echo "║ Editor: http://localhost:${PORT}/ ║"
echo "║ API: http://localhost:${PORT}/api/reasoning ║"
@@ -76,7 +76,7 @@ echo "╚═══════════════════════
echo ""
# ----------------------------------------------------------
# 3) Avvio Node-RED
# 3) Start Node-RED
# ----------------------------------------------------------
exec "$ROOT_DIR/node_modules/.bin/node-red" \
--userDir "$USERDIR" \
speech_to_speech/LLM/README.md
+5 -138
View File
@@ -1,139 +1,6 @@
# LLM Summary
# Language model handlers
## Available LLM backends (`--llm_backend`)
Runtime-supported values in `s2s_pipeline.py`:
- `transformers` → `language_model.py` (Transformers backend)
- `mlx-lm` → `language_model.py` (MLX backend)
- `responses-api` → `responses_api_language_model.py`
## Usage
### 1) Transformers (`--llm_backend transformers`)
- Handler: `LanguageModelHandler`
- Typical use: local GPU/CPU inference using Hugging Face Transformers
- Backend-specific args prefix: `--llm_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
```bash
python s2s_pipeline.py \
--llm_backend transformers \
--model_name Qwen/Qwen3-4B-Instruct-2507 \
--llm_device cuda \
--llm_torch_dtype float16 \
--llm_gen_max_new_tokens 128
```
Common options:
- `--llm_gen_min_new_tokens`
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`
### 2) MLX-LM (`--llm_backend mlx-lm`)
- Handler: `LanguageModelHandler`
- Typical use: Apple Silicon local inference
- Backend-specific args prefix: same as Transformers (`--llm_*`)
```bash
python s2s_pipeline.py \
--llm_backend mlx-lm \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
--llm_device mps \
--llm_gen_max_new_tokens 128
```
Common options:
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`
### 3) OpenAI-compatible API (`--llm_backend responses-api`)
- Handler: `ResponsesApiModelHandler`
- Typical use: remote model serving via OpenAI-compatible endpoints
- Backend-specific args prefix: `--responses_api_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
```bash
python s2s_pipeline.py \
--llm_backend responses-api \
--model_name gpt-5.4-mini \
--responses_api_api_key YOUR_API_KEY \
--responses_api_base_url https://api.example.com/v1 \
--responses_api_stream true
```
Common options:
- `--chat_size`
- `--init_chat_prompt`
- `--user_role`
## LLM Behavior
When STT is set to language auto-detection (`--language auto`), LLM handlers can receive `(text, language_code)` and prepend a language control instruction like:
- `Please reply to my message in <language>.`
This helps the assistant respond in the detected language. The behavior is opt-in via `--enable_lang_prompt` (shared across all backends); it defaults to `False`.
## Setup
### CUDA setup
```bash
python s2s_pipeline.py \
--llm_backend transformers \
--model_name microsoft/Phi-3-mini-4k-instruct
```
### Local Mac setup
```bash
python s2s_pipeline.py \
--local_mac_optimal_settings \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16
```
`--local_mac_optimal_settings` already sets `--llm_backend mlx-lm` and will default the model to `mlx-community/Qwen3-4B-Instruct-2507-bf16` if not overridden.
### Realtime (OpenAI-compatible) setup
Run the server in realtime mode, then connect with the realtime client:
```bash
# 1. Start the pipeline in realtime mode
python s2s_pipeline.py \
--mode realtime \
--llm_backend mlx-lm \
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
--ws_host 0.0.0.0 \
--ws_port 8765
# 2. Connect with the realtime client
python listen_and_play_realtime.py --host 127.0.0.1 --port 8765
```
Or with `--local_mac_optimal_settings` on Apple Silicon:
```bash
python s2s_pipeline.py \
--local_mac_optimal_settings \
--mode realtime \
--ws_host 0.0.0.0 \
--ws_port 8765
```
### Remote API setup
```bash
python s2s_pipeline.py \
--llm_backend responses-api \
--model_name gpt-5.4-mini \
--responses_api_api_key YOUR_API_KEY
```
This directory contains local Transformers/MLX and remote Responses/Chat
Completions handlers. See [language model backends](../../docs/speech/llm.md)
for configuration and [realtime API](../../docs/speech/realtime-api.md) for the
tool-call and conversation flow.
speech_to_speech/LLM/chat_completions_language_model.py
+1 -1
View File
@@ -196,7 +196,7 @@ class ChatCompletionsApiModelHandler(BaseOpenAICompatibleHandler):
if self.stream:
create_kwargs["stream_options"] = {"include_usage": True}
# Svuota i parametri dei tool per evitare il blocco 400 di NVIDIA NIM
# Remove tool parameters to avoid NVIDIA NIM's HTTP 400 response.
create_kwargs.pop("tools", None)
create_kwargs.pop("tool_choice", None)
speech_to_speech/LLM/lm_output_processor.py
+15 -16
View File
@@ -4,7 +4,7 @@ LLM Output Processor
Intercepts LLM output to:
1. Extract tool calls and send them via text_output_queue
2. Forward clean text to TTS pipeline
3. 2026-09-25: SANITIZE APIPILOT internal markers before clients / TTS (anti-leak tecnico)
3. Sanitize ApiPilot internal markers before sending output to clients or TTS
"""
from __future__ import annotations
@@ -17,16 +17,15 @@ from queue import Queue
from speech_to_speech.baseHandler import BaseHandler
from speech_to_speech.pipeline.events import AssistantTextEvent, ResponseFailedEvent, TokenUsageEvent
# 2026-09-25 SANITIZE APIPILOT CALL/RESULT markers in LLM output BEFORE:
# (a) AssistantTextEvent viene mandato ai client WS (e loggato come <<< TEXT:/ASSISTANT:)
# (b) TTSInput viene mandato a Qwen3-TTS (che lo pronuncerebbe ad alta voce).
# Formati osservati reali oggi (upstream Vocero start_pipeline_rag):
# 1) <<<APIPILOT_CALL:NAME=tool,ARGUMENTS=json>>> standard 3x
# Sanitize ApiPilot CALL/RESULT markers before emitting AssistantTextEvent
# to WebSocket clients or TTSInput to Qwen3-TTS, which would speak the markers.
# Observed formats from upstream Vocero start_pipeline_rag:
# 1) <<<APIPILOT_CALL:NAME=tool,ARGUMENTS=json>>> standard triple brackets
# 2) <APIPILOT_CALL:NAME=get_user_briefing,ARGUMENTS={"format":"voice"}
# singolo '<' '>' + args inventati da AI (format=voice)
# 3) <<<APIPILOT_CALL:...}> \n>> spezzato multi-riga
# 4) >>>APIPILOT_CALL:...<<< invertito (errore sintassi AI)
# 5) <<<APIPILOT_RESULT:PASS2>>> + blocco result.
# single brackets and model-generated arguments
# 3) <<<APIPILOT_CALL:...}> \n>> split across lines
# 4) >>>APIPILOT_CALL:...<<< inverted brackets
# 5) <<<APIPILOT_RESULT:PASS2>>> followed by a result block
_APIPILOT_CALL_NORM = re.compile(
r"[<]{1,3}\s*APIPILOT_CALL:NAME=([A-Za-z0-9_.-]+),ARGUMENTS=(.*?)[>]{1,3}",
re.IGNORECASE | re.DOTALL,
@@ -64,7 +63,7 @@ def _sanitize_apipilot_before_client_and_tts(text: str) -> tuple[str, bool, str]
if _APIPILOT_RESULT.search(new_text):
new_text = _APIPILOT_RESULT.sub(" ", new_text)
found_result = True
# safety catchall frammenti marker senza wrapper (se marker troncato mid-stream)
# Catch marker fragments without a wrapper, including mid-stream truncation.
low = new_text.lower()
if "apipilot" in low and ("call" in low or "result" in low or "name=" in low or "arguments=" in low):
new_text = re.sub(r"(?i)\bAPIPILOT(?:_CALL|_RESULT)?\b", " ", new_text)
@@ -184,8 +183,8 @@ class LMOutputProcessor(BaseHandler[LLMOut, TTSIn]):
logger.debug(f"LM processor: text='{lm_output.text}', tools={lm_output.tools}")
# 2026-09-25 SANITIZE SINGOLO PUNTO CENTRALE: PRIMA di AssistantTextEvent (client/log)
# E PRIMA di TTSInput (Qwen3-TTS che pronuncerebbe marker APIPILOT CALL).
# Sanitize centrally before AssistantTextEvent and TTSInput so neither
# clients nor Qwen3-TTS receive internal ApiPilot markers.
clean_text = lm_output.text
any_sanitized = False
sanitize_tag = ""
@@ -193,14 +192,14 @@ class LMOutputProcessor(BaseHandler[LLMOut, TTSIn]):
clean_text, any_sanitized, sanitize_tag = _sanitize_apipilot_before_client_and_tts(clean_text)
if any_sanitized:
logger.warning(
"[LM-OUTPUT-APIPILOT-SANITIZE] %s — tool=%s rawLen=%d cleanLen=%d (rimosso / sostituito con placeholder spagnolo PRIMA di client + TTS).",
"[LM-OUTPUT-APIPILOT-SANITIZE] %s — tool=%s rawLen=%d cleanLen=%d (removed or replaced with the Spanish placeholder before client/TTS output).",
sanitize_tag or "-",
(sanitize_tag if sanitize_tag else "marker"),
len(lm_output.text or ""),
len(clean_text or ""),
)
# evita che lo stesso chunk (marker solo) arrivi a TTS come "" → Hello.
# Se dopo sanitize testo vuoto MA avevamo trovato marker → placeholder.
# Use the placeholder when sanitization removes the entire chunk,
# avoiding empty text that would trigger the TTS greeting fallback.
if not clean_text.strip():
clean_text = _APIPILOT_SPANISH_PLACEHOLDER
speech_to_speech/STT/README.md
+4 -162
View File
@@ -1,163 +1,5 @@
# STT Summary
# Speech-to-text handlers
This document summarizes the Speech-to-Text (STT) implementations in the `STT/` folder, including language support, language abbreviations, and usage in `s2s_pipeline.py`.
## Available STT Modes (`--stt`)
- `whisper` → `STT/whisper_stt_handler.py`
- `whisper-mlx` → `STT/lightning_whisper_mlx_handler.py`
- `mlx-audio-whisper` → `STT/mlx_audio_whisper_handler.py`
- `faster-whisper` → `STT/faster_whisper_handler.py`
- `parakeet-tdt` → `STT/parakeet_tdt_handler.py`
- `paraformer` → `STT/paraformer_handler.py`
## Language Support by Handler
### 1) Whisper (`--stt whisper`)
- Handler: `WhisperSTTHandler`
- Language input flag: `--language` (from shared Whisper args)
- Supports fixed language (e.g. `en`) or `auto`
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- Detects language from token output
- If detected language is outside the supported list, it falls back to the previous language
### 2) Lightning Whisper MLX (`--stt whisper-mlx`)
- Handler: `LightningWhisperSTTHandler`
- Uses same shared `--language` argument as Whisper
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- If `--language auto`, model auto-detects each utterance
- If detected language is unsupported, falls back to last supported language
### 3) MLX Audio Whisper (`--stt mlx-audio-whisper`)
- Handler: `MLXAudioWhisperSTTHandler`
- Model flag: `--mlx_audio_whisper_model_name`
- Language still comes from shared `--language` flag (wired in pipeline)
- Internal supported language list:
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
- Behavior:
- Uses fixed language unless `--language auto`
- Falls back to last known supported language when needed
### 4) Faster-Whisper (`--stt faster-whisper`)
- Handler: `FasterWhisperSTTHandler`
- Language flag: `--faster_whisper_stt_gen_language`
- Default language: `en`
- Note:
- This handler passes generation kwargs directly to `faster_whisper.WhisperModel.transcribe(...)`
- Effective language coverage depends on selected Faster-Whisper/OpenAI Whisper model
### 5) Parakeet TDT (`--stt parakeet-tdt`)
- Handler: `ParakeetTDTSTTHandler`
- Language flag: `--parakeet_tdt_language` (optional)
- Supports auto language detection when language not specified
- Declared supported language list (25 European languages):
- `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `pl`, `ru`, `uk`, `cs`, `sk`, `hu`, `ro`, `bg`, `hr`, `sl`, `sr`, `da`, `no`, `sv`, `fi`, `et`, `lv`, `lt`
- Backend behavior:
- On macOS/MPS: MLX (`mlx-community/parakeet-tdt-0.6b-v3`)
- On CUDA/CPU: nano-parakeet (`nvidia/parakeet-tdt-0.6b-v3`)
### 6) Paraformer (`--stt paraformer`)
- Handler: `ParaformerSTTHandler`
- Model flag: `--paraformer_stt_model_name`
- Default model: `paraformer-zh`
- No dedicated language flag in current args class
- Practical support:
- Depends on selected FunASR model checkpoint
- Default setup is Chinese-oriented (`zh`)
## Language Abbreviations (ISO-style codes seen in STT handlers)
| Code | Language |
|---|---|
| `en` | English |
| `fr` | French |
| `es` | Spanish |
| `zh` | Chinese |
| `ja` | Japanese |
| `ko` | Korean |
| `hi` | Hindi |
| `de` | German |
| `pt` | Portuguese |
| `pl` | Polish |
| `it` | Italian |
| `nl` | Dutch |
| `ru` | Russian |
| `uk` | Ukrainian |
| `cs` | Czech |
| `sk` | Slovak |
| `hu` | Hungarian |
| `ro` | Romanian |
| `bg` | Bulgarian |
| `hr` | Croatian |
| `sl` | Slovenian |
| `sr` | Serbian |
| `da` | Danish |
| `no` | Norwegian |
| `sv` | Swedish |
| `fi` | Finnish |
| `et` | Estonian |
| `lv` | Latvian |
| `lt` | Lithuanian |
| `auto` | Per-utterance automatic language detection |
## Usage Examples
### Whisper (Transformers)
```bash
python s2s_pipeline.py --stt whisper --language en
python s2s_pipeline.py --stt whisper --language auto
```
### Whisper MLX (LightningWhisperMLX)
```bash
python s2s_pipeline.py --stt whisper-mlx --language auto --device mps
```
### MLX Audio Whisper
```bash
python s2s_pipeline.py --stt mlx-audio-whisper \
--mlx_audio_whisper_model_name mlx-community/whisper-large-v3-turbo \
--language auto
```
### Faster-Whisper
```bash
python s2s_pipeline.py --stt faster-whisper \
--faster_whisper_stt_model_name large-v3 \
--faster_whisper_stt_gen_language en
```
### Parakeet TDT
```bash
python s2s_pipeline.py --stt parakeet-tdt --parakeet_tdt_device auto
python s2s_pipeline.py --stt parakeet-tdt --parakeet_tdt_language de
```
With live transcription (MLX or CUDA/nano-parakeet backend):
```bash
python s2s_pipeline.py --stt parakeet-tdt \
--enable_live_transcription \
--live_transcription_update_interval 0.25
```
### Paraformer
```bash
python s2s_pipeline.py --stt paraformer --paraformer_stt_model_name paraformer-zh
```
This directory contains Whisper, Parakeet, and Paraformer handlers and live
transcription helpers. See [speech-to-text backends](../../docs/speech/stt.md)
for supported modes, language codes, and configuration.
speech_to_speech/TTS/README.md
+4 -154
View File
@@ -1,155 +1,5 @@
# TTS Summary
# Text-to-speech handlers
## Available TTS modes (`--tts`)
Runtime-supported values in `s2s_pipeline.py`:
- `chatTTS` → `chatTTS_handler.py`
- `facebookMMS` → `facebookmms_handler.py`
- `pocket` → `pocket_tts_handler.py`
- `kokoro` → `kokoro_handler.py`
- `qwen3` → `qwen3_tts_handler.py`
Deprecated TTS implementations, including MeloTTS, live in [`../../../archive/TTS`](../../../archive/TTS) and are no longer wired into `s2s_pipeline.py`.
## Usage
### 1) ChatTTS (`--tts chatTTS`)
Primary args prefix: `--chat_tts_*`
```bash
python s2s_pipeline.py \
--tts chatTTS \
--chat_tts_device cuda \
--chat_tts_stream true \
--chat_tts_chunk_size 512
```
### 2) Facebook MMS (`--tts facebookMMS`)
Primary args prefix: `--facebook_mms_*` plus `--tts_language`
```bash
python s2s_pipeline.py \
--tts facebookMMS \
--facebook_mms_device cuda \
--tts_language en
```
This handler maps STT language codes (e.g. `en`, `fr`, `es`) to MMS model suffixes (e.g. `eng`, `fra`, `spa`) and reloads the model on language changes.
### 3) Pocket TTS (`--tts pocket`)
Primary args prefix: `--pocket_tts_*`
```bash
python s2s_pipeline.py \
--tts pocket \
--pocket_tts_voice jean \
--pocket_tts_device cpu \
--pocket_tts_sample_rate 16000
```
Available preset voices include:
`alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`, `azelma`.
### 4) Kokoro (`--tts kokoro`)
Primary args prefix: `--kokoro_*`
```bash
python s2s_pipeline.py \
--tts kokoro \
--kokoro_device auto \
--kokoro_voice bm_fable \
--kokoro_lang_code b
```
Behavior:
- Uses MLX backend on Apple Silicon (`mlx-community/Kokoro-82M-bf16`)
- Uses native kokoro pipeline otherwise (`hexgrad/Kokoro-82M`)
- Can auto-switch voice/language based on STT language code mapping
### 5) Qwen3-TTS (`--tts qwen3`)
Primary args prefix: `--qwen3_tts_*`
```bash
python s2s_pipeline.py \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_device cuda \
--qwen3_tts_backend ggml \
--qwen3_tts_speaker Aiden \
--qwen3_tts_language auto \
--qwen3_tts_non_streaming_mode True
```
Behavior:
- Uses `faster-qwen3-tts` on non-macOS platforms, defaulting to the GGML backend. Pass `--qwen3_tts_backend torch` to use the CUDA-graphs backend instead.
- Uses `mlx-audio` on Apple Silicon and auto-maps `Qwen/...` model IDs to `mlx-community/...`, defaulting to the `6bit` MLX variant unless the model name already pins a suffix.
- Supports MLX quantization overrides on Apple Silicon via `--qwen3_tts_mlx_quantization bf16|4bit|6bit|8bit`.
- Keeps the existing voice-clone/custom-voice/voice-design handler flow intact.
- Defaults to the CustomVoice model with speaker `Aiden`, so no reference audio is required. Voice-clone/base models can still use `--qwen3_tts_ref_audio`.
Install notes for Linux GGML:
- The default PyPI `qwentts-cpp-python` wheel targets CUDA 12.8.
- If that wheel does not match your CUDA runtime, install one of the Hugging Face wheelhouse builds before installing `speech-to-speech`.
```bash
pip install "qwentts-cpp-python==0.3.1+cu130" \
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130
pip install speech-to-speech
```
Available wheelhouse directories include `cu124`, `cu128`, `cu130`, and `cpu`.
Example for Apple Silicon using the default 6-bit MLX variant:
```bash
python s2s_pipeline.py \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_speaker Aiden
```
You can override the default and select `bf16`, `4bit`, or `8bit` explicitly:
```bash
python s2s_pipeline.py \
--tts qwen3 \
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--qwen3_tts_mlx_quantization 4bit \
--qwen3_tts_speaker Aiden
```
To benchmark the Apple Silicon MLX variants side by side:
```bash
.venv/bin/python benchmark_tts.py \
--handlers qwen3 \
--iterations 3 \
--qwen3_mlx_quantizations bf16 4bit 6bit 8bit
```
This will run separate benchmark entries for `qwen3[bf16]`, `qwen3[4bit]`, `qwen3[6bit]`, and `qwen3[8bit]`.
## Setup
### Low-latency GPU setup
```bash
python s2s_pipeline.py \
--stt whisper \
--tts pocket
```
### Apple Silicon setup
```bash
python s2s_pipeline.py \
--local_mac_optimal_settings
```
`--tts pocket` and `--tts kokoro` are also valid options on macOS.
This directory contains ChatTTS, Facebook MMS, Pocket, Kokoro, and Qwen3-TTS
handlers. See [text-to-speech backends](../../docs/speech/tts.md) for configuration,
voice selection, and the optional reference-audio example.
speech_to_speech/TTS/qwen3_tts_handler.py
+16 -19
View File
@@ -13,16 +13,15 @@ import re
import tempfile
import unicodedata
# 2026-09-25 SANITIZE APIPILOT MARKERS PRIMA DI TTS (anti-leak pronunciato da Qwen3).
# Lato server S2S Python abbiamo gia' gate + sanitize ma come DOPPIA RETE
# di sicurezza sanitizziamo DIRETTAMENTE il testo che entra in Qwen3-TTS.
# Formati osservati reali oggi:
# (a) <<<APIPILOT_CALL:NAME=tool,ARGUMENTS=json>>> standard 3 parentesi
# Sanitize ApiPilot markers directly before TTS as an additional safeguard
# alongside the S2S relay's audio gating and sanitization.
# Observed formats:
# (a) <<<APIPILOT_CALL:NAME=tool,ARGUMENTS=json>>> standard triple brackets
# (b) <APIPILOT_CALL:NAME=tool,ARGUMENTS={"format":"voice"}
# singolo < > (caso user oggi)
# (c) <<<APIPILOT_CALL:...}> \n>> spezzato piu' righe
# (d) >>>APIPILOT_CALL:...<<< invertito
# Regola: CALL -> placeholder spagnolo (frase naturale utente), RESULT -> strip.
# single brackets
# (c) <<<APIPILOT_CALL:...}> \n>> split across lines
# (d) >>>APIPILOT_CALL:...<<< inverted brackets
# Replace CALL with a natural Spanish utterance; strip RESULT markers.
_APIPILOT_CALL_NORMAL_RE = re.compile(
r"[<]{1,3}\s*APIPILOT_CALL:NAME=([A-Za-z0-9_.-]+),ARGUMENTS=(.*?)[>]{1,3}",
re.IGNORECASE | re.DOTALL,
@@ -39,7 +38,7 @@ _APIPILOT_SPANISH_PLACEHOLDER = "Voy a consultar tus datos, un momento por favor
def _tts_sanitize_apipilot_markers(text: str) -> tuple[str, bool, str]:
"""Rimuove marker APIPILOT dal testo TTS.
"""Remove ApiPilot markers from TTS text.
Returns (sanitized_text, found_any, log_tag).
"""
@@ -48,7 +47,7 @@ def _tts_sanitize_apipilot_markers(text: str) -> tuple[str, bool, str]:
found_call = False
found_result = False
tool_name = ""
# 1) marker CALL standard + invertito -> placeholder spagnolo
# 1) Replace standard/inverted CALL markers with the Spanish placeholder.
def _repl_call(m: re.Match) -> str:
nonlocal found_call, tool_name
found_call = True
@@ -62,11 +61,11 @@ def _tts_sanitize_apipilot_markers(text: str) -> tuple[str, bool, str]:
text2 = _APIPILOT_CALL_NORMAL_RE.sub(_repl_call, text)
if not found_call:
text2 = _APIPILOT_CALL_INVERTED_RE.sub(_repl_call, text2)
# 2) marker RESULT -> strip silenzioso (non sono pronunciabili)
# 2) Strip RESULT markers silently; they must not be spoken.
if _APIPILOT_RESULT_RE.search(text2):
found_result = True
text2 = _APIPILOT_RESULT_RE.sub(" ", text2)
# 3) safety: FRAMMENTI marker singoli che non hanno wrapper completo
# 3) Catch marker fragments without a complete wrapper.
low = text2.lower()
if "apipilot" in low and ("call" in low or "result" in low):
import re as _re_sf
@@ -782,18 +781,16 @@ class Qwen3TTSHandler(BaseHandler[TTSIn, TTSOut]):
coalesced_text, _language_code, _saw_end_of_response = self._coalesce_pending_tts_input(tts_input)
raw_text = coalesced_text or "Hello."
# 2026-09-25 SANITIZE APIPILOT PRIMA DI TTS (doppia rete sicurezza):
# Anche se lato server S2s Python abbiamo gia' gate audio drop + sanitize,
# Qwen3-TTS puo' ricevere marker per race / nuovi formati. Quindi QUI
# eliminiamo / sostituiamo con placeholder PRIMA di chiamare generate().
# A race or a new marker format can bypass relay sanitization. Strip or
# replace markers again before passing text to the TTS model.
text, found_marker, sanitize_tag = _tts_sanitize_apipilot_markers(raw_text)
if found_marker:
logger.warning(
"[TTS-APIPILOT-SANITIZE] marker=%s rawLen=%d sanitizedLen=%d → sostituito con placeholder spagnolo / strip.",
"[TTS-APIPILOT-SANITIZE] marker=%s rawLen=%d sanitizedLen=%d → replaced with the Spanish placeholder or stripped.",
sanitize_tag, len(raw_text or ""), len(text or ""),
)
if not text.strip():
# Safety se il marker era l'unica cosa: ripetiamo placeholder invece di Hello.
# Use the Spanish placeholder when the marker was the entire utterance.
text = _APIPILOT_SPANISH_PLACEHOLDER
model_type = self._model_type()
speech_to_speech/api/openai_realtime/README.md
+5 -215
View File
@@ -1,216 +1,6 @@
## Realtime Engine -- High-Level Architecture
# Realtime API implementation
The server runs as a handler thread inside the existing pipeline. A FastAPI/uvicorn instance serves the WebSocket endpoint at `/v1/realtime`, while the same queue-driven handler chain (VAD, STT, LLM, TTS) processes audio in parallel threads.
```mermaid
flowchart LR
subgraph client [Client]
WS["WebSocket /v1/realtime"]
end
subgraph server [Realtime Server]
Router["WebSocket Router"]
Service["RealtimeService"]
Config["RuntimeConfig"]
end
subgraph pipeline [Pipeline Threads]
VAD["VAD"]
STT["STT"]
TN["TranscriptionNotifier"]
LLM["LLM"]
Proc["LMOutputProcessor"]
TTS["TTS"]
end
WS -- "client events (JSON)" --> Router
Router -- "parse / dispatch" --> Service
Service -- "PCM chunks" --> VAD
Service -- "session config" --> Config
Config -. "read by" .-> VAD & LLM & TTS
VAD -- "speech segments" --> STT
STT -- "transcript" --> TN
TN -- "transcript + notify" --> LLM
LLM -- "text + tools" --> Proc
Proc -- "clean text" --> TTS
TTS -- "PCM audio" --> Router
Proc -- "assistant_text / tools" --> Router
VAD -- "speech_started/stopped" --> Router
TN -- "transcription events" --> Router
Router -- "server events (JSON)" --> WS
```
**Key flow:**
1. **Inbound audio**: Client sends `input_audio_buffer.append` with base64 PCM. `RealtimeService` decodes, resamples to 16 kHz, splits into 512-sample chunks, and puts them on the `recv_audio_chunks_queue` for VAD.
2. **Speech detection**: VAD detects speech boundaries and emits `speech_started` / `speech_stopped` events on the `text_output_queue`. Full utterance audio goes to STT.
3. **Transcription**: STT output passes through `TranscriptionNotifier`, which taps the transcript for `transcription.delta` / `transcription.completed` events before forwarding to the LLM.
4. **Generation**: The LLM generates text (and optional tool calls). `LMOutputProcessor` splits the output: clean text goes to TTS, and `assistant_text` + tool call dicts go to the `text_output_queue`.
5. **Outbound audio**: TTS writes PCM chunks to `send_audio_chunks_queue`. The router's async `_send_loop` drains both queues, encoding PCM as `response.output_audio.delta` events and translating internal messages into protocol events.
6. **Session config**: `session.update` events deep-merge into `RuntimeConfig`, which is a shared Pydantic model read by VAD (turn detection thresholds), LLM (instructions, tools), and TTS (voice) at processing time.
---
## Supported OpenAI Realtime Events
### Client -> Server
| Event | Description |
|---|---|
| `input_audio_buffer.append` | Stream base64 PCM audio. Decoded, resampled to 16 kHz, and chunked for the VAD. |
| `session.update` | Deep-merge session config (instructions, tools, voice, turn detection, audio format). |
| `conversation.item.create` | Inject `input_text` or `function_call_output` into the LLM context without triggering generation. |
| `response.create` | Trigger LLM generation. Supports per-response `instructions` and `tool_choice` overrides. |
| `response.cancel` | Cancel the in-progress response and re-enable listening. |
### Server -> Client
| Event | Description |
|---|---|
| `session.created` | Sent on connection with current session config. |
| `error` | Protocol errors (`session_limit_reached`, `unknown_or_invalid_event`, `invalid_session_type`, `conversation_already_has_active_response`, etc.) |
| `input_audio_buffer.speech_started` | VAD detected user speech. |
| `input_audio_buffer.speech_stopped` | End of user speech segment. |
| `conversation.item.created` | Acknowledges injected `input_text` from `conversation.item.create`. |
| `conversation.item.input_audio_transcription.delta` | Streaming partial transcript (when live transcription is enabled). |
| `conversation.item.input_audio_transcription.completed` | Final transcript for the user turn (with duration usage). |
| `response.created` | Emitted on the first outbound audio chunk (response is `in_progress`). |
| `response.output_audio.delta` | Base64 PCM audio chunk from TTS. |
| `response.output_audio.done` | Audio stream complete for the current output item. |
| `response.output_audio_transcript.done` | Full assistant text transcript for the turn. |
| `response.function_call_arguments.done` | Tool call with `call_id`, `name`, and JSON `arguments`. |
| `response.done` | Response finished (`completed`, `cancelled` with reason `turn_detected` or `client_cancelled`). |
---
## Tool Calling Design
Tool calling works through two distinct paths depending on the LLM backend, but both converge to the same wire protocol for the client.
### Local LLM path (`LanguageModelHandler` -- transformers / mlx-lm)
Tools defined in `session.update` are converted to `FunctionTool` objects. Each tool's JSON Schema `parameters` are turned into a Python `inspect.Signature` (via `signature_from_schema`), and `to_code_prompt()` renders a human-readable `def name(...): """docstring"""` block.
These tool signatures are injected into the system prompt using a Jinja2 template (`tool_prompt.py`) that instructs the model to wrap tool calls in `<code>...</code>` delimiters:
```
<code>
function_name(arg_name_1=value1, arg_name_2='string_value')
</code>
```
After generation, `_extract_tools` uses a regex to find `<code>` blocks, then `extract_function_calls_from_text` parses each `name(kwargs)` call and validates it against registered tools. Valid calls become `ResponseFunctionToolCall` dicts with generated `call_id`s.
### OpenAI API path (`ResponsesApiModelHandler`)
Tools are passed natively as the `tools=` parameter to `client.responses.create`. The API returns structured `function_call` items directly -- no prompt engineering or regex parsing needed. Per-response `tool_choice` overrides from `response.create` are supported.
### Common output path
Both handlers yield `(text, language_code, tools)` tuples. `LMOutputProcessor` forwards clean text to TTS and puts `{"type": "assistant_text", "text": ..., "tools": [...]}` on the `text_output_queue`. The router's `_send_loop` translates these into:
- `response.output_audio_transcript.done` for the text
- `response.function_call_arguments.done` for each tool call
### Tool result flow
1. Client executes the tool and sends `conversation.item.create` with `type: "function_call_output"` and `output: "<result>"`
2. `RealtimeService` appends the tool output to the chat context and emits `conversation.item.created`; this does not trigger generation.
3. If the tool result needs to be spoken to the user, such as camera/search/data results, the client sends `response.create` to trigger follow-up generation.
4. For fire-and-forget robot actions such as dance, emotion, head movement, stop, or idle tools, the client can stop after `conversation.item.created`; the assistant should already have spoken the natural lead-in before the tool call.
---
## Interruption Handling
Barge-in (user speaks while the assistant is playing audio) is handled cooperatively between the VAD, the `_send_loop`, and the LLM/TTS handlers via a shared `CancelScope` object (`cancel_scope.py`).
### CancelScope design
`CancelScope` replaces the old two-signal pattern (`cancel_response` Event + `discard_stale_output` boolean) with a single object that manages:
- **Generation counter** (`cancel_scope.generation`): pipeline threads (LLM, TTS) capture the current generation at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token. When `cancel()` is called, the generation increments and all prior generations become stale -- no timing games required.
- **Discard flag** (`cancel_scope.discarding`): set by `cancel()`, checked by the async `_send_loop` to drop output from superseded generations that arrives between `cancel()` and `response_done()`. Cleared by `response_done(generation)` (only when the sentinel's generation matches the discarded or current one -- sentinels from unrelated older generations are ignored), by `new_response()` on an explicit `response.create`, or by `reset()` on session claim/release.
Pipeline output is **generation-tagged**: `AudioOutput` chunks and `AssistantTextEvent`s carry a `cancel_generation` field stamped by the handler that produced them. The send loop's `_generation_is_discardable` drops an item if its generation is stale, or if `discarding` is set and the item is not from the current generation. Output from the *current* generation always passes through, so a fresh response is never swallowed by a lingering discard window (e.g. a superseded speculative turn whose TTS never emitted a `__RESPONSE_DONE__` sentinel).
```mermaid
sequenceDiagram
participant User
participant VAD
participant SendLoop as _send_loop
participant LLM
participant TTS
participant Client
Note over TTS,Client: Response active or pending (in_response / response_pending)
User->>VAD: speaks
VAD->>SendLoop: speech_started on text_output_queue
SendLoop->>Client: input_audio_buffer.speech_started
SendLoop->>Client: response.done (status=cancelled, reason=turn_detected)
SendLoop->>SendLoop: cancel_scope.cancel() (gen++ & discarding=True)
SendLoop->>SendLoop: flush output_queue + text_output_queue
SendLoop->>SendLoop: response_playing.clear()
LLM->>LLM: is_stale(gen) → True, aborts generation
TTS->>TTS: is_stale(gen) → True, aborts generation
TTS->>SendLoop: __RESPONSE_DONE__ (tagged with gen)
SendLoop->>SendLoop: cancel_scope.response_done(gen) (discarding=False)
Note over VAD,Client: Pipeline is now processing the new user utterance
```
**Step by step:**
1. **VAD detects speech**: puts a `SpeechStartedEvent` on `text_output_queue`.
2. **`_send_loop` processes text events first** (priority over audio): translates `speech_started` into protocol events. If an active response was in progress, `RealtimeService.dispatch_pipeline_event` emits `response.output_audio.done` + `response.done` with `status="cancelled"` and `reason="turn_detected"`.
3. **Cancel + queue flush**: if a response is active (`in_response`) *or* pending (`response_pending` -- accepted `response.create`, no audio yet), and interrupts are enabled (see step 4), the send loop calls `cancel_scope.cancel()` (increments generation, enables discard), clears `response_pending`, drains `output_queue` (preserving `__RESPONSE_DONE__` sentinels) and `text_output_queue` (preserving user-side events: `speech_stopped`, partial/completed transcriptions, token usage), then clears `response_playing`.
4. **Interrupt gating**: the cancel only fires if the `SpeechStartedEvent.interrupt_response` flag is set *and* the session config allows it (`turn_detection.interrupt_response`, read via `RuntimeConfig.interrupt_response_enabled`, default true). When disabled, user speech during a response is transcribed but the response keeps playing.
5. **LLM/TTS cancellation**: handlers capture `gen = cancel_scope.generation` at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token, aborting early when stale.
6. **Discard guard**: while `cancel_scope.discarding` is True, the send loop drops audio chunks and assistant text whose `cancel_generation` is not current (see `_generation_is_discardable` above). The guard clears when a `__RESPONSE_DONE__` with a matching generation arrives (via `cancel_scope.response_done(gen)`), or when an explicit `response.create` starts a new response (`cancel_scope.new_response()`).
7. **Client-initiated cancel**: `response.cancel` calls `cancel_scope.cancel()` (only if a response was active), flushes both queues with the same preservation rules, triggers `finish_response(status="cancelled", reason="client_cancelled")`, re-enables `should_listen`, and clears `response_playing`.
8. **Spurious cancel safety**: if no response is active, `cancel_scope.cancel()` is not called, preventing the discard guard from being set without a `__RESPONSE_DONE__` to clear it.
---
## Testing
### Local LLM with Transformers
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend transformers \
--tts kokoro \
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
### Local LLM with MLX-LM
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend mlx-lm \
--tts kokoro \
--model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" \
--llm_device mps \
--llm_torch_dtype float16 \
--enable_live_transcription
```
### Remote LLM with OpenAI-compatible API
```bash
uv run speech-to-speech \
--mode realtime \
--stt parakeet-tdt \
--llm_backend responses-api \
--tts kokoro \
--model_name "openai/gpt-oss-20b:groq" \
--responses_api_base_url "https://router.huggingface.co/v1" \
--responses_api_api_key "$HF_TOKEN" \
--responses_api_stream \
--enable_live_transcription
```
This directory implements the OpenAI Realtime-compatible WebSocket service,
session configuration, event handlers, and pipeline routing. See the
[realtime API guide](../../../docs/speech/realtime-api.md) for architecture,
supported events, tool calling, cancellation, and startup examples.
speech_to_speech/api/openai_realtime/handlers/session.py
+1 -1
View File
@@ -30,7 +30,7 @@ class SessionHandler(RealtimeBaseHandler):
"""
s = event.session
# Forza il bypass della voce inviata dal browser
# Ignore the voice supplied by the browser.
if hasattr(s, "voice") and s.voice is not None:
s.voice = None
elif isinstance(s, dict) and "voice" in s:
speech_to_speech/api/openai_realtime/service.py
+5 -6
View File
@@ -277,11 +277,10 @@ class RealtimeService:
try:
return model_cls.model_validate(raw) # type: ignore[return-value]
except ValidationError as e:
# ── Retrocompatibilità session.update ─────────────────────────────────
# Il formato standard OpenAI Realtime GA annida instructions/tools/tool_choice
# DENTRO "session": { ... }. Per tollerare anche client che li mandano
# alla RADICE (es. bug prima della Fase 2 MCP), ricostruiamo qui il payload
# corretto e riproviamo la validazione. Funziona SOLO per session.update.
# Backward-compatible session.update parsing.
# OpenAI Realtime GA nests instructions/tools/tool_choice in "session".
# Accept older clients that send them at the root by rebuilding the
# nested payload and retrying validation, only for session.update.
if event_type == "session.update":
try:
raw_dict = dict(raw) if not isinstance(raw, dict) else raw
@@ -290,7 +289,7 @@ class RealtimeService:
normalized_inner = dict(session_inner)
else:
normalized_inner = {}
# Promuovi campi noti dalla root → inner se mancanti (fallback flat)
# Promote known root fields into the inner session when missing.
SESSION_KNOWN_TOP_LEVEL = (
"instructions",
"tools",
speech_to_speech/arguments_classes/chat_tts_arguments.py
+1 -1
View File
@@ -14,6 +14,6 @@ class ChatTTSHandlerArguments:
chat_tts_chunk_size: int = field(
default=512,
metadata={
"help": "Sets the size of the audio data chunk processed per cycle, balancing playback latency and CPU load.. Default is 512。."
"help": "Sets the size of the audio data chunk processed per cycle, balancing playback latency and CPU load. Default is 512."
},
)
start_ui.sh
+7 -7
View File
@@ -13,32 +13,32 @@ fi
S2S_PROXY_TARGET="${S2S_PROXY_TARGET:-ws://ai-server:12345/v1/realtime}"
if ! command -v python3 >/dev/null 2>&1; then
echo "Errore: python3 non trovato."
echo "Error: python3 was not found."
exit 1
fi
if [ ! -d "$VENV_DIR" ]; then
echo "Creazione ambiente virtuale in .venv..."
echo "Creating the virtual environment in .venv..."
python3 -m venv "$VENV_DIR"
fi
source "$VENV_DIR/bin/activate"
if [ ! -f "$REQ_FILE" ]; then
echo "Errore: file requirements non trovato: $REQ_FILE"
echo "Error: requirements file was not found: $REQ_FILE"
exit 1
fi
if ! python -c "import fastapi, uvicorn, httpx, huggingface_hub, dotenv" >/dev/null 2>&1; then
echo "Installazione dipendenze da s2s-client/requirements.txt..."
echo "Installing dependencies from s2s-client/requirements.txt..."
python -m pip install -r "$REQ_FILE"
fi
export S2S_PROXY_TARGET
echo "💻 Avvio dell'interfaccia S2S Client UI (Porta $UI_PORT)..."
echo "🔌 Proxy realtime verso $S2S_PROXY_TARGET"
echo " Premi Ctrl+C in questo terminale per fermarla."
echo "💻 Starting the S2S client UI (port $UI_PORT)..."
echo "🔌 Realtime proxy target: $S2S_PROXY_TARGET"
echo " Press Ctrl+C in this terminal to stop it."
echo "----------------------------------------------------"
cd "$ROOT_DIR"