valenti/vocero-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consolidate English documentation under root docs

Browse Source
This commit is contained in:
alvaro
2026-10-05 10:11:35 -03:00
parent b91762a927
commit 6b8c27fd76
38 changed files with 2261 additions and 5594 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/client/architecture.md
+98
View File
@@ -0,0 +1,98 @@
# Client architecture
The [browser client](../../s2s-client) combines a voice UI with a FastAPI server.
The speech pipeline and optional Node-RED reasoning service run separately.
Runtime configuration comes from environment variables and the root `.env`.
```mermaid
flowchart LR
Browser[Browser UI]
Server[Client FastAPI server]
Speech[Realtime speech backend]
Reasoning[Node-RED or HTTP reasoning provider]
Tools[ApiPilot backend]
Browser -->|HTTP configuration and proxies| Server
Browser <-->|WebSocket via local proxy| Server
Server <-->|Realtime events and audio| Speech
Browser <-->|Optional direct connection| Speech
Server -->|Optional transcript reasoning| Reasoning
Server -->|Tool discovery and execution| Tools
```
## Connection modes
`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
session/queue proxy, or the user can enter a direct URL in settings. See
[usage](usage.md) for the configuration matrix.
The local realtime proxy relays audio and events in both directions. It also
patches selected `session.update` instructions, handles ApiPilot marker-based
tool execution, filters internal text, and can suppress speculative/tool-marker
audio. It is not an unconditional byte-for-byte tunnel.
## Source map
| Source | Responsibility |
|---|---|
| [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes |
| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning |
| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar |
| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion |
| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll |
| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs |
| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers |
| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog |
| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction |
| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state |
| [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier |
| [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation |
| [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging |
## Server routes
| Route | Purpose |
|---|---|
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
| `POST /api/reasoning` | Node-RED or OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
| `GET /api/user/phone` | Configured caller profile lookup |
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
| `POST /api/search` | Serper web search proxy |
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
| `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation |
The root serves the UI. `/docs` remains the FastAPI API explorer, while project
documentation files are served in the `/docs/` subdirectories. Markdown remains
source text, and directories without `index.html` are not browsable pages.
Static HTML, JS and CSS
responses receive `Cache-Control: no-store` during development.
## Audio path
1. The browser captures microphone audio with echo cancellation, noise
suppression and automatic gain-control settings.
2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at 24 kHz is decoded and sent to
[audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers,
resamples and fades playback edges. Input speech can clear playback for barge-in.
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
registration structure intact when editing them. The speech backend's thread
and cancellation model is documented in [realtime API](../speech/realtime-api.md).
## Reasoning and context
The browser triggers external reasoning on completed user transcripts only in
`OPENAI` or `NODE_RED` mode. Node-RED `log_only=true` responses leave native speech
output running; an answer-producing response uses the external-answer path.
The server can independently forward assistant text to Node-RED for paired
conversation logs. These HTTP operations supplement the realtime audio path.
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
[tool calling](tool-calling.md) for the conditions and limitations of each path.
docs/client/context-injection.md
+132
View File
@@ -0,0 +1,132 @@
# Context injection
Context reaches the assistant through browser-built instructions, server-patched
realtime session instructions, and HTTP reasoning context. The implementation is
in [server.py](../../s2s-client/server.py),
[main.js](../../s2s-client/main.js), and the
[WebSocket client](../../s2s-client/ws/s2s-ws-client.js).
## Independent controls
`S2S_INJECT_PROFILE` defaults to false and gates server loading of caller,
episode, report and prior-conversation context. It also controls whether the
browser appends caller profile text to its realtime instructions.
It is not a master switch for every context source. Operator-supplied RAG text
and configured ApiPilot discovery work independently. `S2S_INJECT_TODAY`
defaults to true; date and voice-output guards are built on paths that invoke
instruction augmentation or HTTP context assembly. A WebSocket session with no
profile/episode/report/summary or ApiPilot block skips proxy augmentation.
Direct browser connections bypass server-side WebSocket injection entirely.
| Setting | Default | Effect |
|---|---|---|
| `S2S_INJECT_PROFILE` | false | Enable server-resolved caller, episode and history context |
| `S2S_INJECT_REPORTS` | false | Allow scheduled-report fetch/injection; also requires the profile gate |
| `S2S_REPORTS_INJECT_MODE` | `minimal` | `minimal`: labels/dates; `full`: scripts; `off`/`none`: no report prompt block |
| `S2S_REPORTS_QUICK_HINT_MAX` | 5 | Keep at most this many newest reports, clamped to 1–50 |
| `S2S_INJECT_TODAY` | true | Add the current server-local date guard when its context builder runs |
| `S2S_DEBUG_PROMPT` | false | Log complete prompt/context text instead of only summaries |
## Context sources
| Block | Trigger and source | Size/behavior |
|---|---|---|
| Podcast episode | `episode_id` page/query/body value and profile gate | Up to 24,000 characters of serialized episode content |
| Caller profile | `phoneNumber` and `VOCERO_USER_PHONE_AUTH_URL` | Up to 1,600 serialized characters |
| Scheduled reports | `S2S_REPORT_LIST`, profile gate and report gate | Newest N labels/dates by default; full mode includes scripts with a 12,000-character block limit |
| Latest conversation | `S2S_LATEST_SUMMARY_TEXT`, or `S2S_LATEST_SUMMARY` with a phone number | Up to 1,600 characters; static text overrides HTTP fetch |
| Custom RAG | Operator applies text through the RAG dialog | Up to 12,000 characters; appended in browser instructions and HTTP `user_context` |
| ApiPilot discovery | Configured `S2S_APIPILOT_TOOLS_URL` | Tool names/descriptions/auth requirements and call-marker rules, independent of the profile gate |
### Episode
The current episode fetch uses the deployment-specific URL:
```text
https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}
```
It follows redirects with a 15-second timeout, accepts JSON or plain text,
unwraps a `result` field, and includes arbitrary episode fields. The prompt
labels it as the primary reference for the user's current episode. The endpoint
is hardcoded in the fetch helper rather than selected through an environment
variable.
### Caller profile and login
For caller-specific context, the proxy can first post
`{"phoneNumber":"+56912345678"}` to `S2S_LOGIN_BY_PHONE`, extract a bearer token,
and forward it to subsequent profile/history requests. Profile retrieval uses
`GET VOCERO_USER_PHONE_AUTH_URL?phoneNumber=...` with a 10-second timeout and
accepts an object, optionally wrapped in `result`.
Encode the leading plus sign in page links, for example
`?phoneNumber=%2B56912345678`. Caller information is also fetched through
`/api/user/phone` for the UI. An available profile can update live browser
instructions when profile injection is enabled.
### Scheduled reports
`S2S_INJECT_REPORTS=false` skips report fetching and prompt content, including
the table of contents. When enabled, the report endpoint may be called with or
without a phone number. The resolver keeps dictionary entries, reads
`sourceScheduleLabel` or `sourceSchedule.label`, reads `date` or
`sourceSchedule.date`, and optionally keeps `podcastScript` in full mode.
Entries are sorted newest first using ISO/date/timestamp handling and an
ISO-like string fallback. Empty/unparseable dates sort last. The newest-item
limit applies before constructing the prompt block. `minimal` exposes only the
labels and dates; `full` explicitly opts into script content.
### Latest conversation summary
`S2S_LATEST_SUMMARY_TEXT` takes priority over `S2S_LATEST_SUMMARY` and needs no
phone number or network request. Literal `\n` sequences in the environment value
become newlines. Otherwise the proxy fetches
`S2S_LATEST_SUMMARY?phoneNumber=...` and accepts text or JSON with an optional
`result` wrapper. Both sources pass through the same cap and history wrapper.
### Custom RAG
The operator can paste reference material and explicitly apply it to the
session. The browser appends it to instructions even with profile injection
disabled, includes it in HTTP `user_context`, and updates an open realtime
session. This context does not require an external retrieval endpoint.
## Application timing and order
At local WebSocket connect, the server resolves enabled context and stores it in
`session_ctx`. Each client `session.update` then passes through
`_patch_client_session_update_with_profile()`. Other client frame types are not
modified by this helper. The broader relay still performs ApiPilot handling and
text/audio filtering.
The current augmentation function **keeps base instructions first**, then
appends surviving voice-output guards, date context, episode, profile, reports,
and summary blocks. ApiPilot discovery is appended separately. If a block type
is already present in the base, deduplication may omit its server candidate.
This describes string assembly, not a guarantee of model attention or priority.
Every HTTP reasoning request rebuilds enabled context. Its `user_context`
assembly orders voice guard, date, episode, profile, reports, summary, ApiPilot
and browser-provided context, then deduplicates recognized types and strips
internal markers. Server instruction augmentation remains separate, so avoid
assuming the WebSocket and HTTP prompts have identical ordering.
Context markers support internal deduplication and diagnostics; ordinary
profile/report/episode/history sentinels are stripped before final augmentation.
ApiPilot call/catalog markers have their own handling. Treat the source helpers
as the reference for the exact final payload.
## Failure behavior and inspection
Optional context-fetch errors are logged and the affected context is skipped.
The speech conversation normally continues without that reference material.
This does not mean all API calls ignore errors: missing reasoning configuration,
invalid tool arguments and tool authorization can return explicit errors.
Inspect `/api/config` for active gates, the actual outbound `session.update` in
browser DevTools, and `[INJECT-*]`/`[DEBUG-PROMPT]` server logs. Full prompt
logging includes caller data and conversation text. See
[troubleshooting](troubleshooting.md) for a step-by-step diagnostic flow.
docs/client/design.md
+90
View File
@@ -0,0 +1,90 @@
# Design language
This guide describes the current browser UI. Use
[style.css](../../s2s-client/style.css) as the token/selector reference and
[index.html](../../s2s-client/index.html) for the theme-specific structure.
Earlier documentation described a monochrome-only dark UI; the current design
also uses aurora gradients, glass surfaces, and a separate light theme.
## Themes and tokens
`S2S_THEME=white` selects the white/blue interface. Empty or `dark` selects the
dark aurora interface. The browser sets `html[data-theme]` after configuration
loads; an initial visibility guard prevents the wrong theme flashing on first
paint. Extend the existing theme selectors when adding components.
| Token group | Dark theme values/purpose |
|---|---|
| Canvas | `--bg: #07080d`; translucent `--bg-elev` and `--bg-elev-2` surfaces |
| Glass | `--glass` and `--glass-strong` for translucent floating elements |
| Text | `--text`, `--text-dim`, `--text-faint` |
| Borders | `--border` and `--border-strong` hairlines |
| Brand | `--brand: #8b5cf6`, `--brand-2: #ec4899`, `--brand-3: #22d3ee` |
| Radius | `--radius-sm: 10px`, `--radius-md: 16px`, `--radius-lg: 26px`, `--radius-xl: 34px` |
| Shadows | `--shadow-soft`, `--shadow-glow-brand`, `--shadow-glass` |
Prefer existing variables and `color-mix` over new unrelated colors. The dark
theme's orb is the strongest visual focus, while gradients and restrained role
tints also appear in the surrounding interface. The white theme changes
surfaces, typography contrast, controls and chat presentation through its own
selector overrides rather than a second stylesheet.
## Voice state
| State | Dark-theme color |
|---|---|
| Listening/user speaking | Cyan (`--listening`) |
| Assistant speaking | Violet (`--speaking`) |
| Processing | Amber (`--processing`) |
| Ready/authenticated | Green (`--success`) |
| Error | Red (`--error`) |
`--voice-user`, `--voice-assistant` and `--voice-tool` connect transcript roles
to the same state vocabulary. When adding a state, define its existing control,
caption and transcript behavior together. Light-theme controls can use their
blue palette while retaining recognizable state feedback.
## Typography and copy
Use Inter (`--font-sans`) for human prose, buttons, settings and transcript
bodies. Geist Mono (`--font-mono`) is reserved for machine status, role labels,
tool identifiers and diagnostic JSON. Font stacks have system fallbacks.
Visible labels, accessibility text, permission instructions and recovery
messages default to Spanish. Source identifiers, comments, diagnostic logs and
internal model instructions use English. Exact model/voice/protocol identifiers
and proper names are preserved. Spanish wake phrases are user-input literals.
Use sentence case and concrete verbs. Empty states should explain the next
action; errors should say what failed and how to recover. Keep internal system
details out of ordinary user flows.
## Layout and components
The page uses a column layout: header, voice stage, always-visible inline
conversation, and sticky footer. The conversation is no longer the old
toggleable right overlay. Preserve space below the latest message so the sticky
footer does not obscure it; chat autoscroll retries after layout/reflow changes.
In the dark theme, the reports sidebar is fixed on the left at desktop widths
of at least 1024px. On smaller screens it becomes an off-canvas panel with a
toggle and backdrop. The white theme hides the sidebar, its toggle, and hint.
The user avatar opens a profile dialog; profile fields collapse from two
columns on narrow screens. Some controls are intentionally gated by `?debug=true`.
Use the existing button, icon-button, modal, switch and chat-message classes.
The dark theme uses glass bubbles with role tints, borders and restrained glow;
the light theme uses its white/blue message and avatar treatment. Keep required
chat DOM elements intact; optional historical overlay controls may be absent.
## Motion and accessibility
Microphone RMS and assistant audio levels drive the voice visualization.
Breathing, glow and decorative gradients should remain subordinate to speech
feedback. `body.booting` suppresses initial transitions; new animation should
respect that state and `prefers-reduced-motion`.
For changes, verify both themes, narrow mobile layouts down to 360px, keyboard
focus, dialog dismissal, and readable contrast. Prefer touch targets of at least
44px and respect safe-area padding. A CSS guideline is not proof that every
existing control already meets the target.
docs/client/glossary.md
+40
View File
@@ -0,0 +1,40 @@
# Context glossary
These terms explain the product and its conversation flow. Implementation
details belong in [architecture](architecture.md) and the source. Model/provider
names shown in a deployment are configuration examples, not definitions.
| Term | Meaning |
|---|---|
| Speech-to-speech client | The browser interface used to start a voice conversation, display transcripts, and play the assistant's reply |
| Pipeline | The ordered backend path: VAD → STT → LLM → TTS |
| VAD | Voice activity detection; determines when a user speech segment starts and stops |
| STT | Speech to text; transcribes an audio utterance |
| LLM | Language model; generates response text and optional tool calls |
| VLM | Vision-language model; a language model with image support when that capability is configured |
| TTS | Text to speech; synthesizes response audio with the configured voice/model |
| Realtime backend | The service that processes audio and exposes the `/v1/realtime` WebSocket protocol |
| Client server/proxy | The local FastAPI service serving the page and optional HTTP/WebSocket bridges |
| Builder | A person/team credited with creating the interface or deployment, distinct from a model's authors |
| Powered by | Hosting or inference infrastructure credited by the deployment |
| Tool | A callable operation used on demand; distinct from the pipeline stages that process ordinary turns |
| Native tool declaration | A function name, description and JSON Schema sent in `session.tools` |
| Tool executor | Code that receives a tool call, executes it and returns the result to the model |
| Web search | The browser-supported native tool whose request goes through the server's `/api/search` proxy |
| Search proxy | Same-origin server route holding a deployment search key and calling the external provider |
| ApiPilot | The dynamic-tool discovery/execution bridge, with both HTTP endpoints and model-text call markers |
| Call marker | Internal structured text such as `<<<APIPILOT_CALL:...>>>` used by the ApiPilot bridge; separate from native function-call events |
| Tools panel | Controls for available browser tools and search-key configuration |
| Settings | Connection, voice, audio and instruction controls persisted in browser storage |
| Identity block | The header's product/user identity, separate from configuration dialogs |
| About panel | Introduction and project/pipeline credits |
| Caller profile | Information looked up using the page's `phoneNumber`, used for identity display and optionally model context |
| RAG context | Reference material explicitly applied by the operator for the current session |
| Episode context | Content loaded for the page's `episode_id`, treated as the current episode reference |
| Queue | Users waiting for an available compute slot; waiting is not yet an active conversation |
| Ticket | A held queue position returned by the load balancer |
| Position | The displayed place in the queue; distinct from a time estimate |
| Claim | The transition from a waiting ticket to an allocated realtime session and usage reservation |
| At capacity | A full waiting queue that cannot accept another ticket |
| Barge-in | User speech interrupts an active assistant response, coordinating playback clearing and backend cancellation |
| Log-only reasoning | External transcript processing that observes/logs the turn without replacing native speech output |
docs/client/reasoning.md
+106
View File
@@ -0,0 +1,106 @@
# Reasoning modes
Reasoning determines the assistant's response text. The realtime backend always
provides the speech channel; optional HTTP reasoning can observe a transcript or
produce a separate answer. Set `S2S_REASONING_TYPE` to select the mode.
| Mode | Response owner | HTTP hook |
|---|---|---|
| `PROXY` (default) | Realtime backend | Browser does not call `/api/reasoning`; that endpoint returns 409 |
| `NODE_RED` | Realtime backend for `log_only=true`; external answer otherwise | Server forwards to `S2S_REASONING_NODE_RED_URL/api/reasoning` |
| `OPENAI` | OpenAI-compatible Chat Completions response | Server calls `OPENAI_BASE_URL/chat/completions` |
The mode is normalized case-insensitively. `NODERED` and `NODE-RED` resolve to
`NODE_RED`; unknown values resolve to `PROXY`.
## Shared instruction assembly
In [main.js](../../s2s-client/main.js), `effectiveReasoningInstructions()` starts
with `pinnedPrompt || settings.instructions`. `/api/config` pins
`S2S_PRIMARY_PROMPT` when non-empty; otherwise the settings prompt or
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
profile to choose this base prompt.
The browser adds caller context when `injectProfile` is enabled, custom RAG text
when the operator has applied it, and an available ApiPilot catalog block.
`effectiveInstructions()` adds `OPENAI_RENDER_HINT` only in `OPENAI` mode.
The WebSocket client packages these instructions in `session.update.session`;
the local proxy may then add server-resolved context. Details belong in
[context injection](context-injection.md).
## Native realtime reasoning (`PROXY`)
The realtime backend receives session instructions and enabled native tools,
then processes each voice turn through STT, model generation and TTS.
Native function-call events are executed by the browser. ApiPilot also has a
separate marker-based path handled by the local proxy. These are distinct
protocols, described in [tool calling](tool-calling.md).
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
instructions still apply, but server-side session context and marker execution
require the local `/v1/realtime` route.
## Node-RED reasoning
Configure a Node-RED base URL reachable by the UI server:
```env
S2S_REASONING_TYPE=NODE_RED
S2S_REASONING_NODE_RED_URL=http://localhost:1881
```
`maybeRunOpenAiReasoning()` handles both external modes despite its historical
function name. It runs after a complete user transcript and posts a body with
`transcript`, `instructions`, `language`, `user_context`, `phoneNumber`, and
`episode_id`. The server builds context and forwards the reasoning fields to
Node-RED with a 30-second timeout.
The checked-in Node-RED flow exposes `POST /api/reasoning` and
`POST /api/log/system-response`. Its reasoning response uses `log_only=true` and
an empty answer. The browser returns early in that case, preserving the native
backend's response. Assistant-text logging is forwarded separately by the relay
and is not an audio transport.
An answer-producing flow must return a non-empty `answer` and omit `log_only` or
set it to `false`. That activates the external-answer behavior below.
## OpenAI-compatible HTTP reasoning
```env
S2S_REASONING_TYPE=OPENAI
OPENAI_API_KEY=YOUR_API_KEY
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_REASONING_MODEL=gpt-4.1-mini
```
`OPENAI_ORGANIZATION` and `OPENAI_PROJECT` are optional headers. The model value
is a runtime default, not a recommendation about provider availability.
The server combines instructions and context into model messages, requests a
final answer, and returns `answer` with available usage metadata. ApiPilot can
append discovery instructions, execute a detected call marker, and request a
second response based on the tool result. The endpoint does not forward the
browser's native `session.tools` catalog as OpenAI Chat Completions `tools`.
For a non-log-only answer, the browser cancels/suppresses the active native
response and appends the external assistant text. It uses browser
`SpeechSynthesis` when available. Its compatibility fallback sends
`[VOCERO_RENDER_REPLY]` plus the answer to the realtime backend and requests a
response. The render hint asks the model to read only the answer after that
marker. This instruction is model behavior, not a guarantee of exact playback.
External reasoning adds an HTTP operation to each completed transcript and has
separate provider token usage. It is not the speech pipeline's `responses-api`
or `chat-completions` LLM backend setting; those configure the model inside the
realtime backend instead.
## Where to change behavior
| Goal | Source |
|---|---|
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
| Change browser context assembly | `effectiveReasoningInstructions()` in `main.js` |
| Change HTTP provider messages | `reasoning()` in `server.py` |
| Change external-answer rendering | `maybeRunOpenAiReasoning()` in `main.js` |
| Change Node-RED processing | `s2s-reasoning-engine/flows.json` |
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
docs/client/tool-calling.md
+137
View File
@@ -0,0 +1,137 @@
# Tool calling
The application has two tool mechanisms: native realtime function calling and
ApiPilot calls detected from model text. A spoken promise to fetch something is
not evidence that either mechanism executed a tool.
## Native realtime function calls
The browser sends tool declarations inside the `session` object:
```json
{
"type": "session.update",
"session": {
"type": "realtime",
"tools": [
{
"type": "function",
"name": "web_search",
"description": "Search the web for information.",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"]
}
}
],
"tool_choice": "auto"
}
}
```
Keep configuration inside `session`, not at the event root. The backend has a
legacy flat-payload compatibility path, but new clients should use the nested
shape. The current browser's `activeToolDefs()` declares `web_search` when
enabled and a deploy/user search key is available. It does not load the old
remote agent-profile MCP catalog into native session tools.
For a native call, the backend emits:
```json
{
"type": "response.function_call_arguments.done",
"call_id": "call_example",
"name": "web_search",
"arguments": "{\"query\":\"example query\"}"
}
```
`arguments` is a JSON-encoded string. The
[WebSocket client](../../s2s-client/ws/s2s-ws-client.js) parses it and dispatches
`toolcall`; [main.js](../../s2s-client/main.js) runs the supported executor.
Web search uses `POST /api/search`, keeping the deployment's Serper key on the
server. Tool output returns to the model as:
```json
{
"type": "conversation.item.create",
"item": {
"type": "function_call_output",
"call_id": "call_example",
"output": "Tool result text"
}
}
```
The client requests follow-up generation with `response.create` when the
result should be spoken. A `function_call_output` item alone does not start
generation. See the [realtime API guide](../speech/realtime-api.md) for local
model `<code>` extraction and native remote-provider tool calls.
## ApiPilot discovery and execution
`S2S_APIPILOT_TOOLS_URL` enables the current dynamic-tool bridge. The backend
discovers tools through its `/tools` route and executes through `/execute`.
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
server exposes:
- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
`data.tools`, `data.count`, and `data.include_unauthorized`.
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
a `success/data/error` envelope.
Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
`requires_auth`. Execution looks up the current discovery cache, refreshes once
if the tool is missing, and applies an authorization gate and argument
validation before forwarding. Caller `Authorization` and
`x-demo-session` can be forwarded; configured service credentials and
login-by-phone provide additional server paths. Display `error.message` to the
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.
The discovery cache defaults to 300 seconds server-side
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
also keeps a per-user catalog cache. ApiPilot configuration is independent of
`S2S_INJECT_PROFILE`.
## Marker-based two-pass reasoning
The browser and proxy add an ApiPilot catalog as **instruction text**, including
tool names, descriptions, authentication requirements and required parameters.
The model is instructed to emit a call marker when it chooses a matching tool:
```text
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
```
Tool names are exact identifiers and must not be translated. The local realtime
proxy accumulates assistant text, detects the marker, validates and executes
the ApiPilot call, then injects the result as context and requests a new
response. The HTTP reasoning route has analogous discovery/execution and
second-answer handling. Internal marker text and its speculative audio are
filtered from user output where those handlers run.
This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
discovery is not the same as declaring a native `session.tools` array. Direct
browser connections bypass the local relay's marker executor even if the
browser includes catalog instructions.
## Retired documentation assumptions
The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
Those APIs/rankers are absent from the current client/server path. The retained
agent-profile JSON is an illustrative historical/example shape. Add or modify
current ApiPilot tool definitions in the provider; do not restore obsolete
endpoints solely to match older documentation.
## Verify a tool turn
1. Confirm `/api/config` reports the intended feature and connection mode.
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
separate evidence.
3. For native calls, find `response.function_call_arguments.done`, then the
browser `toolcall` and executor request.
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
5. Confirm the tool result reaches the model and follow-up text/audio reaches
the browser. A spoken acknowledgement alone does not complete the flow.
docs/client/troubleshooting.md
+115
View File
@@ -0,0 +1,115 @@
# Troubleshooting
Diagnose configuration, protocol events, execution and playback separately.
Use [architecture](architecture.md), [tool calling](tool-calling.md) and
[context injection](context-injection.md) to identify which path is active.
## Check configuration and connection first
Inspect the server configuration without printing secret environment files:
```bash
curl -fsS http://localhost:8420/api/config | jq
```
Confirm `s2sUrl`, `lb`, `reasoningType`, `injectProfile`, `apipilot`, `theme`
and `startSessionMode`. A direct WebSocket URL bypasses the local relay's
server-side context injection and marker execution. For a local proxy, confirm
the server can resolve and reach `S2S_PROXY_TARGET`.
In browser DevTools, inspect the WebSocket handshake and `session.created`,
`session.update` and error events. Check that instructions and native tools are
inside `session`. A cached browser module can disagree with current server
code; inspect served asset versions and reload with cache disabled when needed.
## The assistant promises a tool but nothing happens
Follow the actual execution mechanism:
1. For a native tool, confirm it appears in the outbound `session.tools` array.
2. Find `response.function_call_arguments.done` with a tool name, `call_id`, and
JSON-string arguments.
3. Confirm the browser dispatched `toolcall` and the executor sent its request.
4. Confirm `function_call_output` and any required `response.create` were sent.
5. For ApiPilot, inspect discovery, marker detection, authorization/schema
validation, execution, and second-pass result injection instead.
A spoken sentence such as “Voy a buscar el reporte” does not satisfy step 2.
If the function-call event is absent, inspect model/backend tool generation
before debugging a browser executor that has not been invoked.
Earlier local debug notes recorded a reproduction where tools and required
tool policy were sent but the model produced only spoken text. Those notes
refer to the retired MCP ranking path; they do not establish a current ApiPilot
failure. Both local investigation notes remain marked `OPEN`. Their useful
lesson is to collect evidence at each boundary and reproduce against current
code before claiming a root cause is resolved.
## The assistant misses caller or report context
- Verify the page has an encoded `phoneNumber` or the intended `episode_id`.
- Inspect `/api/user/phone` and context-fetch logs for HTTP/JSON failures.
- Confirm profile injection is enabled; a visible profile alone does not prove
realtime instructions contain it.
- Reports need `S2S_INJECT_REPORTS=true`; `minimal` sends only labels and dates.
A historical script will not be in the prompt in that mode.
- Static summary text overrides the HTTP summary endpoint.
- Inspect the final `session.update` rather than only its browser base prompt.
The proxy can append and deduplicate context after the browser sends it.
`S2S_DEBUG_PROMPT=true` enables complete prompt logs, including caller information
and conversation text. Use summaries when full payloads are unnecessary. The
browser's application functions are module-scoped; do not assume names such as
`effectiveInstructions()` are available directly on `window` in DevTools.
## Short gaps or clipped words around tool results
The retained audio investigation found very low local serialization/render
times for one large tool result, while still observing audio interruptions.
That evidence ruled out a UI-render bottleneck for that reproduction; it does
not rule out rendering costs for all future payloads.
Useful checks are:
- Measure tool-result size, prompt length, serialization time and UI render time.
- Inspect whether result injection overlaps an active or pending response.
- Check whether a speculative response starts before its `response.created`
reaches the browser. Cancellation based only on an existing response ID can
miss that window.
- Inspect deferred text/response queues and whether they flush after
`response.done` rather than interrupting the active turn.
- Inspect playback prebuffering and queue underflow, especially when the first
words of every sentence are affected.
- Confirm oversized tool results are summarized or capped on the relevant
path rather than assuming every tool payload gets the same treatment.
The historical investigation introduced a 90ms playback prebuffer and reduced
MCP prompt reinjection. The old 150ms MCP fallback timer is historical, not a
current ApiPilot configuration knob. Verify the live worklet and current relay
before changing audio timing based on those observations.
## External reasoning and startup
`/api/reasoning` returning 409 is expected in `PROXY` mode. In `NODE_RED`, check
that `S2S_REASONING_NODE_RED_URL` is reachable from the server and whether the
response is `log_only=true`. An empty answer is valid only in log-only mode.
In `OPENAI`, check the configured endpoint/model/key and provider logs; HTTP
reasoning does not use the native session's tool array.
For `TALK`/`TALKFIRST`, the two-second warmup does not override browser
microphone/audio permissions. If automatic startup fails, use the unlocker
gesture and inspect the permission error rather than increasing the timer
without evidence.
## Read-only source checks
From the repository root, useful syntax checks are:
```bash
node --check s2s-client/main.js
node --check s2s-client/ws/s2s-ws-client.js
python -m py_compile s2s-client/server.py
```
Syntax checks do not validate microphone access, network services, model output,
or audio quality. Collect a fresh browser/backend reproduction for those paths.
docs/client/usage.md
+153
View File
@@ -0,0 +1,153 @@
# Client usage and configuration
The browser voice client connects to an existing OpenAI Realtime-compatible
speech backend. The UI server can also proxy WebSocket traffic, search,
reasoning, caller context, ApiPilot tools and load-balancer requests.
## Local startup
From the repository root:
```bash
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
```
The launcher loads the root `.env`, creates `.venv` if needed, installs the
client requirements when the dependency check fails, and runs uvicorn on port
8420. `UI_PORT` overrides the UI port. Because `.env` is sourced by the script,
values in that file can replace variables supplied on the command line.
To install and start the UI manually:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r s2s-client/requirements.txt
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime \
.venv/bin/python -m uvicorn --app-dir s2s-client server:app --port 8420
```
Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
access, and speak. Microphone access requires HTTPS or a local browser origin;
use HTTPS when accessing the UI from another machine.
The speech pipeline has separate model/runtime dependencies and no complete
dependency manifest in this checkout. In an already prepared speech
environment, a local realtime server can be started with:
```bash
python -m speech_to_speech.s2s_pipeline \
--mode realtime \
--stt parakeet-tdt \
--llm_backend transformers \
--tts kokoro \
--model_name Qwen/Qwen3-4B-Instruct-2507 \
--llm_device cuda \
--llm_torch_dtype float16 \
--enable_live_transcription
```
Select backends and devices supported by the host; the example model is not a
fixed deployment requirement. The default realtime endpoint is
`ws://localhost:8765/v1/realtime`, configured through `--ws_host` and `--ws_port`.
`--num_pipelines` controls the number of isolated concurrent realtime sessions.
See [LLM](../speech/llm.md), [STT](../speech/stt.md), and [TTS](../speech/tts.md).
## Connection modes
| Configuration | Browser connection | Settings URL | Load-balancer features |
|---|---|---|---|
| `SPEECH_TO_SPEECH_URL` set | Direct to pinned URL | Locked | Disabled |
| `S2S_PROXY_TARGET` set, no direct URL | Same-origin `/v1/realtime` proxy | Locked | Disabled |
| `LOAD_BALANCER_URL` set, neither above set | Per-session compute URL from `/api/session` | Hidden | Queue enabled; metering requires `SPACE_ID` too |
| None set | User-provided realtime URL | Editable | Disabled |
Direct URLs may be full `ws(s)://.../v1/realtime` addresses or host values that
the browser normalizes. A direct URL must be reachable **from the browser**.
`S2S_PROXY_TARGET` must be reachable **from the UI server**. HTTPS deployments
need a secure browser WebSocket URL.
The launcher supplies `ws://ai-server:12345/v1/realtime` as a proxy-target
default when no value is configured. That hostname must resolve from the UI
server; it is not a universally available local service.
## Docker
Use the project root as the build context:
```bash
docker build -f s2s-client/Dockerfile -t s2s-client .
docker run --rm -p 7860:7860 \
-e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client
```
For the proxy path, replace the direct setting with a server-reachable
`S2S_PROXY_TARGET`. If the upstream is another container, attach the UI to the
appropriate existing network and use that service's hostname. An upstream
bound only to host loopback is not reachable via an arbitrary container name.
The image includes `s2s-client` and root `docs`. `/docs` is still the FastAPI API
explorer; documentation lives under `/docs/client/`, `/docs/speech/`,
`/docs/adr/`, `/docs/examples/`, and `/docs/proposals/`. Legacy `/project/` and
`/public/docs.html` redirect to the historical proposal. Markdown is served as a
source file rather than rendered through a separate documentation site. Request
file paths such as `/docs/README.md`; directories without `index.html` are not
browsable pages.
## Settings and tools
Voice, editable system instructions and connection settings are kept in browser
`localStorage` under `s2s.ws.*` keys. A non-empty `S2S_PRIMARY_PROMPT` pins the
instruction field. Restart applies connection/voice changes to a new session.
Model voice names are runtime identifiers and stay unchanged across UI
translations.
Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
tools panel. The server proxies requests through `/api/search`; its own key is
not returned to the browser. Dynamic ApiPilot tools are configured through
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).
Operator RAG text is applied through the RAG dialog and updates live
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
include `episode_id=...`. See [context injection](context-injection.md) for
enabled data sources and limits.
## Theme and session startup
`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
the dark aurora theme. Theme-specific sections remain hidden until configuration
resolves, with a fallback to avoid an invisible page after config failure.
`S2S_START_SESSION_MODE` controls initial microphone/session startup:
| Value | Behavior |
|---|---|
| Empty/default | User taps to start |
| `TALK` | After a 2-second warmup, show an audio unlocker; a user gesture starts the session and greeting |
| `TALKFIRST` | After a 2-second warmup, attempt automatic start and inject the Spanish wake phrase; microphone failure falls back to the unlocker |
The warmup is a timer, not proof that all upstream context requests have
finished. Browser permission and audio policies can still require a gesture.
The greeting uses the same `hola vocero` input as wake-word handling.
## Queue and daily usage
Load-balancer mode proxies session acquisition and queue polling. Waiting does
not reserve conversation time; a daily budget is reserved when a compute slot
is granted. The server enables metering only with both `LOAD_BALANCER_URL` and
`SPACE_ID` and with neither pinned direct/proxy mode active.
| Environment setting | Default | Purpose |
|---|---|---|
| `LIMIT_ANON_SEC` | 300 | Anonymous seconds per UTC day |
| `LIMIT_FREE_SEC` | 600 | Signed-in non-PRO seconds per UTC day |
| `UNLIMITED_ORGS` | Adds to built-in organizations | Additional unlimited organization members |
| `USAGE_HASH_SECRET` | Random if unset | Identity-key hashing and anonymous cookie signing |
PRO members are unlimited. Built-in unlimited organizations are `cerebras`,
`HuggingFaceM4`, `smolagents`, and `pollen-robotics`; organization matching is
case-insensitive. See [auth.py](../../s2s-client/auth.py) and
[limiter.py](../../s2s-client/limiter.py) for the current enforcement behavior.
For external transcript processing, configure the selected
[reasoning mode](reasoning.md). For connection, context and audio diagnostics,
use [troubleshooting](troubleshooting.md).