alvaro
96895996d9
Vocero needs to query AquaPortal on behalf of the caller selected by the existing phone link. This adds server-owned conversations and MCP tool execution for voice and HTTP reasoning while leaving the speech pipeline unchanged. Release notes: - Added browser-bound conversations that keep backend credentials server-side and isolate callers. - Connected the eight-tool AquaPortal catalog through authenticated Streamable HTTP with full input schemas and structured results. - Added bounded multi-step tool execution with cancellation, expiry, and stale-result protection. - Updated browser bootstrap, tool progress, reasoning headers, and conversation cleanup. - Disabled legacy unscoped content preloads in MCP mode and documented configuration and deployment constraints. - Preserved disabled-mode contracts and added local SDK, WebSocket, and browser regression coverage. Reviewed-on: AquaPortal/s2s#1
134 lines
8.1 KiB
Markdown
134 lines
8.1 KiB
Markdown
# Client architecture
|
|
|
|
The [React browser client](../../vocero-frontend) uses the
|
|
[FastAPI backend](../../vocero-api) for HTTP APIs and a speech WebSocket relay.
|
|
`start_ui.sh` serves Vite on the existing public port `8420` and FastAPI on
|
|
loopback port `8421`. Vite forwards backend routes and provides hot reload;
|
|
Uvicorn reloads Python edits. The launcher supervises both processes.
|
|
The speech pipeline runs separately.
|
|
Runtime configuration comes from environment variables and the root `.env`.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Browser[Browser UI]
|
|
Vite[Vite on public port 8420]
|
|
Server[Client FastAPI server]
|
|
Speech[Realtime speech backend]
|
|
Reasoning[OpenAI-compatible HTTP reasoning provider]
|
|
Tools[ApiPilot backend]
|
|
Browser <-->|Frontend and HMR| Vite
|
|
Vite -->|HTTP configuration and proxies| Server
|
|
Vite <-->|Speech WebSocket relay| Server
|
|
Server <-->|Realtime events and audio| Speech
|
|
Browser <-->|Optional direct connection| Speech
|
|
Server -->|Optional transcript reasoning| Reasoning
|
|
Server -->|Tool discovery and execution| Tools
|
|
```
|
|
|
|
## Connection modes
|
|
|
|
`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
|
|
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
|
|
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
|
|
session/queue proxy, or the user can enter a direct URL in settings. See
|
|
[usage](usage.md) for the configuration matrix.
|
|
|
|
The local realtime proxy relays audio and events in both directions. It also
|
|
patches selected `session.update` instructions, handles ApiPilot marker-based
|
|
tool execution, filters internal text, and can suppress speculative/tool-marker
|
|
audio. It is not an unconditional byte-for-byte tunnel.
|
|
|
|
## Source map
|
|
|
|
| Source | Responsibility |
|
|
|---|---|
|
|
| [server.py](../../vocero-api/server.py) | Compatibility entry point for existing `server:app` commands |
|
|
| [main.py](../../vocero-api/vocero_api/main.py) | FastAPI assembly, documentation serving and redirects |
|
|
| [api](../../vocero-api/vocero_api/api) | HTTP route registration, handlers and compatible response validation |
|
|
| [models](../../vocero-api/vocero_api/models) | Pydantic request and response contracts |
|
|
| [core](../../vocero-api/vocero_api/core) | Existing environment parsing, logging, URL helpers and diagnostics |
|
|
| [providers](../../vocero-api/vocero_api/providers) | Caller context and ApiPilot HTTP integrations |
|
|
| [services](../../vocero-api/vocero_api/services) | Context construction, reasoning and tool-context rendering |
|
|
| [realtime](../../vocero-api/vocero_api/realtime) | WebSocket relay, events, markers and audio handling |
|
|
| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown |
|
|
| [vite.config.ts](../../vocero-frontend/vite.config.ts) | Public development endpoint, HMR, allowed hosts, and backend routing |
|
|
| [App.tsx](../../vocero-frontend/src/App.tsx) | React layout, voice controls, dialogs, profile and reports sidebar |
|
|
| [controller.ts](../../vocero-frontend/src/realtime/controller.ts) | Application state, settings, context, queue, tools and external reasoning |
|
|
| [client.ts](../../vocero-frontend/src/realtime/client.ts) | Realtime protocol, session lifecycle, audio and transcript events |
|
|
| [components](../../vocero-frontend/src/components) | Chat, caller profile, settings and voice components |
|
|
| [audio](../../vocero-frontend/src/audio) | TypeScript capture/playback worklets, PCM codec, and visualizer |
|
|
| [theme.css](../../vocero-frontend/src/styles/theme.css) | Tailwind semantic tokens, configured theme variants and responsive variants |
|
|
| [ui.tsx](../../vocero-frontend/src/components/ui.tsx) | Shared Tailwind buttons, fields and dialog controls; layout utilities live with components |
|
|
| [styles](../../vocero-frontend/src/styles) | Minimal base defaults, reduced motion, audio artwork, sanitized Markdown, decorative surfaces and native range controls |
|
|
|
|
## Server routes
|
|
|
|
| Route | Purpose |
|
|
|---|---|
|
|
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
|
|
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
|
|
| `POST /api/reasoning` | OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
|
|
| `POST /api/conversations` | MCP-mode phone login and browser-owned conversation creation |
|
|
| `DELETE /api/conversations/current` | Release the selected MCP conversation |
|
|
| `POST /api/conversations/current/interrupt` | Cancel its matching HTTP reasoning turn |
|
|
| `GET /api/mcp/tools` | Discover the selected conversation's authenticated MCP catalog |
|
|
| `GET /api/user/phone` | Configured caller profile lookup |
|
|
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
|
|
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
|
|
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
|
|
| `POST /api/search` | Serper web search proxy |
|
|
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
|
|
|
|
The public root serves the React UI. `/__vite_hmr` carries Vite hot updates;
|
|
`/v1/realtime` remains the speech connection. `/docs` remains the FastAPI API
|
|
explorer, while project documentation files are served in the `/docs/`
|
|
subdirectories. Markdown remains
|
|
source text, and directories without `index.html` are not browsable pages.
|
|
FastAPI serves backend and documentation routes only. The frontend source and
|
|
assets live in `vocero-frontend` and are served by Vite.
|
|
|
|
The uv project and its Python tools are scoped to `vocero-api/`. Response models
|
|
describe service-owned structure and retain loose upstream values. See
|
|
[development and compatibility](development.md) before tightening a contract.
|
|
|
|
## Audio path
|
|
|
|
1. The browser captures microphone audio with echo cancellation, noise
|
|
suppression and automatic gain-control settings.
|
|
2. [mic-capture.worklet.ts](../../vocero-frontend/src/audio/mic-capture.worklet.ts) resamples to 16 kHz
|
|
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
|
|
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
|
|
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to
|
|
[audio-playback.worklet.ts](../../vocero-frontend/src/audio/audio-playback.worklet.ts), which buffers,
|
|
resamples and fades playback edges. Input speech can clear playback for barge-in.
|
|
The client flushes short final chunks on response completion and remains in
|
|
the speaking state until the worklet reports that playback has drained.
|
|
Sequence numbers reject drain notifications from superseded playback.
|
|
|
|
Microphone meter subscribers are separate from application-state subscribers,
|
|
so level updates do not rerender the main React tree.
|
|
|
|
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
|
|
registration structure intact when editing them. The speech backend's thread
|
|
and cancellation model is documented in [realtime API](../speech/realtime-api.md).
|
|
|
|
## Reasoning and context
|
|
|
|
The browser triggers external reasoning on completed user transcripts only in
|
|
`OPENAI` mode. A non-empty answer replaces the native response through the
|
|
external-answer path. These HTTP operations supplement the realtime audio path.
|
|
New user speech and session teardown cancel pending reasoning. Turn identity
|
|
checks reject stale HTTP results and speech callbacks even if cancellation is late.
|
|
|
|
With MCP enabled, `services/conversations.py` owns bounded server-side caller
|
|
credentials. `providers/mcp.py` owns the official SDK transport, while
|
|
`services/mcp_tools.py` validates selections and enforces per-turn limits.
|
|
`services/mcp_reasoning.py` and `realtime/mcp_proxy.py` share that executor for
|
|
HTTP and voice reasoning respectively. The browser stores only its conversation
|
|
ID in memory and uses an HttpOnly owner cookie. Legacy public content preloads
|
|
are disabled in this mode; the AquaPortal backend remains the authority for
|
|
content access. The speech pipeline and its audio protocol are unchanged.
|
|
|
|
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
|
|
[tool calling](tool-calling.md) for the conditions and limitations of each path.
|