alvaro
4b0108ecf0
Opening a report or podcast link now establishes an authenticated starting reference for the conversation. Users can select other accessible content or compare reports while retaining the original link, and TALKFIRST greetings can name the verified report or podcast. Release notes: - Added authenticated report and episode link resolution with bounded lookups and unavailable-content handling - Added explicit conversation selection without restricting searches or changing the reference during comparisons - Added contextual TALKFIRST greetings through both voice-proxy and HTTP reasoning paths - Preserved backend permission checks, conversation isolation, cancellation, and the existing speech protocol Reviewed-on: AquaPortal/s2s#2
143 lines
8.7 KiB
Markdown
143 lines
8.7 KiB
Markdown
# Client architecture
|
|
|
|
The [React browser client](../../vocero-frontend) uses the
|
|
[FastAPI backend](../../vocero-api) for HTTP APIs and a speech WebSocket relay.
|
|
`start_ui.sh` serves Vite on the existing public port `8420` and FastAPI on
|
|
loopback port `8421`. Vite forwards backend routes and provides hot reload;
|
|
Uvicorn reloads Python edits. The launcher supervises both processes.
|
|
The speech pipeline runs separately.
|
|
Runtime configuration comes from environment variables and the root `.env`.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Browser[Browser UI]
|
|
Vite[Vite on public port 8420]
|
|
Server[Client FastAPI server]
|
|
Speech[Realtime speech backend]
|
|
Reasoning[OpenAI-compatible HTTP reasoning provider]
|
|
Tools[ApiPilot backend]
|
|
Browser <-->|Frontend and HMR| Vite
|
|
Vite -->|HTTP configuration and proxies| Server
|
|
Vite <-->|Speech WebSocket relay| Server
|
|
Server <-->|Realtime events and audio| Speech
|
|
Browser <-->|Optional direct connection| Speech
|
|
Server -->|Optional transcript reasoning| Reasoning
|
|
Server -->|Tool discovery and execution| Tools
|
|
```
|
|
|
|
## Connection modes
|
|
|
|
`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
|
|
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
|
|
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
|
|
session/queue proxy, or the user can enter a direct URL in settings. See
|
|
[usage](usage.md) for the configuration matrix.
|
|
|
|
The local realtime proxy relays audio and events in both directions. It also
|
|
patches selected `session.update` instructions, handles ApiPilot marker-based
|
|
tool execution, filters internal text, and can suppress speculative/tool-marker
|
|
audio. It is not an unconditional byte-for-byte tunnel.
|
|
|
|
## Source map
|
|
|
|
| Source | Responsibility |
|
|
|---|---|
|
|
| [server.py](../../vocero-api/server.py) | Compatibility entry point for existing `server:app` commands |
|
|
| [main.py](../../vocero-api/vocero_api/main.py) | FastAPI assembly, documentation serving and redirects |
|
|
| [api](../../vocero-api/vocero_api/api) | HTTP route registration, handlers and compatible response validation |
|
|
| [models](../../vocero-api/vocero_api/models) | Pydantic request and response contracts |
|
|
| [core](../../vocero-api/vocero_api/core) | Existing environment parsing, logging, URL helpers and diagnostics |
|
|
| [providers](../../vocero-api/vocero_api/providers) | Caller context and ApiPilot HTTP integrations |
|
|
| [services](../../vocero-api/vocero_api/services) | Context construction, reasoning and tool-context rendering |
|
|
| [realtime](../../vocero-api/vocero_api/realtime) | WebSocket relay, events, markers and audio handling |
|
|
| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown |
|
|
| [vite.config.ts](../../vocero-frontend/vite.config.ts) | Public development endpoint, HMR, allowed hosts, and backend routing |
|
|
| [App.tsx](../../vocero-frontend/src/App.tsx) | React layout, voice controls, dialogs, profile and reports sidebar |
|
|
| [controller.ts](../../vocero-frontend/src/realtime/controller.ts) | Application state, settings, context, queue, tools and external reasoning |
|
|
| [client.ts](../../vocero-frontend/src/realtime/client.ts) | Realtime protocol, session lifecycle, audio and transcript events |
|
|
| [components](../../vocero-frontend/src/components) | Chat, caller profile, settings and voice components |
|
|
| [audio](../../vocero-frontend/src/audio) | TypeScript capture/playback worklets, PCM codec, and visualizer |
|
|
| [theme.css](../../vocero-frontend/src/styles/theme.css) | Tailwind semantic tokens, configured theme variants and responsive variants |
|
|
| [ui.tsx](../../vocero-frontend/src/components/ui.tsx) | Shared Tailwind buttons, fields and dialog controls; layout utilities live with components |
|
|
| [styles](../../vocero-frontend/src/styles) | Minimal base defaults, reduced motion, audio artwork, sanitized Markdown, decorative surfaces and native range controls |
|
|
|
|
## Server routes
|
|
|
|
| Route | Purpose |
|
|
|---|---|
|
|
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
|
|
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
|
|
| `POST /api/reasoning` | OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
|
|
| `POST /api/conversations` | MCP-mode phone login and browser-owned conversation creation |
|
|
| `DELETE /api/conversations/current` | Release the selected MCP conversation |
|
|
| `POST /api/conversations/current/interrupt` | Cancel its matching HTTP reasoning turn |
|
|
| `GET /api/mcp/tools` | Discover the selected conversation's authenticated MCP catalog |
|
|
| `GET /api/user/phone` | Configured caller profile lookup |
|
|
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
|
|
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
|
|
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
|
|
| `POST /api/search` | Serper web search proxy |
|
|
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
|
|
|
|
The public root serves the React UI. `/__vite_hmr` carries Vite hot updates;
|
|
`/v1/realtime` remains the speech connection. `/docs` remains the FastAPI API
|
|
explorer, while project documentation files are served in the `/docs/`
|
|
subdirectories. Markdown remains
|
|
source text, and directories without `index.html` are not browsable pages.
|
|
FastAPI serves backend and documentation routes only. The frontend source and
|
|
assets live in `vocero-frontend` and are served by Vite.
|
|
|
|
The uv project and its Python tools are scoped to `vocero-api/`. Response models
|
|
describe service-owned structure and retain loose upstream values. See
|
|
[development and compatibility](development.md) before tightening a contract.
|
|
|
|
## Audio path
|
|
|
|
1. The browser captures microphone audio with echo cancellation, noise
|
|
suppression and automatic gain-control settings.
|
|
2. [mic-capture.worklet.ts](../../vocero-frontend/src/audio/mic-capture.worklet.ts) resamples to 16 kHz
|
|
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
|
|
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
|
|
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to
|
|
[audio-playback.worklet.ts](../../vocero-frontend/src/audio/audio-playback.worklet.ts), which buffers,
|
|
resamples and fades playback edges. Input speech can clear playback for barge-in.
|
|
The client flushes short final chunks on response completion and remains in
|
|
the speaking state until the worklet reports that playback has drained.
|
|
Sequence numbers reject drain notifications from superseded playback.
|
|
|
|
Microphone meter subscribers are separate from application-state subscribers,
|
|
so level updates do not rerender the main React tree.
|
|
|
|
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
|
|
registration structure intact when editing them. The speech backend's thread
|
|
and cancellation model is documented in [realtime API](../speech/realtime-api.md).
|
|
|
|
## Reasoning and context
|
|
|
|
The browser triggers external reasoning on completed user transcripts only in
|
|
`OPENAI` mode. A non-empty answer replaces the native response through the
|
|
external-answer path. These HTTP operations supplement the realtime audio path.
|
|
New user speech and session teardown cancel pending reasoning. Turn identity
|
|
checks reject stale HTTP results and speech callbacks even if cancellation is late.
|
|
|
|
With MCP enabled, `services/conversations.py` owns bounded server-side caller
|
|
credentials. `providers/mcp.py` owns the official SDK transport, while
|
|
`services/mcp_tools.py` validates selections and enforces per-turn limits.
|
|
`services/content_context.py` resolves linked metadata and implements the local
|
|
`select_content` action. Each conversation retains separate original-link and
|
|
current-selection references; reading or comparing reports does not change the
|
|
selection. The current podcast is passed as an MCP request header, with every
|
|
retrieval still subject to backend authorization. TALKFIRST uses the verified
|
|
title in the opening instructions after link resolution completes.
|
|
`services/mcp_reasoning.py` and `realtime/mcp_proxy.py` share that executor for
|
|
HTTP and voice reasoning respectively. The browser stores only its conversation
|
|
ID in memory and uses an HttpOnly owner cookie. Legacy public content preloads
|
|
are disabled in this mode; the AquaPortal backend remains the authority for
|
|
content access. The speech pipeline and its audio protocol are unchanged.
|
|
In OPENAI mode, MCP TALKFIRST greetings enter the HTTP reasoning path after the
|
|
speech connection is ready, and use the existing external-answer playback and
|
|
cancellation lifecycle.
|
|
|
|
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
|
|
[tool calling](tool-calling.md) for the conditions and limitations of each path.
|