vocero-client/docs/client/architecture.md

Client architecture

The React browser client uses the FastAPI backend for HTTP APIs and a speech WebSocket relay. start_ui.sh serves Vite on the existing public port 8420 and FastAPI on loopback port 8421. Vite forwards backend routes and provides hot reload; Uvicorn reloads Python edits. The launcher supervises both processes. The speech pipeline runs separately. Runtime configuration comes from environment variables and the root .env.

flowchart LR
    Browser[Browser UI]
    Vite[Vite on public port 8420]
    Server[Client FastAPI server]
    Speech[Realtime speech backend]
    Reasoning[OpenAI-compatible HTTP reasoning provider]
    Tools[ApiPilot backend]
    Browser <-->|Frontend and HMR| Vite
    Vite -->|HTTP configuration and proxies| Server
    Vite <-->|Speech WebSocket relay| Server
    Server <-->|Realtime events and audio| Speech
    Browser <-->|Optional direct connection| Speech
    Server -->|Optional transcript reasoning| Reasoning
    Server -->|Tool discovery and execution| Tools

Connection modes

SPEECH_TO_SPEECH_URL pins a direct browser connection. Otherwise, S2S_PROXY_TARGET exposes the same-origin /v1/realtime proxy. Either setting disables load-balancer mode. With neither set, LOAD_BALANCER_URL enables the session/queue proxy, or the user can enter a direct URL in settings. See usage for the configuration matrix.

The local realtime proxy relays audio and events in both directions. It also patches selected session.update instructions, handles ApiPilot marker-based tool execution, filters internal text, and can suppress speculative/tool-marker audio. It is not an unconditional byte-for-byte tunnel.

Source map

Source	Responsibility
server.py	Compatibility entry point for existing server:app commands
main.py	FastAPI assembly, documentation serving and redirects
api	HTTP route registration, handlers and compatible response validation
models	Pydantic request and response contracts
core	Existing environment parsing, logging, URL helpers and diagnostics
providers	Caller context and ApiPilot HTTP integrations
services	Context construction, reasoning and tool-context rendering
realtime	WebSocket relay, events, markers and audio handling
start_ui.sh	Dependency setup, ports, readiness, reload processes, and shutdown
vite.config.ts	Public development endpoint, HMR, allowed hosts, and backend routing
App.tsx	React layout, voice controls, dialogs, profile and reports sidebar
controller.ts	Application state, settings, context, queue, tools and external reasoning
client.ts	Realtime protocol, session lifecycle, audio and transcript events
components	Chat, caller profile, settings and voice components
audio	TypeScript capture/playback worklets, PCM codec, and visualizer
theme.css	Tailwind semantic tokens, configured theme variants and responsive variants
ui.tsx	Shared Tailwind buttons, fields and dialog controls; layout utilities live with components
styles	Minimal base defaults, reduced motion, audio artwork, sanitized Markdown, decorative surfaces and native range controls

Server routes

Route	Purpose
GET /api/config	Connection settings, primary prompt, reasoning/theme/start modes, and feature flags
WS /v1/realtime	Local realtime proxy to S2S_PROXY_TARGET
POST /api/reasoning	OpenAI-compatible HTTP reasoning; disabled in PROXY mode
POST /api/conversations	MCP-mode phone login and browser-owned conversation creation
DELETE /api/conversations/current	Release the selected MCP conversation
POST /api/conversations/current/interrupt	Cancel its matching HTTP reasoning turn
GET /api/mcp/tools	Discover the selected conversation's authenticated MCP catalog
GET /api/user/phone	Configured caller profile lookup
GET /api/user/reports	Configured scheduled report lookup, subject to report-injection controls
GET /api/apipilot/tools	Tool discovery with optional authorization and cache refresh
POST /api/apipilot/execute	Validated tool execution returning a success/data/error envelope
POST /api/search	Serper web search proxy
/api/session, /api/queue/*	Load-balancer session and waiting-queue proxies

The public root serves the React UI. /__vite_hmr carries Vite hot updates; /v1/realtime remains the speech connection. /docs remains the FastAPI API explorer, while project documentation files are served in the /docs/ subdirectories. Markdown remains source text, and directories without index.html are not browsable pages. FastAPI serves backend and documentation routes only. The frontend source and assets live in vocero-frontend and are served by Vite.

The uv project and its Python tools are scoped to vocero-api/. Response models describe service-owned structure and retain loose upstream values. See development and compatibility before tightening a contract.

Audio path

1. The browser captures microphone audio with echo cancellation, noise suppression and automatic gain-control settings.
2. mic-capture.worklet.ts resamples to 16 kHz mono PCM16. The browser sends base64 input_audio_buffer.append frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to audio-playback.worklet.ts, which buffers, resamples and fades playback edges. Input speech can clear playback for barge-in. The client flushes short final chunks on response completion and remains in the speaking state until the worklet reports that playback has drained. Sequence numbers reject drain notifications from superseded playback.

Microphone meter subscribers are separate from application-state subscribers, so level updates do not rerender the main React tree.

Audio worklets are loaded through AudioWorklet.addModule; keep their processor registration structure intact when editing them. The speech backend's thread and cancellation model is documented in realtime API.

Reasoning and context

The browser triggers external reasoning on completed user transcripts only in OPENAI mode. A non-empty answer replaces the native response through the external-answer path. These HTTP operations supplement the realtime audio path. New user speech and session teardown cancel pending reasoning. Turn identity checks reject stale HTTP results and speech callbacks even if cancellation is late.

With MCP enabled, services/conversations.py owns bounded server-side caller credentials. providers/mcp.py owns the official SDK transport, while services/mcp_tools.py validates selections and enforces per-turn limits. services/content_context.py resolves linked metadata and implements the local select_content action. Each conversation retains separate original-link and current-selection references; reading or comparing reports does not change the selection. The current podcast is passed as an MCP request header, with every retrieval still subject to backend authorization. TALKFIRST uses the verified title in the opening instructions after link resolution completes. services/mcp_reasoning.py and realtime/mcp_proxy.py share that executor for HTTP and voice reasoning respectively. The browser stores only its conversation ID in memory and uses an HttpOnly owner cookie. Legacy public content preloads are disabled in this mode; the AquaPortal backend remains the authority for content access. The speech pipeline and its audio protocol are unchanged. In OPENAI mode, MCP TALKFIRST greetings enter the HTTP reasoning path after the speech connection is ready, and use the existing external-answer playback and cancellation lifecycle.

Read reasoning, context injection, and tool calling for the conditions and limitations of each path.