vocero-client/docs/client/architecture.md

Client architecture

The React browser client uses the FastAPI backend for HTTP APIs and a speech WebSocket relay. start_ui.sh serves Vite on the existing public port 8420 and FastAPI on loopback port 8421. Vite forwards backend routes and provides hot reload; Uvicorn reloads Python edits. The launcher supervises both processes. The speech pipeline runs separately. Runtime configuration comes from environment variables and the root .env.

flowchart LR
    Browser[Browser UI]
    Vite[Vite on public port 8420]
    Server[Client FastAPI server]
    Speech[Realtime speech backend]
    Reasoning[OpenAI-compatible HTTP reasoning provider]
    Tools[ApiPilot backend]
    Browser <-->|Frontend and HMR| Vite
    Vite -->|HTTP configuration and proxies| Server
    Vite <-->|Speech WebSocket relay| Server
    Server <-->|Realtime events and audio| Speech
    Browser <-->|Optional direct connection| Speech
    Server -->|Optional transcript reasoning| Reasoning
    Server -->|Tool discovery and execution| Tools

Connection modes

SPEECH_TO_SPEECH_URL pins a direct browser connection. Otherwise, S2S_PROXY_TARGET exposes the same-origin /v1/realtime proxy. Either setting disables load-balancer mode. With neither set, LOAD_BALANCER_URL enables the session/queue proxy, or the user can enter a direct URL in settings. See usage for the configuration matrix.

The local realtime proxy relays audio and events in both directions. It also patches selected session.update instructions, handles ApiPilot marker-based tool execution, filters internal text, and can suppress speculative/tool-marker audio. It is not an unconditional byte-for-byte tunnel.

Source map

Source	Responsibility
server.py	Compatibility entry point for existing server:app commands
main.py	FastAPI assembly, documentation serving and redirects
api	HTTP route registration, handlers and compatible response validation
models	Pydantic request and response contracts
core	Existing environment parsing, logging, URL helpers and diagnostics
providers	Caller context and ApiPilot HTTP integrations
services	Context construction, reasoning and tool-context rendering
realtime	WebSocket relay, events, markers and audio handling
start_ui.sh	Dependency setup, ports, readiness, reload processes, and shutdown
vite.config.ts	Public development endpoint, HMR, allowed hosts, and backend routing
App.tsx	React layout, voice controls, dialogs, profile and reports sidebar
controller.ts	Application state, settings, context, queue, tools and external reasoning
client.ts	Realtime protocol, session lifecycle, audio and transcript events
components	Chat, caller profile, settings and voice components
audio	TypeScript capture/playback worklets, PCM codec, and visualizer
legacy-theme.css	Retained themes and responsive visual styles alongside Tailwind

Server routes

Route	Purpose
GET /api/config	Connection settings, primary prompt, reasoning/theme/start modes, and feature flags
WS /v1/realtime	Local realtime proxy to S2S_PROXY_TARGET
POST /api/reasoning	OpenAI-compatible HTTP reasoning; disabled in PROXY mode
GET /api/user/phone	Configured caller profile lookup
GET /api/user/reports	Configured scheduled report lookup, subject to report-injection controls
GET /api/apipilot/tools	Tool discovery with optional authorization and cache refresh
POST /api/apipilot/execute	Validated tool execution returning a success/data/error envelope
POST /api/search	Serper web search proxy
/api/session, /api/queue/*	Load-balancer session and waiting-queue proxies

The public root serves the React UI. /__vite_hmr carries Vite hot updates; /v1/realtime remains the speech connection. /docs remains the FastAPI API explorer, while project documentation files are served in the /docs/ subdirectories. Markdown remains source text, and directories without index.html are not browsable pages. FastAPI serves backend and documentation routes only. The frontend source and assets live in s2s-frontend and are served by Vite.

The uv project and its Python tools are scoped to s2s-client/. Response models describe service-owned structure and retain loose upstream values. See development and compatibility before tightening a contract.

Audio path

1. The browser captures microphone audio with echo cancellation, noise suppression and automatic gain-control settings.
2. mic-capture.worklet.ts resamples to 16 kHz mono PCM16. The browser sends base64 input_audio_buffer.append frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to audio-playback.worklet.ts, which buffers, resamples and fades playback edges. Input speech can clear playback for barge-in.

Audio worklets are loaded through AudioWorklet.addModule; keep their processor registration structure intact when editing them. The speech backend's thread and cancellation model is documented in realtime API.

Reasoning and context

The browser triggers external reasoning on completed user transcripts only in OPENAI mode. A non-empty answer replaces the native response through the external-answer path. These HTTP operations supplement the realtime audio path.

Read reasoning, context injection, and tool calling for the conditions and limitations of each path.