vocero-client/docs/client/architecture.md

Client architecture

The browser client combines a voice UI with a FastAPI server. The speech pipeline and optional Node-RED reasoning service run separately. Runtime configuration comes from environment variables and the root .env.

flowchart LR
    Browser[Browser UI]
    Server[Client FastAPI server]
    Speech[Realtime speech backend]
    Reasoning[Node-RED or HTTP reasoning provider]
    Tools[ApiPilot backend]
    Browser -->|HTTP configuration and proxies| Server
    Browser <-->|WebSocket via local proxy| Server
    Server <-->|Realtime events and audio| Speech
    Browser <-->|Optional direct connection| Speech
    Server -->|Optional transcript reasoning| Reasoning
    Server -->|Tool discovery and execution| Tools

Connection modes

SPEECH_TO_SPEECH_URL pins a direct browser connection. Otherwise, S2S_PROXY_TARGET exposes the same-origin /v1/realtime proxy. Either setting disables load-balancer mode. With neither set, LOAD_BALANCER_URL enables the session/queue proxy, or the user can enter a direct URL in settings. See usage for the configuration matrix.

The local realtime proxy relays audio and events in both directions. It also patches selected session.update instructions, handles ApiPilot marker-based tool execution, filters internal text, and can suppress speculative/tool-marker audio. It is not an unconditional byte-for-byte tunnel.

Source map

Source	Responsibility
server.py	Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes
main.js	UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning
index.html	Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar
style.css	Dark aurora/glass theme, light theme, responsive layout and motion
ui/chat.js	Transcript history, tool results, export and autoscroll
ui/account.js	Account controls, queue and usage dialogs
ui/dom.js	DOM, escaping and error-display helpers
ws/s2s-ws-client.js	Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog
ws/codec.js	PCM/base64 conversion and transcript extraction
ws/orb-visualizer.js	Audio-reactive orb state
auth.py	Hugging Face OAuth and caller identity/tier
limiter.py	SQLite daily talk-time reservations and reconciliation
flows.json	Node-RED reasoning and assistant-response logging

Server routes

Route	Purpose
GET /api/config	Connection settings, primary prompt, reasoning/theme/start modes, and feature flags
WS /v1/realtime	Local realtime proxy to S2S_PROXY_TARGET
POST /api/reasoning	Node-RED or OpenAI-compatible HTTP reasoning; disabled in PROXY mode
GET /api/user/phone	Configured caller profile lookup
GET /api/user/reports	Configured scheduled report lookup, subject to report-injection controls
GET /api/apipilot/tools	Tool discovery with optional authorization and cache refresh
POST /api/apipilot/execute	Validated tool execution returning a success/data/error envelope
POST /api/search	Serper web search proxy
/api/session, /api/queue/*	Load-balancer session and waiting-queue proxies
/api/me, /api/session/heartbeat, /api/session/end	Identity, reservation extension and usage reconciliation

The root serves the UI. /docs remains the FastAPI API explorer, while project documentation files are served in the /docs/ subdirectories. Markdown remains source text, and directories without index.html are not browsable pages. Static HTML, JS and CSS responses receive Cache-Control: no-store during development.

Audio path

1. The browser captures microphone audio with echo cancellation, noise suppression and automatic gain-control settings.
2. mic-capture.js resamples to 16 kHz mono PCM16. The browser sends base64 input_audio_buffer.append frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at 24 kHz is decoded and sent to audio-playback.js, which buffers, resamples and fades playback edges. Input speech can clear playback for barge-in.

Audio worklets are loaded through AudioWorklet.addModule; keep their processor registration structure intact when editing them. The speech backend's thread and cancellation model is documented in realtime API.

Reasoning and context

The browser triggers external reasoning on completed user transcripts only in OPENAI or NODE_RED mode. Node-RED log_only=true responses leave native speech output running; an answer-producing response uses the external-answer path. The server can independently forward assistant text to Node-RED for paired conversation logs. These HTTP operations supplement the realtime audio path.

Read reasoning, context injection, and tool calling for the conditions and limitations of each path.