# Client architecture

The [React browser client](../../vocero-frontend) uses the
[FastAPI backend](../../vocero-api) for HTTP APIs and a speech WebSocket relay.
`start_ui.sh` serves Vite on the existing public port `8420` and FastAPI on
loopback port `8421`. Vite forwards backend routes and provides hot reload;
Uvicorn reloads Python edits. The launcher supervises both processes.
The speech pipeline runs separately.
Runtime configuration comes from environment variables and the root `.env`.

```mermaid
flowchart LR
    Browser[Browser UI]
    Vite[Vite on public port 8420]
    Server[Client FastAPI server]
    Speech[Realtime speech backend]
    Reasoning[OpenAI-compatible HTTP reasoning provider]
    Tools[ApiPilot backend]
    Browser <-->|Frontend and HMR| Vite
    Vite -->|HTTP configuration and proxies| Server
    Vite <-->|Speech WebSocket relay| Server
    Server <-->|Realtime events and audio| Speech
    Browser <-->|Optional direct connection| Speech
    Server -->|Optional transcript reasoning| Reasoning
    Server -->|Tool discovery and execution| Tools
```

## Connection modes

`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
session/queue proxy, or the user can enter a direct URL in settings. See
[usage](usage.md) for the configuration matrix.

The local realtime proxy relays audio and events in both directions. It also
patches selected `session.update` instructions, handles ApiPilot marker-based
tool execution, filters internal text, and can suppress speculative/tool-marker
audio. It is not an unconditional byte-for-byte tunnel.

## Source map

| Source | Responsibility |
|---|---|
| [server.py](../../vocero-api/server.py) | Compatibility entry point for existing `server:app` commands |
| [main.py](../../vocero-api/vocero_api/main.py) | FastAPI assembly, documentation serving and redirects |
| [api](../../vocero-api/vocero_api/api) | HTTP route registration, handlers and compatible response validation |
| [models](../../vocero-api/vocero_api/models) | Pydantic request and response contracts |
| [core](../../vocero-api/vocero_api/core) | Existing environment parsing, logging, URL helpers and diagnostics |
| [providers](../../vocero-api/vocero_api/providers) | Caller context and ApiPilot HTTP integrations |
| [services](../../vocero-api/vocero_api/services) | Context construction, reasoning and tool-context rendering |
| [realtime](../../vocero-api/vocero_api/realtime) | WebSocket relay, events, markers and audio handling |
| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown |
| [vite.config.ts](../../vocero-frontend/vite.config.ts) | Public development endpoint, HMR, allowed hosts, and backend routing |
| [App.tsx](../../vocero-frontend/src/App.tsx) | React layout, voice controls, dialogs, profile and reports sidebar |
| [controller.ts](../../vocero-frontend/src/realtime/controller.ts) | Application state, settings, context, queue, tools and external reasoning |
| [client.ts](../../vocero-frontend/src/realtime/client.ts) | Realtime protocol, session lifecycle, audio and transcript events |
| [components](../../vocero-frontend/src/components) | Chat, caller profile, settings and voice components |
| [audio](../../vocero-frontend/src/audio) | TypeScript capture/playback worklets, PCM codec, and visualizer |
| [legacy-theme.css](../../vocero-frontend/src/styles/legacy-theme.css) | Theme tokens, voice artwork and component appearance |
| [layout.css](../../vocero-frontend/src/styles/layout.css) | Shared responsive layout, transcript sizing and reduced-motion behavior |

## Server routes

| Route | Purpose |
|---|---|
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
| `POST /api/reasoning` | OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
| `GET /api/user/phone` | Configured caller profile lookup |
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
| `POST /api/search` | Serper web search proxy |
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |

The public root serves the React UI. `/__vite_hmr` carries Vite hot updates;
`/v1/realtime` remains the speech connection. `/docs` remains the FastAPI API
explorer, while project documentation files are served in the `/docs/`
subdirectories. Markdown remains
source text, and directories without `index.html` are not browsable pages.
FastAPI serves backend and documentation routes only. The frontend source and
assets live in `vocero-frontend` and are served by Vite.

The uv project and its Python tools are scoped to `vocero-api/`. Response models
describe service-owned structure and retain loose upstream values. See
[development and compatibility](development.md) before tightening a contract.

## Audio path

1. The browser captures microphone audio with echo cancellation, noise
   suppression and automatic gain-control settings.
2. [mic-capture.worklet.ts](../../vocero-frontend/src/audio/mic-capture.worklet.ts) resamples to 16 kHz
   mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to
   [audio-playback.worklet.ts](../../vocero-frontend/src/audio/audio-playback.worklet.ts), which buffers,
   resamples and fades playback edges. Input speech can clear playback for barge-in.
   The client flushes short final chunks on response completion and remains in
   the speaking state until the worklet reports that playback has drained.
   Sequence numbers reject drain notifications from superseded playback.

Microphone meter subscribers are separate from application-state subscribers,
so level updates do not rerender the main React tree.

Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
registration structure intact when editing them. The speech backend's thread
and cancellation model is documented in [realtime API](../speech/realtime-api.md).

## Reasoning and context

The browser triggers external reasoning on completed user transcripts only in
`OPENAI` mode. A non-empty answer replaces the native response through the
external-answer path. These HTTP operations supplement the realtime audio path.
New user speech and session teardown cancel pending reasoning. Turn identity
checks reject stale HTTP results and speech callbacks even if cancellation is late.

Read [reasoning](reasoning.md), [context injection](context-injection.md), and
[tool calling](tool-calling.md) for the conditions and limitations of each path.