# Client architecture

The [browser client](../../s2s-client) combines a voice UI with a FastAPI server.
The speech pipeline and optional Node-RED reasoning service run separately.
Runtime configuration comes from environment variables and the root `.env`.

```mermaid
flowchart LR
    Browser[Browser UI]
    Server[Client FastAPI server]
    Speech[Realtime speech backend]
    Reasoning[Node-RED or HTTP reasoning provider]
    Tools[ApiPilot backend]
    Browser -->|HTTP configuration and proxies| Server
    Browser <-->|WebSocket via local proxy| Server
    Server <-->|Realtime events and audio| Speech
    Browser <-->|Optional direct connection| Speech
    Server -->|Optional transcript reasoning| Reasoning
    Server -->|Tool discovery and execution| Tools
```

## Connection modes

`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
session/queue proxy, or the user can enter a direct URL in settings. See
[usage](usage.md) for the configuration matrix.

The local realtime proxy relays audio and events in both directions. It also
patches selected `session.update` instructions, handles ApiPilot marker-based
tool execution, filters internal text, and can suppress speculative/tool-marker
audio. It is not an unconditional byte-for-byte tunnel.

## Source map

| Source | Responsibility |
|---|---|
| [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes |
| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning |
| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar |
| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion |
| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll |
| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs |
| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers |
| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog |
| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction |
| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state |
| [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier |
| [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation |
| [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging |

## Server routes

| Route | Purpose |
|---|---|
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
| `POST /api/reasoning` | Node-RED or OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
| `GET /api/user/phone` | Configured caller profile lookup |
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
| `POST /api/search` | Serper web search proxy |
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
| `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation |

The root serves the UI. `/docs` remains the FastAPI API explorer, while project
documentation files are served in the `/docs/` subdirectories. Markdown remains
source text, and directories without `index.html` are not browsable pages.
Static HTML, JS and CSS
responses receive `Cache-Control: no-store` during development.

## Audio path

1. The browser captures microphone audio with echo cancellation, noise
   suppression and automatic gain-control settings.
2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz
   mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at 24 kHz is decoded and sent to
   [audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers,
   resamples and fades playback edges. Input speech can clear playback for barge-in.

Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
registration structure intact when editing them. The speech backend's thread
and cancellation model is documented in [realtime API](../speech/realtime-api.md).

## Reasoning and context

The browser triggers external reasoning on completed user transcripts only in
`OPENAI` or `NODE_RED` mode. Node-RED `log_only=true` responses leave native speech
output running; an answer-producing response uses the external-answer path.
The server can independently forward assistant text to Node-RED for paired
conversation logs. These HTTP operations supplement the realtime audio path.

Read [reasoning](reasoning.md), [context injection](context-injection.md), and
[tool calling](tool-calling.md) for the conditions and limitations of each path.