docs(dev): explain the combined Vite and FastAPI launcher
This commit is contained in:
+26
-19
@@ -1,18 +1,24 @@
|
||||
# Client architecture
|
||||
|
||||
The [browser client](../../s2s-client) combines a voice UI with a FastAPI server.
|
||||
The [React browser client](../../frontend-s2s) uses the
|
||||
[FastAPI backend](../../s2s-client) for HTTP APIs and a speech WebSocket relay.
|
||||
`start_ui.sh` serves Vite on the existing public port `8420` and FastAPI on
|
||||
loopback port `8421`. Vite forwards backend routes and provides hot reload;
|
||||
Uvicorn reloads Python edits. The launcher supervises both processes.
|
||||
The speech pipeline and optional Node-RED reasoning service run separately.
|
||||
Runtime configuration comes from environment variables and the root `.env`.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Browser[Browser UI]
|
||||
Vite[Vite on public port 8420]
|
||||
Server[Client FastAPI server]
|
||||
Speech[Realtime speech backend]
|
||||
Reasoning[Node-RED or HTTP reasoning provider]
|
||||
Tools[ApiPilot backend]
|
||||
Browser -->|HTTP configuration and proxies| Server
|
||||
Browser <-->|WebSocket via local proxy| Server
|
||||
Browser <-->|Frontend and HMR| Vite
|
||||
Vite -->|HTTP configuration and proxies| Server
|
||||
Vite <-->|Speech WebSocket relay| Server
|
||||
Server <-->|Realtime events and audio| Speech
|
||||
Browser <-->|Optional direct connection| Speech
|
||||
Server -->|Optional transcript reasoning| Reasoning
|
||||
@@ -37,15 +43,14 @@ audio. It is not an unconditional byte-for-byte tunnel.
|
||||
| Source | Responsibility |
|
||||
|---|---|
|
||||
| [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes |
|
||||
| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning |
|
||||
| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar |
|
||||
| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion |
|
||||
| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll |
|
||||
| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs |
|
||||
| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers |
|
||||
| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog |
|
||||
| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction |
|
||||
| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state |
|
||||
| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown |
|
||||
| [vite.config.ts](../../frontend-s2s/vite.config.ts) | Public development endpoint, HMR, allowed hosts, and backend routing |
|
||||
| [App.tsx](../../frontend-s2s/src/App.tsx) | React layout, voice controls, dialogs, profile and reports sidebar |
|
||||
| [controller.ts](../../frontend-s2s/src/realtime/controller.ts) | Application state, settings, context, queue, tools and external reasoning |
|
||||
| [client.ts](../../frontend-s2s/src/realtime/client.ts) | Realtime protocol, session lifecycle, audio and transcript events |
|
||||
| [components](../../frontend-s2s/src/components) | Chat, account, settings and voice components |
|
||||
| [audio](../../frontend-s2s/src/audio) | TypeScript capture/playback worklets, PCM codec, and visualizer |
|
||||
| [legacy-theme.css](../../frontend-s2s/src/styles/legacy-theme.css) | Retained themes and responsive visual styles alongside Tailwind |
|
||||
| [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier |
|
||||
| [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation |
|
||||
| [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging |
|
||||
@@ -65,21 +70,23 @@ audio. It is not an unconditional byte-for-byte tunnel.
|
||||
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
|
||||
| `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation |
|
||||
|
||||
The root serves the UI. `/docs` remains the FastAPI API explorer, while project
|
||||
documentation files are served in the `/docs/` subdirectories. Markdown remains
|
||||
The public root serves the React UI. `/__vite_hmr` carries Vite hot updates;
|
||||
`/v1/realtime` remains the speech connection. `/docs` remains the FastAPI API
|
||||
explorer, while project documentation files are served in the `/docs/`
|
||||
subdirectories. Markdown remains
|
||||
source text, and directories without `index.html` are not browsable pages.
|
||||
Static HTML, JS and CSS
|
||||
responses receive `Cache-Control: no-store` during development.
|
||||
The existing static client remains in `s2s-client` for direct FastAPI and Docker
|
||||
usage; its static responses retain the existing no-cache headers.
|
||||
|
||||
## Audio path
|
||||
|
||||
1. The browser captures microphone audio with echo cancellation, noise
|
||||
suppression and automatic gain-control settings.
|
||||
2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz
|
||||
2. [mic-capture.worklet.ts](../../frontend-s2s/src/audio/mic-capture.worklet.ts) resamples to 16 kHz
|
||||
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
|
||||
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
|
||||
4. Assistant PCM16 audio at 24 kHz is decoded and sent to
|
||||
[audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers,
|
||||
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to
|
||||
[audio-playback.worklet.ts](../../frontend-s2s/src/audio/audio-playback.worklet.ts), which buffers,
|
||||
resamples and fades playback edges. Input speech can clear playback for barge-in.
|
||||
|
||||
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
|
||||
|
||||
Reference in New Issue
Block a user