valenti/vocero-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(dev): explain the combined Vite and FastAPI launcher

Browse Source
This commit is contained in:
alvaro
2026-10-06 16:53:40 -03:00
parent c8f8aff8c7
commit 99413fc06d
6 changed files with 170 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+15 -7
View File
@@ -6,7 +6,8 @@ documentation, source identifiers, comments, and internal instructions use Engli
| Component | Purpose |
|---|---|
| [s2s-client](s2s-client) | Browser UI, FastAPI server, realtime WebSocket proxy, search and ApiPilot bridges |
| [frontend-s2s](frontend-s2s) | React, TypeScript, and Tailwind browser UI, served by Vite with hot reload |
| [s2s-client](s2s-client) | FastAPI backend, realtime WebSocket proxy, search and ApiPilot bridges; retained static client |
| [speech_to_speech](speech_to_speech) | Queue-driven VAD, speech recognition, language model, and speech synthesis pipeline |
| [s2s-reasoning-engine](s2s-reasoning-engine) | Optional Node-RED HTTP reasoning and conversation logging flows |
@@ -20,16 +21,23 @@ The speech backend needs its own installed model and runtime dependencies;
From the repository root, point the client at an existing realtime backend:
```bash
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh
```
Open <http://localhost:8420/> and allow microphone access. The launcher creates
`.venv`, installs missing UI requirements, loads the root `.env` if present, and
starts the server. For the local WebSocket proxy instead of a direct browser
connection, configure `S2S_PROXY_TARGET` and leave `SPEECH_TO_SPEECH_URL` unset.
`.venv`, installs missing Python requirements and locked frontend dependencies,
and loads the root `.env` if present. Vite listens on `UI_PORT` (default `8420`)
and forwards HTTP/WebSocket backend routes to FastAPI on `127.0.0.1:8421`.
Both servers reload during development; stopping the launcher stops both.
Leave `SPEECH_TO_SPEECH_URL` unset to use the same-origin speech proxy.
To build the UI container, use the repository root as the Docker build context so
the root documentation is included:
The launcher requires Python 3 with venv support, Node.js 22.18+ on the 22.x
line or Node.js 24+, npm, Bash 4.4+, `setsid`, and `curl`. For staging hostnames,
set `S2S_DEV_ALLOWED_HOSTS` and forward the Vite HMR WebSocket through the existing
HTTPS reverse proxy. See [development setup](frontend-s2s/README.md).
The existing Docker image still serves the retained static client. To build it,
use the repository root as the Docker build context so documentation is included:
```bash
docker build -f s2s-client/Dockerfile -t s2s-client .
docs/client/architecture.md
+26 -19
View File
@@ -1,18 +1,24 @@
# Client architecture
The [browser client](../../s2s-client) combines a voice UI with a FastAPI server.
The [React browser client](../../frontend-s2s) uses the
[FastAPI backend](../../s2s-client) for HTTP APIs and a speech WebSocket relay.
`start_ui.sh` serves Vite on the existing public port `8420` and FastAPI on
loopback port `8421`. Vite forwards backend routes and provides hot reload;
Uvicorn reloads Python edits. The launcher supervises both processes.
The speech pipeline and optional Node-RED reasoning service run separately.
Runtime configuration comes from environment variables and the root `.env`.
```mermaid
flowchart LR
Browser[Browser UI]
Vite[Vite on public port 8420]
Server[Client FastAPI server]
Speech[Realtime speech backend]
Reasoning[Node-RED or HTTP reasoning provider]
Tools[ApiPilot backend]
Browser -->|HTTP configuration and proxies| Server
Browser <-->|WebSocket via local proxy| Server
Browser <-->|Frontend and HMR| Vite
Vite -->|HTTP configuration and proxies| Server
Vite <-->|Speech WebSocket relay| Server
Server <-->|Realtime events and audio| Speech
Browser <-->|Optional direct connection| Speech
Server -->|Optional transcript reasoning| Reasoning
@@ -37,15 +43,14 @@ audio. It is not an unconditional byte-for-byte tunnel.
| Source | Responsibility |
|---|---|
| [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes |
| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning |
| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar |
| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion |
| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll |
| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs |
| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers |
| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog |
| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction |
| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state |
| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown |
| [vite.config.ts](../../frontend-s2s/vite.config.ts) | Public development endpoint, HMR, allowed hosts, and backend routing |
| [App.tsx](../../frontend-s2s/src/App.tsx) | React layout, voice controls, dialogs, profile and reports sidebar |
| [controller.ts](../../frontend-s2s/src/realtime/controller.ts) | Application state, settings, context, queue, tools and external reasoning |
| [client.ts](../../frontend-s2s/src/realtime/client.ts) | Realtime protocol, session lifecycle, audio and transcript events |
| [components](../../frontend-s2s/src/components) | Chat, account, settings and voice components |
| [audio](../../frontend-s2s/src/audio) | TypeScript capture/playback worklets, PCM codec, and visualizer |
| [legacy-theme.css](../../frontend-s2s/src/styles/legacy-theme.css) | Retained themes and responsive visual styles alongside Tailwind |
| [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier |
| [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation |
| [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging |
@@ -65,21 +70,23 @@ audio. It is not an unconditional byte-for-byte tunnel.
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
| `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation |
The root serves the UI. `/docs` remains the FastAPI API explorer, while project
documentation files are served in the `/docs/` subdirectories. Markdown remains
The public root serves the React UI. `/__vite_hmr` carries Vite hot updates;
`/v1/realtime` remains the speech connection. `/docs` remains the FastAPI API
explorer, while project documentation files are served in the `/docs/`
subdirectories. Markdown remains
source text, and directories without `index.html` are not browsable pages.
Static HTML, JS and CSS
responses receive `Cache-Control: no-store` during development.
The existing static client remains in `s2s-client` for direct FastAPI and Docker
usage; its static responses retain the existing no-cache headers.
## Audio path
1. The browser captures microphone audio with echo cancellation, noise
suppression and automatic gain-control settings.
2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz
2. [mic-capture.worklet.ts](../../frontend-s2s/src/audio/mic-capture.worklet.ts) resamples to 16 kHz
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
4. Assistant PCM16 audio at 24 kHz is decoded and sent to
[audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers,
4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to
[audio-playback.worklet.ts](../../frontend-s2s/src/audio/audio-playback.worklet.ts), which buffers,
resamples and fades playback edges. Input speech can clear playback for barge-in.
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
docs/client/troubleshooting.md
+18
View File
@@ -22,6 +22,24 @@ In browser DevTools, inspect the WebSocket handshake and `session.created`,
inside `session`. A cached browser module can disagree with current server
code; inspect served asset versions and reload with cache disabled when needed.
## Launcher and hot reload
`start_ui.sh` serves Vite on `8420` and FastAPI on loopback port `8421` by
default. Check both processes in the launcher/PM2 logs. If either exits, the
launcher stops the other and returns a failure. Port conflicts are errors;
use distinct `UI_PORT` and `S2S_BACKEND_PORT` values for additional worktrees.
A Vite "Blocked request" response means the public hostname is missing from
`S2S_DEV_ALLOWED_HOSTS`. Configure it without a scheme or port and restart the
launcher. If the page loads but edits do not appear, check the browser's
`/__vite_hmr` WebSocket and the reverse proxy's upgrade handling. Speech uses
the separate `/v1/realtime` WebSocket on the same public origin.
For a startup dependency error, check Node/npm on the process manager's PATH,
Python venv support, and `setsid`/`curl`. The first start installs dependencies;
later starts reuse them until frontend manifests change. Changes to root
`.env` or launcher settings require a restart.
## The assistant promises a tool but nothing happens
Follow the actual execution mechanism:
docs/client/usage.md
+22 -7
View File
@@ -9,23 +9,35 @@ reasoning, caller context, ApiPilot tools and load-balancer requests.
From the repository root:
```bash
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh
```
The launcher loads the root `.env`, creates `.venv` if needed, installs the
client requirements when the dependency check fails, and runs uvicorn on port
8420. `UI_PORT` overrides the UI port. Because `.env` is sourced by the script,
values in that file can replace variables supplied on the command line.
client requirements when the dependency check fails, and installs the locked
frontend dependencies when needed. Vite serves the React frontend on port
8420 and proxies the backend routes to Uvicorn on `127.0.0.1:8421`.
`UI_PORT` overrides the public port; `S2S_BACKEND_PORT` overrides the internal
port. Both servers reload source edits, and stopping the launcher stops both.
The speech pipeline remains a separate service. Because `.env` is sourced by
the script, values in that file can replace variables supplied on the command line.
To install and start the UI manually:
Use Node.js 22.18+ on the 22.x line or Node.js 24+, npm, Python 3 with venv,
Bash 4.4+, `setsid`, and `curl`. Remote staging access also needs the exact
hostname in `S2S_DEV_ALLOWED_HOSTS`, HTTPS, and WebSocket forwarding for speech
and Vite HMR. See [frontend development setup](../../frontend-s2s/README.md).
To install and start the backend manually:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r s2s-client/requirements.txt
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime \
.venv/bin/python -m uvicorn --app-dir s2s-client server:app --port 8420
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime \
.venv/bin/python -m uvicorn --app-dir s2s-client server:app \
--host 127.0.0.1 --port 8421 --reload --reload-dir s2s-client
```
Then, in a second terminal, run `cd frontend-s2s`, `npm ci`, and `npm run dev`.
Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
access, and speak. Microphone access requires HTTPS or a local browser origin;
use HTTPS when accessing the UI from another machine.
@@ -72,6 +84,9 @@ server; it is not a universally available local service.
## Docker
This existing image serves the retained static client. It does not run the
React/Vite development launcher.
Use the project root as the build context:
```bash
frontend-s2s/README.md
+77 -38
View File
@@ -6,15 +6,72 @@ rendering and state, including the realtime WebSocket and audio integration.
## Run during development
Use Node.js 22.18 or later on the 22.x line, or Node.js 24+. Install dependencies
with npm; the lockfile is committed.
From the repository root, run `./start_ui.sh`. Open `http://localhost:8420`.
Add `?debug=true` to access settings, tools, and custom RAG context. Existing
`phoneNumber` and `episode_id` query parameters are supported.
1. Start the existing backend using `./start_ui.sh` from your configured S2S
checkout, or reuse an already-running instance. It normally listens on port
`8420` and continues serving the original static UI there. Its Python
environment, `.env`, speech service, and upstream services must be configured
as before.
2. In a second terminal, start this frontend:
The launcher requires Python 3 with venv support, Node.js 22.18 or later on the
22.x line or Node.js 24+, npm, Bash 4.4+, `setsid` (util-linux), and `curl`.
It creates `.venv` and installs missing Python dependencies. It runs
`npm ci --include=dev` on first start and when either frontend package manifest
changes. Subsequent restarts reuse the installed dependencies.
| Server | Default address | Purpose |
| ------- | ---------------- | ---------------------------------------------- |
| Vite | `0.0.0.0:8420` | Public React UI, backend proxy, and hot reload |
| FastAPI | `127.0.0.1:8421` | Internal HTTP APIs and speech WebSocket relay |
`UI_PORT` preserves the existing public port override. `UI_HOST` defaults to
`0.0.0.0`; `S2S_BACKEND_PORT` changes the loopback backend port. The two ports
must differ. Occupied ports fail startup rather than silently selecting a new
endpoint. The launcher waits for FastAPI before starting Vite, stops both
process groups on Ctrl+C/SIGTERM, and exits with a failure if either server exits.
Uvicorn watches only `s2s-client`, avoiding frontend and dependency directories.
The Vite child explicitly uses `NODE_ENV=development`, including when a process
manager sets `NODE_ENV=production`, so React Fast Refresh remains enabled.
Vite serves React and Tailwind with hot updates. It proxies `/api`,
`/v1/realtime` (including WebSockets), OAuth, and backend documentation routes
to FastAPI. The browser uses one origin on port `8420`, so there is no separate
API origin or development CORS setup. The proxy preserves the incoming Host
header because FastAPI uses it to construct the speech WebSocket URL.
Runtime UI flags, voices, reasoning mode, prompts, and the speech URL still come from FastAPI's
`/api/config`. A backend explicitly configured with `SPEECH_TO_SPEECH_URL`
continues to return that direct speech endpoint.
The launcher sources the root `.env`, retaining its precedence over shell
variables. The Vite configuration reads this project's environment files and
the inherited `S2S_*` settings. Do not copy the repository's backend secrets
into this directory or into `VITE_*`
variables, which are exposed to browser code.
## Staging with hot reload
PM2 can continue running the root `start_ui.sh` as its existing `s2s_ui` process.
Ensure the Node/npm versions above are on PM2's PATH and deploy `frontend-s2s`
alongside `s2s-client` and `docs`. The reverse proxy still targets port `8420`.
Set the exact public hostname in the root `.env`, without scheme or port:
```dotenv
S2S_DEV_ALLOWED_HOSTS=voice-staging.example.com
```
Replace the example with the existing staging hostname; use commas for multiple
hosts. IP addresses and localhost are accepted without this setting.
The HTTPS reverse proxy must preserve `Host`, `X-Forwarded-Host` and
`X-Forwarded-Proto`, and forward WebSocket upgrades for `/v1/realtime` and
`/__vite_hmr`. HMR uses the browser's public origin, including HTTPS/WSS, so no
second public port is needed. The browser needs HTTPS for microphone access
outside localhost. Restart `s2s_ui` after changing launcher or environment
settings; source edits use the reloaders. The speech service remains separately
managed.
## Run the servers separately
Start FastAPI on `127.0.0.1:8421` as described in the
[client usage guide](../docs/client/usage.md), then run:
```sh
cd frontend-s2s
@@ -22,32 +79,11 @@ with npm; the lockfile is committed.
npm run dev
```
3. Open `http://127.0.0.1:5173`. Add `?debug=true` to access settings, tools,
and custom RAG context. Existing `phoneNumber` and `episode_id` query
parameters are supported.
Vite serves React and Tailwind with hot updates. It proxies `/api`,
`/v1/realtime` (including WebSockets), OAuth, and backend documentation routes
to FastAPI. The browser uses one origin on port `5173`, so there is no separate
API origin or development CORS setup. The proxy preserves the incoming Host
header because FastAPI uses it to construct the speech WebSocket URL.
For a different backend, copy `.env.example` to `.env.local` and set
`S2S_BACKEND_URL`. This is a **Vite server** setting. Runtime UI flags, voices,
reasoning mode, prompts, and the speech URL still come from FastAPI's
`/api/config`. A backend explicitly configured with `SPEECH_TO_SPEECH_URL`
continues to return that direct speech endpoint.
The Vite configuration reads only this project's environment files. Do not
copy the repository's backend secrets into this directory or into `VITE_*`
variables, which are exposed to browser code.
For access through a development reverse proxy, set the exact hostname in
`S2S_DEV_ALLOWED_HOSTS`. Use `npm run dev -- --host 0.0.0.0` only when Vite must
listen beyond localhost. The browser needs HTTPS for microphone access outside
localhost, and that proxy must forward WebSocket upgrades for both speech and
Vite HMR. Wiring the staging entrypoint and launcher is a separate step:
**this port does not change `start_ui.sh`.**
Standalone Vite binds `127.0.0.1:8420`; add `-- --host 0.0.0.0` for remote
access. For a different backend, copy `.env.example` to `.env.local` and set
`S2S_BACKEND_URL`. The combined launcher overrides that setting to point at the
backend it starts. Do not start standalone Vite while the combined launcher
already owns the public port.
Component and CSS changes use Fast Refresh. Editing the session controller or
audio modules can require restarting a conversation or a page reload. Teardown
@@ -61,25 +97,28 @@ React plugins replacing Next.js-specific tooling. Executable configuration
files use TypeScript; TypeScript's own project settings use JSON.
| Command | Purpose |
| -------------------- | ----------------------------------------------------------- |
| `npm run dev` | Vite development server on port 5173, with a fixed port |
| ----------------------- | ----------------------------------------------------------- |
| `npm run dev` | Vite development server on port 8420, with a fixed port |
| `npm run check` | Formatting, strict typed ESLint, type checking, and tests |
| `npm run lint:fix` | Apply ESLint fixes |
| `npm run format` | Format source and sort Tailwind classes |
| `npm run typecheck` | Check application, tests, audio worklets, and configuration |
| `npm test` | Run Vitest and React Testing Library regression tests |
| `npm run test:watch` | Watch tests while editing |
| `npm run test:launcher` | Test launcher supervision and signal handling on Linux |
| `npm run build` | Typecheck and compile the app and worklets into `dist/` |
| `npm run preview` | Inspect the compiled output locally |
No build step is needed before `npm run dev`. `dist/` is generated and ignored
by Git. Build output is available for later serving integration; this change
does not mount it in FastAPI or change the staging launch process.
by Git. The launcher serves source through Vite; it does not use `dist/` or
`vite preview`.
Tests cover session initialization, response sequencing, cumulative transcripts,
interruptions, queue cancellation, microphone-permission cancellation, tool
results, external reasoning and `LOG_ONLY`, late-response cleanup, the idle
timeout, settings, RAG, profiles, and sanitized Markdown/export behavior.
Launcher tests start temporary local children to check readiness, environment
forwarding, failures, and shutdown of complete process groups.
Vitest handles Node 26's experimental web storage behavior internally.
## Source layout
s2s-client/README.md
+9 -4
View File
@@ -10,19 +10,24 @@ short_description: Voice chat over WebSocket against a HF speech-to-speech
hf_oauth: true
---
# Realtime voice client
# Realtime voice backend and retained static client
Browser voice-chat UI and FastAPI server for an OpenAI Realtime-compatible
FastAPI server and retained static voice-chat UI for an OpenAI Realtime-compatible
speech-to-speech backend. The browser streams microphone audio over WebSocket,
plays assistant audio, displays transcripts, and offers Spanish controls.
The root `start_ui.sh` now serves the [React frontend](../frontend-s2s) with
Vite on public port `8420`, proxying backend requests here on loopback port
`8421`. This directory's static UI remains available when FastAPI is run
directly or through the existing Docker image.
The complete [usage guide](../docs/client/usage.md) covers local startup,
connection modes, Docker, settings, and usage limits. See the
[documentation index](../docs/README.md), [architecture](../docs/client/architecture.md),
and [change history](CHANGELOG.md) for implementation details and history.
Assistant replies render Markdown with Marked and DOMPurify. Their pinned,
checked-in browser modules are served from `vendor/`, including upstream
The retained static client's assistant replies render Markdown with Marked and
DOMPurify. Their pinned, checked-in browser modules are served from `vendor/`, including upstream
licenses, so the UI does not require a CDN or a Node.js runtime. To refresh the
browser dependencies with Node.js 20 or newer after updating `package.json`: