From 99413fc06dd30f242c041d4867260275f76507f9 Mon Sep 17 00:00:00 2001 From: Alvaro Parker Date: Tue, 6 Oct 2026 16:53:40 -0300 Subject: [PATCH] docs(dev): explain the combined Vite and FastAPI launcher --- README.md | 22 ++++-- docs/client/architecture.md | 45 ++++++------ docs/client/troubleshooting.md | 18 +++++ docs/client/usage.md | 29 ++++++-- frontend-s2s/README.md | 121 ++++++++++++++++++++++----------- s2s-client/README.md | 13 ++-- 6 files changed, 170 insertions(+), 78 deletions(-) diff --git a/README.md b/README.md index 1105d3a..b27f90d 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,8 @@ documentation, source identifiers, comments, and internal instructions use Engli | Component | Purpose | |---|---| -| [s2s-client](s2s-client) | Browser UI, FastAPI server, realtime WebSocket proxy, search and ApiPilot bridges | +| [frontend-s2s](frontend-s2s) | React, TypeScript, and Tailwind browser UI, served by Vite with hot reload | +| [s2s-client](s2s-client) | FastAPI backend, realtime WebSocket proxy, search and ApiPilot bridges; retained static client | | [speech_to_speech](speech_to_speech) | Queue-driven VAD, speech recognition, language model, and speech synthesis pipeline | | [s2s-reasoning-engine](s2s-reasoning-engine) | Optional Node-RED HTTP reasoning and conversation logging flows | @@ -20,16 +21,23 @@ The speech backend needs its own installed model and runtime dependencies; From the repository root, point the client at an existing realtime backend: ```bash -SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh +S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh ``` Open and allow microphone access. The launcher creates -`.venv`, installs missing UI requirements, loads the root `.env` if present, and -starts the server. For the local WebSocket proxy instead of a direct browser -connection, configure `S2S_PROXY_TARGET` and leave `SPEECH_TO_SPEECH_URL` unset. +`.venv`, installs missing Python requirements and locked frontend dependencies, +and loads the root `.env` if present. Vite listens on `UI_PORT` (default `8420`) +and forwards HTTP/WebSocket backend routes to FastAPI on `127.0.0.1:8421`. +Both servers reload during development; stopping the launcher stops both. +Leave `SPEECH_TO_SPEECH_URL` unset to use the same-origin speech proxy. -To build the UI container, use the repository root as the Docker build context so -the root documentation is included: +The launcher requires Python 3 with venv support, Node.js 22.18+ on the 22.x +line or Node.js 24+, npm, Bash 4.4+, `setsid`, and `curl`. For staging hostnames, +set `S2S_DEV_ALLOWED_HOSTS` and forward the Vite HMR WebSocket through the existing +HTTPS reverse proxy. See [development setup](frontend-s2s/README.md). + +The existing Docker image still serves the retained static client. To build it, +use the repository root as the Docker build context so documentation is included: ```bash docker build -f s2s-client/Dockerfile -t s2s-client . diff --git a/docs/client/architecture.md b/docs/client/architecture.md index 76dd251..9d319ad 100644 --- a/docs/client/architecture.md +++ b/docs/client/architecture.md @@ -1,18 +1,24 @@ # Client architecture -The [browser client](../../s2s-client) combines a voice UI with a FastAPI server. +The [React browser client](../../frontend-s2s) uses the +[FastAPI backend](../../s2s-client) for HTTP APIs and a speech WebSocket relay. +`start_ui.sh` serves Vite on the existing public port `8420` and FastAPI on +loopback port `8421`. Vite forwards backend routes and provides hot reload; +Uvicorn reloads Python edits. The launcher supervises both processes. The speech pipeline and optional Node-RED reasoning service run separately. Runtime configuration comes from environment variables and the root `.env`. ```mermaid flowchart LR Browser[Browser UI] + Vite[Vite on public port 8420] Server[Client FastAPI server] Speech[Realtime speech backend] Reasoning[Node-RED or HTTP reasoning provider] Tools[ApiPilot backend] - Browser -->|HTTP configuration and proxies| Server - Browser <-->|WebSocket via local proxy| Server + Browser <-->|Frontend and HMR| Vite + Vite -->|HTTP configuration and proxies| Server + Vite <-->|Speech WebSocket relay| Server Server <-->|Realtime events and audio| Speech Browser <-->|Optional direct connection| Speech Server -->|Optional transcript reasoning| Reasoning @@ -37,15 +43,14 @@ audio. It is not an unconditional byte-for-byte tunnel. | Source | Responsibility | |---|---| | [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes | -| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning | -| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar | -| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion | -| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll | -| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs | -| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers | -| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog | -| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction | -| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state | +| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown | +| [vite.config.ts](../../frontend-s2s/vite.config.ts) | Public development endpoint, HMR, allowed hosts, and backend routing | +| [App.tsx](../../frontend-s2s/src/App.tsx) | React layout, voice controls, dialogs, profile and reports sidebar | +| [controller.ts](../../frontend-s2s/src/realtime/controller.ts) | Application state, settings, context, queue, tools and external reasoning | +| [client.ts](../../frontend-s2s/src/realtime/client.ts) | Realtime protocol, session lifecycle, audio and transcript events | +| [components](../../frontend-s2s/src/components) | Chat, account, settings and voice components | +| [audio](../../frontend-s2s/src/audio) | TypeScript capture/playback worklets, PCM codec, and visualizer | +| [legacy-theme.css](../../frontend-s2s/src/styles/legacy-theme.css) | Retained themes and responsive visual styles alongside Tailwind | | [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier | | [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation | | [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging | @@ -65,21 +70,23 @@ audio. It is not an unconditional byte-for-byte tunnel. | `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies | | `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation | -The root serves the UI. `/docs` remains the FastAPI API explorer, while project -documentation files are served in the `/docs/` subdirectories. Markdown remains +The public root serves the React UI. `/__vite_hmr` carries Vite hot updates; +`/v1/realtime` remains the speech connection. `/docs` remains the FastAPI API +explorer, while project documentation files are served in the `/docs/` +subdirectories. Markdown remains source text, and directories without `index.html` are not browsable pages. -Static HTML, JS and CSS -responses receive `Cache-Control: no-store` during development. +The existing static client remains in `s2s-client` for direct FastAPI and Docker +usage; its static responses retain the existing no-cache headers. ## Audio path 1. The browser captures microphone audio with echo cancellation, noise suppression and automatic gain-control settings. -2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz +2. [mic-capture.worklet.ts](../../frontend-s2s/src/audio/mic-capture.worklet.ts) resamples to 16 kHz mono PCM16. The browser sends base64 `input_audio_buffer.append` frames. 3. The backend processes VAD → STT → LLM → TTS and emits realtime events. -4. Assistant PCM16 audio at 24 kHz is decoded and sent to - [audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers, +4. Assistant PCM16 audio at the current 16 kHz protocol rate is decoded and sent to + [audio-playback.worklet.ts](../../frontend-s2s/src/audio/audio-playback.worklet.ts), which buffers, resamples and fades playback edges. Input speech can clear playback for barge-in. Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor diff --git a/docs/client/troubleshooting.md b/docs/client/troubleshooting.md index c68c6b1..2e9c5c4 100644 --- a/docs/client/troubleshooting.md +++ b/docs/client/troubleshooting.md @@ -22,6 +22,24 @@ In browser DevTools, inspect the WebSocket handshake and `session.created`, inside `session`. A cached browser module can disagree with current server code; inspect served asset versions and reload with cache disabled when needed. +## Launcher and hot reload + +`start_ui.sh` serves Vite on `8420` and FastAPI on loopback port `8421` by +default. Check both processes in the launcher/PM2 logs. If either exits, the +launcher stops the other and returns a failure. Port conflicts are errors; +use distinct `UI_PORT` and `S2S_BACKEND_PORT` values for additional worktrees. + +A Vite "Blocked request" response means the public hostname is missing from +`S2S_DEV_ALLOWED_HOSTS`. Configure it without a scheme or port and restart the +launcher. If the page loads but edits do not appear, check the browser's +`/__vite_hmr` WebSocket and the reverse proxy's upgrade handling. Speech uses +the separate `/v1/realtime` WebSocket on the same public origin. + +For a startup dependency error, check Node/npm on the process manager's PATH, +Python venv support, and `setsid`/`curl`. The first start installs dependencies; +later starts reuse them until frontend manifests change. Changes to root +`.env` or launcher settings require a restart. + ## The assistant promises a tool but nothing happens Follow the actual execution mechanism: diff --git a/docs/client/usage.md b/docs/client/usage.md index 2c9caf4..cb72387 100644 --- a/docs/client/usage.md +++ b/docs/client/usage.md @@ -9,23 +9,35 @@ reasoning, caller context, ApiPilot tools and load-balancer requests. From the repository root: ```bash -SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh +S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh ``` The launcher loads the root `.env`, creates `.venv` if needed, installs the -client requirements when the dependency check fails, and runs uvicorn on port -8420. `UI_PORT` overrides the UI port. Because `.env` is sourced by the script, -values in that file can replace variables supplied on the command line. +client requirements when the dependency check fails, and installs the locked +frontend dependencies when needed. Vite serves the React frontend on port +8420 and proxies the backend routes to Uvicorn on `127.0.0.1:8421`. +`UI_PORT` overrides the public port; `S2S_BACKEND_PORT` overrides the internal +port. Both servers reload source edits, and stopping the launcher stops both. +The speech pipeline remains a separate service. Because `.env` is sourced by +the script, values in that file can replace variables supplied on the command line. -To install and start the UI manually: +Use Node.js 22.18+ on the 22.x line or Node.js 24+, npm, Python 3 with venv, +Bash 4.4+, `setsid`, and `curl`. Remote staging access also needs the exact +hostname in `S2S_DEV_ALLOWED_HOSTS`, HTTPS, and WebSocket forwarding for speech +and Vite HMR. See [frontend development setup](../../frontend-s2s/README.md). + +To install and start the backend manually: ```bash python3 -m venv .venv .venv/bin/python -m pip install -r s2s-client/requirements.txt -SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime \ - .venv/bin/python -m uvicorn --app-dir s2s-client server:app --port 8420 +S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime \ + .venv/bin/python -m uvicorn --app-dir s2s-client server:app \ + --host 127.0.0.1 --port 8421 --reload --reload-dir s2s-client ``` +Then, in a second terminal, run `cd frontend-s2s`, `npm ci`, and `npm run dev`. + Open , tap the orb/microphone control, grant microphone access, and speak. Microphone access requires HTTPS or a local browser origin; use HTTPS when accessing the UI from another machine. @@ -72,6 +84,9 @@ server; it is not a universally available local service. ## Docker +This existing image serves the retained static client. It does not run the +React/Vite development launcher. + Use the project root as the build context: ```bash diff --git a/frontend-s2s/README.md b/frontend-s2s/README.md index 4a81f81..c02462e 100644 --- a/frontend-s2s/README.md +++ b/frontend-s2s/README.md @@ -6,48 +6,84 @@ rendering and state, including the realtime WebSocket and audio integration. ## Run during development -Use Node.js 22.18 or later on the 22.x line, or Node.js 24+. Install dependencies -with npm; the lockfile is committed. +From the repository root, run `./start_ui.sh`. Open `http://localhost:8420`. +Add `?debug=true` to access settings, tools, and custom RAG context. Existing +`phoneNumber` and `episode_id` query parameters are supported. -1. Start the existing backend using `./start_ui.sh` from your configured S2S - checkout, or reuse an already-running instance. It normally listens on port - `8420` and continues serving the original static UI there. Its Python - environment, `.env`, speech service, and upstream services must be configured - as before. -2. In a second terminal, start this frontend: +The launcher requires Python 3 with venv support, Node.js 22.18 or later on the +22.x line or Node.js 24+, npm, Bash 4.4+, `setsid` (util-linux), and `curl`. +It creates `.venv` and installs missing Python dependencies. It runs +`npm ci --include=dev` on first start and when either frontend package manifest +changes. Subsequent restarts reuse the installed dependencies. - ```sh - cd frontend-s2s - npm ci - npm run dev - ``` +| Server | Default address | Purpose | +| ------- | ---------------- | ---------------------------------------------- | +| Vite | `0.0.0.0:8420` | Public React UI, backend proxy, and hot reload | +| FastAPI | `127.0.0.1:8421` | Internal HTTP APIs and speech WebSocket relay | -3. Open `http://127.0.0.1:5173`. Add `?debug=true` to access settings, tools, - and custom RAG context. Existing `phoneNumber` and `episode_id` query - parameters are supported. +`UI_PORT` preserves the existing public port override. `UI_HOST` defaults to +`0.0.0.0`; `S2S_BACKEND_PORT` changes the loopback backend port. The two ports +must differ. Occupied ports fail startup rather than silently selecting a new +endpoint. The launcher waits for FastAPI before starting Vite, stops both +process groups on Ctrl+C/SIGTERM, and exits with a failure if either server exits. +Uvicorn watches only `s2s-client`, avoiding frontend and dependency directories. +The Vite child explicitly uses `NODE_ENV=development`, including when a process +manager sets `NODE_ENV=production`, so React Fast Refresh remains enabled. Vite serves React and Tailwind with hot updates. It proxies `/api`, `/v1/realtime` (including WebSockets), OAuth, and backend documentation routes -to FastAPI. The browser uses one origin on port `5173`, so there is no separate +to FastAPI. The browser uses one origin on port `8420`, so there is no separate API origin or development CORS setup. The proxy preserves the incoming Host header because FastAPI uses it to construct the speech WebSocket URL. -For a different backend, copy `.env.example` to `.env.local` and set -`S2S_BACKEND_URL`. This is a **Vite server** setting. Runtime UI flags, voices, -reasoning mode, prompts, and the speech URL still come from FastAPI's +Runtime UI flags, voices, reasoning mode, prompts, and the speech URL still come from FastAPI's `/api/config`. A backend explicitly configured with `SPEECH_TO_SPEECH_URL` continues to return that direct speech endpoint. -The Vite configuration reads only this project's environment files. Do not -copy the repository's backend secrets into this directory or into `VITE_*` +The launcher sources the root `.env`, retaining its precedence over shell +variables. The Vite configuration reads this project's environment files and +the inherited `S2S_*` settings. Do not copy the repository's backend secrets +into this directory or into `VITE_*` variables, which are exposed to browser code. -For access through a development reverse proxy, set the exact hostname in -`S2S_DEV_ALLOWED_HOSTS`. Use `npm run dev -- --host 0.0.0.0` only when Vite must -listen beyond localhost. The browser needs HTTPS for microphone access outside -localhost, and that proxy must forward WebSocket upgrades for both speech and -Vite HMR. Wiring the staging entrypoint and launcher is a separate step: -**this port does not change `start_ui.sh`.** +## Staging with hot reload + +PM2 can continue running the root `start_ui.sh` as its existing `s2s_ui` process. +Ensure the Node/npm versions above are on PM2's PATH and deploy `frontend-s2s` +alongside `s2s-client` and `docs`. The reverse proxy still targets port `8420`. + +Set the exact public hostname in the root `.env`, without scheme or port: + +```dotenv +S2S_DEV_ALLOWED_HOSTS=voice-staging.example.com +``` + +Replace the example with the existing staging hostname; use commas for multiple +hosts. IP addresses and localhost are accepted without this setting. +The HTTPS reverse proxy must preserve `Host`, `X-Forwarded-Host` and +`X-Forwarded-Proto`, and forward WebSocket upgrades for `/v1/realtime` and +`/__vite_hmr`. HMR uses the browser's public origin, including HTTPS/WSS, so no +second public port is needed. The browser needs HTTPS for microphone access +outside localhost. Restart `s2s_ui` after changing launcher or environment +settings; source edits use the reloaders. The speech service remains separately +managed. + +## Run the servers separately + +Start FastAPI on `127.0.0.1:8421` as described in the +[client usage guide](../docs/client/usage.md), then run: + +```sh +cd frontend-s2s +npm ci +npm run dev +``` + +Standalone Vite binds `127.0.0.1:8420`; add `-- --host 0.0.0.0` for remote +access. For a different backend, copy `.env.example` to `.env.local` and set +`S2S_BACKEND_URL`. The combined launcher overrides that setting to point at the +backend it starts. Do not start standalone Vite while the combined launcher +already owns the public port. Component and CSS changes use Fast Refresh. Editing the session controller or audio modules can require restarting a conversation or a page reload. Teardown @@ -60,26 +96,29 @@ The scripts and code style follow `frontend-admin` conventions, with Vite and React plugins replacing Next.js-specific tooling. Executable configuration files use TypeScript; TypeScript's own project settings use JSON. -| Command | Purpose | -| -------------------- | ----------------------------------------------------------- | -| `npm run dev` | Vite development server on port 5173, with a fixed port | -| `npm run check` | Formatting, strict typed ESLint, type checking, and tests | -| `npm run lint:fix` | Apply ESLint fixes | -| `npm run format` | Format source and sort Tailwind classes | -| `npm run typecheck` | Check application, tests, audio worklets, and configuration | -| `npm test` | Run Vitest and React Testing Library regression tests | -| `npm run test:watch` | Watch tests while editing | -| `npm run build` | Typecheck and compile the app and worklets into `dist/` | -| `npm run preview` | Inspect the compiled output locally | +| Command | Purpose | +| ----------------------- | ----------------------------------------------------------- | +| `npm run dev` | Vite development server on port 8420, with a fixed port | +| `npm run check` | Formatting, strict typed ESLint, type checking, and tests | +| `npm run lint:fix` | Apply ESLint fixes | +| `npm run format` | Format source and sort Tailwind classes | +| `npm run typecheck` | Check application, tests, audio worklets, and configuration | +| `npm test` | Run Vitest and React Testing Library regression tests | +| `npm run test:watch` | Watch tests while editing | +| `npm run test:launcher` | Test launcher supervision and signal handling on Linux | +| `npm run build` | Typecheck and compile the app and worklets into `dist/` | +| `npm run preview` | Inspect the compiled output locally | No build step is needed before `npm run dev`. `dist/` is generated and ignored -by Git. Build output is available for later serving integration; this change -does not mount it in FastAPI or change the staging launch process. +by Git. The launcher serves source through Vite; it does not use `dist/` or +`vite preview`. Tests cover session initialization, response sequencing, cumulative transcripts, interruptions, queue cancellation, microphone-permission cancellation, tool results, external reasoning and `LOG_ONLY`, late-response cleanup, the idle timeout, settings, RAG, profiles, and sanitized Markdown/export behavior. +Launcher tests start temporary local children to check readiness, environment +forwarding, failures, and shutdown of complete process groups. Vitest handles Node 26's experimental web storage behavior internally. ## Source layout diff --git a/s2s-client/README.md b/s2s-client/README.md index 167b3cc..81053b3 100644 --- a/s2s-client/README.md +++ b/s2s-client/README.md @@ -10,19 +10,24 @@ short_description: Voice chat over WebSocket against a HF speech-to-speech hf_oauth: true --- -# Realtime voice client +# Realtime voice backend and retained static client -Browser voice-chat UI and FastAPI server for an OpenAI Realtime-compatible +FastAPI server and retained static voice-chat UI for an OpenAI Realtime-compatible speech-to-speech backend. The browser streams microphone audio over WebSocket, plays assistant audio, displays transcripts, and offers Spanish controls. +The root `start_ui.sh` now serves the [React frontend](../frontend-s2s) with +Vite on public port `8420`, proxying backend requests here on loopback port +`8421`. This directory's static UI remains available when FastAPI is run +directly or through the existing Docker image. + The complete [usage guide](../docs/client/usage.md) covers local startup, connection modes, Docker, settings, and usage limits. See the [documentation index](../docs/README.md), [architecture](../docs/client/architecture.md), and [change history](CHANGELOG.md) for implementation details and history. -Assistant replies render Markdown with Marked and DOMPurify. Their pinned, -checked-in browser modules are served from `vendor/`, including upstream +The retained static client's assistant replies render Markdown with Marked and +DOMPurify. Their pinned, checked-in browser modules are served from `vendor/`, including upstream licenses, so the UI does not require a CDN or a Node.js runtime. To refresh the browser dependencies with Node.js 20 or newer after updating `package.json`: