154 lines
6.8 KiB
Markdown
154 lines
6.8 KiB
Markdown
# Client usage and configuration
|
|
|
|
The browser voice client connects to an existing OpenAI Realtime-compatible
|
|
speech backend. The UI server can also proxy WebSocket traffic, search,
|
|
reasoning, caller context, ApiPilot tools and load-balancer requests.
|
|
|
|
## Local startup
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
|
|
```
|
|
|
|
The launcher loads the root `.env`, creates `.venv` if needed, installs the
|
|
client requirements when the dependency check fails, and runs uvicorn on port
|
|
8420. `UI_PORT` overrides the UI port. Because `.env` is sourced by the script,
|
|
values in that file can replace variables supplied on the command line.
|
|
|
|
To install and start the UI manually:
|
|
|
|
```bash
|
|
python3 -m venv .venv
|
|
.venv/bin/python -m pip install -r s2s-client/requirements.txt
|
|
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime \
|
|
.venv/bin/python -m uvicorn --app-dir s2s-client server:app --port 8420
|
|
```
|
|
|
|
Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
|
|
access, and speak. Microphone access requires HTTPS or a local browser origin;
|
|
use HTTPS when accessing the UI from another machine.
|
|
|
|
The speech pipeline has separate model/runtime dependencies and no complete
|
|
dependency manifest in this checkout. In an already prepared speech
|
|
environment, a local realtime server can be started with:
|
|
|
|
```bash
|
|
python -m speech_to_speech.s2s_pipeline \
|
|
--mode realtime \
|
|
--stt parakeet-tdt \
|
|
--llm_backend transformers \
|
|
--tts kokoro \
|
|
--model_name Qwen/Qwen3-4B-Instruct-2507 \
|
|
--llm_device cuda \
|
|
--llm_torch_dtype float16 \
|
|
--enable_live_transcription
|
|
```
|
|
|
|
Select backends and devices supported by the host; the example model is not a
|
|
fixed deployment requirement. The default realtime endpoint is
|
|
`ws://localhost:8765/v1/realtime`, configured through `--ws_host` and `--ws_port`.
|
|
`--num_pipelines` controls the number of isolated concurrent realtime sessions.
|
|
See [LLM](../speech/llm.md), [STT](../speech/stt.md), and [TTS](../speech/tts.md).
|
|
|
|
## Connection modes
|
|
|
|
| Configuration | Browser connection | Settings URL | Load-balancer features |
|
|
|---|---|---|---|
|
|
| `SPEECH_TO_SPEECH_URL` set | Direct to pinned URL | Locked | Disabled |
|
|
| `S2S_PROXY_TARGET` set, no direct URL | Same-origin `/v1/realtime` proxy | Locked | Disabled |
|
|
| `LOAD_BALANCER_URL` set, neither above set | Per-session compute URL from `/api/session` | Hidden | Queue enabled; metering requires `SPACE_ID` too |
|
|
| None set | User-provided realtime URL | Editable | Disabled |
|
|
|
|
Direct URLs may be full `ws(s)://.../v1/realtime` addresses or host values that
|
|
the browser normalizes. A direct URL must be reachable **from the browser**.
|
|
`S2S_PROXY_TARGET` must be reachable **from the UI server**. HTTPS deployments
|
|
need a secure browser WebSocket URL.
|
|
|
|
The launcher supplies `ws://ai-server:12345/v1/realtime` as a proxy-target
|
|
default when no value is configured. That hostname must resolve from the UI
|
|
server; it is not a universally available local service.
|
|
|
|
## Docker
|
|
|
|
Use the project root as the build context:
|
|
|
|
```bash
|
|
docker build -f s2s-client/Dockerfile -t s2s-client .
|
|
docker run --rm -p 7860:7860 \
|
|
-e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client
|
|
```
|
|
|
|
For the proxy path, replace the direct setting with a server-reachable
|
|
`S2S_PROXY_TARGET`. If the upstream is another container, attach the UI to the
|
|
appropriate existing network and use that service's hostname. An upstream
|
|
bound only to host loopback is not reachable via an arbitrary container name.
|
|
|
|
The image includes `s2s-client` and root `docs`. `/docs` is still the FastAPI API
|
|
explorer; documentation lives under `/docs/client/`, `/docs/speech/`,
|
|
`/docs/adr/`, `/docs/examples/`, and `/docs/proposals/`. Legacy `/project/` and
|
|
`/public/docs.html` redirect to the historical proposal. Markdown is served as a
|
|
source file rather than rendered through a separate documentation site. Request
|
|
file paths such as `/docs/README.md`; directories without `index.html` are not
|
|
browsable pages.
|
|
|
|
## Settings and tools
|
|
|
|
Voice, editable system instructions and connection settings are kept in browser
|
|
`localStorage` under `s2s.ws.*` keys. A non-empty `S2S_PRIMARY_PROMPT` pins the
|
|
instruction field. Restart applies connection/voice changes to a new session.
|
|
Model voice names are runtime identifiers and stay unchanged across UI
|
|
translations.
|
|
|
|
Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
|
|
tools panel. The server proxies requests through `/api/search`; its own key is
|
|
not returned to the browser. Dynamic ApiPilot tools are configured through
|
|
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).
|
|
|
|
Operator RAG text is applied through the RAG dialog and updates live
|
|
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
|
|
include `episode_id=...`. See [context injection](context-injection.md) for
|
|
enabled data sources and limits.
|
|
|
|
## Theme and session startup
|
|
|
|
`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
|
|
the dark aurora theme. Theme-specific sections remain hidden until configuration
|
|
resolves, with a fallback to avoid an invisible page after config failure.
|
|
|
|
`S2S_START_SESSION_MODE` controls initial microphone/session startup:
|
|
|
|
| Value | Behavior |
|
|
|---|---|
|
|
| Empty/default | User taps to start |
|
|
| `TALK` | After a 2-second warmup, show an audio unlocker; a user gesture starts the session and greeting |
|
|
| `TALKFIRST` | After a 2-second warmup, attempt automatic start and inject the Spanish wake phrase; microphone failure falls back to the unlocker |
|
|
|
|
The warmup is a timer, not proof that all upstream context requests have
|
|
finished. Browser permission and audio policies can still require a gesture.
|
|
The greeting uses the same `hola vocero` input as wake-word handling.
|
|
|
|
## Queue and daily usage
|
|
|
|
Load-balancer mode proxies session acquisition and queue polling. Waiting does
|
|
not reserve conversation time; a daily budget is reserved when a compute slot
|
|
is granted. The server enables metering only with both `LOAD_BALANCER_URL` and
|
|
`SPACE_ID` and with neither pinned direct/proxy mode active.
|
|
|
|
| Environment setting | Default | Purpose |
|
|
|---|---|---|
|
|
| `LIMIT_ANON_SEC` | 300 | Anonymous seconds per UTC day |
|
|
| `LIMIT_FREE_SEC` | 600 | Signed-in non-PRO seconds per UTC day |
|
|
| `UNLIMITED_ORGS` | Adds to built-in organizations | Additional unlimited organization members |
|
|
| `USAGE_HASH_SECRET` | Random if unset | Identity-key hashing and anonymous cookie signing |
|
|
|
|
PRO members are unlimited. Built-in unlimited organizations are `cerebras`,
|
|
`HuggingFaceM4`, `smolagents`, and `pollen-robotics`; organization matching is
|
|
case-insensitive. See [auth.py](../../s2s-client/auth.py) and
|
|
[limiter.py](../../s2s-client/limiter.py) for the current enforcement behavior.
|
|
|
|
For external transcript processing, configure the selected
|
|
[reasoning mode](reasoning.md). For connection, context and audio diagnostics,
|
|
use [troubleshooting](troubleshooting.md).
|