vocero-client/docs/client/usage.md

# Client usage and configuration

The browser voice client connects to an existing OpenAI Realtime-compatible
speech backend. The UI server can also proxy WebSocket traffic, search,
reasoning, caller context, ApiPilot tools and load-balancer requests.

## Local startup

From the repository root:

```bash
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh
```

The launcher loads the root `.env`, creates `.venv` if needed, installs the
client requirements when the dependency check fails, and installs the locked
frontend dependencies when needed. Vite serves the React frontend on port
8420 and proxies the backend routes to Uvicorn on `127.0.0.1:8421`.
`UI_PORT` overrides the public port; `S2S_BACKEND_PORT` overrides the internal
port. Both servers reload source edits, and stopping the launcher stops both.
The speech pipeline remains a separate service. Because `.env` is sourced by
the script, values in that file can replace variables supplied on the command line.

Use Node.js 22.18+ on the 22.x line or Node.js 24+, npm, Python 3 with venv,
Bash 4.4+, `setsid`, and `curl`. Remote staging access also needs the exact
hostname in `S2S_DEV_ALLOWED_HOSTS`, HTTPS, and WebSocket forwarding for speech
and Vite HMR. See [frontend development setup](../../frontend-s2s/README.md).

To install and start the backend manually:

```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r s2s-client/requirements.txt
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime \
  .venv/bin/python -m uvicorn --app-dir s2s-client server:app \
  --host 127.0.0.1 --port 8421 --reload --reload-dir s2s-client
```

Then, in a second terminal, run `cd frontend-s2s`, `npm ci`, and `npm run dev`.

Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
access, and speak. Microphone access requires HTTPS or a local browser origin;
use HTTPS when accessing the UI from another machine.

The speech pipeline has separate model/runtime dependencies and no complete
dependency manifest in this checkout. In an already prepared speech
environment, a local realtime server can be started with:

```bash
python -m speech_to_speech.s2s_pipeline \
  --mode realtime \
  --stt parakeet-tdt \
  --llm_backend transformers \
  --tts kokoro \
  --model_name Qwen/Qwen3-4B-Instruct-2507 \
  --llm_device cuda \
  --llm_torch_dtype float16 \
  --enable_live_transcription
```

Select backends and devices supported by the host; the example model is not a
fixed deployment requirement. The default realtime endpoint is
`ws://localhost:8765/v1/realtime`, configured through `--ws_host` and `--ws_port`.
`--num_pipelines` controls the number of isolated concurrent realtime sessions.
See [LLM](../speech/llm.md), [STT](../speech/stt.md), and [TTS](../speech/tts.md).

## Connection modes

| Configuration | Browser connection | Settings URL | Load-balancer features |
|---|---|---|---|
| `SPEECH_TO_SPEECH_URL` set | Direct to pinned URL | Locked | Disabled |
| `S2S_PROXY_TARGET` set, no direct URL | Same-origin `/v1/realtime` proxy | Locked | Disabled |
| `LOAD_BALANCER_URL` set, neither above set | Per-session compute URL from `/api/session` | Hidden | Queue enabled; metering requires `SPACE_ID` too |
| None set | User-provided realtime URL | Editable | Disabled |

Direct URLs may be full `ws(s)://.../v1/realtime` addresses or host values that
the browser normalizes. A direct URL must be reachable **from the browser**.
`S2S_PROXY_TARGET` must be reachable **from the UI server**. HTTPS deployments
need a secure browser WebSocket URL.

The launcher supplies `ws://ai-server:12345/v1/realtime` as a proxy-target
default when no value is configured. That hostname must resolve from the UI
server; it is not a universally available local service.

## Docker

This existing image serves the retained static client. It does not run the
React/Vite development launcher.

Use the project root as the build context:

```bash
docker build -f s2s-client/Dockerfile -t s2s-client .
docker run --rm -p 7860:7860 \
  -e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client
```

For the proxy path, replace the direct setting with a server-reachable
`S2S_PROXY_TARGET`. If the upstream is another container, attach the UI to the
appropriate existing network and use that service's hostname. An upstream
bound only to host loopback is not reachable via an arbitrary container name.

The image includes `s2s-client` and root `docs`. `/docs` is still the FastAPI API
explorer; documentation lives under `/docs/client/`, `/docs/speech/`,
`/docs/adr/`, `/docs/examples/`, and `/docs/proposals/`. Legacy `/project/` and
`/public/docs.html` redirect to the historical proposal. Markdown is served as a
source file rather than rendered through a separate documentation site. Request
file paths such as `/docs/README.md`; directories without `index.html` are not
browsable pages.

## Settings and tools

Voice, editable system instructions and connection settings are kept in browser
`localStorage` under `s2s.ws.*` keys. A non-empty `S2S_PRIMARY_PROMPT` pins the
instruction field. Restart applies connection/voice changes to a new session.
Model voice names are runtime identifiers and stay unchanged across UI
translations.

Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
tools panel. The server proxies requests through `/api/search`; its own key is
not returned to the browser. Dynamic ApiPilot tools are configured through
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).

Operator RAG text is applied through the RAG dialog and updates live
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
include `episode_id=...`. See [context injection](context-injection.md) for
enabled data sources and limits.

## Theme and session startup

`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
the dark aurora theme. Theme-specific sections remain hidden until configuration
resolves, with a fallback to avoid an invisible page after config failure.

`S2S_START_SESSION_MODE` controls initial microphone/session startup:

| Value | Behavior |
|---|---|
| Empty/default | User taps to start |
| `TALK` | After a 2-second warmup, show an audio unlocker; a user gesture starts the session and greeting |
| `TALKFIRST` | After a 2-second warmup, attempt automatic start and inject the Spanish wake phrase; microphone failure falls back to the unlocker |

The warmup is a timer, not proof that all upstream context requests have
finished. Browser permission and audio policies can still require a gesture.
The greeting uses the same `hola vocero` input as wake-word handling.

## Queue and daily usage

Load-balancer mode proxies session acquisition and queue polling. Waiting does
not reserve conversation time; a daily budget is reserved when a compute slot
is granted. The server enables metering only with both `LOAD_BALANCER_URL` and
`SPACE_ID` and with neither pinned direct/proxy mode active.

| Environment setting | Default | Purpose |
|---|---|---|
| `LIMIT_ANON_SEC` | 300 | Anonymous seconds per UTC day |
| `LIMIT_FREE_SEC` | 600 | Signed-in non-PRO seconds per UTC day |
| `UNLIMITED_ORGS` | Adds to built-in organizations | Additional unlimited organization members |
| `USAGE_HASH_SECRET` | Random if unset | Identity-key hashing and anonymous cookie signing |

PRO members are unlimited. Built-in unlimited organizations are `cerebras`,
`HuggingFaceM4`, `smolagents`, and `pollen-robotics`; organization matching is
case-insensitive. See [auth.py](../../s2s-client/auth.py) and
[limiter.py](../../s2s-client/limiter.py) for the current enforcement behavior.

For external transcript processing, configure the selected
[reasoning mode](reasoning.md). For connection, context and audio diagnostics,
use [troubleshooting](troubleshooting.md).