vocero-client/docs/client/usage.md

Client usage and configuration

The browser voice client connects to an existing OpenAI Realtime-compatible speech backend. The UI server can also proxy WebSocket traffic, search, reasoning, caller context, ApiPilot tools and load-balancer requests.

Local startup

From the repository root:

S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh

The launcher loads the root .env, creates .venv if needed, installs the client requirements when the dependency check fails, and installs the locked frontend dependencies when needed. Vite serves the React frontend on port 8420 and proxies the backend routes to Uvicorn on 127.0.0.1:8421. UI_PORT overrides the public port; S2S_BACKEND_PORT overrides the internal port. Both servers reload source edits, and stopping the launcher stops both. The speech pipeline remains a separate service. Because .env is sourced by the script, values in that file can replace variables supplied on the command line.

Use Node.js 22.18+ on the 22.x line or Node.js 24+, npm, Python 3 with venv, Bash 4.4+, setsid, and curl. Remote staging access also needs the exact hostname in S2S_DEV_ALLOWED_HOSTS, HTTPS, and WebSocket forwarding for speech and Vite HMR. See frontend development setup.

To install and start the backend manually:

python3 -m venv .venv
.venv/bin/python -m pip install -r s2s-client/requirements.txt
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime \
  .venv/bin/python -m uvicorn --app-dir s2s-client server:app \
  --host 127.0.0.1 --port 8421 --reload --reload-dir s2s-client

Then, in a second terminal, run cd s2s-frontend, npm ci, and npm run dev.

Open http://localhost:8420/, tap the orb/microphone control, grant microphone access, and speak. Microphone access requires HTTPS or a local browser origin; use HTTPS when accessing the UI from another machine.

The speech pipeline has separate model/runtime dependencies and no complete dependency manifest in this checkout. In an already prepared speech environment, a local realtime server can be started with:

python -m speech_to_speech.s2s_pipeline \
  --mode realtime \
  --stt parakeet-tdt \
  --llm_backend transformers \
  --tts kokoro \
  --model_name Qwen/Qwen3-4B-Instruct-2507 \
  --llm_device cuda \
  --llm_torch_dtype float16 \
  --enable_live_transcription

Select backends and devices supported by the host; the example model is not a fixed deployment requirement. The default realtime endpoint is ws://localhost:8765/v1/realtime, configured through --ws_host and --ws_port. --num_pipelines controls the number of isolated concurrent realtime sessions. See LLM, STT, and TTS.

Connection modes

Configuration	Browser connection	Settings URL	Load-balancer features
SPEECH_TO_SPEECH_URL set	Direct to pinned URL	Locked	Disabled
S2S_PROXY_TARGET set, no direct URL	Same-origin /v1/realtime proxy	Locked	Disabled
LOAD_BALANCER_URL set, neither above set	Per-session compute URL from /api/session	Hidden	Queue enabled; metering requires SPACE_ID too
None set	User-provided realtime URL	Editable	Disabled

Direct URLs may be full ws(s)://.../v1/realtime addresses or host values that the browser normalizes. A direct URL must be reachable from the browser. S2S_PROXY_TARGET must be reachable from the UI server. HTTPS deployments need a secure browser WebSocket URL.

The launcher supplies ws://ai-server:12345/v1/realtime as a proxy-target default when no value is configured. That hostname must resolve from the UI server; it is not a universally available local service.

Docker

This existing image serves the retained static client. It does not run the React/Vite development launcher.

Use the project root as the build context:

docker build -f s2s-client/Dockerfile -t s2s-client .
docker run --rm -p 7860:7860 \
  -e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client

For the proxy path, replace the direct setting with a server-reachable S2S_PROXY_TARGET. If the upstream is another container, attach the UI to the appropriate existing network and use that service's hostname. An upstream bound only to host loopback is not reachable via an arbitrary container name.

The image includes s2s-client and root docs. /docs is still the FastAPI API explorer; documentation lives under /docs/client/, /docs/speech/, /docs/adr/, /docs/examples/, and /docs/proposals/. Legacy /project/ and /public/docs.html redirect to the historical proposal. Markdown is served as a source file rather than rendered through a separate documentation site. Request file paths such as /docs/README.md; directories without index.html are not browsable pages.

Settings and tools

Voice, editable system instructions and connection settings are kept in browser localStorage under s2s.ws.* keys. A non-empty S2S_PRIMARY_PROMPT pins the instruction field. Restart applies connection/voice changes to a new session. Model voice names are runtime identifiers and stay unchanged across UI translations.

Web search needs SERPER_API_KEY on the server or a user-supplied key in the tools panel. The server proxies requests through /api/search; its own key is not returned to the browser. Dynamic ApiPilot tools are configured through S2S_APIPILOT_TOOLS_URL; see tool calling.

Operator RAG text is applied through the RAG dialog and updates live instructions. Caller links may include phoneNumber=%2B...; episode links may include episode_id=.... See context injection for enabled data sources and limits.

Theme and session startup

S2S_THEME=white selects the light interface. dark or an empty value selects the dark aurora theme. Theme-specific sections remain hidden until configuration resolves, with a fallback to avoid an invisible page after config failure.

S2S_START_SESSION_MODE controls initial microphone/session startup:

Value	Behavior
Empty/default	User taps to start
TALK	After a 2-second warmup, show an audio unlocker; a user gesture starts the session and greeting
TALKFIRST	After a 2-second warmup, attempt automatic start and inject the Spanish wake phrase; microphone failure falls back to the unlocker

The warmup is a timer, not proof that all upstream context requests have finished. Browser permission and audio policies can still require a gesture. The greeting uses the same hola vocero input as wake-word handling.

Queue and daily usage

Load-balancer mode proxies session acquisition and queue polling. Waiting does not reserve conversation time; a daily budget is reserved when a compute slot is granted. The server enables metering only with both LOAD_BALANCER_URL and SPACE_ID and with neither pinned direct/proxy mode active.

Environment setting	Default	Purpose
LIMIT_ANON_SEC	300	Anonymous seconds per UTC day
LIMIT_FREE_SEC	600	Signed-in non-PRO seconds per UTC day
UNLIMITED_ORGS	Adds to built-in organizations	Additional unlimited organization members
USAGE_HASH_SECRET	Random if unset	Identity-key hashing and anonymous cookie signing

PRO members are unlimited. Built-in unlimited organizations are cerebras, HuggingFaceM4, smolagents, and pollen-robotics; organization matching is case-insensitive. See auth.py and limiter.py for the current enforcement behavior.

For external transcript processing, configure the selected reasoning mode. For connection, context and audio diagnostics, use troubleshooting.