vocero-client/docs/client/usage.md

Client usage and configuration

The browser voice client connects to an existing OpenAI Realtime-compatible speech backend. The UI server can also proxy WebSocket traffic, search, reasoning, caller context, authenticated AquaPortal MCP tools and load-balancer requests.

Local startup

From the repository root:

S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh

The launcher loads the root .env, syncs vocero-api/.venv from the committed vocero-api/uv.lock, and installs the locked frontend dependencies when needed. Vite serves the React frontend on port 8420 and proxies the backend routes to Uvicorn on 127.0.0.1:8421. UI_PORT overrides the public port; S2S_BACKEND_PORT overrides the internal port. Both servers reload source edits, and stopping the launcher stops both. The speech pipeline remains a separate service. Because .env is sourced by the script, values in that file can replace variables supplied on the command line.

Use Node.js 22.18+ on the 22.x line or Node.js 24+, npm, uv 0.12.5, Bash 4.4+, setsid, and curl. Remote staging access also needs the exact hostname in S2S_DEV_ALLOWED_HOSTS, HTTPS, and WebSocket forwarding for speech and Vite HMR. See frontend development setup.

To install and start the backend manually:

uv sync --project vocero-api --locked
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime \
  uv run --project vocero-api --locked uvicorn --app-dir vocero-api vocero_api.main:app \
  --host 127.0.0.1 --port 8421 --reload --reload-dir vocero-api

Then, in a second terminal, run cd vocero-frontend, npm ci, and npm run dev.

uv provisions Python 3.12 when needed. The Python environment belongs only to vocero-api; an existing root .venv for speech dependencies is left alone. See development and compatibility for checks and schema rules.

Open http://localhost:8420/, tap the orb/microphone control, grant microphone access, and speak. Microphone access requires HTTPS or a local browser origin; use HTTPS when accessing the UI from another machine.

The speech pipeline has separate model/runtime dependencies and no complete dependency manifest in this checkout. In an already prepared speech environment, a local realtime server can be started with:

python -m speech_to_speech.s2s_pipeline \
  --mode realtime \
  --stt parakeet-tdt \
  --llm_backend transformers \
  --tts kokoro \
  --model_name Qwen/Qwen3-4B-Instruct-2507 \
  --llm_device cuda \
  --llm_torch_dtype float16 \
  --enable_live_transcription

Select backends and devices supported by the host; the example model is not a fixed deployment requirement. The default realtime endpoint is ws://localhost:8765/v1/realtime, configured through --ws_host and --ws_port. --num_pipelines controls the number of isolated concurrent realtime sessions. See LLM, STT, and TTS.

Connection modes

Configuration	Browser connection	Settings URL	Load-balancer features
SPEECH_TO_SPEECH_URL set	Direct to pinned URL	Locked	Disabled
S2S_PROXY_TARGET set, no direct URL	Same-origin /v1/realtime proxy	Locked	Disabled
LOAD_BALANCER_URL set, neither above set	Per-session compute URL from /api/session	Hidden	Allocation, queue polling and cancellation
None set	User-provided realtime URL	Editable	Disabled

Direct URLs may be full ws(s)://.../v1/realtime addresses or host values that the browser normalizes. A direct URL must be reachable from the browser. S2S_PROXY_TARGET must be reachable from the UI server. HTTPS deployments need a secure browser WebSocket URL.

The launcher supplies ws://ai-server:12345/v1/realtime as a proxy-target default when no value is configured. That hostname must resolve from the UI server; it is not a universally available local service.

Docker

This image serves FastAPI backend routes and shared documentation. Run the React frontend separately with Vite and point S2S_BACKEND_URL at the exposed backend address. The image does not run the development launcher or serve a UI.

Use the project root as the build context:

docker build -f vocero-api/Dockerfile -t vocero-api .
docker run --rm -p 7860:7860 \
  -e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime vocero-api

For the proxy path, replace the direct setting with a server-reachable S2S_PROXY_TARGET. If the upstream is another container, attach the UI to the appropriate existing network and use that service's hostname. An upstream bound only to host loopback is not reachable via an arbitrary container name.

The image includes vocero-api and root docs. /docs is still the FastAPI API explorer; documentation lives under /docs/client/, /docs/speech/, /docs/adr/, /docs/examples/, and /docs/proposals/. Legacy /project/ and /public/docs.html redirect to the historical proposal. Markdown is served as a source file rather than rendered through a separate documentation site. Request file paths such as /docs/README.md; directories without index.html are not browsable pages.

Settings and tools

Voice, editable system instructions and connection settings are kept in browser localStorage under s2s.ws.* keys. A non-empty S2S_PRIMARY_PROMPT pins the instruction field. Restart applies connection/voice changes to a new session. Settings and tool preferences use a draft: Guardar applies them and Cancelar or dialog dismissal discards unsaved edits. Voice and instructions update the live session on Save; connection changes apply on the next start. Noise-gate calibration is explicitly live and saves immediately. Model voice names are runtime identifiers and stay unchanged across UI translations.

Web search needs SERPER_API_KEY on the server or a user-supplied key in the tools panel. The server proxies requests through /api/search; its own key is not returned to the browser. AquaPortal content tools use authenticated MCP; see tool calling.

Operator RAG text is applied through the RAG dialog and updates live instructions. Caller links may include phoneNumber=%2B...; episode links may include episode_id=...; MCP report links may include report_id=.... See context injection for enabled data sources and limits.

Authenticated AquaPortal MCP

S2S_MCP_URL enables the MCP client transport. Set it to the backend's complete Streamable HTTP endpoint, for example https://backend.example.test/api/v1/mcp/. Set S2S_LOGIN_BY_PHONE to that backend's POST /api/v1/auth/phone-login URL. The phone parameter remains an identity selector without proof of ownership. MCP uses the returned opaque sessionToken for that user, with no service-token override.

For voice reasoning (S2S_REASONING_TYPE=PROXY), set S2S_PROXY_TARGET to the speech service and leave SPEECH_TO_SPEECH_URL and LOAD_BALANCER_URL unset. The browser must connect to Vocero's same-origin /v1/realtime relay so tool selections reach its executor. An incompatible direct configuration returns 503 from /api/config. OPENAI reasoning uses the authenticated HTTP loop and can retain a direct or load-balanced speech connection. No speech-service code or audio-format change is required.

The backend must expose the eight-tool catalog and phone-login contract described in tool calling. Leave S2S_MCP_URL unset to use the existing caller-context paths without MCP tools. Setting it makes MCP the exclusive AquaPortal content path: legacy profile, report, latest-summary, and episode preloads are skipped. Legacy caller HTTP routes return 409 while MCP is enabled.

ApiPilot has been retired. /api/apipilot/tools and /api/apipilot/execute return 404, and /api/config no longer includes apipilot. Remove obsolete S2S_APIPILOT_TOOLS_URL, S2S_APIPILOT_TOOLS_EXECUTE_URL, S2S_APIPILOT_SERVICE_TOKEN, S2S_APIPILOT_API_KEY, S2S_APIPILOT_TTL_SEC, and S2S_APIPILOT_TIMEOUT_SEC from deployment configuration; they are ignored. Keep S2S_LOGIN_BY_PHONE, which is still required for MCP authentication. Deploy the API and frontend together and refresh existing browser sessions. The separately deployed speech_to_speech/ service is unchanged by this removal; its existing legacy marker sanitizers remain, with no speech restart required.

Setting	Default	Meaning
S2S_MCP_URL	Empty, disabled	Backend MCP endpoint
S2S_MCP_TIMEOUT_SEC	20	Entire MCP operation timeout, clamped to 1–120 seconds
S2S_MCP_MAX_CALLS_PER_TURN	6	Tool attempts per user turn, clamped to 1–20
S2S_CONVERSATION_TTL_SEC	3600	Absolute conversation lifetime, clamped to 60–86400 seconds
S2S_MAX_CONVERSATIONS	1000	Process-local conversation capacity, clamped to 1–10000

POST /api/conversations accepts {"phoneNumber":"+56000000000"} with optional report_id and episode_id fields, and returns conversationId, expiresIn, and a minimal caller profile. An HttpOnly, SameSite=Strict cookie binds the browser to its conversations; HTTPS sets Secure. Backend credentials stay in memory on the API server. HTTP conversation requests use X-Vocero-Conversation; the same-origin WebSocket uses conversation_id. The conversation ID alone is insufficient without its owning browser cookie. Different tabs can own different caller identities without replacing one another.

GET /api/mcp/tools discovers the authenticated catalog. POST /api/reasoning requires an increasing positive X-Vocero-Turn header in addition to the conversation header in MCP mode. POST /api/conversations/current/interrupt cancels the matching X-Vocero-Turn, and DELETE /api/conversations/current clears that conversation. These endpoints require the cookie and conversation header. They do not change the existing load-balancer /api/session contract. Stopping a conversation drops its local credentials; it does not revoke backend sessions globally.

Conversation state is bounded, process-local, and cleared on restart. Use one API worker, as in the launcher and Docker command. Multiple workers or replicas require a shared conversation store before enabling this integration. Browser sessions must restart after expiry or an API reload. Backend access is checked on each MCP request. There is no shared content or authorization cache. HTTP reasoning retains up to six completed turns, bounded to 96,000 characters, within that conversation; the speech service keeps its existing conversation history. These histories are cleared when their sessions end.

The browser waits for phone login, linked-content resolution, and tool discovery before starting an MCP conversation. Login or discovery failure blocks startup and displays a retryable error. Link resolution is limited to ten seconds and uses only authenticated MCP tools. Missing, inaccessible, or temporarily unavailable linked content permits a generic conversation; an expired backend session still prevents startup. The browser allows thirty seconds for login and link resolution together.

For example, ?phoneNumber=%2B56000000000&report_id=report-example selects the report and its podcast, report:report-example, when available. The full report ID is preserved. An episode_id selects a daily podcast, daily:<episode_id>. When both are present, the explicit daily episode determines the podcast and report_id remains a separate report reference. No unavailable episode is silently replaced with a different one.

The link provides the initial subject, not a search filter or an access grant. Users can ask about any accessible report, compare multiple reports, or choose another subject. The local select_content action validates that choice and updates the current reference while preserving the original link for follow-ups. Merely searching or reading content does not change the selection. A new conversation, including after restart, resolves the page's selectors again. Changing browser-supplied profile, phone, report, or episode values on a reasoning request cannot change the stored identity or selection.

Theme and session startup

S2S_THEME=white selects the light interface. dark or an empty value selects the dark aurora theme. Theme-specific sections remain hidden until configuration resolves, with a fallback to avoid an invisible page after config failure. Configuration failures offer Reintentar and block session startup until a retry succeeds. With MCP disabled, reports load independently and caller context gets an eight-second loading window before the interface allows a generic session.

S2S_START_SESSION_MODE controls initial microphone/session startup:

Value	Behavior
Empty/default	User taps to start
TALK	Attempt automatic start and inject the Spanish wake phrase as soon as configuration and caller context are ready
TALKFIRST	Same startup trigger as TALK; with MCP, greet using the linked report or podcast title after authenticated resolution

Automatic startup runs once per page load, without a startup dialog or artificial delay; optional reports and tools can still be loading. Browser permission and audio policies can still require a gesture. When audio is blocked, an inline hint asks the user to tap the microphone button. Permission or connection failures show the error and leave the same button available to retry. Stopping a session does not start it again automatically. The greeting uses the same hola vocero input as wake-word handling. With MCP and TALKFIRST, the greeting names the linked report, or the podcast when no report is available, and invites questions about this or other reports. It does not summarize automatically. A missing reference gets a brief explanation; links without a content selector receive a generic greeting.

In PROXY, the relay supplies the verified metadata in the initial speech instructions. In OPENAI, the controller sends the opening input through the authenticated HTTP reasoning path after the speech connection is ready, then uses the existing external-answer playback. Speaking first or stopping the session cancels a pending greeting. Neither path changes the speech service.

On browsers with speech recognition, Activar con «hola vocero» opts into wake-word listening for the current page session. The interface shows when the microphone is listening for that phrase. It is off by default, pauses while a dialog or voice session is active, and turns off when the conversation stops.

The transcript keeps your reading position when new messages arrive. Ir al último mensaje resumes following the conversation. Names and Santiago-local times appear with each message; downloaded JSON and text retain raw timestamps and message text. The interface respects the browser's reduced-motion setting.

Queue and session lifecycle

Load-balancer mode proxies POST /api/session. A free compute slot returns a grant containing session_id, connect_url, and pending_timeout_s. Otherwise the browser receives a queue ticket and polls GET /api/queue/{queue_id} until a slot is granted. The browser offers a timed join action after waiting.

DELETE /api/queue/{queue_id} cancels a waiting ticket. On session stop or page exit, the browser uses POST /api/queue/end through sendBeacon, falling back to a keepalive fetch. Expired tickets and missed join deadlines require a new attempt. Full queues return 503 with state: "at_capacity".

Once connected, session teardown closes the microphone, audio graph and WebSocket. The 30-second inactivity timeout counts time waiting for user input; processing, external reasoning and buffered playback do not consume it. Caller-profile lookup and upstream authorization operate independently of allocation and use their existing configured endpoints.

For external transcript processing, configure the selected reasoning mode. For connection, context and audio diagnostics, use troubleshooting.