vocero-client/docs/client/usage.md

# Client usage and configuration

The browser voice client connects to an existing OpenAI Realtime-compatible
speech backend. The UI server can also proxy WebSocket traffic, search,
reasoning, caller context, ApiPilot tools and load-balancer requests.

## Local startup

From the repository root:

```bash
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime ./start_ui.sh
```

The launcher loads the root `.env`, syncs `vocero-api/.venv` from the committed
`vocero-api/uv.lock`, and installs the locked
frontend dependencies when needed. Vite serves the React frontend on port
8420 and proxies the backend routes to Uvicorn on `127.0.0.1:8421`.
`UI_PORT` overrides the public port; `S2S_BACKEND_PORT` overrides the internal
port. Both servers reload source edits, and stopping the launcher stops both.
The speech pipeline remains a separate service. Because `.env` is sourced by
the script, values in that file can replace variables supplied on the command line.

Use Node.js 22.18+ on the 22.x line or Node.js 24+, npm, uv 0.12.5,
Bash 4.4+, `setsid`, and `curl`. Remote staging access also needs the exact
hostname in `S2S_DEV_ALLOWED_HOSTS`, HTTPS, and WebSocket forwarding for speech
and Vite HMR. See [frontend development setup](../../vocero-frontend/README.md).

To install and start the backend manually:

```bash
uv sync --project vocero-api --locked
S2S_PROXY_TARGET=ws://localhost:8765/v1/realtime \
  uv run --project vocero-api --locked uvicorn --app-dir vocero-api vocero_api.main:app \
  --host 127.0.0.1 --port 8421 --reload --reload-dir vocero-api
```

Then, in a second terminal, run `cd vocero-frontend`, `npm ci`, and `npm run dev`.

uv provisions Python 3.12 when needed. The Python environment belongs only to
`vocero-api`; an existing root `.venv` for speech dependencies is left alone.
See [development and compatibility](development.md) for checks and schema rules.

Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
access, and speak. Microphone access requires HTTPS or a local browser origin;
use HTTPS when accessing the UI from another machine.

The speech pipeline has separate model/runtime dependencies and no complete
dependency manifest in this checkout. In an already prepared speech
environment, a local realtime server can be started with:

```bash
python -m speech_to_speech.s2s_pipeline \
  --mode realtime \
  --stt parakeet-tdt \
  --llm_backend transformers \
  --tts kokoro \
  --model_name Qwen/Qwen3-4B-Instruct-2507 \
  --llm_device cuda \
  --llm_torch_dtype float16 \
  --enable_live_transcription
```

Select backends and devices supported by the host; the example model is not a
fixed deployment requirement. The default realtime endpoint is
`ws://localhost:8765/v1/realtime`, configured through `--ws_host` and `--ws_port`.
`--num_pipelines` controls the number of isolated concurrent realtime sessions.
See [LLM](../speech/llm.md), [STT](../speech/stt.md), and [TTS](../speech/tts.md).

## Connection modes

| Configuration | Browser connection | Settings URL | Load-balancer features |
|---|---|---|---|
| `SPEECH_TO_SPEECH_URL` set | Direct to pinned URL | Locked | Disabled |
| `S2S_PROXY_TARGET` set, no direct URL | Same-origin `/v1/realtime` proxy | Locked | Disabled |
| `LOAD_BALANCER_URL` set, neither above set | Per-session compute URL from `/api/session` | Hidden | Allocation, queue polling and cancellation |
| None set | User-provided realtime URL | Editable | Disabled |

Direct URLs may be full `ws(s)://.../v1/realtime` addresses or host values that
the browser normalizes. A direct URL must be reachable **from the browser**.
`S2S_PROXY_TARGET` must be reachable **from the UI server**. HTTPS deployments
need a secure browser WebSocket URL.

The launcher supplies `ws://ai-server:12345/v1/realtime` as a proxy-target
default when no value is configured. That hostname must resolve from the UI
server; it is not a universally available local service.

## Docker

This image serves FastAPI backend routes and shared documentation. Run the
React frontend separately with Vite and point `S2S_BACKEND_URL` at the exposed
backend address. The image does not run the development launcher or serve a UI.

Use the project root as the build context:

```bash
docker build -f vocero-api/Dockerfile -t vocero-api .
docker run --rm -p 7860:7860 \
  -e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime vocero-api
```

For the proxy path, replace the direct setting with a server-reachable
`S2S_PROXY_TARGET`. If the upstream is another container, attach the UI to the
appropriate existing network and use that service's hostname. An upstream
bound only to host loopback is not reachable via an arbitrary container name.

The image includes `vocero-api` and root `docs`. `/docs` is still the FastAPI API
explorer; documentation lives under `/docs/client/`, `/docs/speech/`,
`/docs/adr/`, `/docs/examples/`, and `/docs/proposals/`. Legacy `/project/` and
`/public/docs.html` redirect to the historical proposal. Markdown is served as a
source file rather than rendered through a separate documentation site. Request
file paths such as `/docs/README.md`; directories without `index.html` are not
browsable pages.

## Settings and tools

Voice, editable system instructions and connection settings are kept in browser
`localStorage` under `s2s.ws.*` keys. A non-empty `S2S_PRIMARY_PROMPT` pins the
instruction field. Restart applies connection/voice changes to a new session.
Settings and tool preferences use a draft: **Guardar** applies them and
**Cancelar** or dialog dismissal discards unsaved edits. Voice and instructions
update the live session on Save; connection changes apply on the next start.
Noise-gate calibration is explicitly live and saves immediately.
Model voice names are runtime identifiers and stay unchanged across UI
translations.

Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
tools panel. The server proxies requests through `/api/search`; its own key is
not returned to the browser. Dynamic ApiPilot tools are configured through
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).

Operator RAG text is applied through the RAG dialog and updates live
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
include `episode_id=...`; MCP report links may include `report_id=...`.
See [context injection](context-injection.md) for
enabled data sources and limits.

### Authenticated AquaPortal MCP

`S2S_MCP_URL` enables the MCP client transport. Set it to the backend's complete
Streamable HTTP endpoint, for example `https://backend.example.test/api/v1/mcp/`.
Set `S2S_LOGIN_BY_PHONE` to that backend's `POST /api/v1/auth/phone-login` URL.
The phone parameter remains an identity selector without proof of ownership.
MCP uses the returned opaque `sessionToken`, never an ApiPilot API key or service token.

For voice reasoning (`S2S_REASONING_TYPE=PROXY`), set `S2S_PROXY_TARGET` to the
speech service and leave `SPEECH_TO_SPEECH_URL` and `LOAD_BALANCER_URL` unset.
The browser must connect to Vocero's same-origin `/v1/realtime` relay so tool
selections reach its executor. An incompatible direct configuration returns
503 from `/api/config`. `OPENAI` reasoning uses the authenticated HTTP loop and
can retain a direct or load-balanced speech connection. No speech-service code
or audio-format change is required.

The backend must expose the eight-tool catalog and phone-login contract described
in [tool calling](tool-calling.md#authenticated-aquaportal-mcp). Leave
`S2S_MCP_URL` unset to retain the existing ApiPilot and caller-context paths.
Setting it makes MCP the exclusive AquaPortal content path: legacy profile,
report, latest-summary, and episode preloads are skipped. Existing ApiPilot
and caller HTTP routes return 409 while MCP is enabled.

| Setting | Default | Meaning |
|---|---|---|
| `S2S_MCP_URL` | Empty, disabled | Backend MCP endpoint |
| `S2S_MCP_TIMEOUT_SEC` | `20` | Entire MCP operation timeout, clamped to 1–120 seconds |
| `S2S_MCP_MAX_CALLS_PER_TURN` | `6` | Tool attempts per user turn, clamped to 1–20 |
| `S2S_CONVERSATION_TTL_SEC` | `3600` | Absolute conversation lifetime, clamped to 60–86400 seconds |
| `S2S_MAX_CONVERSATIONS` | `1000` | Process-local conversation capacity, clamped to 1–10000 |

`POST /api/conversations` accepts `{"phoneNumber":"+56000000000"}` with optional
`report_id` and `episode_id` fields, and returns
`conversationId`, `expiresIn`, and a minimal caller profile. An HttpOnly,
SameSite=Strict cookie binds the browser to its conversations; HTTPS sets Secure.
Backend credentials stay in memory on the API server. HTTP conversation requests
use `X-Vocero-Conversation`; the same-origin WebSocket uses `conversation_id`.
The conversation ID alone is insufficient without its owning browser cookie.
Different tabs can own different caller identities without replacing one another.

`GET /api/mcp/tools` discovers the authenticated catalog.
`POST /api/reasoning` requires an increasing positive `X-Vocero-Turn` header
in addition to the conversation header in MCP mode.
`POST /api/conversations/current/interrupt` cancels the matching `X-Vocero-Turn`, and
`DELETE /api/conversations/current` clears that conversation. These endpoints
require the cookie and conversation header. They do not change the existing
load-balancer `/api/session` contract. Stopping a conversation drops its local
credentials; it does not revoke backend sessions globally.

Conversation state is bounded, process-local, and cleared on restart. Use one
API worker, as in the launcher and Docker command. Multiple workers or replicas
require a shared conversation store before enabling this integration. Browser
sessions must restart after expiry or an API reload. Backend access is checked
on each MCP request. There is no shared content or authorization cache. HTTP
reasoning retains up to six completed turns, bounded to 96,000 characters,
within that conversation; the speech service keeps its existing conversation
history. These histories are cleared when their sessions end.

The browser waits for phone login, linked-content resolution, and tool discovery
before starting an MCP conversation. Login or discovery failure blocks startup
and displays a retryable error. Link resolution is limited to ten seconds and
uses only authenticated MCP tools. Missing, inaccessible, or temporarily
unavailable linked content permits a generic conversation; an expired backend
session still prevents startup. The browser allows thirty seconds for login
and link resolution together.

For example, `?phoneNumber=%2B56000000000&report_id=report-example` selects the
report and its podcast, `report:report-example`, when available. The full report
ID is preserved. An `episode_id` selects a daily podcast, `daily:<episode_id>`.
When both are present, the explicit daily episode determines the podcast and
`report_id` remains a separate report reference. No unavailable episode is
silently replaced with a different one.

The link provides the initial subject, not a search filter or an access grant.
Users can ask about any accessible report, compare multiple reports, or choose
another subject. The local `select_content` action validates that choice and
updates the current reference while preserving the original link for follow-ups.
Merely searching or reading content does not change the selection. A new
conversation, including after restart, resolves the page's selectors again.
Changing browser-supplied profile, phone, report, or episode values on a
reasoning request cannot change the stored identity or selection.

## Theme and session startup

`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
the dark aurora theme. Theme-specific sections remain hidden until configuration
resolves, with a fallback to avoid an invisible page after config failure.
Configuration failures offer **Reintentar** and block session startup until a
retry succeeds. With MCP disabled, reports load independently and caller context
gets an eight-second loading window before the interface allows a generic session.

`S2S_START_SESSION_MODE` controls initial microphone/session startup:

| Value | Behavior |
|---|---|
| Empty/default | User taps to start |
| `TALK` | Attempt automatic start and inject the Spanish wake phrase as soon as configuration and caller context are ready |
| `TALKFIRST` | Same startup trigger as `TALK`; with MCP, greet using the linked report or podcast title after authenticated resolution |

Automatic startup runs once per page load, without a startup dialog or artificial
delay; optional reports and tools can still be loading. Browser permission and
audio policies can still require a gesture. When audio is blocked, an inline hint
asks the user to tap the microphone button. Permission or connection failures
show the error and leave the same button available to retry. Stopping a session
does not start it again automatically.
The greeting uses the same `hola vocero` input as wake-word handling. With MCP
and TALKFIRST, the greeting names the linked report, or the podcast when no
report is available, and invites questions about this or other reports. It does
not summarize automatically. A missing reference gets a brief explanation;
links without a content selector receive a generic greeting.

In `PROXY`, the relay supplies the verified metadata in the initial speech
instructions. In `OPENAI`, the controller sends the opening input through the
authenticated HTTP reasoning path after the speech connection is ready, then
uses the existing external-answer playback. Speaking first or stopping the
session cancels a pending greeting. Neither path changes the speech service.

On browsers with speech recognition, **Activar con «hola vocero»** opts into
wake-word listening for the current page session. The interface shows when the
microphone is listening for that phrase. It is off by default, pauses while a
dialog or voice session is active, and turns off when the conversation stops.

The transcript keeps your reading position when new messages arrive. **Ir al
último mensaje** resumes following the conversation. Names and Santiago-local
times appear with each message; downloaded JSON and text retain raw timestamps
and message text. The interface respects the browser's reduced-motion setting.

## Queue and session lifecycle

Load-balancer mode proxies `POST /api/session`. A free compute slot returns a
grant containing `session_id`, `connect_url`, and `pending_timeout_s`. Otherwise
the browser receives a queue ticket and polls `GET /api/queue/{queue_id}` until
a slot is granted. The browser offers a timed join action after waiting.

`DELETE /api/queue/{queue_id}` cancels a waiting ticket. On session stop or page
exit, the browser uses `POST /api/queue/end` through `sendBeacon`, falling back
to a keepalive fetch. Expired tickets and missed join deadlines require a new
attempt. Full queues return `503` with `state: "at_capacity"`.

Once connected, session teardown closes the microphone, audio graph and
WebSocket. The 30-second inactivity timeout counts time waiting for user input;
processing, external reasoning and buffered playback do not consume it.
Caller-profile lookup and upstream authorization operate
independently of allocation and use their existing configured endpoints.

For external transcript processing, configure the selected
[reasoning mode](reasoning.md). For connection, context and audio diagnostics,
use [troubleshooting](troubleshooting.md).