vocero-client/AGENTS.md

Repository guidance

Scope and working conventions

This repository is Vocero: a browser voice client, its FastAPI integration server, and a speech pipeline. Keep this guide self-contained and base its instructions on this checkout's code, configuration, documentation, and tools.

All file and directory references in this guide must point within the repository and be relative to its root. Commands may use the project's command-line tools, such as rg, git, uv, npm, Docker, and PM2. HTTP routes mentioned below are URLs, not filesystem paths.

• Check the current branch, worktree, and working-tree changes before editing. Honor the checkout requested by the user, including direct work on main. Use an isolated worktree when requested and preserve other worktrees.
• An analysis-only or planning request authorizes inspection, not implementation. Return findings with source locations. Present plans in the conversation unless the user asks to save a file; follow any explicit approval gate.
• Write clear explanations and ask one conceptual question at a time. Carry out already authorized work without repeatedly asking for permission.
• Use the native patch/edit tool for manual changes. Prefer rg, fd, jq, git, and standard CLI tools for inspection; check availability before assuming a tool is missing. Do not install system-wide packages unnecessarily.
• When commits are requested, make coherent, scoped Conventional Commits as work progresses. Avoid excessive micro-commits and unrelated cleanup.
• For a requested pull request, describe the problem, resulting behavior, validation, and relevant limitations. Create issues or post comments only when requested.
• When asked to merge locally, use the requested rebase/fast-forward flow; preserve a recovery ref before rewriting and verify retained patches. A local merge does not imply a push, deployment, or restart. Report those separately.
• Treat historical proposals in docs/proposals/ as context. Verify their claims against current code and distinguish proposed authentication or deployment changes from implemented behavior.

Project map

Location	Responsibility
vocero-api/	The complete uv project: manifests, environment, Python tooling, tests, and FastAPI package
vocero-api/vocero_api/main.py	Application assembly, documentation mount, and retained redirects
vocero-api/vocero_api/api/	HTTP routes, route registration, and response validation helper
vocero-api/vocero_api/models/	Pydantic request and response contracts
vocero-api/vocero_api/core/	Environment parsing, logging, public URLs, and startup diagnostics
vocero-api/vocero_api/providers/	ApiPilot and caller/profile/report HTTP integrations
vocero-api/vocero_api/services/	Prompt/context construction, reasoning, and tool-context rendering
vocero-api/vocero_api/realtime/	WebSocket relay, event handling, marker filtering, and audio forwarding
vocero-frontend/	React, TypeScript, Tailwind, and Vite browser application
speech_to_speech/	Separately run VAD → STT → LLM → TTS pipeline and realtime API
start_ui.sh	Dependency sync and supervision of FastAPI plus Vite
docs/	Maintained technical documentation, examples, and historical proposals

vocero-api/server.py is a compatibility shim for server:app; new backend implementation belongs in the package. Keep HTTP handlers focused on transport and reuse providers/services for shared behavior.

The old static browser site, daily talk-time quotas, quota database/accounting, and Hugging Face account UI were deliberately removed. Do not restore their files, routes, dependencies, or compatibility layers as incidental cleanup. Load-balancer allocation/queues and upstream caller authentication remain.

Tooling and checks

The uv project must stay inside vocero-api/, including vocero-api/pyproject.toml, vocero-api/uv.lock, vocero-api/.python-version, vocero-api/.venv/, vocero-api/Makefile, and vocero-api/.pre-commit-config.yaml. Use Python 3.12 and the uv version required by tool.uv.required-version (currently 0.12.5). Update dependencies through uv and commit the lockfile; do not use pip to modify this environment or move it to the repository root. Keep uv pins in vocero-api/Dockerfile and setup documentation aligned when changing the required version.

The frontend uses npm and vocero-frontend/package-lock.json. Its Node requirement is ^22.18.0 || >=24.0.0; consult vocero-frontend/package.json for current scripts. Keep application code and supported configuration files in TypeScript.

Commands below run from the repository root:

Command	Purpose
make -C vocero-api install	Sync the locked Python environment and development tools
make -C vocero-api format	Apply Ruff fixes and formatting
make -C vocero-api check	Check lock freshness, formatting, Ruff lint, mypy, and pytest
make -C vocero-api test-launcher	Test readiness, failures, dependency sync, and process-group shutdown
make -C vocero-api test-image	Build and smoke-test the actual backend Docker image
npm --prefix vocero-frontend ci	Install locked frontend dependencies
npm --prefix vocero-frontend run check	Check formatting, lint, TypeScript, Vitest, and launcher integration
npm --prefix vocero-frontend run build	Type-check and build frontend assets

Python uses Ruff with a 120-character line limit, double quotes, import ordering, and typed application functions checked by mypy. Frontend formatting and lint rules live in vocero-frontend/prettier.config.ts and vocero-frontend/eslint.config.ts; use those tools rather than introducing competing conventions.

Run checks appropriate to the change: backend checks for Python/API work, frontend checks and build for browser work, launcher integration for supervisor changes, and image tests for packaging/Docker changes. Documentation-only work normally needs link/path and diff checks, not service startup. Wait for commands to finish before reporting success, and state skipped coverage or blockers.

Tests should use mock upstream HTTP and local WebSocket fixtures, isolating environment configuration. Do not rely on deployed services, real credentials, or model downloads for client tests. The client gate does not test the separate speech pipeline, microphone hardware, or real model output. That pipeline has no complete dependency manifest or general test gate in this checkout; inspect the selected backend and explain the verification limits for speech changes.

API and compatibility rules

Read backend development and compatibility before refactoring the server or changing schemas. Existing wire behavior is the baseline, including behavior that looks unsafe, inconsistent, or buggy. Intentional bug fixes or contract changes need their own scope and regression evidence; do not bundle them into tooling or structural refactors.

• Preserve paths, methods, statuses, error shapes, headers, field names, omissions, coercion, authentication precedence, and outgoing provider calls. Trace both frontend consumers and provider parsing; a permissive consumer is not evidence that an upstream schema can safely be tightened.
• Retain existing request-model defaults and extra-field behavior. Do not make new fields required, forbid previously accepted extras, or normalize loosely parsed provider values without evidence and an intentional behavior change.
• Response models validate fields constructed by this service. Upstream-owned values remain permissive where the old parser was permissive: Any, optional fields, open objects, and arbitrary profile JSON are deliberate boundaries.
• validated_json_response in vocero-api/vocero_api/api/responses.py validates with Pydantic, then serializes the original payload. Do not substitute model_dump() or automatic response-model serialization: those can insert defaults, filter unknown keys, coerce values, or alter serialization. Keep OpenAPI declarations aligned with the actual validation boundary.
• Preserve environment/dotenv parsing, default flags, and import-time behavior during refactors. Do not introduce stricter configuration validation or change authentication behavior as part of a tooling change.
• vocero-api/tests/fixtures/legacy-contracts.json captures the server before the migration at 3d42df5. vocero-api/tests/test_compatibility.py compares HTTP response bytes, statuses, selected headers, outgoing requests, request schemas, and realtime frames. Do not regenerate captures merely to make a refactor pass.
• Known reasoning failures, ApiPilot authentication/cache quirks, and relay behavior are documented in the compatibility guide. Preserve narrow lint/type exceptions that retain those behaviors until a deliberate fix is in scope.
• The pytest warning filter is only for Starlette TestClient's exact deprecated AnyIO BlockingPortal alias. Do not broaden warning suppression or upgrade runtime frameworks just to silence that warning without a compatibility review.

Browser and realtime behavior

• React renders the UI; vocero-frontend/src/realtime/controller.ts owns observable application state, vocero-frontend/src/realtime/client.ts owns protocol/session lifecycle, and vocero-frontend/src/audio/ owns the audio graph and worklets. Keep high-frequency audio/WebSocket work outside React rendering and preserve cleanup across unmounts and reconnects.
• Preserve the existing 16 kHz PCM16 audio path, worklet registration, interruption behavior, and teardown. Session startup initializes audio, waits for session.created, and sends session.update before audio or the greeting. A session.updated acknowledgment is optional; requiring it previously broke startup with this speech service.
• Preserve query parameters such as phoneNumber and episode_id, existing s2s.ws.* preference keys, session-only RAG, and TALK/TALKFIRST behavior unless the task explicitly changes them. Context comes from browser instructions and server/provider calls; do not assume a vector database or a completed magic-link authentication flow exists.
• Assistant Markdown is parsed with Marked and sanitized with DOMPurify in vocero-frontend/src/lib/messages.ts. User text stays literal, and exports retain raw message text. Keep sanitization and unsafe-link regression coverage when changing it.
• Reuse the Tailwind tokens in vocero-frontend/src/styles/theme.css and shared controls in vocero-frontend/src/components/ui.tsx. Keep specialized artwork and sanitized Markdown styles scoped under vocero-frontend/src/styles/. Verify both light and dark themes, narrow screens, keyboard focus, dialog dismissal, and voice-state feedback for UI changes. Preserve intentional ?debug=true controls and browser audio-permission flows.
• User-visible times use America/Santiago where no saved zone exists; avoid browser-local timezone assumptions. Keep raw timestamps in their existing API and export formats.

Startup and deployment boundaries

./start_ui.sh loads the optional root .env, syncs vocero-api/.venv/ with uv sync --project vocero-api --locked, and installs frontend packages when the manifests change. It starts Vite on 0.0.0.0:8420 and FastAPI on 127.0.0.1:8421, with UI_HOST, UI_PORT, and S2S_BACKEND_PORT overrides. Use distinct ports for concurrent worktrees. Preserve readiness checks, failure propagation, and shutdown of both process groups, including Uvicorn's reload worker.

Staging intentionally retains Vite HMR and Uvicorn reload. The Vite child uses NODE_ENV=development even under PM2. Do not replace this with static production serving unless that deployment change is requested. Vite proxies /api, /v1/realtime, and documentation routes to FastAPI; preserve public Host headers used to construct WebSocket URLs. Remote staging needs explicit S2S_DEV_ALLOWED_HOSTS hostnames and reverse-proxy WebSocket forwarding for both /v1/realtime and /__vite_hmr.

The established PM2 process name is s2s_ui. When it points to this launcher's deployed checkout, pm2 restart s2s_ui --update-env also performs the locked uv sync; a separate sync is optional to prepare dependencies before restarting. Ensure the required uv and Node/npm are on that process's PATH. Verify the live PM2 script path and environment before making claims about a deployment.

The launcher does not start or restart speech_to_speech/. A configured speech URL does not prove which source revision or implementation runs there. SPEECH_TO_SPEECH_URL pins a direct browser connection; S2S_PROXY_TARGET enables the local relay. Either disables load-balancer mode. Reasoning supports PROXY and OPENAI; unknown S2S_REASONING_TYPE values fall back to PROXY.

The backend Docker image serves APIs, WebSockets, and root docs/, not the React UI. Build with the repository root as context and vocero-api/Dockerfile; the image listens on 7860. FastAPI's root remains 404, /docs is the API explorer, and project documentation is served below /docs/ with retained legacy redirects.

Language, documentation, and secrets

• Use English for code, identifiers, comments, logs, internal instructions, and developer documentation. Spanish is the default for visible UI, spoken output, and user-facing API messages. Preserve literal wake phrases and protocol names; translating an existing response or prompt can change a contract.
• Keep maintained documentation under docs/client/, docs/speech/, docs/adr/, docs/examples/, or docs/proposals/. Update existing documents first and keep docs/README.md current when adding a document. Use one canonical English file without language suffixes. Historical proposals do not define current behavior.
• Reserve root Markdown for established files such as README.md and AGENTS.md. Keep task summaries in the conversation unless a saved artifact is requested. Record notable changes under Unreleased in vocero-api/CHANGELOG.md.
• Keep credentials and deployment values out of commits and examples. Never copy the root .env into the frontend or expose backend secrets through VITE_*. The launcher loads .env when present, so its values can override shell values; direct Python startup uses load_dotenv(..., override=False). Preserve both behaviors and avoid importing live configuration when a test can isolate it.

Start with usage, architecture, development, and troubleshooting. For speech work, read the realtime protocol and the relevant STT, LLM, or TTS guide before changing pipeline behavior.