vocero-client/AGENTS.md

Repository guidance

Scope and working conventions

This repository is Vocero: a browser voice client, its FastAPI integration server, and a speech pipeline. Use the code, commands, configuration, and documentation within this checkout. Do not assume other repositories or developer-specific directories are available.

All file and directory references in this guide are relative to the repository root. HTTP routes mentioned below are URLs, not filesystem paths.

• Check the current branch, worktree, and working-tree changes before editing. Honor the checkout requested by the user, including direct work on main. Use an isolated worktree when requested and preserve other worktrees.
• An analysis-only or planning request authorizes inspection, not implementation. Return findings with source locations. Present plans in the conversation unless the user asks to save a file; follow any explicit approval gate.
• Write clear explanations and ask one conceptual question at a time. Carry out already authorized work without repeatedly asking for permission.
• Use the native patch/edit tool for manual changes. Prefer rg, fd, jq, git, and standard CLI tools for inspection; check availability before assuming a tool is missing. Do not install system-wide packages unnecessarily.
• When commits are requested, make coherent, scoped Conventional Commits as work progresses. Avoid excessive micro-commits and unrelated cleanup.
• Invoke work-on-feature only when explicitly requested. For a requested PR, follow the available pr-description skill exactly and put verification details in the handoff when its format excludes them. Issues are optional; create native Gitea issues or post comments only when requested.
• When asked to merge locally, use the requested rebase/fast-forward flow; preserve a recovery ref before rewriting and verify retained patches. A local merge does not imply a push, deployment, or restart. Report those separately.
• Treat old conversations and proposals as context. Verify claims against the current code and distinguish proposed authentication/deployment changes from implemented behavior.

Project map

Location	Responsibility
s2s-client/	The complete uv project: manifests, environment, Python tooling, tests, and FastAPI package
s2s-client/s2s_client/main.py	Application assembly, documentation mount, and retained redirects
s2s-client/s2s_client/api/	HTTP routes, route registration, and response validation helper
s2s-client/s2s_client/models/	Pydantic request and response contracts
s2s-client/s2s_client/core/	Environment parsing, logging, public URLs, and startup diagnostics
s2s-client/s2s_client/providers/	ApiPilot and caller/profile/report HTTP integrations
s2s-client/s2s_client/services/	Prompt/context construction, reasoning, and tool-context rendering
s2s-client/s2s_client/realtime/	WebSocket relay, event handling, marker filtering, and audio forwarding
s2s-frontend/	React, TypeScript, Tailwind, and Vite browser application
speech_to_speech/	Separately run VAD → STT → LLM → TTS pipeline and realtime API
start_ui.sh	Dependency sync and supervision of FastAPI plus Vite
docs/	Maintained technical documentation, examples, and historical proposals

s2s-client/server.py is a compatibility shim for server:app; new backend implementation belongs in the package. Keep HTTP handlers focused on transport and reuse providers/services for shared behavior.

The old static browser site, daily talk-time quotas, quota database/accounting, and Hugging Face account UI were deliberately removed. Do not restore their files, routes, dependencies, or compatibility layers as incidental cleanup. Load-balancer allocation/queues and upstream caller authentication remain.

Tooling and checks

The uv project must stay inside s2s-client/, including s2s-client/pyproject.toml, s2s-client/uv.lock, s2s-client/.python-version, s2s-client/.venv/, s2s-client/Makefile, and s2s-client/.pre-commit-config.yaml. Use Python 3.12 and the uv version required by tool.uv.required-version (currently 0.12.5). Update dependencies through uv and commit the lockfile; do not use pip to modify this environment or move it to the repository root. Keep uv pins in s2s-client/Dockerfile and setup documentation aligned when changing the required version.

The frontend uses npm and s2s-frontend/package-lock.json. Its Node requirement is ^22.18.0 || >=24.0.0; consult s2s-frontend/package.json for current scripts. Keep application code and supported configuration files in TypeScript.

Commands below run from the repository root:

Command	Purpose
make -C s2s-client install	Sync the locked Python environment and development tools
make -C s2s-client format	Apply Ruff fixes and formatting
make -C s2s-client check	Check lock freshness, formatting, Ruff lint, mypy, and pytest
make -C s2s-client test-launcher	Test readiness, failures, dependency sync, and process-group shutdown
make -C s2s-client test-image	Build and smoke-test the actual backend Docker image
npm --prefix s2s-frontend ci	Install locked frontend dependencies
npm --prefix s2s-frontend run check	Check formatting, lint, TypeScript, Vitest, and launcher integration
npm --prefix s2s-frontend run build	Type-check and build frontend assets

Python uses Ruff with a 120-character line limit, double quotes, import ordering, and typed application functions checked by mypy. Frontend formatting and lint rules live in s2s-frontend/prettier.config.ts and s2s-frontend/eslint.config.ts; use those tools rather than introducing competing conventions.

Run checks appropriate to the change: backend checks for Python/API work, frontend checks and build for browser work, launcher integration for supervisor changes, and image tests for packaging/Docker changes. Documentation-only work normally needs link/path and diff checks, not service startup. Wait for commands to finish before reporting success, and state skipped coverage or blockers.

Tests should use mock upstream HTTP and local WebSocket fixtures, isolating environment configuration. Do not rely on deployed services, real credentials, or model downloads for client tests. The client gate does not test the separate speech pipeline, microphone hardware, or real model output. That pipeline has no complete dependency manifest or general test gate in this checkout; inspect the selected backend and explain the verification limits for speech changes.

API and compatibility rules

Read backend development and compatibility before refactoring the server or changing schemas. Existing wire behavior is the baseline, including behavior that looks unsafe, inconsistent, or buggy. Intentional bug fixes or contract changes need their own scope and regression evidence; do not bundle them into tooling or structural refactors.

• Preserve paths, methods, statuses, error shapes, headers, field names, omissions, coercion, authentication precedence, and outgoing provider calls. Trace both frontend consumers and provider parsing; a permissive consumer is not evidence that an upstream schema can safely be tightened.
• Retain existing request-model defaults and extra-field behavior. Do not make new fields required, forbid previously accepted extras, or normalize loosely parsed provider values without evidence and an intentional behavior change.
• Response models validate fields constructed by this service. Upstream-owned values remain permissive where the old parser was permissive: Any, optional fields, open objects, and arbitrary profile JSON are deliberate boundaries.
• validated_json_response in s2s-client/s2s_client/api/responses.py validates with Pydantic, then serializes the original payload. Do not substitute model_dump() or automatic response-model serialization: those can insert defaults, filter unknown keys, coerce values, or alter serialization. Keep OpenAPI declarations aligned with the actual validation boundary.
• Preserve environment/dotenv parsing, default flags, and import-time behavior during refactors. Do not introduce stricter configuration validation or change authentication behavior as part of a tooling change.
• s2s-client/tests/fixtures/legacy-contracts.json captures the server before the migration at 3d42df5. s2s-client/tests/test_compatibility.py compares HTTP response bytes, statuses, selected headers, outgoing requests, request schemas, and realtime frames. Do not regenerate captures merely to make a refactor pass.
• Known reasoning failures, ApiPilot authentication/cache quirks, and relay behavior are documented in the compatibility guide. Preserve narrow lint/type exceptions that retain those behaviors until a deliberate fix is in scope.
• The pytest warning filter is only for Starlette TestClient's exact deprecated AnyIO BlockingPortal alias. Do not broaden warning suppression or upgrade runtime frameworks just to silence that warning without a compatibility review.

Browser and realtime behavior

• React renders the UI; s2s-frontend/src/realtime/controller.ts owns observable application state, s2s-frontend/src/realtime/client.ts owns protocol/session lifecycle, and s2s-frontend/src/audio/ owns the audio graph and worklets. Keep high-frequency audio/WebSocket work outside React rendering and preserve cleanup across unmounts and reconnects.
• Preserve the existing 16 kHz PCM16 audio path, worklet registration, interruption behavior, and teardown. Session startup initializes audio, waits for session.created, and sends session.update before audio or the greeting. A session.updated acknowledgment is optional; requiring it previously broke startup with this speech service.
• Preserve query parameters such as phoneNumber and episode_id, existing s2s.ws.* preference keys, session-only RAG, and TALK/TALKFIRST behavior unless the task explicitly changes them. Context comes from browser instructions and server/provider calls; do not assume a vector database or a completed magic-link authentication flow exists.
• Assistant Markdown is parsed with Marked and sanitized with DOMPurify in s2s-frontend/src/lib/messages.ts. User text stays literal, and exports retain raw message text. Keep sanitization and unsafe-link regression coverage when changing it.
• Reuse the existing design tokens, Tailwind integration, and s2s-frontend/src/styles/legacy-theme.css. Verify both light and dark themes, narrow screens, keyboard focus, dialog dismissal, and voice-state feedback for UI changes. Preserve intentional ?debug=true controls and browser audio-permission flows.
• User-visible times use America/Santiago where no saved zone exists; avoid browser-local timezone assumptions. Keep raw timestamps in their existing API and export formats.

Startup and deployment boundaries

./start_ui.sh sources .env, syncs s2s-client/.venv/ with uv sync --project s2s-client --locked, and installs frontend packages when the manifests change. It starts Vite on 0.0.0.0:8420 and FastAPI on 127.0.0.1:8421, with UI_HOST, UI_PORT, and S2S_BACKEND_PORT overrides. Use distinct ports for concurrent worktrees. Preserve readiness checks, failure propagation, and shutdown of both process groups, including Uvicorn's reload worker.

Staging intentionally retains Vite HMR and Uvicorn reload. The Vite child uses NODE_ENV=development even under PM2. Do not replace this with static production serving unless that deployment change is requested. Vite proxies /api, /v1/realtime, and documentation routes to FastAPI; preserve public Host headers used to construct WebSocket URLs. Remote staging needs explicit S2S_DEV_ALLOWED_HOSTS hostnames and reverse-proxy WebSocket forwarding for both /v1/realtime and /__vite_hmr.

The established PM2 process name is s2s_ui. When it points to this launcher's deployed checkout, pm2 restart s2s_ui --update-env also performs the locked uv sync; a separate sync is optional to prepare dependencies before restarting. Ensure the required uv and Node/npm are on that process's PATH. Verify the live PM2 script path and environment before making claims about a deployment.

The launcher does not start or restart speech_to_speech/. A configured speech URL does not prove which source revision or implementation runs there. SPEECH_TO_SPEECH_URL pins a direct browser connection; S2S_PROXY_TARGET enables the local relay. Either disables load-balancer mode. Reasoning supports PROXY and OPENAI; unknown S2S_REASONING_TYPE values fall back to PROXY.

The backend Docker image serves APIs, WebSockets, and root docs/, not the React UI. Build with the repository root as context and s2s-client/Dockerfile; the image listens on 7860. FastAPI's root remains 404, /docs is the API explorer, and project documentation is served below /docs/ with retained legacy redirects.

Language, documentation, and secrets

• Use English for code, identifiers, comments, logs, internal instructions, and developer documentation. Spanish is the default for visible UI, spoken output, and user-facing API messages. Preserve literal wake phrases and protocol names; translating an existing response or prompt can change a contract.
• Keep maintained documentation under docs/client/, docs/speech/, docs/adr/, docs/examples/, or docs/proposals/. Update existing documents first and keep docs/README.md current when adding a document. Use one canonical English file without language suffixes. Historical proposals do not define current behavior.
• Reserve root Markdown for established files such as README.md and AGENTS.md. Keep task summaries in the conversation unless a saved artifact is requested. Record notable changes under Unreleased in s2s-client/CHANGELOG.md.
• Keep credentials and deployment values out of commits and examples. Never copy the root .env into the frontend or expose backend secrets through VITE_*. The launcher sources .env before startup, so it can override shell values; direct Python startup uses load_dotenv(..., override=False). Preserve both behaviors and avoid importing live configuration when a test can isolate it.

Start with usage, architecture, development, and troubleshooting. For speech work, read the realtime protocol and the relevant STT, LLM, or TTS guide before changing pipeline behavior.