vocero-client/docs/client/reasoning.md

Reasoning modes

Reasoning determines the assistant's response text. The realtime backend always provides the speech channel; optional HTTP reasoning produces a separate answer from a transcript. Set S2S_REASONING_TYPE to select the mode.

Mode	Response owner	HTTP hook
PROXY (default)	Realtime backend	Browser does not call /api/reasoning; that endpoint returns 409
OPENAI	OpenAI-compatible Chat Completions response	Server calls OPENAI_BASE_URL/chat/completions

The mode is normalized case-insensitively. Unknown values resolve to PROXY.

Shared instruction assembly

In context.ts, buildInstructions() starts with config.primaryPrompt || settings.instructions. /api/config pins S2S_PRIMARY_PROMPT when non-empty; otherwise the settings prompt or DEFAULT_INSTRUCTIONS is used. The current server does not fetch a remote agent profile to choose this base prompt.

The browser adds caller context when injectProfile is enabled, custom RAG text when the operator has applied it, and an available ApiPilot catalog block. speechInstructions() adds the speech-rendering hint only in OPENAI mode. The WebSocket client packages these instructions in session.update.session; the local proxy may then add server-resolved context. Details belong in context injection.

Native realtime reasoning (PROXY)

The realtime backend receives session instructions and enabled native tools, then processes each voice turn through STT, model generation and TTS. Native function-call events are executed by the browser. ApiPilot also has a separate marker-based path handled by the local proxy. These are distinct protocols, described in tool calling.

Using a direct browser URL bypasses local WebSocket proxy processing. Browser instructions still apply, but server-side session context and marker execution require the local /v1/realtime route.

OpenAI-compatible HTTP reasoning

S2S_REASONING_TYPE=OPENAI
OPENAI_API_KEY=YOUR_API_KEY
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_REASONING_MODEL=gpt-4.1-mini

OPENAI_ORGANIZATION and OPENAI_PROJECT are optional headers. The model value is a runtime default, not a recommendation about provider availability.

VoiceController.reason() in controller.ts runs after a complete user transcript and posts a body with transcript, instructions, language, user_context, phoneNumber, and episode_id. The server combines instructions and context into model messages, requests a final answer with a 30-second timeout, and returns answer. ApiPilot can append discovery instructions, execute a detected call marker, and request a second response based on the tool result, including available usage_pass2 metadata. The endpoint does not forward the browser's native session.tools catalog as OpenAI Chat Completions tools.

For an external answer, the browser cancels/suppresses the active native response and appends the external assistant text. It uses browser SpeechSynthesis when available. The React client adds the answer to the realtime conversation as an assistant item; without browser speech synthesis it remains visible as text.

External reasoning adds an HTTP operation to each completed transcript and has separate provider token usage. It is not the speech pipeline's responses-api or chat-completions LLM backend setting; those configure the model inside the realtime backend instead.

Where to change behavior

Goal	Source
Change the base agent prompt	S2S_PRIMARY_PROMPT or browser settings when unpinned
Change browser context assembly	buildInstructions() in s2s-frontend/src/lib/context.ts
Change HTTP provider messages	reasoning() in s2s-client/s2s_client/services/reasoning.py
Change external-answer rendering	VoiceController.reason() and RealtimeClient.renderExternalAnswer()
Change native model/tool generation	speech_to_speech/LLM and realtime handlers