vocero-client/docs/client/reasoning.md

Reasoning modes

Reasoning determines the assistant's response text. The realtime backend always provides the speech channel; optional HTTP reasoning can observe a transcript or produce a separate answer. Set S2S_REASONING_TYPE to select the mode.

Mode	Response owner	HTTP hook
PROXY (default)	Realtime backend	Browser does not call /api/reasoning; that endpoint returns 409
NODE_RED	Realtime backend for log_only=true; external answer otherwise	Server forwards to S2S_REASONING_NODE_RED_URL/api/reasoning
OPENAI	OpenAI-compatible Chat Completions response	Server calls OPENAI_BASE_URL/chat/completions

The mode is normalized case-insensitively. NODERED and NODE-RED resolve to NODE_RED; unknown values resolve to PROXY.

Shared instruction assembly

In context.ts, buildInstructions() starts with config.primaryPrompt || settings.instructions. /api/config pins S2S_PRIMARY_PROMPT when non-empty; otherwise the settings prompt or DEFAULT_INSTRUCTIONS is used. The current server does not fetch a remote agent profile to choose this base prompt.

The browser adds caller context when injectProfile is enabled, custom RAG text when the operator has applied it, and an available ApiPilot catalog block. speechInstructions() adds the speech-rendering hint only in OPENAI mode. The WebSocket client packages these instructions in session.update.session; the local proxy may then add server-resolved context. Details belong in context injection.

Native realtime reasoning (PROXY)

The realtime backend receives session instructions and enabled native tools, then processes each voice turn through STT, model generation and TTS. Native function-call events are executed by the browser. ApiPilot also has a separate marker-based path handled by the local proxy. These are distinct protocols, described in tool calling.

Using a direct browser URL bypasses local WebSocket proxy processing. Browser instructions still apply, but server-side session context and marker execution require the local /v1/realtime route.

Node-RED reasoning

Configure a Node-RED base URL reachable by the UI server:

S2S_REASONING_TYPE=NODE_RED
S2S_REASONING_NODE_RED_URL=http://localhost:1881

VoiceController.reason() in controller.ts handles both external modes. It runs after a complete user transcript and posts a body with transcript, instructions, language, user_context, phoneNumber, and episode_id. The server builds context and forwards the reasoning fields to Node-RED with a 30-second timeout.

The checked-in Node-RED flow exposes POST /api/reasoning and POST /api/log/system-response. Its reasoning response uses log_only=true and an empty answer. The browser returns early in that case, preserving the native backend's response. Assistant-text logging is forwarded separately by the relay and is not an audio transport.

An answer-producing flow must return a non-empty answer and omit log_only or set it to false. That activates the external-answer behavior below.

OpenAI-compatible HTTP reasoning

S2S_REASONING_TYPE=OPENAI
OPENAI_API_KEY=YOUR_API_KEY
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_REASONING_MODEL=gpt-4.1-mini

OPENAI_ORGANIZATION and OPENAI_PROJECT are optional headers. The model value is a runtime default, not a recommendation about provider availability.

The server combines instructions and context into model messages, requests a final answer, and returns answer with available usage metadata. ApiPilot can append discovery instructions, execute a detected call marker, and request a second response based on the tool result. The endpoint does not forward the browser's native session.tools catalog as OpenAI Chat Completions tools.

For a non-log-only answer, the browser cancels/suppresses the active native response and appends the external assistant text. It uses browser SpeechSynthesis when available. The React client adds the answer to the realtime conversation as an assistant item; without browser speech synthesis it remains visible as text.

External reasoning adds an HTTP operation to each completed transcript and has separate provider token usage. It is not the speech pipeline's responses-api or chat-completions LLM backend setting; those configure the model inside the realtime backend instead.

Where to change behavior

Goal	Source
Change the base agent prompt	S2S_PRIMARY_PROMPT or browser settings when unpinned
Change browser context assembly	buildInstructions() in s2s-frontend/src/lib/context.ts
Change HTTP provider messages	reasoning() in server.py
Change external-answer rendering	VoiceController.reason() and RealtimeClient.renderExternalAnswer()
Change Node-RED processing	s2s-reasoning-engine/flows.json
Change native model/tool generation	speech_to_speech/LLM and realtime handlers