alvaro
dfed03ac7c
Vocero now retrieves AquaPortal content through authenticated MCP. This removes the obsolete ApiPilot integration from the API and browser while preserving the separately deployed speech pipeline. Release notes: - Removed ApiPilot endpoints, configuration, credential overrides, and HTTP/realtime tool execution. - Removed browser ApiPilot catalog loading and prompt generation while retaining authenticated MCP discovery. - Updated regression coverage and operating documentation for the retired routes and settings. Reviewed-on: AquaPortal/s2s#3
4.5 KiB
Reasoning modes
Reasoning determines the assistant's response text. The realtime backend always
provides the speech channel; optional HTTP reasoning produces a separate answer
from a transcript. Set S2S_REASONING_TYPE to select the mode.
| Mode | Response owner | HTTP hook |
|---|---|---|
PROXY (default) |
Realtime backend | Browser does not call /api/reasoning; that endpoint returns 409 |
OPENAI |
OpenAI-compatible Chat Completions response | Server calls OPENAI_BASE_URL/chat/completions |
The mode is normalized case-insensitively. Unknown values resolve to PROXY.
Shared instruction assembly
In context.ts, buildInstructions() starts
with config.primaryPrompt || settings.instructions. /api/config pins
S2S_PRIMARY_PROMPT when non-empty; otherwise the settings prompt or
DEFAULT_INSTRUCTIONS is used. The current server does not fetch a remote agent
profile to choose this base prompt.
The browser adds caller context when injectProfile is enabled and custom RAG
text when the operator has applied it. In MCP mode, the server constructs tool
instructions from the authenticated conversation's catalog and content selection.
speechInstructions() adds the speech-rendering hint only in OPENAI mode.
The WebSocket client packages these instructions in session.update.session;
the local proxy may then add server-resolved context. Details belong in
context injection.
Native realtime reasoning (PROXY)
The realtime backend receives session instructions and enabled native tools, then processes each voice turn through STT, model generation and TTS. Native function-call events are executed by the browser. Authenticated MCP has a separate marker-based path handled by the local proxy. These are distinct protocols, described in tool calling.
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
instructions still apply, but server-side session context and marker execution
require the local /v1/realtime route. MCP in PROXY mode requires this relay;
configuration rejects a pinned direct URL or load-balancer bypass.
OpenAI-compatible HTTP reasoning
S2S_REASONING_TYPE=OPENAI
OPENAI_API_KEY=YOUR_API_KEY
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_REASONING_MODEL=gpt-4.1-mini
OPENAI_ORGANIZATION and OPENAI_PROJECT are optional headers. The model value
is a runtime default, not a recommendation about provider availability.
VoiceController.reason() in controller.ts
runs after a complete user transcript and posts a body with transcript,
instructions, language, user_context, phoneNumber, and episode_id.
Without MCP, the server combines instructions and context into model messages,
requests a final answer with a 30-second timeout, and returns answer.
With MCP, the browser also sends conversation and turn headers. The server uses
that conversation's backend token, linked/current content, and bounded history
to run tool selections before returning answer. This path has a 180-second
total timeout and configured per-operation limits. Retired ApiPilot execution
metadata is no longer returned. The endpoint does not forward the browser's
native session.tools catalog as OpenAI Chat Completions tools.
For an external answer, the browser cancels/suppresses the active native
response and appends the external assistant text. It uses browser
SpeechSynthesis when available. The React client adds the answer to the realtime
conversation as an assistant item; without browser speech synthesis it remains
visible as text.
External reasoning adds an HTTP operation to each completed transcript and has
separate provider token usage. It is not the speech pipeline's responses-api
or chat-completions LLM backend setting; those configure the model inside the
realtime backend instead.
Where to change behavior
| Goal | Source |
|---|---|
| Change the base agent prompt | S2S_PRIMARY_PROMPT or browser settings when unpinned |
| Change browser context assembly | buildInstructions() in vocero-frontend/src/lib/context.ts |
| Change HTTP provider messages | reasoning() in vocero-api/vocero_api/services/reasoning.py |
| Change MCP reasoning or tool context | vocero-api/vocero_api/services/mcp_reasoning.py and vocero-api/vocero_api/services/mcp_tools.py |
| Change external-answer rendering | VoiceController.reason() and RealtimeClient.renderExternalAnswer() |
| Change native model/tool generation | speech_to_speech/LLM and realtime handlers |