alvaro
dfed03ac7c
Vocero now retrieves AquaPortal content through authenticated MCP. This removes the obsolete ApiPilot integration from the API and browser while preserving the separately deployed speech pipeline. Release notes: - Removed ApiPilot endpoints, configuration, credential overrides, and HTTP/realtime tool execution. - Removed browser ApiPilot catalog loading and prompt generation while retaining authenticated MCP discovery. - Updated regression coverage and operating documentation for the retired routes and settings. Reviewed-on: AquaPortal/s2s#3
88 lines
4.5 KiB
Markdown
88 lines
4.5 KiB
Markdown
# Reasoning modes
|
|
|
|
Reasoning determines the assistant's response text. The realtime backend always
|
|
provides the speech channel; optional HTTP reasoning produces a separate answer
|
|
from a transcript. Set `S2S_REASONING_TYPE` to select the mode.
|
|
|
|
| Mode | Response owner | HTTP hook |
|
|
|---|---|---|
|
|
| `PROXY` (default) | Realtime backend | Browser does not call `/api/reasoning`; that endpoint returns 409 |
|
|
| `OPENAI` | OpenAI-compatible Chat Completions response | Server calls `OPENAI_BASE_URL/chat/completions` |
|
|
|
|
The mode is normalized case-insensitively. Unknown values resolve to `PROXY`.
|
|
|
|
## Shared instruction assembly
|
|
|
|
In [context.ts](../../vocero-frontend/src/lib/context.ts), `buildInstructions()` starts
|
|
with `config.primaryPrompt || settings.instructions`. `/api/config` pins
|
|
`S2S_PRIMARY_PROMPT` when non-empty; otherwise the settings prompt or
|
|
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
|
|
profile to choose this base prompt.
|
|
|
|
The browser adds caller context when `injectProfile` is enabled and custom RAG
|
|
text when the operator has applied it. In MCP mode, the server constructs tool
|
|
instructions from the authenticated conversation's catalog and content selection.
|
|
`speechInstructions()` adds the speech-rendering hint only in `OPENAI` mode.
|
|
The WebSocket client packages these instructions in `session.update.session`;
|
|
the local proxy may then add server-resolved context. Details belong in
|
|
[context injection](context-injection.md).
|
|
|
|
## Native realtime reasoning (`PROXY`)
|
|
|
|
The realtime backend receives session instructions and enabled native tools,
|
|
then processes each voice turn through STT, model generation and TTS.
|
|
Native function-call events are executed by the browser. Authenticated MCP has a
|
|
separate marker-based path handled by the local proxy. These are distinct
|
|
protocols, described in [tool calling](tool-calling.md).
|
|
|
|
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
|
|
instructions still apply, but server-side session context and marker execution
|
|
require the local `/v1/realtime` route. MCP in `PROXY` mode requires this relay;
|
|
configuration rejects a pinned direct URL or load-balancer bypass.
|
|
|
|
## OpenAI-compatible HTTP reasoning
|
|
|
|
```env
|
|
S2S_REASONING_TYPE=OPENAI
|
|
OPENAI_API_KEY=YOUR_API_KEY
|
|
OPENAI_BASE_URL=https://api.openai.com/v1
|
|
OPENAI_REASONING_MODEL=gpt-4.1-mini
|
|
```
|
|
|
|
`OPENAI_ORGANIZATION` and `OPENAI_PROJECT` are optional headers. The model value
|
|
is a runtime default, not a recommendation about provider availability.
|
|
|
|
`VoiceController.reason()` in [controller.ts](../../vocero-frontend/src/realtime/controller.ts)
|
|
runs after a complete user transcript and posts a body with `transcript`,
|
|
`instructions`, `language`, `user_context`, `phoneNumber`, and `episode_id`.
|
|
Without MCP, the server combines instructions and context into model messages,
|
|
requests a final answer with a 30-second timeout, and returns `answer`.
|
|
With MCP, the browser also sends conversation and turn headers. The server uses
|
|
that conversation's backend token, linked/current content, and bounded history
|
|
to run tool selections before returning `answer`. This path has a 180-second
|
|
total timeout and configured per-operation limits. Retired ApiPilot execution
|
|
metadata is no longer returned. The endpoint does not forward the browser's
|
|
native `session.tools` catalog as OpenAI Chat Completions `tools`.
|
|
|
|
For an external answer, the browser cancels/suppresses the active native
|
|
response and appends the external assistant text. It uses browser
|
|
`SpeechSynthesis` when available. The React client adds the answer to the realtime
|
|
conversation as an assistant item; without browser speech synthesis it remains
|
|
visible as text.
|
|
|
|
External reasoning adds an HTTP operation to each completed transcript and has
|
|
separate provider token usage. It is not the speech pipeline's `responses-api`
|
|
or `chat-completions` LLM backend setting; those configure the model inside the
|
|
realtime backend instead.
|
|
|
|
## Where to change behavior
|
|
|
|
| Goal | Source |
|
|
|---|---|
|
|
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
|
|
| Change browser context assembly | `buildInstructions()` in `vocero-frontend/src/lib/context.ts` |
|
|
| Change HTTP provider messages | `reasoning()` in `vocero-api/vocero_api/services/reasoning.py` |
|
|
| Change MCP reasoning or tool context | `vocero-api/vocero_api/services/mcp_reasoning.py` and `vocero-api/vocero_api/services/mcp_tools.py` |
|
|
| Change external-answer rendering | `VoiceController.reason()` and `RealtimeClient.renderExternalAnswer()` |
|
|
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
|