alvaro
abd03ed07d
Rename the project directories, Python distribution and import package, and npm package. Update launcher, Docker, tooling, tests, lockfiles, and documentation while preserving existing API contracts. Make AGENTS.md self-contained and remove external skill and workflow references.
83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
# Reasoning modes
|
|
|
|
Reasoning determines the assistant's response text. The realtime backend always
|
|
provides the speech channel; optional HTTP reasoning produces a separate answer
|
|
from a transcript. Set `S2S_REASONING_TYPE` to select the mode.
|
|
|
|
| Mode | Response owner | HTTP hook |
|
|
|---|---|---|
|
|
| `PROXY` (default) | Realtime backend | Browser does not call `/api/reasoning`; that endpoint returns 409 |
|
|
| `OPENAI` | OpenAI-compatible Chat Completions response | Server calls `OPENAI_BASE_URL/chat/completions` |
|
|
|
|
The mode is normalized case-insensitively. Unknown values resolve to `PROXY`.
|
|
|
|
## Shared instruction assembly
|
|
|
|
In [context.ts](../../vocero-frontend/src/lib/context.ts), `buildInstructions()` starts
|
|
with `config.primaryPrompt || settings.instructions`. `/api/config` pins
|
|
`S2S_PRIMARY_PROMPT` when non-empty; otherwise the settings prompt or
|
|
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
|
|
profile to choose this base prompt.
|
|
|
|
The browser adds caller context when `injectProfile` is enabled, custom RAG text
|
|
when the operator has applied it, and an available ApiPilot catalog block.
|
|
`speechInstructions()` adds the speech-rendering hint only in `OPENAI` mode.
|
|
The WebSocket client packages these instructions in `session.update.session`;
|
|
the local proxy may then add server-resolved context. Details belong in
|
|
[context injection](context-injection.md).
|
|
|
|
## Native realtime reasoning (`PROXY`)
|
|
|
|
The realtime backend receives session instructions and enabled native tools,
|
|
then processes each voice turn through STT, model generation and TTS.
|
|
Native function-call events are executed by the browser. ApiPilot also has a
|
|
separate marker-based path handled by the local proxy. These are distinct
|
|
protocols, described in [tool calling](tool-calling.md).
|
|
|
|
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
|
|
instructions still apply, but server-side session context and marker execution
|
|
require the local `/v1/realtime` route.
|
|
|
|
## OpenAI-compatible HTTP reasoning
|
|
|
|
```env
|
|
S2S_REASONING_TYPE=OPENAI
|
|
OPENAI_API_KEY=YOUR_API_KEY
|
|
OPENAI_BASE_URL=https://api.openai.com/v1
|
|
OPENAI_REASONING_MODEL=gpt-4.1-mini
|
|
```
|
|
|
|
`OPENAI_ORGANIZATION` and `OPENAI_PROJECT` are optional headers. The model value
|
|
is a runtime default, not a recommendation about provider availability.
|
|
|
|
`VoiceController.reason()` in [controller.ts](../../vocero-frontend/src/realtime/controller.ts)
|
|
runs after a complete user transcript and posts a body with `transcript`,
|
|
`instructions`, `language`, `user_context`, `phoneNumber`, and `episode_id`.
|
|
The server combines instructions and context into model messages, requests a
|
|
final answer with a 30-second timeout, and returns `answer`. ApiPilot can
|
|
append discovery instructions, execute a detected call marker, and request a
|
|
second response based on the tool result, including available `usage_pass2`
|
|
metadata. The endpoint does not forward the browser's native `session.tools`
|
|
catalog as OpenAI Chat Completions `tools`.
|
|
|
|
For an external answer, the browser cancels/suppresses the active native
|
|
response and appends the external assistant text. It uses browser
|
|
`SpeechSynthesis` when available. The React client adds the answer to the realtime
|
|
conversation as an assistant item; without browser speech synthesis it remains
|
|
visible as text.
|
|
|
|
External reasoning adds an HTTP operation to each completed transcript and has
|
|
separate provider token usage. It is not the speech pipeline's `responses-api`
|
|
or `chat-completions` LLM backend setting; those configure the model inside the
|
|
realtime backend instead.
|
|
|
|
## Where to change behavior
|
|
|
|
| Goal | Source |
|
|
|---|---|
|
|
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
|
|
| Change browser context assembly | `buildInstructions()` in `vocero-frontend/src/lib/context.ts` |
|
|
| Change HTTP provider messages | `reasoning()` in `vocero-api/vocero_api/services/reasoning.py` |
|
|
| Change external-answer rendering | `VoiceController.reason()` and `RealtimeClient.renderExternalAnswer()` |
|
|
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
|