106 lines
5.0 KiB
Markdown
106 lines
5.0 KiB
Markdown
# Reasoning modes
|
|
|
|
Reasoning determines the assistant's response text. The realtime backend always
|
|
provides the speech channel; optional HTTP reasoning can observe a transcript or
|
|
produce a separate answer. Set `S2S_REASONING_TYPE` to select the mode.
|
|
|
|
| Mode | Response owner | HTTP hook |
|
|
|---|---|---|
|
|
| `PROXY` (default) | Realtime backend | Browser does not call `/api/reasoning`; that endpoint returns 409 |
|
|
| `NODE_RED` | Realtime backend for `log_only=true`; external answer otherwise | Server forwards to `S2S_REASONING_NODE_RED_URL/api/reasoning` |
|
|
| `OPENAI` | OpenAI-compatible Chat Completions response | Server calls `OPENAI_BASE_URL/chat/completions` |
|
|
|
|
The mode is normalized case-insensitively. `NODERED` and `NODE-RED` resolve to
|
|
`NODE_RED`; unknown values resolve to `PROXY`.
|
|
|
|
## Shared instruction assembly
|
|
|
|
In [context.ts](../../s2s-frontend/src/lib/context.ts), `buildInstructions()` starts
|
|
with `config.primaryPrompt || settings.instructions`. `/api/config` pins
|
|
`S2S_PRIMARY_PROMPT` when non-empty; otherwise the settings prompt or
|
|
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
|
|
profile to choose this base prompt.
|
|
|
|
The browser adds caller context when `injectProfile` is enabled, custom RAG text
|
|
when the operator has applied it, and an available ApiPilot catalog block.
|
|
`speechInstructions()` adds the speech-rendering hint only in `OPENAI` mode.
|
|
The WebSocket client packages these instructions in `session.update.session`;
|
|
the local proxy may then add server-resolved context. Details belong in
|
|
[context injection](context-injection.md).
|
|
|
|
## Native realtime reasoning (`PROXY`)
|
|
|
|
The realtime backend receives session instructions and enabled native tools,
|
|
then processes each voice turn through STT, model generation and TTS.
|
|
Native function-call events are executed by the browser. ApiPilot also has a
|
|
separate marker-based path handled by the local proxy. These are distinct
|
|
protocols, described in [tool calling](tool-calling.md).
|
|
|
|
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
|
|
instructions still apply, but server-side session context and marker execution
|
|
require the local `/v1/realtime` route.
|
|
|
|
## Node-RED reasoning
|
|
|
|
Configure a Node-RED base URL reachable by the UI server:
|
|
|
|
```env
|
|
S2S_REASONING_TYPE=NODE_RED
|
|
S2S_REASONING_NODE_RED_URL=http://localhost:1881
|
|
```
|
|
|
|
`VoiceController.reason()` in [controller.ts](../../s2s-frontend/src/realtime/controller.ts)
|
|
handles both external modes. It runs after a complete user transcript and posts a body with
|
|
`transcript`, `instructions`, `language`, `user_context`, `phoneNumber`, and
|
|
`episode_id`. The server builds context and forwards the reasoning fields to
|
|
Node-RED with a 30-second timeout.
|
|
|
|
The checked-in Node-RED flow exposes `POST /api/reasoning` and
|
|
`POST /api/log/system-response`. Its reasoning response uses `log_only=true` and
|
|
an empty answer. The browser returns early in that case, preserving the native
|
|
backend's response. Assistant-text logging is forwarded separately by the relay
|
|
and is not an audio transport.
|
|
|
|
An answer-producing flow must return a non-empty `answer` and omit `log_only` or
|
|
set it to `false`. That activates the external-answer behavior below.
|
|
|
|
## OpenAI-compatible HTTP reasoning
|
|
|
|
```env
|
|
S2S_REASONING_TYPE=OPENAI
|
|
OPENAI_API_KEY=YOUR_API_KEY
|
|
OPENAI_BASE_URL=https://api.openai.com/v1
|
|
OPENAI_REASONING_MODEL=gpt-4.1-mini
|
|
```
|
|
|
|
`OPENAI_ORGANIZATION` and `OPENAI_PROJECT` are optional headers. The model value
|
|
is a runtime default, not a recommendation about provider availability.
|
|
|
|
The server combines instructions and context into model messages, requests a
|
|
final answer, and returns `answer` with available usage metadata. ApiPilot can
|
|
append discovery instructions, execute a detected call marker, and request a
|
|
second response based on the tool result. The endpoint does not forward the
|
|
browser's native `session.tools` catalog as OpenAI Chat Completions `tools`.
|
|
|
|
For a non-log-only answer, the browser cancels/suppresses the active native
|
|
response and appends the external assistant text. It uses browser
|
|
`SpeechSynthesis` when available. The React client adds the answer to the realtime
|
|
conversation as an assistant item; without browser speech synthesis it remains
|
|
visible as text.
|
|
|
|
External reasoning adds an HTTP operation to each completed transcript and has
|
|
separate provider token usage. It is not the speech pipeline's `responses-api`
|
|
or `chat-completions` LLM backend setting; those configure the model inside the
|
|
realtime backend instead.
|
|
|
|
## Where to change behavior
|
|
|
|
| Goal | Source |
|
|
|---|---|
|
|
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
|
|
| Change browser context assembly | `buildInstructions()` in `s2s-frontend/src/lib/context.ts` |
|
|
| Change HTTP provider messages | `reasoning()` in `server.py` |
|
|
| Change external-answer rendering | `VoiceController.reason()` and `RealtimeClient.renderExternalAnswer()` |
|
|
| Change Node-RED processing | `s2s-reasoning-engine/flows.json` |
|
|
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
|