139 lines
5.4 KiB
Markdown
139 lines
5.4 KiB
Markdown
# Tool calling
|
|
|
|
The application has two tool mechanisms: native realtime function calling and
|
|
ApiPilot calls detected from model text. A spoken promise to fetch something is
|
|
not evidence that either mechanism executed a tool.
|
|
|
|
## Native realtime function calls
|
|
|
|
The browser sends tool declarations inside the `session` object:
|
|
|
|
```json
|
|
{
|
|
"type": "session.update",
|
|
"session": {
|
|
"type": "realtime",
|
|
"tools": [
|
|
{
|
|
"type": "function",
|
|
"name": "web_search",
|
|
"description": "Search the web for information.",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {"query": {"type": "string"}},
|
|
"required": ["query"]
|
|
}
|
|
}
|
|
],
|
|
"tool_choice": "auto"
|
|
}
|
|
}
|
|
```
|
|
|
|
Keep configuration inside `session`, not at the event root. The backend has a
|
|
legacy flat-payload compatibility path, but new clients should use the nested
|
|
shape. The current browser's `activeToolDefs()` declares `web_search` when
|
|
enabled and a deploy/user search key is available. It does not load the old
|
|
remote agent-profile MCP catalog into native session tools.
|
|
|
|
For a native call, the backend emits:
|
|
|
|
```json
|
|
{
|
|
"type": "response.function_call_arguments.done",
|
|
"call_id": "call_example",
|
|
"name": "web_search",
|
|
"arguments": "{\"query\":\"example query\"}"
|
|
}
|
|
```
|
|
|
|
`arguments` is a JSON-encoded string. The
|
|
[WebSocket client](../../s2s-frontend/src/realtime/client.ts) validates the event and
|
|
dispatches `tool`; [controller.ts](../../s2s-frontend/src/realtime/controller.ts)
|
|
parses the arguments and runs the supported executor.
|
|
Web search uses `POST /api/search`, keeping the deployment's Serper key on the
|
|
server. Tool output returns to the model as:
|
|
|
|
```json
|
|
{
|
|
"type": "conversation.item.create",
|
|
"item": {
|
|
"type": "function_call_output",
|
|
"call_id": "call_example",
|
|
"output": "Tool result text"
|
|
}
|
|
}
|
|
```
|
|
|
|
The client requests follow-up generation with `response.create` when the
|
|
result should be spoken. A `function_call_output` item alone does not start
|
|
generation. See the [realtime API guide](../speech/realtime-api.md) for local
|
|
model `<code>` extraction and native remote-provider tool calls.
|
|
|
|
## ApiPilot discovery and execution
|
|
|
|
`S2S_APIPILOT_TOOLS_URL` enables the current dynamic-tool bridge. The backend
|
|
discovers tools through its `/tools` route and executes through `/execute`.
|
|
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
|
|
server exposes:
|
|
|
|
- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
|
|
`data.tools`, `data.count`, and `data.include_unauthorized`.
|
|
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
|
|
a `success/data/error` envelope.
|
|
|
|
Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
|
|
`requires_auth`. Execution looks up the current discovery cache, refreshes once
|
|
if the tool is missing, and applies an authorization gate and argument
|
|
validation before forwarding. Caller `Authorization` and
|
|
`x-demo-session` can be forwarded; configured service credentials and
|
|
login-by-phone provide additional server paths. Display `error.message` to the
|
|
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.
|
|
|
|
The discovery cache defaults to 300 seconds server-side
|
|
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
|
|
also keeps a per-user catalog cache. ApiPilot configuration is independent of
|
|
`S2S_INJECT_PROFILE`.
|
|
|
|
## Marker-based two-pass reasoning
|
|
|
|
The browser and proxy add an ApiPilot catalog as **instruction text**, including
|
|
tool names, descriptions, authentication requirements and required parameters.
|
|
The model is instructed to emit a call marker when it chooses a matching tool:
|
|
|
|
```text
|
|
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
|
|
```
|
|
|
|
Tool names are exact identifiers and must not be translated. The local realtime
|
|
proxy accumulates assistant text, detects the marker, validates and executes
|
|
the ApiPilot call, then injects the result as context and requests a new
|
|
response. The HTTP reasoning route has analogous discovery/execution and
|
|
second-answer handling. Internal marker text and its speculative audio are
|
|
filtered from user output where those handlers run.
|
|
|
|
This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
|
|
discovery is not the same as declaring a native `session.tools` array. Direct
|
|
browser connections bypass the local relay's marker executor even if the
|
|
browser includes catalog instructions.
|
|
|
|
## Retired documentation assumptions
|
|
|
|
The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
|
|
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
|
|
Those APIs/rankers are absent from the current client/server path. The retained
|
|
agent-profile JSON is an illustrative historical/example shape. Add or modify
|
|
current ApiPilot tool definitions in the provider; do not restore obsolete
|
|
endpoints solely to match older documentation.
|
|
|
|
## Verify a tool turn
|
|
|
|
1. Confirm `/api/config` reports the intended feature and connection mode.
|
|
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
|
|
separate evidence.
|
|
3. For native calls, find `response.function_call_arguments.done`, then the
|
|
browser `toolcall` and executor request.
|
|
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
|
|
5. Confirm the tool result reaches the model and follow-up text/audio reaches
|
|
the browser. A spoken acknowledgement alone does not complete the flow.
|