# Tool calling

The application has two tool mechanisms: native realtime function calling and
ApiPilot calls detected from model text. A spoken promise to fetch something is
not evidence that either mechanism executed a tool.

## Native realtime function calls

The browser sends tool declarations inside the `session` object:

```json
{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "tools": [
      {
        "type": "function",
        "name": "web_search",
        "description": "Search the web for information.",
        "parameters": {
          "type": "object",
          "properties": {"query": {"type": "string"}},
          "required": ["query"]
        }
      }
    ],
    "tool_choice": "auto"
  }
}
```

Keep configuration inside `session`, not at the event root. The backend has a
legacy flat-payload compatibility path, but new clients should use the nested
shape. The current browser's `activeToolDefs()` declares `web_search` when
enabled and a deploy/user search key is available. It does not load the old
remote agent-profile MCP catalog into native session tools.

For a native call, the backend emits:

```json
{
  "type": "response.function_call_arguments.done",
  "call_id": "call_example",
  "name": "web_search",
  "arguments": "{\"query\":\"example query\"}"
}
```

`arguments` is a JSON-encoded string. The
[WebSocket client](../../s2s-frontend/src/realtime/client.ts) validates the event and
dispatches `tool`; [controller.ts](../../s2s-frontend/src/realtime/controller.ts)
parses the arguments and runs the supported executor.
Web search uses `POST /api/search`, keeping the deployment's Serper key on the
server. Tool output returns to the model as:

```json
{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_example",
    "output": "Tool result text"
  }
}
```

The client requests follow-up generation with `response.create` when the
result should be spoken. A `function_call_output` item alone does not start
generation. See the [realtime API guide](../speech/realtime-api.md) for local
model `<code>` extraction and native remote-provider tool calls.

## ApiPilot discovery and execution

`S2S_APIPILOT_TOOLS_URL` enables the current dynamic-tool bridge. The backend
discovers tools through its `/tools` route and executes through `/execute`.
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
server exposes:

- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
  `data.tools`, `data.count`, and `data.include_unauthorized`.
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
  a `success/data/error` envelope.

Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
`requires_auth`. Execution looks up the current discovery cache, refreshes once
if the tool is missing, and applies an authorization gate and argument
validation before forwarding. Caller `Authorization` and
`x-demo-session` can be forwarded; configured service credentials and
login-by-phone provide additional server paths. Display `error.message` to the
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.

The discovery cache defaults to 300 seconds server-side
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
also keeps a per-user catalog cache. ApiPilot configuration is independent of
`S2S_INJECT_PROFILE`.

## Marker-based two-pass reasoning

The browser and proxy add an ApiPilot catalog as **instruction text**, including
tool names, descriptions, authentication requirements and required parameters.
The model is instructed to emit a call marker when it chooses a matching tool:

```text
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
```

Tool names are exact identifiers and must not be translated. The local realtime
proxy accumulates assistant text, detects the marker, validates and executes
the ApiPilot call, then injects the result as context and requests a new
response. The HTTP reasoning route has analogous discovery/execution and
second-answer handling. Internal marker text and its speculative audio are
filtered from user output where those handlers run.

This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
discovery is not the same as declaring a native `session.tools` array. Direct
browser connections bypass the local relay's marker executor even if the
browser includes catalog instructions.

## Retired documentation assumptions

The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
Those APIs/rankers are absent from the current client/server path. The retained
agent-profile JSON is an illustrative historical/example shape. Add or modify
current ApiPilot tool definitions in the provider; do not restore obsolete
endpoints solely to match older documentation.

## Verify a tool turn

1. Confirm `/api/config` reports the intended feature and connection mode.
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
   separate evidence.
3. For native calls, find `response.function_call_arguments.done`, then the
   browser `toolcall` and executor request.
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
5. Confirm the tool result reaches the model and follow-up text/audio reaches
   the browser. A spoken acknowledgement alone does not complete the flow.