Tool calling

The application has two tool mechanisms: native realtime function calling and ApiPilot calls detected from model text. A spoken promise to fetch something is not evidence that either mechanism executed a tool.

Native realtime function calls

The browser sends tool declarations inside the session object:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "tools": [
      {
        "type": "function",
        "name": "web_search",
        "description": "Search the web for information.",
        "parameters": {
          "type": "object",
          "properties": {"query": {"type": "string"}},
          "required": ["query"]
        }
      }
    ],
    "tool_choice": "auto"
  }
}

Keep configuration inside session, not at the event root. The backend has a legacy flat-payload compatibility path, but new clients should use the nested shape. The current browser's activeToolDefs() declares web_search when enabled and a deploy/user search key is available. It does not load the old remote agent-profile MCP catalog into native session tools.

For a native call, the backend emits:

{
  "type": "response.function_call_arguments.done",
  "call_id": "call_example",
  "name": "web_search",
  "arguments": "{\"query\":\"example query\"}"
}

arguments is a JSON-encoded string. The WebSocket client validates the event and dispatches tool; controller.ts parses the arguments and runs the supported executor. Web search uses POST /api/search, keeping the deployment's Serper key on the server. Tool output returns to the model as:

{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_example",
    "output": "Tool result text"
  }
}

The client requests follow-up generation with response.create when the result should be spoken. A function_call_output item alone does not start generation. See the realtime API guide for local model <code> extraction and native remote-provider tool calls.

ApiPilot discovery and execution

S2S_APIPILOT_TOOLS_URL enables the current dynamic-tool bridge. The backend discovers tools through its /tools route and executes through /execute. S2S_APIPILOT_TOOLS_EXECUTE_URL can override the execution target. The same-origin UI server exposes:

GET /api/apipilot/tools?include_unauthorized=true: discovery envelope with data.tools, data.count, and data.include_unauthorized.
POST /api/apipilot/execute: body {"name":"tool_name","arguments":{}} and a success/data/error envelope.

Tool definitions contain name, description, JSON Schema parameters, and requires_auth. Execution looks up the current discovery cache, refreshes once if the tool is missing, and applies an authorization gate and argument validation before forwarding. Caller Authorization and x-demo-session can be forwarded; configured service credentials and login-by-phone provide additional server paths. Display error.message to the user; keep error.code, error.status_code, and error.raw for diagnostics.

The discovery cache defaults to 300 seconds server-side (S2S_APIPILOT_TTL_SEC) and can be refreshed with refresh=true. The browser also keeps a per-user catalog cache. ApiPilot configuration is independent of S2S_INJECT_PROFILE.

Marker-based two-pass reasoning

The browser and proxy add an ApiPilot catalog as instruction text, including tool names, descriptions, authentication requirements and required parameters. The model is instructed to emit a call marker when it chooses a matching tool:

<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>

Tool names are exact identifiers and must not be translated. The local realtime proxy accumulates assistant text, detects the marker, validates and executes the ApiPilot call, then injects the result as context and requests a new response. The HTTP reasoning route has analogous discovery/execution and second-answer handling. Internal marker text and its speculative audio are filtered from user output where those handlers run.

This marker is not response.function_call_arguments.done, and ApiPilot prompt discovery is not the same as declaring a native session.tools array. Direct browser connections bypass the local relay's marker executor even if the browser includes catalog instructions.

Retired documentation assumptions

The former HOWITWORKS.md described /api/mcp/tool, remote S2S_AGENT_PROFILE loading, and Spanish/Italian-to-English client ranking. Those APIs/rankers are absent from the current client/server path. The retained agent-profile JSON is an illustrative historical/example shape. Add or modify current ApiPilot tool definitions in the provider; do not restore obsolete endpoints solely to match older documentation.

Verify a tool turn

Confirm /api/config reports the intended feature and connection mode.
Inspect session.update: native tools and ApiPilot instruction text are separate evidence.
For native calls, find response.function_call_arguments.done, then the browser toolcall and executor request.
For ApiPilot, find the discovery, call-marker detection and execution logs.
Confirm the tool result reaches the model and follow-up text/audio reaches the browser. A spoken acknowledgement alone does not complete the flow.

5.4 KiB Raw Blame History