vocero-client/docs/client/tool-calling.md

# Tool calling

The application supports native realtime function calling and server-side
tool selection from model text. AquaPortal content uses authenticated MCP when
`S2S_MCP_URL` is set; otherwise the legacy ApiPilot bridge remains available.
A spoken promise to fetch something is not evidence that a tool executed.

## Authenticated AquaPortal MCP

Vocero uses the official MCP Python SDK with Streamable HTTP. Each discovery
or execution initializes a client with that conversation's opaque backend token
as `Authorization: Bearer ...`. JSON and SSE responses are supported. Neither
browser values nor model arguments choose the authenticated user. There is no
service-token override, anonymous retry, redirect following, or public REST
fallback on this path. AquaPortal checks current permissions on every request,
including its administrator and assigned report-manager access rules.

The backend phone login must return `sessionToken` plus `user.id` and `user.name`.
The allowed backend tool names are:

| Tool | Content |
|---|---|
| `list_reports` | Accessible published reports with name, center, and date filters |
| `get_report` | One report or an explicit list of ambiguous selections |
| `list_insights` | Published findings, priorities, recommendations, and sources |
| `get_metric_history` | Dated metric values, units, centers, cages, and source reports |
| `list_podcasts` | Daily and report-associated episodes |
| `get_podcast` | Selected episode summary, highlights, and available media |
| `read_podcast_transcript` | Paginated transcript or report-script passages |
| `get_my_briefing` | Compact overview with actual source dates |

Discovery preserves the complete input schema, including nested `request`,
optional filters, references, and pagination. The executor validates arguments
against that schema and supplies `structuredContent` to reasoning when present,
falling back to MCP text content. It preserves `get_report`'s `result` union and
list `next_offset` values. Tool errors are data for the model to explain; expired
authentication stops the conversation. Tool text is labeled as reference data,
never as instructions or authorization.

Both reasoning paths can discover reports, retrieve details, and follow up with
another tool before answering. The HTTP loop retains bounded completed-turn
history. The realtime relay injects results into the speech service's existing
conversation. Attempts are bounded per user turn (six by default), identical
calls within a turn are rejected, and tool results are limited to 64,000
characters individually and 96,000 per turn. Each MCP operation has a timeout;
HTTP reasoning also has a 180-second total timeout.

Selection uses an internal text marker, not a new speech protocol event:

```text
<<<VOCERO_MCP_CALL:NAME=get_report,ARGUMENTS={"request":{"report_id":"example-id"}}>>>
```

The distinct marker is necessary because the current speech pipeline removes
`APIPILOT_CALL` before emitting assistant text. The relay buffers undecided
text/audio, handles transcripts emitted before `response.created`, executes
the MCP selection, and requests another response. Normal speech continues using
the existing PCM16 protocol. This leaves `speech_to_speech/` unchanged, although
the speech service may spend time synthesizing audio that the relay suppresses.
Malformed selections fail with a user-facing error rather than permissive parsing.

New speech, cancellation, disconnect, and expiry invalidate pending work.
Generation checks reject late results even if the transport does not stop in
time. The browser receives `vocero.tool.started` / `vocero.tool.failed` status
events, without backend credentials or raw tool arguments.

Conversation creation accepts optional `report_id` and `episode_id` selectors.
The server resolves them using the authenticated MCP client, retaining only
verified IDs, titles, dates, and podcast availability. Report podcasts use
`report:<full-report-id>`; daily episodes use `daily:<episode-id>`. When both
selectors are present, the explicit daily episode determines the podcast and
the report remains a separate reference. A missing daily episode never falls
back to the report podcast. Link resolution has a ten-second total budget;
unavailable content permits a generic conversation, while invalid authentication
still prevents startup. No public episode endpoint is used.

The original link and current selection are separate, immutable metadata
objects owned by each conversation. The model uses the current selection for
"this report/episode" and the original link for "the report from the link".
`get_podcast` and `read_podcast_transcript` receive the current podcast through
`X-Vocero-Podcast-Id`; explicit `podcast_id` arguments override the header.
Every retrieval still checks backend permissions. Search and comparison calls
receive only their requested filters and do not change the current selection.

Vocero adds one local tool, `select_content`, to the discovered catalog:

```json
{"request":{"kind":"report","id":"report-example"}}
```

`kind` is `report` or `podcast`; podcast selections use the prefixed ID. This
action performs an authorized lookup before replacing the current selection.
It preserves the original link and changes no backend data. Choosing a report
without a podcast or a daily podcast clears the previous, unrelated reference.
Failed, ambiguous, or cancelled lookups do not replace the selection. Reading
another report for comparison alone never invokes this action. Both reasoning
paths refresh their instructions after a successful selection.

TALKFIRST opening instructions mention the linked report's verified title, or
the linked podcast's title when no report is available. They invite questions
about this or other accessible content without an unsolicited summary. Missing
content gets a brief explanation without exposing an inaccessible title. The
opening instruction is consumed after the first completed answer. TALK and
manual startup retain their existing greeting behavior.

See [usage](usage.md#authenticated-aquaportal-mcp) for configuration, conversation
endpoints, single-worker constraints, and the retained unsafe phone selector.

## Native realtime function calls

The browser sends tool declarations inside the `session` object:

```json
{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "tools": [
      {
        "type": "function",
        "name": "web_search",
        "description": "Search the web for information.",
        "parameters": {
          "type": "object",
          "properties": {"query": {"type": "string"}},
          "required": ["query"]
        }
      }
    ],
    "tool_choice": "auto"
  }
}
```

Keep configuration inside `session`, not at the event root. The backend has a
legacy flat-payload compatibility path, but new clients should use the nested
shape. The current browser's `activeToolDefs()` declares `web_search` when
enabled and a deploy/user search key is available. It does not load the old
remote agent-profile MCP catalog into native session tools.

For a native call, the backend emits:

```json
{
  "type": "response.function_call_arguments.done",
  "call_id": "call_example",
  "name": "web_search",
  "arguments": "{\"query\":\"example query\"}"
}
```

`arguments` is a JSON-encoded string. The
[WebSocket client](../../vocero-frontend/src/realtime/client.ts) validates the event and
dispatches `tool`; [controller.ts](../../vocero-frontend/src/realtime/controller.ts)
parses the arguments and runs the supported executor.
Web search uses `POST /api/search`, keeping the deployment's Serper key on the
server. Tool output returns to the model as:

```json
{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_example",
    "output": "Tool result text"
  }
}
```

The client requests follow-up generation with `response.create` when the
result should be spoken. A `function_call_output` item alone does not start
generation. See the [realtime API guide](../speech/realtime-api.md) for local
model `<code>` extraction and native remote-provider tool calls.

## ApiPilot discovery and execution

When MCP is disabled, `S2S_APIPILOT_TOOLS_URL` enables the legacy dynamic-tool bridge. The backend
discovers tools through its `/tools` route and executes through `/execute`.
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
server exposes:

- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
  `data.tools`, `data.count`, and `data.include_unauthorized`.
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
  a `success/data/error` envelope.

Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
`requires_auth`. Execution looks up the current discovery cache, refreshes once
if the tool is missing, and applies an authorization gate and argument
validation before forwarding. Caller `Authorization` and
`x-demo-session` can be forwarded; configured service credentials and
login-by-phone provide additional server paths. Display `error.message` to the
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.

The discovery cache defaults to 300 seconds server-side
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
also keeps a per-user catalog cache. ApiPilot configuration is independent of
`S2S_INJECT_PROFILE`.

## Marker-based two-pass reasoning

The browser and proxy add an ApiPilot catalog as **instruction text**, including
tool names, descriptions, authentication requirements and required parameters.
The model is instructed to emit a call marker when it chooses a matching tool:

```text
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
```

Tool names are exact identifiers and must not be translated. The local realtime
proxy accumulates assistant text, detects the marker, validates and executes
the ApiPilot call, then injects the result as context and requests a new
response. The HTTP reasoning route has analogous discovery/execution and
second-answer handling. Internal marker text and its speculative audio are
filtered from user output where those handlers run.

This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
discovery is not the same as declaring a native `session.tools` array. Direct
browser connections bypass the local relay's marker executor even if the
browser includes catalog instructions.

## Retired documentation assumptions

The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
Those APIs/rankers are absent from the current client/server path. The retained
agent-profile JSON is an illustrative historical/example shape. Add or modify
current ApiPilot tool definitions in the provider; do not restore obsolete
endpoints solely to match older documentation.

## Verify a tool turn

1. Confirm `/api/config` reports the intended feature and connection mode.
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
   separate evidence.
3. For native calls, find `response.function_call_arguments.done`, then the
   browser `toolcall` and executor request.
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
5. Confirm the tool result reaches the model and follow-up text/audio reaches
   the browser. A spoken acknowledgement alone does not complete the flow.