vocero-client/docs/client/tool-calling.md

Tool calling

The application supports native realtime function calling and server-side tool selection from model text. AquaPortal content uses authenticated MCP when S2S_MCP_URL is set. Without MCP, the native browser tools remain available. A spoken promise to fetch something is not evidence that a tool executed.

Authenticated AquaPortal MCP

Vocero uses the official MCP Python SDK with Streamable HTTP. Each discovery or execution initializes a client with that conversation's opaque backend token as Authorization: Bearer .... JSON and SSE responses are supported. Neither browser values nor model arguments choose the authenticated user. There is no service-token override, anonymous retry, redirect following, or public REST fallback on this path. AquaPortal checks current permissions on every request, including its administrator and assigned report-manager access rules.

The backend phone login must return sessionToken plus user.id and user.name. The allowed backend tool names are:

Tool	Content
list_reports	Accessible published reports with name, center, and date filters
get_report	One report or an explicit list of ambiguous selections
list_insights	Published findings, priorities, recommendations, and sources
get_metric_history	Dated metric values, units, centers, cages, and source reports
list_podcasts	Daily and report-associated episodes
get_podcast	Selected episode summary, highlights, and available media
read_podcast_transcript	Paginated transcript or report-script passages
get_my_briefing	Compact overview with actual source dates

Discovery preserves the complete input schema, including nested request, optional filters, references, and pagination. The executor validates arguments against that schema and supplies structuredContent to reasoning when present, falling back to MCP text content. It preserves get_report's result union and list next_offset values. Tool errors are data for the model to explain; expired authentication stops the conversation. Tool text is labeled as reference data, never as instructions or authorization.

Both reasoning paths can discover reports, retrieve details, and follow up with another tool before answering. The HTTP loop retains bounded completed-turn history. The realtime relay injects results into the speech service's existing conversation. Attempts are bounded per user turn (six by default), identical calls within a turn are rejected, and tool results are limited to 64,000 characters individually and 96,000 per turn. Each MCP operation has a timeout; HTTP reasoning also has a 180-second total timeout.

Selection uses an internal text marker, not a new speech protocol event:

<<<VOCERO_MCP_CALL:NAME=get_report,ARGUMENTS={"request":{"report_id":"example-id"}}>>>

The distinct marker is necessary because the current speech pipeline removes APIPILOT_CALL before emitting assistant text. The relay buffers undecided text/audio, handles transcripts emitted before response.created, executes the MCP selection, and requests another response. Normal speech continues using the existing PCM16 protocol. This leaves speech_to_speech/ unchanged, although the speech service may spend time synthesizing audio that the relay suppresses. Malformed selections fail with a user-facing error rather than permissive parsing.

New speech, cancellation, disconnect, and expiry invalidate pending work. Generation checks reject late results even if the transport does not stop in time. The browser receives vocero.tool.started / vocero.tool.failed status events, without backend credentials or raw tool arguments.

Conversation creation accepts optional report_id and episode_id selectors. The server resolves them using the authenticated MCP client, retaining only verified IDs, titles, dates, and podcast availability. Report podcasts use report:<full-report-id>; daily episodes use daily:<episode-id>. When both selectors are present, the explicit daily episode determines the podcast and the report remains a separate reference. A missing daily episode never falls back to the report podcast. Link resolution has a ten-second total budget; unavailable content permits a generic conversation, while invalid authentication still prevents startup. No public episode endpoint is used.

The original link and current selection are separate, immutable metadata objects owned by each conversation. The model uses the current selection for "this report/episode" and the original link for "the report from the link". get_podcast and read_podcast_transcript receive the current podcast through X-Vocero-Podcast-Id; explicit podcast_id arguments override the header. Every retrieval still checks backend permissions. Search and comparison calls receive only their requested filters and do not change the current selection.

Vocero adds one local tool, select_content, to the discovered catalog:

{"request":{"kind":"report","id":"report-example"}}

kind is report or podcast; podcast selections use the prefixed ID. This action performs an authorized lookup before replacing the current selection. It preserves the original link and changes no backend data. Choosing a report without a podcast or a daily podcast clears the previous, unrelated reference. Failed, ambiguous, or cancelled lookups do not replace the selection. Reading another report for comparison alone never invokes this action. Both reasoning paths refresh their instructions after a successful selection.

TALKFIRST opening instructions mention the linked report's verified title, or the linked podcast's title when no report is available. They invite questions about this or other accessible content without an unsolicited summary. Missing content gets a brief explanation without exposing an inaccessible title. The opening instruction is consumed after the first completed answer. TALK and manual startup retain their existing greeting behavior.

See usage for configuration, conversation endpoints, single-worker constraints, and the retained unsafe phone selector.

Native realtime function calls

The browser sends tool declarations inside the session object:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "tools": [
      {
        "type": "function",
        "name": "web_search",
        "description": "Search the web for information.",
        "parameters": {
          "type": "object",
          "properties": {"query": {"type": "string"}},
          "required": ["query"]
        }
      }
    ],
    "tool_choice": "auto"
  }
}

Keep configuration inside session, not at the event root. The backend has a legacy flat-payload compatibility path, but new clients should use the nested shape. The current browser's activeToolDefs() declares web_search when enabled and a deploy/user search key is available. It does not load the old remote agent-profile MCP catalog into native session tools.

For a native call, the backend emits:

{
  "type": "response.function_call_arguments.done",
  "call_id": "call_example",
  "name": "web_search",
  "arguments": "{\"query\":\"example query\"}"
}

arguments is a JSON-encoded string. The WebSocket client validates the event and dispatches tool; controller.ts parses the arguments and runs the supported executor. Web search uses POST /api/search, keeping the deployment's Serper key on the server. Tool output returns to the model as:

{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_example",
    "output": "Tool result text"
  }
}

The client requests follow-up generation with response.create when the result should be spoken. A function_call_output item alone does not start generation. See the realtime API guide for local model <code> extraction and native remote-provider tool calls.

Retired documentation assumptions

ApiPilot discovery/execution endpoints, catalog injection, credential overrides, and two-pass marker handling have been removed from Vocero. Its former HTTP routes return 404 regardless of MCP configuration. The separate speech service still has its existing legacy marker sanitizers; no ApiPilot requests can be made through Vocero. See usage for deployment configuration cleanup.

The former HOWITWORKS.md described /api/mcp/tool, remote S2S_AGENT_PROFILE loading, and Spanish/Italian-to-English client ranking. Those APIs/rankers are absent from the current client/server path. The retained agent-profile JSON is an illustrative historical/example shape. Content tool definitions and permissions belong to the AquaPortal MCP backend; Vocero's provider enforces its allowlist. Do not restore obsolete endpoints solely to match older documentation.

Verify a tool turn

1. Confirm /api/config reports the intended feature and connection mode.
2. For MCP, confirm conversation creation and authenticated /api/mcp/tools discovery succeed. Inspect the relay's session.update for MCP instructions; these are separate from native session.tools declarations.
3. For native calls, find response.function_call_arguments.done, then the browser toolcall and executor request.
4. For MCP, find mcp_tool_completed for the expected user/tool and any vocero.tool.started / vocero.tool.failed voice events.
5. Confirm the tool result reaches the model and follow-up text/audio reaches the browser. A spoken acknowledgement alone does not complete the flow.