feat(mcp): connect authenticated AquaPortal tools to Vocero (#1)
Vocero needs to query AquaPortal on behalf of the caller selected by the existing phone link. This adds server-owned conversations and MCP tool execution for voice and HTTP reasoning while leaving the speech pipeline unchanged. Release notes: - Added browser-bound conversations that keep backend credentials server-side and isolate callers. - Connected the eight-tool AquaPortal catalog through authenticated Streamable HTTP with full input schemas and structured results. - Added bounded multi-step tool execution with cancellation, expiry, and stale-result protection. - Updated browser bootstrap, tool progress, reasoning headers, and conversation cleanup. - Disabled legacy unscoped content preloads in MCP mode and documented configuration and deployment constraints. - Preserved disabled-mode contracts and added local SDK, WebSocket, and browser regression coverage. Reviewed-on: AquaPortal/s2s#1
This commit is contained in:
@@ -68,6 +68,10 @@ audio. It is not an unconditional byte-for-byte tunnel.
|
||||
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
|
||||
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
|
||||
| `POST /api/reasoning` | OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
|
||||
| `POST /api/conversations` | MCP-mode phone login and browser-owned conversation creation |
|
||||
| `DELETE /api/conversations/current` | Release the selected MCP conversation |
|
||||
| `POST /api/conversations/current/interrupt` | Cancel its matching HTTP reasoning turn |
|
||||
| `GET /api/mcp/tools` | Discover the selected conversation's authenticated MCP catalog |
|
||||
| `GET /api/user/phone` | Configured caller profile lookup |
|
||||
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
|
||||
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
|
||||
@@ -116,5 +120,14 @@ external-answer path. These HTTP operations supplement the realtime audio path.
|
||||
New user speech and session teardown cancel pending reasoning. Turn identity
|
||||
checks reject stale HTTP results and speech callbacks even if cancellation is late.
|
||||
|
||||
With MCP enabled, `services/conversations.py` owns bounded server-side caller
|
||||
credentials. `providers/mcp.py` owns the official SDK transport, while
|
||||
`services/mcp_tools.py` validates selections and enforces per-turn limits.
|
||||
`services/mcp_reasoning.py` and `realtime/mcp_proxy.py` share that executor for
|
||||
HTTP and voice reasoning respectively. The browser stores only its conversation
|
||||
ID in memory and uses an HttpOnly owner cookie. Legacy public content preloads
|
||||
are disabled in this mode; the AquaPortal backend remains the authority for
|
||||
content access. The speech pipeline and its audio protocol are unchanged.
|
||||
|
||||
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
|
||||
[tool calling](tool-calling.md) for the conditions and limitations of each path.
|
||||
|
||||
@@ -1,8 +1,74 @@
|
||||
# Tool calling
|
||||
|
||||
The application has two tool mechanisms: native realtime function calling and
|
||||
ApiPilot calls detected from model text. A spoken promise to fetch something is
|
||||
not evidence that either mechanism executed a tool.
|
||||
The application supports native realtime function calling and server-side
|
||||
tool selection from model text. AquaPortal content uses authenticated MCP when
|
||||
`S2S_MCP_URL` is set; otherwise the legacy ApiPilot bridge remains available.
|
||||
A spoken promise to fetch something is not evidence that a tool executed.
|
||||
|
||||
## Authenticated AquaPortal MCP
|
||||
|
||||
Vocero uses the official MCP Python SDK with Streamable HTTP. Each discovery
|
||||
or execution initializes a client with that conversation's opaque backend token
|
||||
as `Authorization: Bearer ...`. JSON and SSE responses are supported. Neither
|
||||
browser values nor model arguments choose the authenticated user. There is no
|
||||
service-token override, anonymous retry, redirect following, or public REST
|
||||
fallback on this path. AquaPortal checks current permissions on every request,
|
||||
including its administrator and assigned report-manager access rules.
|
||||
|
||||
The backend phone login must return `sessionToken` plus `user.id` and `user.name`.
|
||||
The only allowed tool names are:
|
||||
|
||||
| Tool | Content |
|
||||
|---|---|
|
||||
| `list_reports` | Accessible published reports with name, center, and date filters |
|
||||
| `get_report` | One report or an explicit list of ambiguous selections |
|
||||
| `list_insights` | Published findings, priorities, recommendations, and sources |
|
||||
| `get_metric_history` | Dated metric values, units, centers, cages, and source reports |
|
||||
| `list_podcasts` | Daily and report-associated episodes |
|
||||
| `get_podcast` | Selected episode summary, highlights, and available media |
|
||||
| `read_podcast_transcript` | Paginated transcript or report-script passages |
|
||||
| `get_my_briefing` | Compact overview with actual source dates |
|
||||
|
||||
Discovery preserves the complete input schema, including nested `request`,
|
||||
optional filters, references, and pagination. The executor validates arguments
|
||||
against that schema and supplies `structuredContent` to reasoning when present,
|
||||
falling back to MCP text content. It preserves `get_report`'s `result` union and
|
||||
list `next_offset` values. Tool errors are data for the model to explain; expired
|
||||
authentication stops the conversation. Tool text is labeled as reference data,
|
||||
never as instructions or authorization.
|
||||
|
||||
Both reasoning paths can discover reports, retrieve details, and follow up with
|
||||
another tool before answering. The HTTP loop retains bounded completed-turn
|
||||
history. The realtime relay injects results into the speech service's existing
|
||||
conversation. Attempts are bounded per user turn (six by default), identical
|
||||
calls within a turn are rejected, and tool results are limited to 64,000
|
||||
characters individually and 96,000 per turn. Each MCP operation has a timeout;
|
||||
HTTP reasoning also has a 180-second total timeout.
|
||||
|
||||
Selection uses an internal text marker, not a new speech protocol event:
|
||||
|
||||
```text
|
||||
<<<VOCERO_MCP_CALL:NAME=get_report,ARGUMENTS={"request":{"report_id":"example-id"}}>>>
|
||||
```
|
||||
|
||||
The distinct marker is necessary because the current speech pipeline removes
|
||||
`APIPILOT_CALL` before emitting assistant text. The relay buffers undecided
|
||||
text/audio, handles transcripts emitted before `response.created`, executes
|
||||
the MCP selection, and requests another response. Normal speech continues using
|
||||
the existing PCM16 protocol. This leaves `speech_to_speech/` unchanged, although
|
||||
the speech service may spend time synthesizing audio that the relay suppresses.
|
||||
Malformed selections fail with a user-facing error rather than permissive parsing.
|
||||
|
||||
New speech, cancellation, disconnect, and expiry invalidate pending work.
|
||||
Generation checks reject late results even if the transport does not stop in
|
||||
time. The browser receives `vocero.tool.started` / `vocero.tool.failed` status
|
||||
events, without backend credentials or raw tool arguments.
|
||||
|
||||
Automatic current-episode context from `episode_id` is not wired yet. The model
|
||||
must resolve a podcast with the catalog rather than fetching an unauthenticated
|
||||
episode URL. See [usage](usage.md#authenticated-aquaportal-mcp) for configuration,
|
||||
conversation endpoints, single-worker constraints, and the retained unsafe
|
||||
phone selector.
|
||||
|
||||
## Native realtime function calls
|
||||
|
||||
@@ -72,7 +138,7 @@ model `<code>` extraction and native remote-provider tool calls.
|
||||
|
||||
## ApiPilot discovery and execution
|
||||
|
||||
`S2S_APIPILOT_TOOLS_URL` enables the current dynamic-tool bridge. The backend
|
||||
When MCP is disabled, `S2S_APIPILOT_TOOLS_URL` enables the legacy dynamic-tool bridge. The backend
|
||||
discovers tools through its `/tools` route and executes through `/execute`.
|
||||
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
|
||||
server exposes:
|
||||
|
||||
+66
-2
@@ -134,14 +134,78 @@ instructions. Caller links may include `phoneNumber=%2B...`; episode links may
|
||||
include `episode_id=...`. See [context injection](context-injection.md) for
|
||||
enabled data sources and limits.
|
||||
|
||||
### Authenticated AquaPortal MCP
|
||||
|
||||
`S2S_MCP_URL` enables the MCP client transport. Set it to the backend's complete
|
||||
Streamable HTTP endpoint, for example `https://backend.example.test/api/v1/mcp`.
|
||||
Set `S2S_LOGIN_BY_PHONE` to that backend's `POST /api/v1/auth/phone-login` URL.
|
||||
The phone parameter remains an identity selector without proof of ownership.
|
||||
MCP uses the returned opaque `sessionToken`, never an ApiPilot API key or service token.
|
||||
|
||||
For voice reasoning (`S2S_REASONING_TYPE=PROXY`), set `S2S_PROXY_TARGET` to the
|
||||
speech service and leave `SPEECH_TO_SPEECH_URL` and `LOAD_BALANCER_URL` unset.
|
||||
The browser must connect to Vocero's same-origin `/v1/realtime` relay so tool
|
||||
selections reach its executor. An incompatible direct configuration returns
|
||||
503 from `/api/config`. `OPENAI` reasoning uses the authenticated HTTP loop and
|
||||
can retain a direct or load-balanced speech connection. No speech-service code
|
||||
or audio-format change is required.
|
||||
|
||||
The backend must expose the eight-tool catalog and phone-login contract described
|
||||
in [tool calling](tool-calling.md#authenticated-aquaportal-mcp). Leave
|
||||
`S2S_MCP_URL` unset to retain the existing ApiPilot and caller-context paths.
|
||||
Setting it makes MCP the exclusive AquaPortal content path: legacy profile,
|
||||
report, latest-summary, and episode preloads are skipped. Existing ApiPilot
|
||||
and caller HTTP routes return 409 while MCP is enabled.
|
||||
|
||||
| Setting | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `S2S_MCP_URL` | Empty, disabled | Backend MCP endpoint |
|
||||
| `S2S_MCP_TIMEOUT_SEC` | `20` | Entire MCP operation timeout, clamped to 1–120 seconds |
|
||||
| `S2S_MCP_MAX_CALLS_PER_TURN` | `6` | Tool attempts per user turn, clamped to 1–20 |
|
||||
| `S2S_CONVERSATION_TTL_SEC` | `3600` | Absolute conversation lifetime, clamped to 60–86400 seconds |
|
||||
| `S2S_MAX_CONVERSATIONS` | `1000` | Process-local conversation capacity, clamped to 1–10000 |
|
||||
|
||||
`POST /api/conversations` accepts `{"phoneNumber":"+56000000000"}` and returns
|
||||
`conversationId`, `expiresIn`, and a minimal caller profile. An HttpOnly,
|
||||
SameSite=Strict cookie binds the browser to its conversations; HTTPS sets Secure.
|
||||
Backend credentials stay in memory on the API server. HTTP conversation requests
|
||||
use `X-Vocero-Conversation`; the same-origin WebSocket uses `conversation_id`.
|
||||
The conversation ID alone is insufficient without its owning browser cookie.
|
||||
Different tabs can own different caller identities without replacing one another.
|
||||
|
||||
`GET /api/mcp/tools` discovers the authenticated catalog.
|
||||
`POST /api/reasoning` requires an increasing positive `X-Vocero-Turn` header
|
||||
in addition to the conversation header in MCP mode.
|
||||
`POST /api/conversations/current/interrupt` cancels the matching `X-Vocero-Turn`, and
|
||||
`DELETE /api/conversations/current` clears that conversation. These endpoints
|
||||
require the cookie and conversation header. They do not change the existing
|
||||
load-balancer `/api/session` contract. Stopping a conversation drops its local
|
||||
credentials; it does not revoke backend sessions globally.
|
||||
|
||||
Conversation state is bounded, process-local, and cleared on restart. Use one
|
||||
API worker, as in the launcher and Docker command. Multiple workers or replicas
|
||||
require a shared conversation store before enabling this integration. Browser
|
||||
sessions must restart after expiry or an API reload. Backend access is checked
|
||||
on each MCP request. There is no shared content or authorization cache. HTTP
|
||||
reasoning retains up to six completed turns, bounded to 96,000 characters,
|
||||
within that conversation; the speech service keeps its existing conversation
|
||||
history. These histories are cleared when their sessions end.
|
||||
|
||||
The browser waits for successful phone login and discovery before starting an
|
||||
MCP conversation. Failure blocks startup and displays a retryable error. Phone
|
||||
and `episode_id` page parameters remain supported, but automatic selection of
|
||||
the episode from the link is deferred: resolve it explicitly with the podcast
|
||||
tools for now. Changing browser-supplied profile, phone, or episode values on a
|
||||
reasoning request cannot change the stored backend identity.
|
||||
|
||||
## Theme and session startup
|
||||
|
||||
`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
|
||||
the dark aurora theme. Theme-specific sections remain hidden until configuration
|
||||
resolves, with a fallback to avoid an invisible page after config failure.
|
||||
Configuration failures offer **Reintentar** and block session startup until a
|
||||
retry succeeds. Reports load independently. Caller context gets an eight-second
|
||||
loading window before the interface allows a generic session.
|
||||
retry succeeds. With MCP disabled, reports load independently and caller context
|
||||
gets an eight-second loading window before the interface allows a generic session.
|
||||
|
||||
`S2S_START_SESSION_MODE` controls initial microphone/session startup:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user