feat(vocero): add linked content and TALKFIRST greetings (#2)
Opening a report or podcast link now establishes an authenticated starting reference for the conversation. Users can select other accessible content or compare reports while retaining the original link, and TALKFIRST greetings can name the verified report or podcast. Release notes: - Added authenticated report and episode link resolution with bounded lookups and unavailable-content handling - Added explicit conversation selection without restricting searches or changing the reference during comparisons - Added contextual TALKFIRST greetings through both voice-proxy and HTTP reasoning paths - Preserved backend permission checks, conversation isolation, cancellation, and the existing speech protocol Reviewed-on: AquaPortal/s2s#2
This commit is contained in:
@@ -123,11 +123,20 @@ checks reject stale HTTP results and speech callbacks even if cancellation is la
|
||||
With MCP enabled, `services/conversations.py` owns bounded server-side caller
|
||||
credentials. `providers/mcp.py` owns the official SDK transport, while
|
||||
`services/mcp_tools.py` validates selections and enforces per-turn limits.
|
||||
`services/content_context.py` resolves linked metadata and implements the local
|
||||
`select_content` action. Each conversation retains separate original-link and
|
||||
current-selection references; reading or comparing reports does not change the
|
||||
selection. The current podcast is passed as an MCP request header, with every
|
||||
retrieval still subject to backend authorization. TALKFIRST uses the verified
|
||||
title in the opening instructions after link resolution completes.
|
||||
`services/mcp_reasoning.py` and `realtime/mcp_proxy.py` share that executor for
|
||||
HTTP and voice reasoning respectively. The browser stores only its conversation
|
||||
ID in memory and uses an HttpOnly owner cookie. Legacy public content preloads
|
||||
are disabled in this mode; the AquaPortal backend remains the authority for
|
||||
content access. The speech pipeline and its audio protocol are unchanged.
|
||||
In OPENAI mode, MCP TALKFIRST greetings enter the HTTP reasoning path after the
|
||||
speech connection is ready, and use the existing external-answer playback and
|
||||
cancellation lifecycle.
|
||||
|
||||
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
|
||||
[tool calling](tool-calling.md) for the conditions and limitations of each path.
|
||||
|
||||
@@ -7,6 +7,19 @@ in the server's [context services](../../vocero-api/vocero_api/services/context.
|
||||
[context.ts](../../vocero-frontend/src/lib/context.ts), and the
|
||||
[WebSocket client](../../vocero-frontend/src/realtime/client.ts).
|
||||
|
||||
## Authenticated MCP context
|
||||
|
||||
When `S2S_MCP_URL` is configured, the browser submits the page's `report_id` and
|
||||
`episode_id` during conversation creation. The server resolves them through the
|
||||
authenticated MCP client and keeps compact verified references separately from
|
||||
the current conversational selection. No legacy profile or episode preload is
|
||||
used. TALKFIRST can mention the verified title after this preparation finishes.
|
||||
The link never restricts searches or access to other permitted content.
|
||||
|
||||
See [tool calling](tool-calling.md#authenticated-aquaportal-mcp) for selection,
|
||||
authorization, timeout, and missing-content behavior. The controls and public
|
||||
episode fetch below describe the legacy path used when MCP is disabled.
|
||||
|
||||
## Independent controls
|
||||
|
||||
`S2S_INJECT_PROFILE` defaults to false and gates server loading of caller,
|
||||
|
||||
@@ -16,7 +16,7 @@ fallback on this path. AquaPortal checks current permissions on every request,
|
||||
including its administrator and assigned report-manager access rules.
|
||||
|
||||
The backend phone login must return `sessionToken` plus `user.id` and `user.name`.
|
||||
The only allowed tool names are:
|
||||
The allowed backend tool names are:
|
||||
|
||||
| Tool | Content |
|
||||
|---|---|
|
||||
@@ -64,11 +64,47 @@ Generation checks reject late results even if the transport does not stop in
|
||||
time. The browser receives `vocero.tool.started` / `vocero.tool.failed` status
|
||||
events, without backend credentials or raw tool arguments.
|
||||
|
||||
Automatic current-episode context from `episode_id` is not wired yet. The model
|
||||
must resolve a podcast with the catalog rather than fetching an unauthenticated
|
||||
episode URL. See [usage](usage.md#authenticated-aquaportal-mcp) for configuration,
|
||||
conversation endpoints, single-worker constraints, and the retained unsafe
|
||||
phone selector.
|
||||
Conversation creation accepts optional `report_id` and `episode_id` selectors.
|
||||
The server resolves them using the authenticated MCP client, retaining only
|
||||
verified IDs, titles, dates, and podcast availability. Report podcasts use
|
||||
`report:<full-report-id>`; daily episodes use `daily:<episode-id>`. When both
|
||||
selectors are present, the explicit daily episode determines the podcast and
|
||||
the report remains a separate reference. A missing daily episode never falls
|
||||
back to the report podcast. Link resolution has a ten-second total budget;
|
||||
unavailable content permits a generic conversation, while invalid authentication
|
||||
still prevents startup. No public episode endpoint is used.
|
||||
|
||||
The original link and current selection are separate, immutable metadata
|
||||
objects owned by each conversation. The model uses the current selection for
|
||||
"this report/episode" and the original link for "the report from the link".
|
||||
`get_podcast` and `read_podcast_transcript` receive the current podcast through
|
||||
`X-Vocero-Podcast-Id`; explicit `podcast_id` arguments override the header.
|
||||
Every retrieval still checks backend permissions. Search and comparison calls
|
||||
receive only their requested filters and do not change the current selection.
|
||||
|
||||
Vocero adds one local tool, `select_content`, to the discovered catalog:
|
||||
|
||||
```json
|
||||
{"request":{"kind":"report","id":"report-example"}}
|
||||
```
|
||||
|
||||
`kind` is `report` or `podcast`; podcast selections use the prefixed ID. This
|
||||
action performs an authorized lookup before replacing the current selection.
|
||||
It preserves the original link and changes no backend data. Choosing a report
|
||||
without a podcast or a daily podcast clears the previous, unrelated reference.
|
||||
Failed, ambiguous, or cancelled lookups do not replace the selection. Reading
|
||||
another report for comparison alone never invokes this action. Both reasoning
|
||||
paths refresh their instructions after a successful selection.
|
||||
|
||||
TALKFIRST opening instructions mention the linked report's verified title, or
|
||||
the linked podcast's title when no report is available. They invite questions
|
||||
about this or other accessible content without an unsolicited summary. Missing
|
||||
content gets a brief explanation without exposing an inaccessible title. The
|
||||
opening instruction is consumed after the first completed answer. TALK and
|
||||
manual startup retain their existing greeting behavior.
|
||||
|
||||
See [usage](usage.md#authenticated-aquaportal-mcp) for configuration, conversation
|
||||
endpoints, single-worker constraints, and the retained unsafe phone selector.
|
||||
|
||||
## Native realtime function calls
|
||||
|
||||
|
||||
@@ -66,6 +66,16 @@ code before claiming a root cause is resolved.
|
||||
|
||||
## The assistant misses caller or report context
|
||||
|
||||
With MCP enabled, check that `/api/conversations` receives `report_id` or
|
||||
`episode_id` from the page and completes before voice startup. Report IDs retain
|
||||
their full prefix: `report-example` selects podcast `report:report-example`.
|
||||
Link resolution uses the same user's backend session and has a ten-second
|
||||
budget. An inaccessible or unavailable link produces a generic conversation,
|
||||
not a fallback to the latest report. Login or catalog failures still block
|
||||
startup. A report may exist without a podcast. To change the conversational
|
||||
subject, the agent uses `select_content`; search and comparison calls do not
|
||||
change it. MCP ignores the legacy injection flags below.
|
||||
|
||||
- Verify the page has an encoded `phoneNumber` or the intended `episode_id`.
|
||||
- Inspect `/api/user/phone` and context-fetch logs for HTTP/JSON failures.
|
||||
- Confirm profile injection is enabled; a visible profile alone does not prove
|
||||
|
||||
+40
-11
@@ -131,13 +131,14 @@ not returned to the browser. Dynamic ApiPilot tools are configured through
|
||||
|
||||
Operator RAG text is applied through the RAG dialog and updates live
|
||||
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
|
||||
include `episode_id=...`. See [context injection](context-injection.md) for
|
||||
include `episode_id=...`; MCP report links may include `report_id=...`.
|
||||
See [context injection](context-injection.md) for
|
||||
enabled data sources and limits.
|
||||
|
||||
### Authenticated AquaPortal MCP
|
||||
|
||||
`S2S_MCP_URL` enables the MCP client transport. Set it to the backend's complete
|
||||
Streamable HTTP endpoint, for example `https://backend.example.test/api/v1/mcp`.
|
||||
Streamable HTTP endpoint, for example `https://backend.example.test/api/v1/mcp/`.
|
||||
Set `S2S_LOGIN_BY_PHONE` to that backend's `POST /api/v1/auth/phone-login` URL.
|
||||
The phone parameter remains an identity selector without proof of ownership.
|
||||
MCP uses the returned opaque `sessionToken`, never an ApiPilot API key or service token.
|
||||
@@ -165,7 +166,8 @@ and caller HTTP routes return 409 while MCP is enabled.
|
||||
| `S2S_CONVERSATION_TTL_SEC` | `3600` | Absolute conversation lifetime, clamped to 60–86400 seconds |
|
||||
| `S2S_MAX_CONVERSATIONS` | `1000` | Process-local conversation capacity, clamped to 1–10000 |
|
||||
|
||||
`POST /api/conversations` accepts `{"phoneNumber":"+56000000000"}` and returns
|
||||
`POST /api/conversations` accepts `{"phoneNumber":"+56000000000"}` with optional
|
||||
`report_id` and `episode_id` fields, and returns
|
||||
`conversationId`, `expiresIn`, and a minimal caller profile. An HttpOnly,
|
||||
SameSite=Strict cookie binds the browser to its conversations; HTTPS sets Secure.
|
||||
Backend credentials stay in memory on the API server. HTTP conversation requests
|
||||
@@ -191,12 +193,29 @@ reasoning retains up to six completed turns, bounded to 96,000 characters,
|
||||
within that conversation; the speech service keeps its existing conversation
|
||||
history. These histories are cleared when their sessions end.
|
||||
|
||||
The browser waits for successful phone login and discovery before starting an
|
||||
MCP conversation. Failure blocks startup and displays a retryable error. Phone
|
||||
and `episode_id` page parameters remain supported, but automatic selection of
|
||||
the episode from the link is deferred: resolve it explicitly with the podcast
|
||||
tools for now. Changing browser-supplied profile, phone, or episode values on a
|
||||
reasoning request cannot change the stored backend identity.
|
||||
The browser waits for phone login, linked-content resolution, and tool discovery
|
||||
before starting an MCP conversation. Login or discovery failure blocks startup
|
||||
and displays a retryable error. Link resolution is limited to ten seconds and
|
||||
uses only authenticated MCP tools. Missing, inaccessible, or temporarily
|
||||
unavailable linked content permits a generic conversation; an expired backend
|
||||
session still prevents startup. The browser allows thirty seconds for login
|
||||
and link resolution together.
|
||||
|
||||
For example, `?phoneNumber=%2B56000000000&report_id=report-example` selects the
|
||||
report and its podcast, `report:report-example`, when available. The full report
|
||||
ID is preserved. An `episode_id` selects a daily podcast, `daily:<episode_id>`.
|
||||
When both are present, the explicit daily episode determines the podcast and
|
||||
`report_id` remains a separate report reference. No unavailable episode is
|
||||
silently replaced with a different one.
|
||||
|
||||
The link provides the initial subject, not a search filter or an access grant.
|
||||
Users can ask about any accessible report, compare multiple reports, or choose
|
||||
another subject. The local `select_content` action validates that choice and
|
||||
updates the current reference while preserving the original link for follow-ups.
|
||||
Merely searching or reading content does not change the selection. A new
|
||||
conversation, including after restart, resolves the page's selectors again.
|
||||
Changing browser-supplied profile, phone, report, or episode values on a
|
||||
reasoning request cannot change the stored identity or selection.
|
||||
|
||||
## Theme and session startup
|
||||
|
||||
@@ -213,7 +232,7 @@ gets an eight-second loading window before the interface allows a generic sessio
|
||||
|---|---|
|
||||
| Empty/default | User taps to start |
|
||||
| `TALK` | Attempt automatic start and inject the Spanish wake phrase as soon as configuration and caller context are ready |
|
||||
| `TALKFIRST` | Same automatic startup behavior as `TALK` |
|
||||
| `TALKFIRST` | Same startup trigger as `TALK`; with MCP, greet using the linked report or podcast title after authenticated resolution |
|
||||
|
||||
Automatic startup runs once per page load, without a startup dialog or artificial
|
||||
delay; optional reports and tools can still be loading. Browser permission and
|
||||
@@ -221,7 +240,17 @@ audio policies can still require a gesture. When audio is blocked, an inline hin
|
||||
asks the user to tap the microphone button. Permission or connection failures
|
||||
show the error and leave the same button available to retry. Stopping a session
|
||||
does not start it again automatically.
|
||||
The greeting uses the same `hola vocero` input as wake-word handling.
|
||||
The greeting uses the same `hola vocero` input as wake-word handling. With MCP
|
||||
and TALKFIRST, the greeting names the linked report, or the podcast when no
|
||||
report is available, and invites questions about this or other reports. It does
|
||||
not summarize automatically. A missing reference gets a brief explanation;
|
||||
links without a content selector receive a generic greeting.
|
||||
|
||||
In `PROXY`, the relay supplies the verified metadata in the initial speech
|
||||
instructions. In `OPENAI`, the controller sends the opening input through the
|
||||
authenticated HTTP reasoning path after the speech connection is ready, then
|
||||
uses the existing external-answer playback. Speaking first or stopping the
|
||||
session cancels a pending greeting. Neither path changes the speech service.
|
||||
|
||||
On browsers with speech recognition, **Activar con «hola vocero»** opts into
|
||||
wake-word listening for the current page session. The interface shows when the
|
||||
|
||||
Reference in New Issue
Block a user