valenti/vocero-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(vocero): remove the legacy ApiPilot integration (#3)

Browse Source 
Vocero now retrieves AquaPortal content through authenticated MCP. This removes the obsolete ApiPilot integration from the API and browser while preserving the separately deployed speech pipeline.

Release notes:

- Removed ApiPilot endpoints, configuration, credential overrides, and HTTP/realtime tool execution.
- Removed browser ApiPilot catalog loading and prompt generation while retaining authenticated MCP discovery.
- Updated regression coverage and operating documentation for the retired routes and settings.

Reviewed-on: AquaPortal/s2s#3
This commit is contained in:
alvaro
2026-10-08 11:23:41 -04:00
parent 0311d3f1ff
commit dfed03ac7c
39 changed files with 198 additions and 3314 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/client/architecture.md
+5 -7
View File
@@ -15,14 +15,14 @@ flowchart LR
Server[Client FastAPI server]
Speech[Realtime speech backend]
Reasoning[OpenAI-compatible HTTP reasoning provider]
Tools[ApiPilot backend]
Tools[AquaPortal MCP backend]
Browser <-->|Frontend and HMR| Vite
Vite -->|HTTP configuration and proxies| Server
Vite <-->|Speech WebSocket relay| Server
Server <-->|Realtime events and audio| Speech
Browser <-->|Optional direct connection| Speech
Server -->|Optional transcript reasoning| Reasoning
Server -->|Tool discovery and execution| Tools
Server -->|Authenticated tool discovery and execution| Tools
```
## Connection modes
@@ -34,8 +34,8 @@ session/queue proxy, or the user can enter a direct URL in settings. See
[usage](usage.md) for the configuration matrix.
The local realtime proxy relays audio and events in both directions. It also
patches selected `session.update` instructions, handles ApiPilot marker-based
tool execution, filters internal text, and can suppress speculative/tool-marker
patches selected `session.update` instructions, handles authenticated MCP
tool execution when enabled, filters internal text, and suppresses MCP tool-marker
audio. It is not an unconditional byte-for-byte tunnel.
## Source map
@@ -47,7 +47,7 @@ audio. It is not an unconditional byte-for-byte tunnel.
| [api](../../vocero-api/vocero_api/api) | HTTP route registration, handlers and compatible response validation |
| [models](../../vocero-api/vocero_api/models) | Pydantic request and response contracts |
| [core](../../vocero-api/vocero_api/core) | Existing environment parsing, logging, URL helpers and diagnostics |
| [providers](../../vocero-api/vocero_api/providers) | Caller context and ApiPilot HTTP integrations |
| [providers](../../vocero-api/vocero_api/providers) | Authenticated MCP transport, backend login, and caller HTTP integrations |
| [services](../../vocero-api/vocero_api/services) | Context construction, reasoning and tool-context rendering |
| [realtime](../../vocero-api/vocero_api/realtime) | WebSocket relay, events, markers and audio handling |
| [start_ui.sh](../../start_ui.sh) | Dependency setup, ports, readiness, reload processes, and shutdown |
@@ -74,8 +74,6 @@ audio. It is not an unconditional byte-for-byte tunnel.
| `GET /api/mcp/tools` | Discover the selected conversation's authenticated MCP catalog |
| `GET /api/user/phone` | Configured caller profile lookup |
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
| `POST /api/search` | Serper web search proxy |
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
docs/client/context-injection.md
+9 -10
View File
@@ -27,10 +27,10 @@ episode, report and prior-conversation context. It also controls whether the
browser appends caller profile text to its realtime instructions.
It is not a master switch for every context source. Operator-supplied RAG text
and configured ApiPilot discovery work independently. `S2S_INJECT_TODAY`
and authenticated MCP orchestration work independently. `S2S_INJECT_TODAY`
defaults to true; date and voice-output guards are built on paths that invoke
instruction augmentation or HTTP context assembly. A WebSocket session with no
profile/episode/report/summary or ApiPilot block skips proxy augmentation.
profile/episode/report/summary block skips non-MCP proxy augmentation.
Direct browser connections bypass server-side WebSocket injection entirely.
| Setting | Default | Effect |
@@ -51,7 +51,6 @@ Direct browser connections bypass server-side WebSocket injection entirely.
| Scheduled reports | `S2S_REPORT_LIST`, profile gate and report gate | Newest N labels/dates by default; full mode includes scripts with a 12,000-character block limit |
| Latest conversation | `S2S_LATEST_SUMMARY_TEXT`, or `S2S_LATEST_SUMMARY` with a phone number | Up to 1,600 characters; static text overrides HTTP fetch |
| Custom RAG | Operator applies text through the RAG dialog | Up to 12,000 characters; appended in browser instructions and HTTP `user_context` |
| ApiPilot discovery | Configured `S2S_APIPILOT_TOOLS_URL` | Tool names/descriptions/auth requirements and call-marker rules, independent of the profile gate |
### Episode
@@ -113,25 +112,25 @@ session. This context does not require an external retrieval endpoint.
At local WebSocket connect, the server resolves enabled context and stores it in
`session_ctx`. Each client `session.update` then passes through
`_patch_client_session_update_with_profile()`. Other client frame types are not
modified by this helper. The broader relay still performs ApiPilot handling and
text/audio filtering.
modified by this helper. The non-MCP relay retains its general assistant-text
filtering and forwards audio without a tool-execution pass.
The current augmentation function **keeps base instructions first**, then
appends surviving voice-output guards, date context, episode, profile, reports,
and summary blocks. ApiPilot discovery is appended separately. If a block type
is already present in the base, deduplication may omit its server candidate.
and summary blocks. If a block type is already present in the base, deduplication
may omit its server candidate.
This describes string assembly, not a guarantee of model attention or priority.
Every HTTP reasoning request rebuilds enabled context. Its `user_context`
assembly orders voice guard, date, episode, profile, reports, summary, ApiPilot
assembly orders voice guard, date, episode, profile, reports, summary,
and browser-provided context, then deduplicates recognized types and strips
internal markers. Server instruction augmentation remains separate, so avoid
assuming the WebSocket and HTTP prompts have identical ordering.
Context markers support internal deduplication and diagnostics; ordinary
profile/report/episode/history sentinels are stripped before final augmentation.
ApiPilot call/catalog markers have their own handling. Treat the source helpers
as the reference for the exact final payload.
MCP uses its separate conversation context and call-marker handling described
above. Treat the source helpers as the reference for the exact final payload.
## Failure behavior and inspection
docs/client/development.md
+19 -10
View File
@@ -45,7 +45,7 @@ and the retained `server:app` shim; use `--app-dir vocero-api` from the root.
## Schema boundaries
`models/requests.py` retains the existing search, reasoning, and ApiPilot
`models/requests.py` retains the existing search and reasoning
request models, including their defaults, coercion, and ignored extra fields.
The queue beacon still accepts malformed JSON and non-object bodies as before;
its Pydantic view is applied only after the original object check, and
@@ -54,9 +54,10 @@ its Pydantic view is applied only after the original object check, and
`models/responses.py` validates structure constructed by this server. Fields
owned by external services remain `Any` where the old parser imposed no type
constraint. Object models allow extra fields. Caller profiles can be arbitrary
JSON; load-balancer fields remain optional; execution results and error details
remain open. Existing provider parsing, authentication precedence, argument
filtering, environment parsing, and WebSocket frames are unchanged.
JSON; load-balancer fields remain optional. Existing provider parsing,
authentication precedence, environment parsing, and WebSocket frames are
preserved outside explicitly retired integrations. Authenticated conversation
and MCP contracts live in `models/mcp.py` and `models/content.py`.
HTTP handlers call `validated_json_response`: Pydantic validates the payload,
then `JSONResponse` serializes the original object. Returning a dumped model
@@ -69,19 +70,23 @@ existing error shapes; there is no new common error envelope.
## Compatibility evidence
`tests/fixtures/legacy-contracts.json` was captured from the original monolithic
server at commit `3d42df5`, before refactoring. Tests compare 82 HTTP scenarios,
server at commit `3d42df5`, before refactoring. Tests compare 66 HTTP scenarios,
including exact response text, statuses, selected headers, and outgoing mock
HTTP requests. They also compare request OpenAPI definitions and representative
realtime frames. Additional tests cover response validation and permissive
upstream values; the original local HTTP/WebSocket tests remain in the suite.
Four captures belonging exclusively to the retired workflow reasoning service
were removed with that integration; all retained captures are unchanged.
were removed with that integration. ApiPilot retirement removed another 16 HTTP
captures, its two OpenAPI operations, and its request model. The four bootstrap
captures now omit the retired `apipilot` flag. One reasoning failure case was
renamed because removing ApiPilot eliminated its token access; it still exposes
the existing uninitialized profile failure. All other retained capture values
are unchanged.
The captures deliberately preserve unknown fields, omitted fields, unusual JSON
values, queue beacon edge cases, permissive tool arguments, and known failures.
Examples include reasoning's existing uninitialized local variables in certain
configuration combinations, API-key-only tool precheck behavior, the catalog
cache's existing key, and relay termination after a suppressed invalid
values, queue beacon edge cases, and known failures. Examples include reasoning's
existing uninitialized profile in certain non-MCP configuration combinations
and relay termination after a suppressed invalid
`session.update` error. Fix those in separately reviewed behavior changes;
do not regenerate golden captures to make a refactor pass.
@@ -90,3 +95,7 @@ provider responses, live speech models, or production configuration. No deployed
services or real credentials are needed. The Docker check verifies the actual
runtime image, API configuration, documentation, OpenAPI, and the backend's
existing root 404; it removes its temporary container and image afterward.
ApiPilot removal deliberately excludes `speech_to_speech/`. The MCP orchestration
test still exercises its existing pure marker sanitizer without importing model
dependencies, verifying that `VOCERO_MCP_CALL` reaches the Vocero relay unchanged.
docs/client/glossary.md
+2 -2
View File
@@ -22,8 +22,8 @@ names shown in a deployment are configuration examples, not definitions.
| Tool executor | Code that receives a tool call, executes it and returns the result to the model |
| Web search | The browser-supported native tool whose request goes through the server's `/api/search` proxy |
| Search proxy | Same-origin server route holding a deployment search key and calling the external provider |
| ApiPilot | The dynamic-tool discovery/execution bridge, with both HTTP endpoints and model-text call markers |
| Call marker | Internal structured text such as `<<<APIPILOT_CALL:...>>>` used by the ApiPilot bridge; separate from native function-call events |
| AquaPortal MCP | Authenticated tool discovery and execution using the current conversation's backend session |
| Call marker | Internal structured text such as `<<<VOCERO_MCP_CALL:...>>>` used by the MCP orchestrator; separate from native function-call events |
| Tools panel | Controls for available browser tools and search-key configuration |
| Settings | Connection, voice, audio and instruction controls persisted in browser storage |
| Identity block | The header's product/user identity, separate from configuration dialogs |
docs/client/reasoning.md
+15 -10
View File
@@ -19,8 +19,9 @@ with `config.primaryPrompt || settings.instructions`. `/api/config` pins
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
profile to choose this base prompt.
The browser adds caller context when `injectProfile` is enabled, custom RAG text
when the operator has applied it, and an available ApiPilot catalog block.
The browser adds caller context when `injectProfile` is enabled and custom RAG
text when the operator has applied it. In MCP mode, the server constructs tool
instructions from the authenticated conversation's catalog and content selection.
`speechInstructions()` adds the speech-rendering hint only in `OPENAI` mode.
The WebSocket client packages these instructions in `session.update.session`;
the local proxy may then add server-resolved context. Details belong in
@@ -30,13 +31,14 @@ the local proxy may then add server-resolved context. Details belong in
The realtime backend receives session instructions and enabled native tools,
then processes each voice turn through STT, model generation and TTS.
Native function-call events are executed by the browser. ApiPilot also has a
Native function-call events are executed by the browser. Authenticated MCP has a
separate marker-based path handled by the local proxy. These are distinct
protocols, described in [tool calling](tool-calling.md).
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
instructions still apply, but server-side session context and marker execution
require the local `/v1/realtime` route.
require the local `/v1/realtime` route. MCP in `PROXY` mode requires this relay;
configuration rejects a pinned direct URL or load-balancer bypass.
## OpenAI-compatible HTTP reasoning
@@ -53,12 +55,14 @@ is a runtime default, not a recommendation about provider availability.
`VoiceController.reason()` in [controller.ts](../../vocero-frontend/src/realtime/controller.ts)
runs after a complete user transcript and posts a body with `transcript`,
`instructions`, `language`, `user_context`, `phoneNumber`, and `episode_id`.
The server combines instructions and context into model messages, requests a
final answer with a 30-second timeout, and returns `answer`. ApiPilot can
append discovery instructions, execute a detected call marker, and request a
second response based on the tool result, including available `usage_pass2`
metadata. The endpoint does not forward the browser's native `session.tools`
catalog as OpenAI Chat Completions `tools`.
Without MCP, the server combines instructions and context into model messages,
requests a final answer with a 30-second timeout, and returns `answer`.
With MCP, the browser also sends conversation and turn headers. The server uses
that conversation's backend token, linked/current content, and bounded history
to run tool selections before returning `answer`. This path has a 180-second
total timeout and configured per-operation limits. Retired ApiPilot execution
metadata is no longer returned. The endpoint does not forward the browser's
native `session.tools` catalog as OpenAI Chat Completions `tools`.
For an external answer, the browser cancels/suppresses the active native
response and appends the external assistant text. It uses browser
@@ -78,5 +82,6 @@ realtime backend instead.
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
| Change browser context assembly | `buildInstructions()` in `vocero-frontend/src/lib/context.ts` |
| Change HTTP provider messages | `reasoning()` in `vocero-api/vocero_api/services/reasoning.py` |
| Change MCP reasoning or tool context | `vocero-api/vocero_api/services/mcp_reasoning.py` and `vocero-api/vocero_api/services/mcp_tools.py` |
| Change external-answer rendering | `VoiceController.reason()` and `RealtimeClient.renderExternalAnswer()` |
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
docs/client/tool-calling.md
+17 -54
View File
@@ -2,7 +2,7 @@
The application supports native realtime function calling and server-side
tool selection from model text. AquaPortal content uses authenticated MCP when
`S2S_MCP_URL` is set; otherwise the legacy ApiPilot bridge remains available.
`S2S_MCP_URL` is set. Without MCP, the native browser tools remain available.
A spoken promise to fetch something is not evidence that a tool executed.
## Authenticated AquaPortal MCP
@@ -172,69 +172,32 @@ result should be spoken. A `function_call_output` item alone does not start
generation. See the [realtime API guide](../speech/realtime-api.md) for local
model `<code>` extraction and native remote-provider tool calls.
## ApiPilot discovery and execution
When MCP is disabled, `S2S_APIPILOT_TOOLS_URL` enables the legacy dynamic-tool bridge. The backend
discovers tools through its `/tools` route and executes through `/execute`.
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
server exposes:
- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
`data.tools`, `data.count`, and `data.include_unauthorized`.
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
a `success/data/error` envelope.
Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
`requires_auth`. Execution looks up the current discovery cache, refreshes once
if the tool is missing, and applies an authorization gate and argument
validation before forwarding. Caller `Authorization` and
`x-demo-session` can be forwarded; configured service credentials and
login-by-phone provide additional server paths. Display `error.message` to the
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.
The discovery cache defaults to 300 seconds server-side
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
also keeps a per-user catalog cache. ApiPilot configuration is independent of
`S2S_INJECT_PROFILE`.
## Marker-based two-pass reasoning
The browser and proxy add an ApiPilot catalog as **instruction text**, including
tool names, descriptions, authentication requirements and required parameters.
The model is instructed to emit a call marker when it chooses a matching tool:
```text
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
```
Tool names are exact identifiers and must not be translated. The local realtime
proxy accumulates assistant text, detects the marker, validates and executes
the ApiPilot call, then injects the result as context and requests a new
response. The HTTP reasoning route has analogous discovery/execution and
second-answer handling. Internal marker text and its speculative audio are
filtered from user output where those handlers run.
This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
discovery is not the same as declaring a native `session.tools` array. Direct
browser connections bypass the local relay's marker executor even if the
browser includes catalog instructions.
## Retired documentation assumptions
ApiPilot discovery/execution endpoints, catalog injection, credential overrides,
and two-pass marker handling have been removed from Vocero. Its former HTTP
routes return 404 regardless of MCP configuration. The separate speech service
still has its existing legacy marker sanitizers; no ApiPilot requests can be
made through Vocero. See [usage](usage.md#authenticated-aquaportal-mcp) for
deployment configuration cleanup.
The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
Those APIs/rankers are absent from the current client/server path. The retained
agent-profile JSON is an illustrative historical/example shape. Add or modify
current ApiPilot tool definitions in the provider; do not restore obsolete
endpoints solely to match older documentation.
agent-profile JSON is an illustrative historical/example shape. Content tool
definitions and permissions belong to the AquaPortal MCP backend; Vocero's
provider enforces its allowlist. Do not restore obsolete endpoints solely to
match older documentation.
## Verify a tool turn
1. Confirm `/api/config` reports the intended feature and connection mode.
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
separate evidence.
2. For MCP, confirm conversation creation and authenticated `/api/mcp/tools`
discovery succeed. Inspect the relay's `session.update` for MCP instructions;
these are separate from native `session.tools` declarations.
3. For native calls, find `response.function_call_arguments.done`, then the
browser `toolcall` and executor request.
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
4. For MCP, find `mcp_tool_completed` for the expected user/tool and any
`vocero.tool.started` / `vocero.tool.failed` voice events.
5. Confirm the tool result reaches the model and follow-up text/audio reaches
the browser. A spoken acknowledgement alone does not complete the flow.
docs/client/troubleshooting.md
+7 -5
View File
@@ -12,7 +12,7 @@ Inspect the server configuration without printing secret environment files:
curl -fsS http://localhost:8420/api/config | jq
```
Confirm `s2sUrl`, `lb`, `reasoningType`, `injectProfile`, `apipilot`, `theme`
Confirm `s2sUrl`, `lb`, `reasoningType`, `injectProfile`, `mcp`, `theme`
and `startSessionMode`. A direct WebSocket URL bypasses the local relay's
server-side context injection and marker execution. For a local proxy, confirm
the server can resolve and reach `S2S_PROXY_TARGET`.
@@ -50,8 +50,10 @@ Follow the actual execution mechanism:
JSON-string arguments.
3. Confirm the browser dispatched `toolcall` and the executor sent its request.
4. Confirm `function_call_output` and any required `response.create` were sent.
5. For ApiPilot, inspect discovery, marker detection, authorization/schema
validation, execution, and second-pass result injection instead.
5. For MCP, inspect conversation creation, `/api/mcp/tools` discovery,
`mcp_tool_completed` logs, and the returned tool data/follow-up response.
Verify the conversation belongs to the browser and backend authorization
succeeds; do not retry with shared credentials or legacy content endpoints.
A spoken sentence such as “Voy a buscar el reporte” does not satisfy step 2.
If the function-call event is absent, inspect model/backend tool generation
@@ -59,7 +61,7 @@ before debugging a browser executor that has not been invoked.
Earlier local debug notes recorded a reproduction where tools and required
tool policy were sent but the model produced only spoken text. Those notes
refer to the retired MCP ranking path; they do not establish a current ApiPilot
refer to the retired MCP ranking path; they do not establish a current AquaPortal MCP
failure. Both local investigation notes remain marked `OPEN`. Their useful
lesson is to collect evidence at each boundary and reproduce against current
code before claiming a root cause is resolved.
@@ -114,7 +116,7 @@ Useful checks are:
The historical investigation introduced a 90ms playback prebuffer and reduced
MCP prompt reinjection. The old 150ms MCP fallback timer is historical, not a
current ApiPilot configuration knob. Verify the live worklet and current relay
current MCP configuration knob. Verify the live worklet and current relay
before changing audio timing based on those observations.
## External reasoning and startup
docs/client/usage.md
+18 -7
View File
@@ -2,7 +2,8 @@
The browser voice client connects to an existing OpenAI Realtime-compatible
speech backend. The UI server can also proxy WebSocket traffic, search,
reasoning, caller context, ApiPilot tools and load-balancer requests.
reasoning, caller context, authenticated AquaPortal MCP tools and load-balancer
requests.
## Local startup
@@ -126,8 +127,8 @@ translations.
Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
tools panel. The server proxies requests through `/api/search`; its own key is
not returned to the browser. Dynamic ApiPilot tools are configured through
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).
not returned to the browser. AquaPortal content tools use authenticated MCP;
see [tool calling](tool-calling.md).
Operator RAG text is applied through the RAG dialog and updates live
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
@@ -141,7 +142,7 @@ enabled data sources and limits.
Streamable HTTP endpoint, for example `https://backend.example.test/api/v1/mcp/`.
Set `S2S_LOGIN_BY_PHONE` to that backend's `POST /api/v1/auth/phone-login` URL.
The phone parameter remains an identity selector without proof of ownership.
MCP uses the returned opaque `sessionToken`, never an ApiPilot API key or service token.
MCP uses the returned opaque `sessionToken` for that user, with no service-token override.
For voice reasoning (`S2S_REASONING_TYPE=PROXY`), set `S2S_PROXY_TARGET` to the
speech service and leave `SPEECH_TO_SPEECH_URL` and `LOAD_BALANCER_URL` unset.
@@ -153,10 +154,20 @@ or audio-format change is required.
The backend must expose the eight-tool catalog and phone-login contract described
in [tool calling](tool-calling.md#authenticated-aquaportal-mcp). Leave
`S2S_MCP_URL` unset to retain the existing ApiPilot and caller-context paths.
`S2S_MCP_URL` unset to use the existing caller-context paths without MCP tools.
Setting it makes MCP the exclusive AquaPortal content path: legacy profile,
report, latest-summary, and episode preloads are skipped. Existing ApiPilot
and caller HTTP routes return 409 while MCP is enabled.
report, latest-summary, and episode preloads are skipped. Legacy caller HTTP
routes return 409 while MCP is enabled.
ApiPilot has been retired. `/api/apipilot/tools` and `/api/apipilot/execute`
return 404, and `/api/config` no longer includes `apipilot`. Remove obsolete
`S2S_APIPILOT_TOOLS_URL`, `S2S_APIPILOT_TOOLS_EXECUTE_URL`,
`S2S_APIPILOT_SERVICE_TOKEN`, `S2S_APIPILOT_API_KEY`, `S2S_APIPILOT_TTL_SEC`,
and `S2S_APIPILOT_TIMEOUT_SEC` from deployment configuration; they are ignored.
Keep `S2S_LOGIN_BY_PHONE`, which is still required for MCP authentication.
Deploy the API and frontend together and refresh existing browser sessions.
The separately deployed `speech_to_speech/` service is unchanged by this removal;
its existing legacy marker sanitizers remain, with no speech restart required.
| Setting | Default | Meaning |
|---|---|---|