valenti/vocero-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first

Browse Source
This commit is contained in:
valenti
2026-08-26 17:42:00 -04:00
commit ad95d530ad
134 changed files with 31750 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
debug-mcp-tool-not-called.md
+75
View File
@@ -0,0 +1,75 @@
# Debug Session: mcp-tool-not-called
Status: [OPEN]
## Symptom
- The voice assistant says it will call an MCP tool, but no MCP tool call is observed.
- No useful runtime log or call attempt is visible from the current instrumentation.
## Expected
- When the model decides to use an MCP tool, the client should emit a tool-call event, execute the MCP proxy request, and produce visible debug evidence for each stage.
## Hypotheses
1. The realtime model is not actually emitting `response.function_call_arguments.done`, so the spoken promise to use a tool is plain text, not a real function call.
2. The MCP tools are not present in the effective session state at the moment of the request, so the model cannot legally call them.
3. The session instructions are updated too late or with the wrong content, so the model receives guidance text but not usable tool declarations.
4. The browser receives a tool-call event, but the handler path fails before logging or before `execRemoteMcpTool()` starts.
5. The MCP proxy request starts, but the current logs do not surface pre-call or failure details, making the attempt invisible.
## Evidence Plan
- Instrument session bootstrap, tool declaration updates, transcript-based MCP ranking, tool-call event handling, and MCP proxy fetch lifecycle.
- Reproduce with a prompt that should match one MCP tool by description.
- Compare whether the model emits only assistant transcript text or a real tool-call event.
## Notes
- First code change in this session must remain instrumentation only.
## Evidence Summary
- `api/config` loads correctly with `searchMcpEnabled=true`, `agentToolCount=16`, and a non-empty primary prompt.
- Transcript-time MCP ranking is active and produces ranked candidates such as `get_api_info`, `get_report_detail`, and `get_public_day_report` for relevant spoken requests.
- No `response.function_call_arguments.done` event was recorded in the current reproduction.
- No `toolcall` event, no `runTool()` start, and no MCP proxy fetch were recorded in the current reproduction.
## Hypothesis Status
| ID | Hypothesis | Status | Evidence |
|----|------------|--------|----------|
| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | The log file contains transcript/ranking events only; no `function_call_arguments.done` entry was produced in the reproduced turn. |
| B | MCP tools are missing from runtime state | PARTIAL | `api/config` proves the remote profile exposes 16 tools, but the current run lacks direct evidence from live `session.update`. |
| C | MCP guidance reaches the session too late for the same voice turn | LIKELY | Ranking is computed from the live user transcript; current code only pushes updated instructions on the final transcript, which can miss the active turn. |
| D | The browser gets the toolcall event but fails before execution | REJECTED (current run) | There is no `toolcall` event at all in the current reproduction. |
| E | The MCP proxy request starts but is invisible | REJECTED (current run) | No pre-fetch or post-fetch MCP proxy log was emitted. |
## Minimal Fix
- Push the computed MCP-guidance instructions to the live session as soon as ranked tool suggestions appear on partial user transcripts, instead of waiting only for the final transcript.
- Track the last live instruction snapshot to avoid redundant `session.update` spam.
- Add client-construction instrumentation so the next run proves which tools/instructions are present at session start.
## Post-Fix Evidence
- The live session is created with 17 tools and non-empty instructions.
- During matching turns, the client now narrows the tool set down to the strongest MCP candidates and sends `tool_choice="required"`.
- Example evidence from the latest run:
- `get_api_info` is selected as the top candidate with strong score (`29`).
- The client sends `setTools(..., "required")` with `["get_api_info", ...]`.
- The client sends updated instructions immediately after.
- Even with `tool_choice="required"` and a narrowed tool list, no `response.function_call_arguments.done` event is emitted.
- Therefore no browser `toolcall`, no `runTool()`, and no MCP proxy request can happen.
## Updated Hypothesis Status
| ID | Hypothesis | Status | Evidence |
|----|------------|--------|----------|
| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | Even after forcing `tool_choice="required"` and narrowing the tool list, no `function_call_arguments.done` event appears. |
| B | MCP tools are missing from runtime state | REJECTED | The client sends 17 initial tools and later sends narrowed MCP-only tool lists, including `tool_choice="required"`. |
| C | MCP guidance reaches the session too late for the same voice turn | REJECTED | The client now pushes guidance and tool policy during partial transcripts, before the turn finishes. |
| D | The browser gets the toolcall event but fails before execution | REJECTED | No upstream function-call event is emitted at all. |
| E | The MCP proxy request starts but is invisible | REJECTED | The MCP proxy is never reached because the model never initiates a tool call. |
## Current Root Cause
- The speech-to-speech realtime model path in the current backend stack is not performing actual function calling for these MCP tools in this conversation flow, even when tools are declared and `tool_choice="required"` is sent.
- The spoken assistant text is therefore behaving like a promise in natural language, not a real tool invocation.
## Backend Branch B
- Added relay-side instrumentation in `demo/server.py` to report:
- Outbound client text events forwarded to upstream, including `session.update` summaries with `toolCount`, `toolNames`, `toolChoice`, and instruction length.
- Inbound upstream text events forwarded back to the client, including `session.updated`, `response.function_call_arguments.done`, and `error` summaries when present.
- Binary frame forwarding in both directions and client disconnects.
- No new runtime evidence has been collected from this instrumentation yet; a fresh reproduction is required with the patched relay active.