From b91762a927cf1123b513ff45d302e8131a81c7c9 Mon Sep 17 00:00:00 2001 From: Alvaro Parker Date: Mon, 5 Oct 2026 09:34:41 -0300 Subject: [PATCH] chore: remove debug markdown files --- .gitignore | 1 + debug-mcp-audio-stutter.md | 45 ---------------------- debug-mcp-tool-not-called.md | 75 ------------------------------------ 3 files changed, 1 insertion(+), 120 deletions(-) delete mode 100644 debug-mcp-audio-stutter.md delete mode 100644 debug-mcp-tool-not-called.md diff --git a/.gitignore b/.gitignore index 3893c32..128ca90 100644 --- a/.gitignore +++ b/.gitignore @@ -139,3 +139,4 @@ speech_to_speech/*.log speech_to_speech/sessions/ speech_to_speech/.cache/ s2s-reasoning-engine/*.log +debug_files diff --git a/debug-mcp-audio-stutter.md b/debug-mcp-audio-stutter.md deleted file mode 100644 index d4cb57d..0000000 --- a/debug-mcp-audio-stutter.md +++ /dev/null @@ -1,45 +0,0 @@ -# Debug Session: mcp-audio-stutter -- **Status**: [OPEN] -- **Issue**: Piccole interruzioni audio quando una risposta MCP lunga viene eseguita o reiniettata nella conversazione live. - -## Reproduction Steps -1. Avvia la UI realtime. -2. Esegui una richiesta che attiva un tool MCP con output grande. -3. Ascolta la risposta audio dell'assistente. -4. Verifica se compaiono micro-interruzioni durante o subito dopo l'iniezione del risultato MCP. - -## Hypotheses & Verification -| ID | Hypothesis | Likelihood | Effort | Evidence | -|----|------------|------------|--------|----------| -| A | Inviare al modello il payload MCP quasi completo degrada la fluidità della risposta audio anche senza bloccare il main thread. | High | Low | Confirmed | -| B | Esiste una race: il fallback MCP decide il turno prima che `response.created` arrivi, quindi la prima response speculativa parte comunque e viene tagliata. | High | Medium | Confirmed | -| C | La chat/UI fa render pesanti quando arriva il risultato MCP grande. | Medium | Medium | Rejected | -| D | I log/debug del payload MCP grande aggravano il problema sul main thread. | Medium | Low | Rejected | -| E | L'attacco audio di ogni frase soffre di jitter iniziale perché il playback parte senza prebuffer sul primo chunk disponibile. | High | Medium | Confirmed | - -## Log Evidence -- `execRemoteMcpTool` riceve un result JSON di `5738` byte e produce un output testo di `3262` caratteri senza tempi anomali. -- `built live prompt from MCP output` mostra `promptLength` ~`3400` e `buildMs: 0`. -- `rendered tool result row` e `rendered MCP tool result in chat` mostrano `renderMs: 0-1`. -- `sending conversation.item.create payload` mostra `payloadLength` ~`3790` e `stringifyMs: 0`. -- Quindi serializzazione, render UI e stringify non spiegano i micro-tagli. -- Nel run post-fix `sendUserText` mostra `responseActive: false`, `openResponses: 0`, `createInFlight: false`: non è un classico barge-in su response già attiva. -- Il sintomo residuo si manifesta solo sulla prima frase speculativa; questo è coerente con una race prima di `response.created`. -- Il browser può decidere il fallback MCP prima che il client abbia un `responseId` da silenziare: per questo ora viene marcata anche la **next response** come sopprimibile. -- L'ultimo sintomo riferito dall'utente cambia ancora: le micro-interruzioni si sentono sulle prime parole di ogni frase, non solo su quella speculativa. -- Questo è coerente con il worklet `audio-playback` che parte subito sul primo chunk ricevuto, senza accumulare buffer iniziale. - -## Minimal Fix -- Accodata la `sendUserText()` del payload MCP quando esiste una response attiva. -- Flush del testo differito solo dopo `response.done`, prima del `response.create` già in coda. -- Riassunto/troncamento del payload MCP prima di reiniettarlo nella conversazione live. -- Ridotto il delay del fallback MCP a `150ms` per match forti e senza argomenti obbligatori. -- Soppressione della **next speculative response** se il fallback MCP vince il turno prima che `response.created` sia arrivato. -- Introdotto un piccolo prebuffer di avvio (`90ms`) nel worklet `audio-playback` prima di iniziare ogni frase. -- Obiettivo: evitare sia il barge-in quando presente, sia il carico eccessivo sul modello con JSON troppo grande. -- Obiettivo aggiuntivo: ridurre la finestra in cui il modello può iniziare una frase speculativa prima che parta il tool MCP. -- Obiettivo finale: evitare il clipping/jitter sulle prime parole di ogni frase. - -## Verification Conclusion -- Pre-fix: il browser non mostrava alcun collo di bottiglia locale; il prompt inviato al modello restava però molto grande. -- Post-fix: il testo MCP viene accodato se c'è una risposta attiva, il modello riceve solo un summary compatto, il fallback parte quasi subito, la next response speculativa viene silenziata se necessario e il playback aspetta un buffer iniziale prima di iniziare ogni frase. diff --git a/debug-mcp-tool-not-called.md b/debug-mcp-tool-not-called.md deleted file mode 100644 index efb0b64..0000000 --- a/debug-mcp-tool-not-called.md +++ /dev/null @@ -1,75 +0,0 @@ -# Debug Session: mcp-tool-not-called - -Status: [OPEN] - -## Symptom -- The voice assistant says it will call an MCP tool, but no MCP tool call is observed. -- No useful runtime log or call attempt is visible from the current instrumentation. - -## Expected -- When the model decides to use an MCP tool, the client should emit a tool-call event, execute the MCP proxy request, and produce visible debug evidence for each stage. - -## Hypotheses -1. The realtime model is not actually emitting `response.function_call_arguments.done`, so the spoken promise to use a tool is plain text, not a real function call. -2. The MCP tools are not present in the effective session state at the moment of the request, so the model cannot legally call them. -3. The session instructions are updated too late or with the wrong content, so the model receives guidance text but not usable tool declarations. -4. The browser receives a tool-call event, but the handler path fails before logging or before `execRemoteMcpTool()` starts. -5. The MCP proxy request starts, but the current logs do not surface pre-call or failure details, making the attempt invisible. - -## Evidence Plan -- Instrument session bootstrap, tool declaration updates, transcript-based MCP ranking, tool-call event handling, and MCP proxy fetch lifecycle. -- Reproduce with a prompt that should match one MCP tool by description. -- Compare whether the model emits only assistant transcript text or a real tool-call event. - -## Notes -- First code change in this session must remain instrumentation only. - -## Evidence Summary -- `api/config` loads correctly with `searchMcpEnabled=true`, `agentToolCount=16`, and a non-empty primary prompt. -- Transcript-time MCP ranking is active and produces ranked candidates such as `get_api_info`, `get_report_detail`, and `get_public_day_report` for relevant spoken requests. -- No `response.function_call_arguments.done` event was recorded in the current reproduction. -- No `toolcall` event, no `runTool()` start, and no MCP proxy fetch were recorded in the current reproduction. - -## Hypothesis Status -| ID | Hypothesis | Status | Evidence | -|----|------------|--------|----------| -| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | The log file contains transcript/ranking events only; no `function_call_arguments.done` entry was produced in the reproduced turn. | -| B | MCP tools are missing from runtime state | PARTIAL | `api/config` proves the remote profile exposes 16 tools, but the current run lacks direct evidence from live `session.update`. | -| C | MCP guidance reaches the session too late for the same voice turn | LIKELY | Ranking is computed from the live user transcript; current code only pushes updated instructions on the final transcript, which can miss the active turn. | -| D | The browser gets the toolcall event but fails before execution | REJECTED (current run) | There is no `toolcall` event at all in the current reproduction. | -| E | The MCP proxy request starts but is invisible | REJECTED (current run) | No pre-fetch or post-fetch MCP proxy log was emitted. | - -## Minimal Fix -- Push the computed MCP-guidance instructions to the live session as soon as ranked tool suggestions appear on partial user transcripts, instead of waiting only for the final transcript. -- Track the last live instruction snapshot to avoid redundant `session.update` spam. -- Add client-construction instrumentation so the next run proves which tools/instructions are present at session start. - -## Post-Fix Evidence -- The live session is created with 17 tools and non-empty instructions. -- During matching turns, the client now narrows the tool set down to the strongest MCP candidates and sends `tool_choice="required"`. -- Example evidence from the latest run: - - `get_api_info` is selected as the top candidate with strong score (`29`). - - The client sends `setTools(..., "required")` with `["get_api_info", ...]`. - - The client sends updated instructions immediately after. -- Even with `tool_choice="required"` and a narrowed tool list, no `response.function_call_arguments.done` event is emitted. -- Therefore no browser `toolcall`, no `runTool()`, and no MCP proxy request can happen. - -## Updated Hypothesis Status -| ID | Hypothesis | Status | Evidence | -|----|------------|--------|----------| -| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | Even after forcing `tool_choice="required"` and narrowing the tool list, no `function_call_arguments.done` event appears. | -| B | MCP tools are missing from runtime state | REJECTED | The client sends 17 initial tools and later sends narrowed MCP-only tool lists, including `tool_choice="required"`. | -| C | MCP guidance reaches the session too late for the same voice turn | REJECTED | The client now pushes guidance and tool policy during partial transcripts, before the turn finishes. | -| D | The browser gets the toolcall event but fails before execution | REJECTED | No upstream function-call event is emitted at all. | -| E | The MCP proxy request starts but is invisible | REJECTED | The MCP proxy is never reached because the model never initiates a tool call. | - -## Current Root Cause -- The speech-to-speech realtime model path in the current backend stack is not performing actual function calling for these MCP tools in this conversation flow, even when tools are declared and `tool_choice="required"` is sent. -- The spoken assistant text is therefore behaving like a promise in natural language, not a real tool invocation. - -## Backend Branch B -- Added relay-side instrumentation in `demo/server.py` to report: -- Outbound client text events forwarded to upstream, including `session.update` summaries with `toolCount`, `toolNames`, `toolChoice`, and instruction length. -- Inbound upstream text events forwarded back to the client, including `session.updated`, `response.function_call_arguments.done`, and `error` summaries when present. -- Binary frame forwarding in both directions and client disconnects. -- No new runtime evidence has been collected from this instrumentation yet; a fresh reproduction is required with the patched relay active.