chore: remove debug markdown files
This commit is contained in:
@@ -139,3 +139,4 @@ speech_to_speech/*.log
|
|||||||
speech_to_speech/sessions/
|
speech_to_speech/sessions/
|
||||||
speech_to_speech/.cache/
|
speech_to_speech/.cache/
|
||||||
s2s-reasoning-engine/*.log
|
s2s-reasoning-engine/*.log
|
||||||
|
debug_files
|
||||||
|
|||||||
@@ -1,45 +0,0 @@
|
|||||||
# Debug Session: mcp-audio-stutter
|
|
||||||
- **Status**: [OPEN]
|
|
||||||
- **Issue**: Piccole interruzioni audio quando una risposta MCP lunga viene eseguita o reiniettata nella conversazione live.
|
|
||||||
|
|
||||||
## Reproduction Steps
|
|
||||||
1. Avvia la UI realtime.
|
|
||||||
2. Esegui una richiesta che attiva un tool MCP con output grande.
|
|
||||||
3. Ascolta la risposta audio dell'assistente.
|
|
||||||
4. Verifica se compaiono micro-interruzioni durante o subito dopo l'iniezione del risultato MCP.
|
|
||||||
|
|
||||||
## Hypotheses & Verification
|
|
||||||
| ID | Hypothesis | Likelihood | Effort | Evidence |
|
|
||||||
|----|------------|------------|--------|----------|
|
|
||||||
| A | Inviare al modello il payload MCP quasi completo degrada la fluidità della risposta audio anche senza bloccare il main thread. | High | Low | Confirmed |
|
|
||||||
| B | Esiste una race: il fallback MCP decide il turno prima che `response.created` arrivi, quindi la prima response speculativa parte comunque e viene tagliata. | High | Medium | Confirmed |
|
|
||||||
| C | La chat/UI fa render pesanti quando arriva il risultato MCP grande. | Medium | Medium | Rejected |
|
|
||||||
| D | I log/debug del payload MCP grande aggravano il problema sul main thread. | Medium | Low | Rejected |
|
|
||||||
| E | L'attacco audio di ogni frase soffre di jitter iniziale perché il playback parte senza prebuffer sul primo chunk disponibile. | High | Medium | Confirmed |
|
|
||||||
|
|
||||||
## Log Evidence
|
|
||||||
- `execRemoteMcpTool` riceve un result JSON di `5738` byte e produce un output testo di `3262` caratteri senza tempi anomali.
|
|
||||||
- `built live prompt from MCP output` mostra `promptLength` ~`3400` e `buildMs: 0`.
|
|
||||||
- `rendered tool result row` e `rendered MCP tool result in chat` mostrano `renderMs: 0-1`.
|
|
||||||
- `sending conversation.item.create payload` mostra `payloadLength` ~`3790` e `stringifyMs: 0`.
|
|
||||||
- Quindi serializzazione, render UI e stringify non spiegano i micro-tagli.
|
|
||||||
- Nel run post-fix `sendUserText` mostra `responseActive: false`, `openResponses: 0`, `createInFlight: false`: non è un classico barge-in su response già attiva.
|
|
||||||
- Il sintomo residuo si manifesta solo sulla prima frase speculativa; questo è coerente con una race prima di `response.created`.
|
|
||||||
- Il browser può decidere il fallback MCP prima che il client abbia un `responseId` da silenziare: per questo ora viene marcata anche la **next response** come sopprimibile.
|
|
||||||
- L'ultimo sintomo riferito dall'utente cambia ancora: le micro-interruzioni si sentono sulle prime parole di ogni frase, non solo su quella speculativa.
|
|
||||||
- Questo è coerente con il worklet `audio-playback` che parte subito sul primo chunk ricevuto, senza accumulare buffer iniziale.
|
|
||||||
|
|
||||||
## Minimal Fix
|
|
||||||
- Accodata la `sendUserText()` del payload MCP quando esiste una response attiva.
|
|
||||||
- Flush del testo differito solo dopo `response.done`, prima del `response.create` già in coda.
|
|
||||||
- Riassunto/troncamento del payload MCP prima di reiniettarlo nella conversazione live.
|
|
||||||
- Ridotto il delay del fallback MCP a `150ms` per match forti e senza argomenti obbligatori.
|
|
||||||
- Soppressione della **next speculative response** se il fallback MCP vince il turno prima che `response.created` sia arrivato.
|
|
||||||
- Introdotto un piccolo prebuffer di avvio (`90ms`) nel worklet `audio-playback` prima di iniziare ogni frase.
|
|
||||||
- Obiettivo: evitare sia il barge-in quando presente, sia il carico eccessivo sul modello con JSON troppo grande.
|
|
||||||
- Obiettivo aggiuntivo: ridurre la finestra in cui il modello può iniziare una frase speculativa prima che parta il tool MCP.
|
|
||||||
- Obiettivo finale: evitare il clipping/jitter sulle prime parole di ogni frase.
|
|
||||||
|
|
||||||
## Verification Conclusion
|
|
||||||
- Pre-fix: il browser non mostrava alcun collo di bottiglia locale; il prompt inviato al modello restava però molto grande.
|
|
||||||
- Post-fix: il testo MCP viene accodato se c'è una risposta attiva, il modello riceve solo un summary compatto, il fallback parte quasi subito, la next response speculativa viene silenziata se necessario e il playback aspetta un buffer iniziale prima di iniziare ogni frase.
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
# Debug Session: mcp-tool-not-called
|
|
||||||
|
|
||||||
Status: [OPEN]
|
|
||||||
|
|
||||||
## Symptom
|
|
||||||
- The voice assistant says it will call an MCP tool, but no MCP tool call is observed.
|
|
||||||
- No useful runtime log or call attempt is visible from the current instrumentation.
|
|
||||||
|
|
||||||
## Expected
|
|
||||||
- When the model decides to use an MCP tool, the client should emit a tool-call event, execute the MCP proxy request, and produce visible debug evidence for each stage.
|
|
||||||
|
|
||||||
## Hypotheses
|
|
||||||
1. The realtime model is not actually emitting `response.function_call_arguments.done`, so the spoken promise to use a tool is plain text, not a real function call.
|
|
||||||
2. The MCP tools are not present in the effective session state at the moment of the request, so the model cannot legally call them.
|
|
||||||
3. The session instructions are updated too late or with the wrong content, so the model receives guidance text but not usable tool declarations.
|
|
||||||
4. The browser receives a tool-call event, but the handler path fails before logging or before `execRemoteMcpTool()` starts.
|
|
||||||
5. The MCP proxy request starts, but the current logs do not surface pre-call or failure details, making the attempt invisible.
|
|
||||||
|
|
||||||
## Evidence Plan
|
|
||||||
- Instrument session bootstrap, tool declaration updates, transcript-based MCP ranking, tool-call event handling, and MCP proxy fetch lifecycle.
|
|
||||||
- Reproduce with a prompt that should match one MCP tool by description.
|
|
||||||
- Compare whether the model emits only assistant transcript text or a real tool-call event.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
- First code change in this session must remain instrumentation only.
|
|
||||||
|
|
||||||
## Evidence Summary
|
|
||||||
- `api/config` loads correctly with `searchMcpEnabled=true`, `agentToolCount=16`, and a non-empty primary prompt.
|
|
||||||
- Transcript-time MCP ranking is active and produces ranked candidates such as `get_api_info`, `get_report_detail`, and `get_public_day_report` for relevant spoken requests.
|
|
||||||
- No `response.function_call_arguments.done` event was recorded in the current reproduction.
|
|
||||||
- No `toolcall` event, no `runTool()` start, and no MCP proxy fetch were recorded in the current reproduction.
|
|
||||||
|
|
||||||
## Hypothesis Status
|
|
||||||
| ID | Hypothesis | Status | Evidence |
|
|
||||||
|----|------------|--------|----------|
|
|
||||||
| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | The log file contains transcript/ranking events only; no `function_call_arguments.done` entry was produced in the reproduced turn. |
|
|
||||||
| B | MCP tools are missing from runtime state | PARTIAL | `api/config` proves the remote profile exposes 16 tools, but the current run lacks direct evidence from live `session.update`. |
|
|
||||||
| C | MCP guidance reaches the session too late for the same voice turn | LIKELY | Ranking is computed from the live user transcript; current code only pushes updated instructions on the final transcript, which can miss the active turn. |
|
|
||||||
| D | The browser gets the toolcall event but fails before execution | REJECTED (current run) | There is no `toolcall` event at all in the current reproduction. |
|
|
||||||
| E | The MCP proxy request starts but is invisible | REJECTED (current run) | No pre-fetch or post-fetch MCP proxy log was emitted. |
|
|
||||||
|
|
||||||
## Minimal Fix
|
|
||||||
- Push the computed MCP-guidance instructions to the live session as soon as ranked tool suggestions appear on partial user transcripts, instead of waiting only for the final transcript.
|
|
||||||
- Track the last live instruction snapshot to avoid redundant `session.update` spam.
|
|
||||||
- Add client-construction instrumentation so the next run proves which tools/instructions are present at session start.
|
|
||||||
|
|
||||||
## Post-Fix Evidence
|
|
||||||
- The live session is created with 17 tools and non-empty instructions.
|
|
||||||
- During matching turns, the client now narrows the tool set down to the strongest MCP candidates and sends `tool_choice="required"`.
|
|
||||||
- Example evidence from the latest run:
|
|
||||||
- `get_api_info` is selected as the top candidate with strong score (`29`).
|
|
||||||
- The client sends `setTools(..., "required")` with `["get_api_info", ...]`.
|
|
||||||
- The client sends updated instructions immediately after.
|
|
||||||
- Even with `tool_choice="required"` and a narrowed tool list, no `response.function_call_arguments.done` event is emitted.
|
|
||||||
- Therefore no browser `toolcall`, no `runTool()`, and no MCP proxy request can happen.
|
|
||||||
|
|
||||||
## Updated Hypothesis Status
|
|
||||||
| ID | Hypothesis | Status | Evidence |
|
|
||||||
|----|------------|--------|----------|
|
|
||||||
| A | The model speaks about tool use but does not emit a real function call | CONFIRMED | Even after forcing `tool_choice="required"` and narrowing the tool list, no `function_call_arguments.done` event appears. |
|
|
||||||
| B | MCP tools are missing from runtime state | REJECTED | The client sends 17 initial tools and later sends narrowed MCP-only tool lists, including `tool_choice="required"`. |
|
|
||||||
| C | MCP guidance reaches the session too late for the same voice turn | REJECTED | The client now pushes guidance and tool policy during partial transcripts, before the turn finishes. |
|
|
||||||
| D | The browser gets the toolcall event but fails before execution | REJECTED | No upstream function-call event is emitted at all. |
|
|
||||||
| E | The MCP proxy request starts but is invisible | REJECTED | The MCP proxy is never reached because the model never initiates a tool call. |
|
|
||||||
|
|
||||||
## Current Root Cause
|
|
||||||
- The speech-to-speech realtime model path in the current backend stack is not performing actual function calling for these MCP tools in this conversation flow, even when tools are declared and `tool_choice="required"` is sent.
|
|
||||||
- The spoken assistant text is therefore behaving like a promise in natural language, not a real tool invocation.
|
|
||||||
|
|
||||||
## Backend Branch B
|
|
||||||
- Added relay-side instrumentation in `demo/server.py` to report:
|
|
||||||
- Outbound client text events forwarded to upstream, including `session.update` summaries with `toolCount`, `toolNames`, `toolChoice`, and instruction length.
|
|
||||||
- Inbound upstream text events forwarded back to the client, including `session.updated`, `response.function_call_arguments.done`, and `error` summaries when present.
|
|
||||||
- Binary frame forwarding in both directions and client disconnects.
|
|
||||||
- No new runtime evidence has been collected from this instrumentation yet; a fresh reproduction is required with the patched relay active.
|
|
||||||
Reference in New Issue
Block a user