Technical architecture document
+Vocero S2S
+Target technical architecture and implementation plan
+ +Vocero S2S
+Target technical architecture and implementation plan
+Document status: proposal for technical review
+Source cutoff date: July 20, 2026
+Scope: a decoupled, local Speech-to-Speech conversational platform extensible through MCP
+Audience: architecture, platform, security, MLOps, development, QA and operations
++Traceability convention
++
+- [REQ] Supplied requirement: comes directly from the Vocero S2S description provided by the requester.
+- [REC] Recommended decision: a design proposal that must be approved through an ADR.
+- [BENCH] Pending assumption: a hypothesis requiring measurement, load testing, legal assessment or validation in the target environment.
+
1. Executive summary
+Vocero S2S is proposed as a real-time conversational platform separating four responsibilities: web experience, session control, speech and language inference, and enterprise integration through Model Context Protocol (MCP). [REQ] The initial implementation must use Parakeet TDT 0.6B for STT/VAD, LLaMA 3.1 8B Instruct for reasoning and tool calls, Qwen3-TTS 12Hz 1.7B Base in FP16 for synthesis and zero-shot cloning, and an NVIDIA GB10 node initially accessible through port 12345.
The target architecture retains this decoupled pipeline while avoiding placing all logic in the WebSocket connection. [REC] The proposal includes a Control Plane on the UI & Relay server for authentication, authorization, agent profiles, voice policy, MCP orchestration and auditing; a Realtime Data Plane for audio and events with ephemeral session state and priority cancellation; and a local Inference Plane that exposes versioned adapters for STT, LLM and TTS. This separation allows models, transport or infrastructure to be replaced without changing business contracts.
+For web clients, WebRTC offers operational advantages for media on variable networks, while WebSocket remains appropriate for server-to-server links and the proposed first version. Realtime platform documentation recommends WebRTC for browser/media and WebSocket for server pipelines 1. [REC] The plan therefore adopts WebSocket as the initial compatible transport and defines WebRTC as a controlled evolution rather than an MVP prerequisite.
+Interruption management is a first-class capability: when user speech is detected during a response, the system must cancel still-reversible generation, stop audio, flush unplayed buffers and truncate history to what was actually heard. Realtime systems document this pattern through cancellation and truncation of conversation items 2, as do agent frameworks that adjust history to the audio actually played 3.
+MCP is integrated through an enterprise gateway with policy, schema validation, delegated identity, limits, auditing and version adapters. Critical timing decision: as of July 20, 2026, the announcement for 2026-07-28 explicitly describes a Release Candidate whose final standard is expected on July 28, 2026 and includes breaking changes 4. Therefore:
-
+
- [REC] the initial stable compatibility version will be
2025-11-25, verified in the official transport documentation 5; this decision follows the requested source cutoff and does not contradict the stateless technical direction proposed by the RC;
+ - [REC]
2026-07-28will be implemented only behind an experimental adapter and feature flag;
+ - [REC] the domain design will remain ready for the RC's stateless model: self-contained requests, application state through explicit handles, and routing independent of affinity; +
- [REC] promotion will require official ratification, conformance tests and an approved migration; +
- this document will not assume it is a final standard. +
Finally, running STT, LLM and TTS concurrently on a single GB10 has not been demonstrated through a benchmark of the complete pipeline. [BENCH] The first engineering deliverable must be a reproducible harness measuring end-to-end latency, memory, bandwidth, degradation under concurrency, interruption accuracy and cloning quality before committing to capacity or a contractual SLO.
+2. Scope
+2.1 Objectives
+-
+
- [REQ] Deliver bidirectional, low-latency voice conversation from a browser.
- [REQ] Identify users through the customer API and personalize sessions by identity or role.
- [REQ] Dynamically load the agent profile, instructions, rules and tools.
- [REQ] Preserve a corporate cloned voice against defaults sent by the client.
- [REQ] Run STT, reasoning, tool calls and TTS in a decoupled pipeline.
- [REQ] Integrate enterprise systems through an MCP gateway without changing the core.
- [REC] Include barge-in, cancellation, backpressure, timeouts, isolation, traceability and failure recovery as core capabilities.
- [REC] Maintain versioned contracts that allow transports, models, runtimes and MCP servers to be replaced.
- [REC] Provide operational evidence for sizing, SLOs and auditing. +
2.2 Non-goals
+- Train STT, LLM or TTS models from scratch.
- Replace ERP, CRM, catalogs or other systems of record.
- Allow the LLM to execute directly against databases or APIs without a gateway and policy.
- Make the browser a repository for enterprise credentials or master voice material.
- Guarantee capacity, latency or concurrency before benchmarking on GB10.
- Declare MCP version
2026-07-28a final standard before ratification. - Design a complete contact center, universal regulatory recording or SIP telephony at this stage; these may be added as later domains.
3. Architecture principles
+- Separate control and media. Audio has real-time requirements; identity, policy and auditing require consistency and traceability.
- Contracts before implementations. Each boundary publishes versioned events and schemas; models remain behind adapters.
- Priority interruption. Cancellation and audio stopping take precedence over generating more content.
- Explicit state with ownership. Vocero owns conversational sessions; the protocol adapter or explicit handles own MCP state.
- Zero trust between planes. Each hop authenticates the service, authorizes the action, limits scope and propagates audited identity.
- Least privilege for tools. A tool receives only the strictly necessary arguments, identity and scopes.
- Privacy by default. Audio, transcripts, prompts, responses and voices are sensitive data; retention is explicit and minimized.
- Conversational failure handling. Technical failures become deterministic events and safe responses without leaking internal details.
- Measure before promising. Stage latency and percentiles govern the experience, rather than isolated averages.
- Versioned MCP compatibility. The gateway negotiates a stable version and encapsulates future candidates.
4. Context and container view
+4.1 Logical view by plane
+Experience Plane
- Web client, microphone capture, echo cancellation, playback and consent controls.
- Retains only required visual state and cursors; does not retain secrets or master voice material.
Realtime Data Plane
- WebSocket or WebRTC termination, media relay and event multiplexing.
- Enforces buffer, sequence and size limits; discards stale data when generation changes.
- Separates a high-priority control queue —
cancel,barge_in,close— from the audio/data queue.
Conversation Control Plane
- Resolves identity, profile, permissions and effective configuration.
- Maintains the session and turn state machine.
- Builds context, coordinates inference and decides whether a tool can be executed, confirmed or rejected.
Inference Plane
- Hides runtime details and exposes uniform capabilities:
transcribe,generate,synthesize,cancel,health,capacity. - Correlates each output with
session_id,turn_idandgeneration_idto discard late responses.
Enterprise Integration Plane
- MCP gateway with version adapters, allowlists, validation, authorization, service secrets, idempotency and auditing.
- No tool response enters the context without size, classification and sanitization checks.
5. Component responsibilities
+| Component | Responsibilities | Must not |
|---|---|---|
| Web client | Capture/play audio; AEC/NS/AGC where appropriate; show state; send confirmations; report playout progress | Authorize tools; store secrets; decide the effective voice |
| Realtime edge | TLS; connection authentication; limits; routing; DoS protection; WS upgrade or WebRTC termination | Interpret prompts or MCP arguments |
| Realtime Relay | Multiplex audio/events; order sequences; backpressure; sanitize session.update; prioritize interruptions | Retain business logic or downstream credentials |
| Session Orchestrator | State machine; identity; profile; context; turns; cancellation; degradation; correlation | Execute SQL, ERP or APIs directly |
| Policy Engine | Evaluate identity, role, tool, risk, scope, consent and data policy | Silently rewrite business arguments without a record |
| MCP Gateway | Negotiate versions; discover tools; validate JSON Schema; delegate identity; apply timeouts/retries; audit | Expose credentials to the LLM or browser |
| Inference Gateway | Admit/cancel jobs; adapt protocols; measure stages; health/capacity | Decide enterprise permissions |
| STT/VAD | Emit partial/final transcripts and speech signals with timestamps | Persist audio indefinitely |
| LLM | Generate structured text or tool intent within allowed schemas | Treat itself as the authority to execute actions |
| TTS | Synthesize a stream tied to a generation and authorized voice profile | Accept arbitrary ref_audio from the client |
| Voice Vault | Version, encrypt and govern voice references and consent | Serve files without service and tenant authorization |
| Stores | Ephemeral state with TTL, immutable auditing and versioned configuration | Mix operational PII with unrestricted telemetry |
| Observability | Metrics, structured logs, traces, alerts and dashboards | Log audio, tokens or sensitive arguments by default |
6. Deployment view
+[REC] Port 12345 must be internal, protected by a firewall, mTLS and service authentication; it must not be exposed directly to the Internet. [REC] Relay and Orchestrator are deployed with at least two logical instances in production. This does not require physical high availability from the first day of the MVP, but contracts must not depend exclusively on local memory.
[BENCH] The GB10 is initially a single failure and capacity domain. Before production, define the spare-node strategy, admission queue, session limit, shedding policy and model recovery procedure.
+7. State machine and session management
+7.1 Proposed states
+7.2 Identifiers
+connection_id: transport connection; may change on reconnection.session_id: stable, opaque conversational session.turn_id: committed user input.generation_id: agent response; changes after cancellation or regeneration.tool_call_id: unique invocation; must be idempotent when supported by the tool.voice_profile_id+voice_version: immutable reference to the authorized voice.trace_id: distributed correlation; does not replace functional identifiers.
7.3 Effective session state
+[REC] Calculated in this order, without allowing the client to escalate privileges:
- secure platform defaults;
- tenant policy;
- versioned agent profile;
- authorized user attributes;
- allowed session preferences;
- explicitly allowlisted per-response overrides.
The sanitizer for session.update removes or rejects fields that could replace voice_profile_id, tools, system instructions, limits or scopes. Any difference between requested and effective configuration is recorded in a session.configuration_applied event with coded reasons.
[REC] Ephemeral state is stored with TTL and optimistic version control. Resumption requires a short-lived opaque token, the same identity and a profile-version check. Uncommitted audio and generation jobs are not resumed: they are canceled and the client returns to LISTENING.
8. Execution flows
8.1 Normal conversational and MCP flow
8.2 Tool call with confirmation
+- The LLM emits intent, never direct execution.
- Orchestrator validates the name against the allowlist and current schema.
- Policy Engine classifies the action: read, reversible write, irreversible action or prohibited.
- If consent is required, emit
tool.confirmation_requiredwith a readable summary and nonsensitive fields. - The user confirms; create a
confirmation_idbound to a hash of tool + arguments + identity + expiration. - MCP Gateway executes with a delegated token of minimum scope.
- Validate, limit and classify the result before returning it to the LLM.
- Record the decision, not secrets or complete sensitive payloads.
For stable compatibility 2025-11-25, the adapter maintains the protocol lifecycle and session according to the official transport 5. RC 2026-07-28 proposes eliminating the handshake and protocol session, using self-contained requests, routing headers and InputRequiredResult; these capabilities remain only in the experimental adapter until ratification 4.
8.3 Barge-in
Rules:
- An interruption event bypasses token and audio queues.
- Each subsequent chunk from
G1is discarded because of a stalegeneration_id. - The client cursor is the source of truth for played audio; when missing, use the last conservative ACK.
- A read-only tool can be canceled; an already confirmed, irreversible action is not undone by barge-in. Report its status and avoid repeating it.
- Apply AEC and adaptive thresholds to reduce self-interruptions; test brief interjections as a specific case.
8.4 Errors and degradation
+| Condition | Automatic action | User experience | Auditing/operations |
|---|---|---|---|
| Authentication denied | Close with a stable code | Unauthorized access message | Reason, tenant, subject hash |
| Profile unavailable | Bounded retry; fallback only if approved | “I cannot start the assistant right now” | Dependency alert |
| Unhealthy STT | Pause admission or enable text mode if available | Report degradation | Metric and circuit breaker |
| LLM timeout | Cancel generation; retry once only if no tool side effect occurred | Brief, safe response | Span with timeout |
| TTS fails | Keep text; do not replace the voice without a policy | Show text or request a retry | Voice/model status |
| MCP timeout | Cancel or mark status unknown | “I could not confirm the operation” | Do not retry writes without idempotency |
| Invalid MCP response | Reject the payload; do not send it to the LLM | Controlled error | Schema violation |
| Saturated buffer | Slow the producer; discard stale partials; never lose cancellation | Possible network indicator | Drop/backpressure counters |
| Client disconnected | Short lease; cancel media; retain minimal resumable state | Controlled retry | Close reason |
| GPU at capacity | Early rejection or bounded queue | “Service busy, please try again” | Saturation, queue and shedding |
9. Illustrative contracts and events
[REC] The following schemas are illustrative. They must become versioned JSON Schema with compatibility tests and size limits. They do not represent a final contract.
9.1 Common envelope
{
+ "specversion": "1.0",
+ "type": "vocero.audio.input.delta.v1",
+ "source": "web-client",
+ "id": "evt_opaque",
+ "time": "RFC3339 timestamp",
+ "subject": "session/{session_id}",
+ "traceparent": "00-...",
+ "data": {
+ "session_id": "ses_opaque",
+ "turn_id": "turn_opaque",
+ "sequence": 42,
+ "payload": {}
+ }
+}9.2 Minimum event catalog
Client → platform
session.connect.v1session.update_requested.v1audio.input.delta.v1audio.input.commit.v1input.speech_started.v1output.playout_progress.v1generation.cancel_requested.v1tool.confirmation_submitted.v1session.close.v1
Platform → client
session.configuration_applied.v1transcript.partial.v1transcript.final.v1response.text_delta.v1response.audio_delta.v1response.completed.v1output.stop_and_flush.v1tool.confirmation_required.v1tool.status.v1error.v1
9.3 Example of session.update_requested
{
+ "type": "vocero.session.update_requested.v1",
+ "data": {
+ "session_id": "ses_opaque",
+ "client_capabilities": {
+ "audio_codecs": ["opus", "pcm16"],
+ "supports_playout_ack": true
+ },
+ "preferences": {
+ "locale": "es-CL",
+ "turn_mode": "server_vad"
+ }
+ }
+}Fields such as system_prompt, tools, mcp_credentials, ref_audio, ref_text or voice_profile_id are not accepted through this public contract. If received, the relay rejects or removes them according to policy and records the reason.
9.4 Example of internal tool intent
{
+ "tool_call_id": "tc_opaque",
+ "session_id": "ses_opaque",
+ "turn_id": "turn_opaque",
+ "tool": {
+ "name": "production.report.get",
+ "schema_version": "1.2.0"
+ },
+ "arguments": {
+ "site_id": "site_authorized",
+ "period": "value_from_user_or_context"
+ },
+ "execution": {
+ "mode": "read",
+ "idempotency_key": "idem_opaque",
+ "deadline_ms": "configured_value"
+ },
+ "principal": {
+ "subject_ref": "opaque_subject",
+ "tenant_id": "tenant_opaque",
+ "scopes": ["report:read"]
+ }
+}9.5 Errors
{
+ "type": "vocero.error.v1",
+ "data": {
+ "code": "MCP_TOOL_TIMEOUT",
+ "category": "dependency",
+ "retryable": false,
+ "user_message_key": "tool.temporarily_unavailable",
+ "correlation_id": "corr_opaque",
+ "details_ref": "internal_log_reference"
+ }
+}Public messages do not contain stack traces, internal endpoints, prompts, tokens, sensitive arguments or secret names.
+9.6 MCP compatibility
McpAdapter
+├── StableAdapter_2025_11_25 # default
+├── CandidateAdapter_2026_07_28 # feature flag, not enabled in production by default
+└── ConformanceHarness
+ ├── discovery
+ ├── tools/list
+ ├── tools/call
+ ├── cancellation
+ ├── schema validation
+ ├── auth propagation
+ └── error mappingThe official MCP architecture separates a JSON-RPC data layer from a transport layer and includes lifecycle, primitives and notifications 6. [REC] Vocero will preserve this separation inside the adapter so transport details do not contaminate conversational logic.
Candidate capabilities being prepared but not activated as a final standard: the RC 2026-07-28 introduces server/discover, metadata ttlMs and cacheScope in list/read results, and headers Mcp-Method/Mcp-Name for routing 4. [REC] The gateway will model discovery and caching through neutral interfaces: the stable adapter will resolve its mechanism according to 2025-11-25, while the candidate adapter can map the new capabilities. The cache will never share results beyond the allowed identity/tenant scope and will always respect revocation and policy changes.
10. Voice cloning and governance
10.1 Operating model
[REQ] Qwen3-TTS must use ref_audio and ref_text for zero-shot cloning and preserve a corporate voice. The Qwen3-TTS 1.7B Base model card confirms streaming, Spanish support and fast cloning from short audio 7. Repository documentation describes streaming generation and voice cloning 8.
Modes to evaluate:
- Full ICL — recommended for maximum fidelity: uses
ref_audioand an exact, aligned transcriptref_text. [REC] This will be the default candidate mode for approved corporate voices. - x-vector (
x_vector_only_mode=True): allows operation withoutref_text, but research reports lower timbre and prosody fidelity. [BENCH] It can be promoted for a profile only after meeting the defined quality and risk thresholds.
[REC] The client never sends master references during a session. The agent profile contains only voice_profile_id and voice_version; TTS resolves the reference from Voice Vault using service identity. The relay removes standard voices proposed by the client, and Orchestrator applies the authorized effective voice.
10.2 Lifecycle
- Request: voice owner, purpose, tenant, languages and usage period.
- Verifiable consent: document or authorization evidence; legal review for the applicable jurisdiction.
- Controlled ingestion: antivirus, normalization, noise assessment and aligned transcription.
- Evaluation: fidelity, intelligibility, pronunciation, stability and impersonation risk.
- Four-eyes approval: business owner plus security/legal or equivalent roles.
- Versioning: hash of
ref_audio, hash ofref_text, model, parameters and effective date. - Publication: only approved profiles can be used in production.
- Monitoring: usage by tenant/agent, anomalies and complaints.
- Revocation: immediately block new sessions; rotate cache; provide deletion evidence.
10.3 Controls
- Encryption in transit and at rest using managed keys.
- Isolation by tenant and environment; do not copy real voices into development.
- Short-lived signed URLs or direct reading through workload identity.
- Access logging without including audio in logs.
- Provenance markers or disclosure when required by corporate policy or regulation.
- Abuse tests: offensive text, fraud, social engineering, reference extraction and cross-tenant use.
- [BENCH] Compare Full ICL with
ref_audio+ref_textagainst text-free mode; do not assume equivalent quality. - [BENCH] Legal review of code and weight licenses before commercial use; collected evidence must not replace legal advice.
11. Security
11.1 Zones and trust view
11.2 Main threats and mitigations
+| Threat | Recommended mitigations |
|---|---|
| User impersonation | OIDC/OAuth, ephemeral tokens, issuer/audience validation, reauthentication for sensitive actions |
| Prompt injection from users or tool results | Instruction separation, tool allowlist, output validation, untrusted-content labeling, authorization outside the LLM |
| Tool poisoning or malicious schema | Approved-server registry, version/hash pinning, schema review, depth/size limits, cache with secure invalidation |
| Confused deputy | Propagate subject and tenant; token exchange; per-tool scopes; backend resource checks |
| Exfiltration through tools | Egress allowlist, DLP/classification, response minimization, limits and redaction before context insertion |
| Repeated writes | Idempotency key, hash-bound confirmation, execution ledger, no retries for unknown status |
| Voice theft | Vault, encryption, isolation, consent, revocation, no browser download |
| Session hijacking/replay | Short-lived connection tokens, nonce, expiration, identity binding, reconnection limits |
| Audio/GPU DoS | Quotas, bitrate/chunk limits, admission control, bounded queues, circuit breakers, shedding |
| Leaks through logs/traces | Redaction, attribute classification, sampling, limited retention, RBAC access |
| Lateral movement | Segmentation, mTLS, workload identities, deny-by-default and controlled egress |
11.3 Tool authorization
The decision evaluates:
permit = policy(
+ tenant,
+ principal,
+ agent_profile_version,
+ tool_name,
+ tool_schema_version,
+ action_class,
+ resource_scope,
+ argument_classification,
+ confirmation,
+ session_risk
+)The LLM can propose tool_name and arguments, but not principal, scopes, credentials, action classification or policy outcomes. [REC] Irreversible actions require specific human confirmation or a documented corporate exception.
11.4 Data and retention
Define a matrix by data class:
- raw audio;
- voice references;
- transcripts;
- prompt/context;
- model responses;
- tool arguments and results;
- telemetry and auditing.
For each class, approve purpose, legal basis, location, encryption, retention, deletion, export, access, training use and redaction. Pending: the requester did not supply corporate policies or jurisdictions; this document therefore sets no retention periods.
+12. Observability and initial SLOs
12.1 Telemetry model
Traces: one trace per turn with spans for capture, relay, STT, endpointing, LLM, policy, MCP, TTS, network and playout. The future RC proposes standardized names for propagating W3C Trace Context in _meta, but this form remains a candidate until ratification 4. In the stable adapter, Vocero will propagate traceparent through compatible internal metadata without requiring future domain changes.
Experience metrics
turn_endpointing_ms: acoustic endpoint to turn commit.stt_partial_msandstt_final_ms.llm_ttft_ms: commit to first token.tts_ttfa_ms: available text to first audio.ttfs_ms: end of speech to first playable sound.barge_in_stop_ms: interruption detection to effective silence.tool_latency_msby tool and outcome.turn_e2e_msby direct or tool-assisted path.- reconnection, cancellation, false barge-in and abandonment rates.
TTFT and end-to-end latency are recognized as distinct, relevant dimensions for voice/realtime workloads 9.
Quality metrics
- WER/CER by language, accent, noise and domain.
- early/late endpointing rate.
- intelligibility and human assessment of naturalness.
- speaker similarity and voice stability.
- accuracy of tool selection, arguments, policy and task completion.
Resource metrics
- GPU utilization and memory; KV cache; per-model queues; tokens/s; audio seconds/s.
- buffer depth and age; drops; retransmissions; jitter; playout underruns.
- active connections, admitted/rejected sessions and dependency errors.
12.2 Initial targets — not observed performance
+[REC/BENCH] These values are provisional engineering targets, not measured figures, guarantees or a contractual SLO. They must be ratified after benchmarking with real audio, networks, tools and concurrency.
| Indicator | Proposed initial target | Window/percentile | Ratification condition |
|---|---|---|---|
| Conversational service availability | ≥ 99,5 % | Monthly | Exclude agreed windows and define the GPU dependency |
Response audio start ttfs_ms | ≤ 1,200 ms | p95, direct turn | Target corporate network and warm session |
| Barge-in stopping | ≤ 300 ms | p95 | Measured through playout silence, not only ACK |
| Turns without platform errors | ≥ 99,0 % | Monthly | Separate dependency errors from invalid input |
| MCP read executions accepted by the gateway | ≥ 99,0 % | Monthly | Exclude policy rejections and downstream timeouts |
| Loss of accepted control events | 0 | Every window | Verified by sequence and ACK |
| Unauthorized voice use | 0 | Every window | Mandatory alert and audit |
Provisional budget for ttfs_ms for analysis, not a commitment:
endpointing + STT final + scheduling + LLM TTFT + TTS primer audio + transporte/playoutNo fixed milliseconds are allocated per stage before benchmarking; the dashboard must show contributions and percentiles separately.
+12.3 Initial alerts
- Availability and successful-turn burn rates.
- Sustained queue growth or buffer age.
- GPU OOM, model restart or degraded health.
- Divergence between
generation.canceland the last emitted chunk. - MCP error/latency by tool, tenant and dependency.
- Denied access or voice anomalies.
- Missing traces or uncontrolled cardinality.
13. Scalability, backpressure and resilience
13.1 Scaling
- Edge/Relay: horizontal scaling by connections; minimal state; drain before deployment.
- Orchestrator: horizontal scaling by session with a store lease; optional affinity as an optimization rather than a correctness requirement.
- MCP Gateway: horizontal scaling; per-downstream pools; per-tenant/tool bulkheads.
- Inference: admission control based on actual capacity, not only CPU; separate STT, LLM and TTS queues; prioritize cancellation and active audio.
- Voice cache: encrypted cache bounded by approved voices; invalidate on version changes/revocation.
13.2 Backpressure
[REC] Each link publishes high_watermark, low_watermark and an overflow policy. The LLM can produce text faster than TTS or the network can consume it; the system must throttle tokens, group pronounceable units and limit buffered audio. Realtime pipelines document the general problem of fast producers and slow consumers 10.
Policies:
- Control events never share a blocking queue with audio.
- Stale partial STT transcripts may be coalesced; final transcripts and commits may not.
- Discard unsynthesized tokens from a canceled generation.
- Discard generated but unplayed audio on interruption.
- A slow connection cannot consume unbounded memory.
- The client reports
buffered_msand playout; the server adjusts chunking or pauses. - Queues have deadlines; expired work is not processed merely for completeness.
13.3 Resilience
- Per-stage timeouts and a propagated total deadline.
- Circuit breakers by dependency and model.
- Retries with jitter only for idempotent operations.
- Bulkheads by tenant and tool to avoid cascading failures.
- Separate process health, model readiness and capacity signals.
- Verified warmup before accepting sessions.
- Deploy with connection draining and safe cancellation.
- Configuration snapshots and profile/model rollback.
- Runbooks for GPU OOM, STT/TTS degradation, slow MCP, unavailable stores and voice compromise.
[BENCH] The ability to host the entire pipeline simultaneously on one GB10, and the number of sustainable sessions, must be measured. Parakeet TDT 0.6B v3 has documented streaming support and timestamps 11; this validates functional capability rather than the complete system's concurrent capacity.
+14. Phased implementation plan
Phases include no dates or durations because team availability, environments, procurement and change windows were not supplied. Each phase ends with an explicit decision to proceed, correct or stop.
+Phase 0 — Discovery, contracts and risks
Deliverables
- Inventory of functional requirements, data, tenants, roles, tools and sensitive actions.
- Initial ADRs and threat model.
- Event catalog v1 and audio/control schemas.
- License matrix and voice governance.
- Benchmark corpus definition and measurement plan.
Dependencies
- Access to auth/profile APIs and the customer's MCP definition.
- Security, legal, business and operations owners.
- GB10 hardware and versioned model artifacts.
Acceptance criteria
- Labeled and approved requirements; outstanding items have owners.
- No secrets or real voices required for early development.
- ADR-001 through ADR-006 have status
ProposedorAccepted. - Critical threats have an associated control and test.
Risks
- Ambiguous identity/roles.
- Unresolved licensing or voice consent.
- MCP tools without idempotency or risk classification.
Phase 1 — Local vertical slice without write tools
Deliverables
- Web client + WebSocket, Relay, Orchestrator and Inference Gateway.
- Streaming STT → LLM → TTS with an authorized test voice.
- State machine, IDs, cancellation and structured logs.
- Deterministic STT/LLM/TTS simulators for CI.
Dependencies
- Loadable models and compatible runtime.
- Service certificates and an isolated environment.
Acceptance criteria
- Reproducible normal conversation.
- Reconnection does not duplicate turns.
- Canceling a generation prevents late chunks from reaching playout.
- The client-requested voice does not replace the effective voice.
- First latency and resource baseline recorded.
Risks
- GPU contention.
- Endpointing latency.
- Runtime instability or FP16 incompatibility.
Phase 2 — Barge-in, backpressure and robust sessions
Deliverables
- Priority control queue and
output.stop_and_flush. - Playout ACK/cursor; history truncation.
- Buffer limits, shedding, deadlines and circuit breakers.
- Tests with noise, echo, interjections and degraded networks.
Dependencies
- Per-stage instrumentation.
- Dataset of representative conversations and acoustic conditions.
Acceptance criteria
- No stale-generation audio after the approved interruption limit.
- No unbounded memory growth with a slow client.
- No duplicate response after reconnection/cancellation.
- False barge-in measured and within the threshold defined by the business.
Risks
- Echo triggers VAD.
- Inaccurate playout cursor.
- Races between cancellation, TTS and network delivery.
Phase 3 — Read-only MCP gateway
Deliverables
StableAdapter_2025_11_25.- Allowlisted discovery, schema validation and read-only
tools/call. - Policy Engine, delegated identity, limits, auditing and idempotency.
- Simulated MCP server and contract tests.
Dependencies
- Customer MCP server and authorization schema.
- Tool and data classification.
Acceptance criteria
- A tool outside the allowlist is rejected before reaching downstream.
- Invalid arguments and results do not reach the LLM/systems.
- Every tool call correlates with user, session, turn and policy.
- Timeouts do not cause unsafe retries.
Risks
- Prompt injection in results.
- Overprivileged credentials.
- Unstable downstream contracts.
Phase 4 — Writes, confirmation and voice governance
Deliverables
- Confirmation flows bound to hashes and idempotency keys.
- Action ledger and unknown status.
- Voice Vault, approval workflow, revocation and secure cache.
- Abuse tests and incident runbooks.
Dependencies
- Legal approval of voices/licenses.
- Idempotency semantics in enterprise systems.
Acceptance criteria
- No sensitive action executes without the required policy/confirmation.
- Repeated events do not repeat side effects.
- Revoking a voice blocks new synthesis and clears caches.
- Cross-tenant access fails and is audited.
Risks
- Non-idempotent actions.
- Incomplete consent.
- Voice exfiltration or misuse.
Phase 5 — Preproduction, SLOs and resilience
Deliverables
- Load, soak, chaos and recovery tests.
- Dashboards, alerts, ratified SLOs and runbooks.
- Capacity model and admission limits.
- DR/backup for configuration, auditing and Voice Vault.
- Security review and go-live checklist.
Dependencies
- Representative traffic/profiles.
- Production-equivalent environment.
- Defined on-call model and ownership.
Acceptance criteria
- Experience targets ratified or renegotiated with evidence.
- Demonstrated recovery from critical failures.
- No open critical vulnerabilities.
- Operations can detect, diagnose and mitigate runbook scenarios.
Risks
- Single GB10 as a SPOF.
- Insufficient capacity under a real traffic mix.
- Differences between test and field networks.
Phase 6 — Progressive rollout and evolution
Deliverables
- Canary by user/tenant and automatic rollback.
- Post-pilot review of quality, latency, security and task completion.
- WebRTC roadmap if WebSocket does not meet targets.
CandidateAdapter_2026_07_28only if the specification has been ratified.
Acceptance criteria
- Canary without SLO/security regression.
- Validated support and operational escalation.
- MCP migration passes the conformance suite and promotion ADR.
15. Testing strategy
15.1 Pyramid
Unit tests
- state machine, sanitization, policy, error mapping, deduplication, expiration and redaction.
Contract tests
- Event JSON Schema, backward compatibility, MCP adapters, profile and identity APIs.
Integration tests
- Real STT/LLM/TTS; simulated and real MCP; Voice Vault; KMS; stores.
End-to-end
- audio to audio with measured playout, direct paths, tools, confirmation, cancellation and reconnection.
Load and soak
- concurrent sessions, mixed turns, long audio, slow clients, slow tools, connection churn and warm/cold models.
Chaos
- dead process, GPU OOM, slow store, network loss, late response, expired certificate, partial dependency failure and restart during a tool call.
Security
- authn/authz, tenant isolation, replay, prompt injection, schema bombs, SSRF, egress, secrets, DLP, voice abuse and supply chain.
15.2 Minimum scenario matrix
| Dimension | Cases |
|---|---|
| Audio | silence, noise, echo, clipping, variable bitrate, distant microphone |
| Speech | target accents, speed, numbers, proper names, domain terms |
| Turns | long pause, overlap, backchannel, interruption, push-to-talk |
| Network | latency, jitter, loss, reconnection, slow client |
| Tools | success, empty, invalid, timeout, 4xx/5xx, confirmed side effect, unknown status |
| Models | cold start, OOM, late response, excessive tokens/audio, cancellation |
| Security | prohibited tool, insufficient scope, injection, cross-tenant access, revoked voice |
15.3 Test oracles and reproducibility
- Versioned test audio with consent and labels.
- Deterministic fakes for races and errors.
- Recorded generation seeds/parameters where applicable.
- Golden transcripts and schemas with defined acoustic tolerances.
- Capture monotonic timestamps at every boundary.
- Report percentiles and conditions, not only averages.
- A/B comparison of VAD, chunking and voice configurations.
16. Decision matrix and ADRs
| ADR | Decision | Initial status | Alternatives | Evidence for acceptance |
|---|---|---|---|---|
| ADR-001 | Decoupled STT → LLM → TTS pipeline | Accepted by requirement | Native S2S model | Benchmark and per-component control |
| ADR-002 | Initial WebSocket; WebRTC as an evolution | Proposed | WebRTC from MVP; both | Network testing and operational complexity |
| ADR-003 | Separate Relay and Session Orchestrator | Proposed | Monolithic service | Load profile, isolation and maintainability |
| ADR-004 | Stable MCP 2025-11-25 + versioned adapters | Proposed | Adopt a future RC | Ratification, conformance and compatibility |
| ADR-005 | Voice Vault and server-side references | Proposed | Send ref_audio per session | Threat model, cache latency and governance |
| ADR-006 | External ephemeral state with TTL and leases | Proposed | Local memory/sticky sessions | Recovery and load |
| ADR-007 | Priority control queue for barge-in | Proposed | Single FIFO queue | Race tests and interruption target |
| ADR-008 | Policy Engine outside the LLM | Proposed | Prompt-only controls | Bypass tests and auditing |
| ADR-009 | Confirmation + idempotency for writes | Proposed | Immediate execution | Downstream semantics and UX |
| ADR-010 | One GB10 in the MVP with admission control | Proposed | Cluster from the start | Capacity/cost/risk benchmark |
| ADR-011 | OpenTelemetry end-to-end | Proposed | Uncorrelated logs | Trace coverage and cost/cardinality |
| ADR-012 | Versioned event contracts v1 | Proposed | Ad hoc provider payloads | Compatibility tests |
Each ADR must include context, decision, alternatives, consequences, risks, status and review condition. Reversible decisions may start as Proposed; decisions affecting security, data, licenses or external compatibility require explicit approval.
17. Outstanding items and open questions
Product and experience
- Which channels are included: browser only, mobile, telephony or all?
- Which languages, accents, vocabulary and acoustic conditions take priority?
- What behavior is expected when voice fails: text, an alternative voice or closure?
- Which actions require verbal confirmation, visual confirmation or stronger authentication?
Identity and authorization
- Actual contract for
/api/v1/auth/user-by-phoneand authentication mechanism. - Roles, tenants, scopes and data-isolation model.
- Token exchange/delegation to MCP and downstream systems.
MCP
- Inventory of servers, tools, schemas, side effects, idempotency and downstream SLOs.
- Stable version actually supported by the customer's server.
- Discovery cache/refresh strategy.
- Future ratification of
2026-07-28and SDK/conformance-suite availability.
Inference and capacity
- Exact runtime for LLaMA, memory allocation and KV cache policy.
- Sustainable STT + LLM + TTS concurrency on GB10.
- Effects of context length, agent prompts and tool results on TTFT.
- Cold start, model loading, fragmentation and OOM recovery.
- Parakeet quality on real audio and domain content.
Voice and legal considerations
- Owner and consent evidence for every voice.
- Applicable licensing for weights, code, redistribution and commercial use.
- Disclosure, retention, localization and revocation-right rules.
- Fidelity/security metric and acceptance threshold.
Operations
- Environments, topology, certificates, secrets, KMS, stores and backups.
- RTO/RPO and on-call model.
- Logging, sampling, redaction and retention policy.
- Spare-node capacity or alternative service.
18. Consolidated risks
| Risk | Initial likelihood | Impact | Treatment |
|---|---|---|---|
| Complete pipeline exceeds GB10 capacity | Unknown | High | Early benchmark; admission control; separation options |
| Late or false barge-in | Unknown | High | Priority queue, AEC, playout ACK, noise corpus |
| MCP permits an improper action | Medium until tested | Critical | External policy, scopes, confirmation, auditing, idempotency |
| Voice compromise or unauthorized use | Medium until governed | Critical | Voice Vault, consent, isolation, revocation |
| Dependency on a candidate MCP version | Avoidable | High | Stable adapter by default and feature flag |
| Tool result injects instructions | Medium | High | Treat content as untrusted; sanitize, limit and evaluate |
| Inconsistent state after reconnection | Medium | High | IDs, versioning, leases, deduplication and cancellation |
| Observability leaks PII | Medium | High | Redaction, attribute allowlist, access and retention |
| Single inference node fails | High in the initial topology | High | Runbook, spare/failover, degradation and communication |
| Incomplete licensing/consent | Unknown | Critical | Legal gate before production |
Nonnumeric likelihoods reflect missing data; update them after testing and organizational review.
+19. Pilot readiness criteria
The solution is ready for a pilot only when:
- requirements and critical ADRs are approved;
- the stable MCP path is tested and the future candidate remains isolated;
- the end-to-end benchmark demonstrates capacity for the pilot load;
- barge-in and backpressure pass acoustic and network tests;
- sensitive tools have policy, confirmation and idempotency;
- the voice is authorized, versioned and revocable;
- security has no open critical findings;
- dashboards, alerts and runbooks enable service operation;
- provisional SLOs were ratified or replaced with evidence;
- tested application, profile, model and configuration rollback exists.
20. Sources
- [1] OpenAI — Realtime and audio — https://developers.openai.com/api/docs/guides/realtime
- [2] Microsoft Azure — Realtime audio — https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/realtime-audio
- [3] LiveKit — Turns overview — https://docs.livekit.io/agents/logic/turns/
- [4] Model Context Protocol — 2026-07-28 Release Candidate — https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/
- [5] Model Context Protocol — Transport specification 2025-11-25 — https://modelcontextprotocol.io/specification/2025-11-25/basic/transports
- [6] Model Context Protocol — Architecture overview — https://modelcontextprotocol.io/docs/learn/architecture
- [7] Qwen — Qwen3-TTS 12Hz 1.7B Base model card — https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
- [8] QwenLM — Qwen3-TTS repository — https://github.com/QwenLM/Qwen3-TTS
- [9] Telnyx — Inference Benchmark: TTFT vs E2E Latency — https://telnyx.com/resources/inference-benchmark-ttft-vs-e2e
- [10] Arun Baby — Real-Time Agent Pipelines — https://www.arunbaby.com/ai-agents/0016-real-time-agent-pipelines/
- [11] NVIDIA — Parakeet TDT 0.6B v3 model card — https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
Appendix A — Implementation checklist
- [ ] Requirements [REQ], recommendations [REC] and assumptions [BENCH] reviewed.
- [ ] Critical ADRs approved.
- [ ] Models and hashes pinned; SBOM and licenses reviewed.
- [ ] Identity, roles, scopes and tenants defined.
- [ ] Event schemas v1 published and contract-tested.
- [ ] Stable MCP adapter tested against
2025-11-25. - [ ] Candidate MCP adapter disabled by default.
- [ ] Voice Vault, consent, rotation and revocation operational.
- [ ] Admission control and buffer limits configured.
- [ ] Barge-in measured through actual playout.
- [ ] Tool confirmation and idempotency verified.
- [ ] Telemetry redacted and correlated.
- [ ] SLOs ratified through benchmarking.
- [ ] Runbooks and rollback rehearsed.
- [ ] Security review and go-live completed.