Vocero S2S

Target technical architecture and implementation plan

Document status: proposal for technical review
Source cutoff date: July 20, 2026
Scope: a decoupled, local Speech-to-Speech conversational platform extensible through MCP
Audience: architecture, platform, security, MLOps, development, QA and operations

Traceability convention

• [REQ] Supplied requirement: comes directly from the Vocero S2S description provided by the requester.
• [REC] Recommended decision: a design proposal that must be approved through an ADR.
• [BENCH] Pending assumption: a hypothesis requiring measurement, load testing, legal assessment or validation in the target environment.

1. Executive summary

Vocero S2S is proposed as a real-time conversational platform separating four responsibilities: web experience, session control, speech and language inference, and enterprise integration through Model Context Protocol (MCP). [REQ] The initial implementation must use Parakeet TDT 0.6B for STT/VAD, LLaMA 3.1 8B Instruct for reasoning and tool calls, Qwen3-TTS 12Hz 1.7B Base in FP16 for synthesis and zero-shot cloning, and an NVIDIA GB10 node initially accessible through port 12345.

The target architecture retains this decoupled pipeline while avoiding placing all logic in the WebSocket connection. [REC] The proposal includes a Control Plane on the UI & Relay server for authentication, authorization, agent profiles, voice policy, MCP orchestration and auditing; a Realtime Data Plane for audio and events with ephemeral session state and priority cancellation; and a local Inference Plane that exposes versioned adapters for STT, LLM and TTS. This separation allows models, transport or infrastructure to be replaced without changing business contracts.

For web clients, WebRTC offers operational advantages for media on variable networks, while WebSocket remains appropriate for server-to-server links and the proposed first version. Realtime platform documentation recommends WebRTC for browser/media and WebSocket for server pipelines 1. [REC] The plan therefore adopts WebSocket as the initial compatible transport and defines WebRTC as a controlled evolution rather than an MVP prerequisite.

Interruption management is a first-class capability: when user speech is detected during a response, the system must cancel still-reversible generation, stop audio, flush unplayed buffers and truncate history to what was actually heard. Realtime systems document this pattern through cancellation and truncation of conversation items 2, as do agent frameworks that adjust history to the audio actually played 3.

MCP is integrated through an enterprise gateway with policy, schema validation, delegated identity, limits, auditing and version adapters. Critical timing decision: as of July 20, 2026, the announcement for 2026-07-28 explicitly describes a Release Candidate whose final standard is expected on July 28, 2026 and includes breaking changes 4. Therefore:

1. [REC] the initial stable compatibility version will be 2025-11-25, verified in the official transport documentation 5; this decision follows the requested source cutoff and does not contradict the stateless technical direction proposed by the RC;
2. [REC] 2026-07-28 will be implemented only behind an experimental adapter and feature flag;
3. [REC] the domain design will remain ready for the RC's stateless model: self-contained requests, application state through explicit handles, and routing independent of affinity;
4. [REC] promotion will require official ratification, conformance tests and an approved migration;
5. this document will not assume it is a final standard.

Finally, running STT, LLM and TTS concurrently on a single GB10 has not been demonstrated through a benchmark of the complete pipeline. [BENCH] The first engineering deliverable must be a reproducible harness measuring end-to-end latency, memory, bandwidth, degradation under concurrency, interruption accuracy and cloning quality before committing to capacity or a contractual SLO.

2. Scope

2.1 Objectives

• [REQ] Deliver bidirectional, low-latency voice conversation from a browser.
• [REQ] Identify users through the customer API and personalize sessions by identity or role.
• [REQ] Dynamically load the agent profile, instructions, rules and tools.
• [REQ] Preserve a corporate cloned voice against defaults sent by the client.
• [REQ] Run STT, reasoning, tool calls and TTS in a decoupled pipeline.
• [REQ] Integrate enterprise systems through an MCP gateway without changing the core.
• [REC] Include barge-in, cancellation, backpressure, timeouts, isolation, traceability and failure recovery as core capabilities.
• [REC] Maintain versioned contracts that allow transports, models, runtimes and MCP servers to be replaced.
• [REC] Provide operational evidence for sizing, SLOs and auditing.

2.2 Non-goals

• Train STT, LLM or TTS models from scratch.
• Replace ERP, CRM, catalogs or other systems of record.
• Allow the LLM to execute directly against databases or APIs without a gateway and policy.
• Make the browser a repository for enterprise credentials or master voice material.
• Guarantee capacity, latency or concurrency before benchmarking on GB10.
• Declare MCP version 2026-07-28 a final standard before ratification.
• Design a complete contact center, universal regulatory recording or SIP telephony at this stage; these may be added as later domains.

3. Architecture principles

1. Separate control and media. Audio has real-time requirements; identity, policy and auditing require consistency and traceability.
2. Contracts before implementations. Each boundary publishes versioned events and schemas; models remain behind adapters.
3. Priority interruption. Cancellation and audio stopping take precedence over generating more content.
4. Explicit state with ownership. Vocero owns conversational sessions; the protocol adapter or explicit handles own MCP state.
5. Zero trust between planes. Each hop authenticates the service, authorizes the action, limits scope and propagates audited identity.
6. Least privilege for tools. A tool receives only the strictly necessary arguments, identity and scopes.
7. Privacy by default. Audio, transcripts, prompts, responses and voices are sensitive data; retention is explicit and minimized.
8. Conversational failure handling. Technical failures become deterministic events and safe responses without leaking internal details.
9. Measure before promising. Stage latency and percentiles govern the experience, rather than isolated averages.
10. Versioned MCP compatibility. The gateway negotiates a stable version and encapsulates future candidates.

4. Context and container view

4.1 Logical view by plane

Experience Plane

• Web client, microphone capture, echo cancellation, playback and consent controls.
• Retains only required visual state and cursors; does not retain secrets or master voice material.

Realtime Data Plane

• WebSocket or WebRTC termination, media relay and event multiplexing.
• Enforces buffer, sequence and size limits; discards stale data when generation changes.
• Separates a high-priority control queue —cancel, barge_in, close— from the audio/data queue.

Conversation Control Plane

• Resolves identity, profile, permissions and effective configuration.
• Maintains the session and turn state machine.
• Builds context, coordinates inference and decides whether a tool can be executed, confirmed or rejected.

Inference Plane

• Hides runtime details and exposes uniform capabilities: transcribe, generate, synthesize, cancel, health, capacity.
• Correlates each output with session_id, turn_id and generation_id to discard late responses.

Enterprise Integration Plane

• MCP gateway with version adapters, allowlists, validation, authorization, service secrets, idempotency and auditing.
• No tool response enters the context without size, classification and sanitization checks.

5. Component responsibilities

6. Deployment view

[REC] Port 12345 must be internal, protected by a firewall, mTLS and service authentication; it must not be exposed directly to the Internet. [REC] Relay and Orchestrator are deployed with at least two logical instances in production. This does not require physical high availability from the first day of the MVP, but contracts must not depend exclusively on local memory.

[BENCH] The GB10 is initially a single failure and capacity domain. Before production, define the spare-node strategy, admission queue, session limit, shedding policy and model recovery procedure.

7. State machine and session management

7.1 Proposed states

7.2 Identifiers

• connection_id: transport connection; may change on reconnection.
• session_id: stable, opaque conversational session.
• turn_id: committed user input.
• generation_id: agent response; changes after cancellation or regeneration.
• tool_call_id: unique invocation; must be idempotent when supported by the tool.
• voice_profile_id + voice_version: immutable reference to the authorized voice.
• trace_id: distributed correlation; does not replace functional identifiers.

7.3 Effective session state

[REC] Calculated in this order, without allowing the client to escalate privileges:

1. secure platform defaults;
2. tenant policy;
3. versioned agent profile;
4. authorized user attributes;
5. allowed session preferences;
6. explicitly allowlisted per-response overrides.

The sanitizer for session.update removes or rejects fields that could replace voice_profile_id, tools, system instructions, limits or scopes. Any difference between requested and effective configuration is recorded in a session.configuration_applied event with coded reasons.

[REC] Ephemeral state is stored with TTL and optimistic version control. Resumption requires a short-lived opaque token, the same identity and a profile-version check. Uncommitted audio and generation jobs are not resumed: they are canceled and the client returns to LISTENING.

8. Execution flows

8.1 Normal conversational and MCP flow

8.2 Tool call with confirmation

1. The LLM emits intent, never direct execution.
2. Orchestrator validates the name against the allowlist and current schema.
3. Policy Engine classifies the action: read, reversible write, irreversible action or prohibited.
4. If consent is required, emit tool.confirmation_required with a readable summary and nonsensitive fields.
5. The user confirms; create a confirmation_id bound to a hash of tool + arguments + identity + expiration.
6. MCP Gateway executes with a delegated token of minimum scope.
7. Validate, limit and classify the result before returning it to the LLM.
8. Record the decision, not secrets or complete sensitive payloads.

For stable compatibility 2025-11-25, the adapter maintains the protocol lifecycle and session according to the official transport 5. RC 2026-07-28 proposes eliminating the handshake and protocol session, using self-contained requests, routing headers and InputRequiredResult; these capabilities remain only in the experimental adapter until ratification 4.

8.3 Barge-in

Rules:

• An interruption event bypasses token and audio queues.
• Each subsequent chunk from G1 is discarded because of a stale generation_id .
• The client cursor is the source of truth for played audio; when missing, use the last conservative ACK.
• A read-only tool can be canceled; an already confirmed, irreversible action is not undone by barge-in. Report its status and avoid repeating it.
• Apply AEC and adaptive thresholds to reduce self-interruptions; test brief interjections as a specific case.

8.4 Errors and degradation

9. Illustrative contracts and events

[REC] The following schemas are illustrative. They must become versioned JSON Schema with compatibility tests and size limits. They do not represent a final contract.

9.1 Common envelope

{
  "specversion": "1.0",
  "type": "vocero.audio.input.delta.v1",
  "source": "web-client",
  "id": "evt_opaque",
  "time": "RFC3339 timestamp",
  "subject": "session/{session_id}",
  "traceparent": "00-...",
  "data": {
    "session_id": "ses_opaque",
    "turn_id": "turn_opaque",
    "sequence": 42,
    "payload": {}
  }
}

9.2 Minimum event catalog

Client → platform

• session.connect.v1
• session.update_requested.v1
• audio.input.delta.v1
• audio.input.commit.v1
• input.speech_started.v1
• output.playout_progress.v1
• generation.cancel_requested.v1
• tool.confirmation_submitted.v1
• session.close.v1

Platform → client

• session.configuration_applied.v1
• transcript.partial.v1
• transcript.final.v1
• response.text_delta.v1
• response.audio_delta.v1
• response.completed.v1
• output.stop_and_flush.v1
• tool.confirmation_required.v1
• tool.status.v1
• error.v1

9.3 Example of session.update_requested

{
  "type": "vocero.session.update_requested.v1",
  "data": {
    "session_id": "ses_opaque",
    "client_capabilities": {
      "audio_codecs": ["opus", "pcm16"],
      "supports_playout_ack": true
    },
    "preferences": {
      "locale": "es-CL",
      "turn_mode": "server_vad"
    }
  }
}

Fields such as system_prompt, tools, mcp_credentials, ref_audio, ref_text or voice_profile_id are not accepted through this public contract. If received, the relay rejects or removes them according to policy and records the reason.

9.4 Example of internal tool intent

{
  "tool_call_id": "tc_opaque",
  "session_id": "ses_opaque",
  "turn_id": "turn_opaque",
  "tool": {
    "name": "production.report.get",
    "schema_version": "1.2.0"
  },
  "arguments": {
    "site_id": "site_authorized",
    "period": "value_from_user_or_context"
  },
  "execution": {
    "mode": "read",
    "idempotency_key": "idem_opaque",
    "deadline_ms": "configured_value"
  },
  "principal": {
    "subject_ref": "opaque_subject",
    "tenant_id": "tenant_opaque",
    "scopes": ["report:read"]
  }
}

9.5 Errors

{
  "type": "vocero.error.v1",
  "data": {
    "code": "MCP_TOOL_TIMEOUT",
    "category": "dependency",
    "retryable": false,
    "user_message_key": "tool.temporarily_unavailable",
    "correlation_id": "corr_opaque",
    "details_ref": "internal_log_reference"
  }
}

Public messages do not contain stack traces, internal endpoints, prompts, tokens, sensitive arguments or secret names.

9.6 MCP compatibility

McpAdapter
├── StableAdapter_2025_11_25   # default
├── CandidateAdapter_2026_07_28 # feature flag, not enabled in production by default
└── ConformanceHarness
    ├── discovery
    ├── tools/list
    ├── tools/call
    ├── cancellation
    ├── schema validation
    ├── auth propagation
    └── error mapping

The official MCP architecture separates a JSON-RPC data layer from a transport layer and includes lifecycle, primitives and notifications 6. [REC] Vocero will preserve this separation inside the adapter so transport details do not contaminate conversational logic.

Candidate capabilities being prepared but not activated as a final standard: the RC 2026-07-28 introduces server/discover, metadata ttlMs and cacheScope in list/read results, and headers Mcp-Method/Mcp-Name for routing 4. [REC] The gateway will model discovery and caching through neutral interfaces: the stable adapter will resolve its mechanism according to 2025-11-25, while the candidate adapter can map the new capabilities. The cache will never share results beyond the allowed identity/tenant scope and will always respect revocation and policy changes.

10. Voice cloning and governance

10.1 Operating model

[REQ] Qwen3-TTS must use ref_audio and ref_text for zero-shot cloning and preserve a corporate voice. The Qwen3-TTS 1.7B Base model card confirms streaming, Spanish support and fast cloning from short audio 7. Repository documentation describes streaming generation and voice cloning 8.

Modes to evaluate:

• Full ICL — recommended for maximum fidelity: uses ref_audio and an exact, aligned transcript ref_text . [REC] This will be the default candidate mode for approved corporate voices.
• x-vector (x_vector_only_mode=True): allows operation without ref_text, but research reports lower timbre and prosody fidelity. [BENCH] It can be promoted for a profile only after meeting the defined quality and risk thresholds.

[REC] The client never sends master references during a session. The agent profile contains only voice_profile_id and voice_version; TTS resolves the reference from Voice Vault using service identity. The relay removes standard voices proposed by the client, and Orchestrator applies the authorized effective voice.

10.2 Lifecycle

1. Request: voice owner, purpose, tenant, languages and usage period.
2. Verifiable consent: document or authorization evidence; legal review for the applicable jurisdiction.
3. Controlled ingestion: antivirus, normalization, noise assessment and aligned transcription.
4. Evaluation: fidelity, intelligibility, pronunciation, stability and impersonation risk.
5. Four-eyes approval: business owner plus security/legal or equivalent roles.
6. Versioning: hash of ref_audio, hash of ref_text, model, parameters and effective date.
7. Publication: only approved profiles can be used in production.
8. Monitoring: usage by tenant/agent, anomalies and complaints.
9. Revocation: immediately block new sessions; rotate cache; provide deletion evidence.

10.3 Controls

• Encryption in transit and at rest using managed keys.
• Isolation by tenant and environment; do not copy real voices into development.
• Short-lived signed URLs or direct reading through workload identity.
• Access logging without including audio in logs.
• Provenance markers or disclosure when required by corporate policy or regulation.
• Abuse tests: offensive text, fraud, social engineering, reference extraction and cross-tenant use.
• [BENCH] Compare Full ICL with ref_audio + ref_text against text-free mode; do not assume equivalent quality.
• [BENCH] Legal review of code and weight licenses before commercial use; collected evidence must not replace legal advice.

11. Security

11.1 Zones and trust view

11.2 Main threats and mitigations

11.3 Tool authorization

The decision evaluates:

permit = policy(
  tenant,
  principal,
  agent_profile_version,
  tool_name,
  tool_schema_version,
  action_class,
  resource_scope,
  argument_classification,
  confirmation,
  session_risk
)

The LLM can propose tool_name and arguments, but not principal, scopes, credentials, action classification or policy outcomes. [REC] Irreversible actions require specific human confirmation or a documented corporate exception.

11.4 Data and retention

Define a matrix by data class:

• raw audio;
• voice references;
• transcripts;
• prompt/context;
• model responses;
• tool arguments and results;
• telemetry and auditing.

For each class, approve purpose, legal basis, location, encryption, retention, deletion, export, access, training use and redaction. Pending: the requester did not supply corporate policies or jurisdictions; this document therefore sets no retention periods.

12. Observability and initial SLOs

12.1 Telemetry model

Traces: one trace per turn with spans for capture, relay, STT, endpointing, LLM, policy, MCP, TTS, network and playout. The future RC proposes standardized names for propagating W3C Trace Context in _meta, but this form remains a candidate until ratification 4. In the stable adapter, Vocero will propagate traceparent through compatible internal metadata without requiring future domain changes.

Experience metrics

• turn_endpointing_ms: acoustic endpoint to turn commit.
• stt_partial_ms and stt_final_ms.
• llm_ttft_ms: commit to first token.
• tts_ttfa_ms: available text to first audio.
• ttfs_ms: end of speech to first playable sound.
• barge_in_stop_ms: interruption detection to effective silence.
• tool_latency_ms by tool and outcome.
• turn_e2e_ms by direct or tool-assisted path.
• reconnection, cancellation, false barge-in and abandonment rates.

TTFT and end-to-end latency are recognized as distinct, relevant dimensions for voice/realtime workloads 9.

Quality metrics

• WER/CER by language, accent, noise and domain.
• early/late endpointing rate.
• intelligibility and human assessment of naturalness.
• speaker similarity and voice stability.
• accuracy of tool selection, arguments, policy and task completion.

Resource metrics

• GPU utilization and memory; KV cache; per-model queues; tokens/s; audio seconds/s.
• buffer depth and age; drops; retransmissions; jitter; playout underruns.
• active connections, admitted/rejected sessions and dependency errors.

12.2 Initial targets — not observed performance

[REC/BENCH] These values are provisional engineering targets, not measured figures, guarantees or a contractual SLO. They must be ratified after benchmarking with real audio, networks, tools and concurrency.

Provisional budget for ttfs_ms for analysis, not a commitment:

endpointing + STT final + scheduling + LLM TTFT + TTS primer audio + transporte/playout

No fixed milliseconds are allocated per stage before benchmarking; the dashboard must show contributions and percentiles separately.

12.3 Initial alerts

• Availability and successful-turn burn rates.
• Sustained queue growth or buffer age.
• GPU OOM, model restart or degraded health.
• Divergence between generation.cancel and the last emitted chunk.
• MCP error/latency by tool, tenant and dependency.
• Denied access or voice anomalies.
• Missing traces or uncontrolled cardinality.

13. Scalability, backpressure and resilience

13.1 Scaling

• Edge/Relay: horizontal scaling by connections; minimal state; drain before deployment.
• Orchestrator: horizontal scaling by session with a store lease; optional affinity as an optimization rather than a correctness requirement.
• MCP Gateway: horizontal scaling; per-downstream pools; per-tenant/tool bulkheads.
• Inference: admission control based on actual capacity, not only CPU; separate STT, LLM and TTS queues; prioritize cancellation and active audio.
• Voice cache: encrypted cache bounded by approved voices; invalidate on version changes/revocation.

13.2 Backpressure

[REC] Each link publishes high_watermark, low_watermark and an overflow policy. The LLM can produce text faster than TTS or the network can consume it; the system must throttle tokens, group pronounceable units and limit buffered audio. Realtime pipelines document the general problem of fast producers and slow consumers 10.

Policies:

1. Control events never share a blocking queue with audio.
2. Stale partial STT transcripts may be coalesced; final transcripts and commits may not.
3. Discard unsynthesized tokens from a canceled generation.
4. Discard generated but unplayed audio on interruption.
5. A slow connection cannot consume unbounded memory.
6. The client reports buffered_ms and playout; the server adjusts chunking or pauses.
7. Queues have deadlines; expired work is not processed merely for completeness.

13.3 Resilience

• Per-stage timeouts and a propagated total deadline.
• Circuit breakers by dependency and model.
• Retries with jitter only for idempotent operations.
• Bulkheads by tenant and tool to avoid cascading failures.
• Separate process health, model readiness and capacity signals.
• Verified warmup before accepting sessions.
• Deploy with connection draining and safe cancellation.
• Configuration snapshots and profile/model rollback.
• Runbooks for GPU OOM, STT/TTS degradation, slow MCP, unavailable stores and voice compromise.

[BENCH] The ability to host the entire pipeline simultaneously on one GB10, and the number of sustainable sessions, must be measured. Parakeet TDT 0.6B v3 has documented streaming support and timestamps 11; this validates functional capability rather than the complete system's concurrent capacity.

14. Phased implementation plan

Phases include no dates or durations because team availability, environments, procurement and change windows were not supplied. Each phase ends with an explicit decision to proceed, correct or stop.

Phase 0 — Discovery, contracts and risks

Deliverables

• Inventory of functional requirements, data, tenants, roles, tools and sensitive actions.
• Initial ADRs and threat model.
• Event catalog v1 and audio/control schemas.
• License matrix and voice governance.
• Benchmark corpus definition and measurement plan.

Dependencies

• Access to auth/profile APIs and the customer's MCP definition.
• Security, legal, business and operations owners.
• GB10 hardware and versioned model artifacts.

Acceptance criteria

• Labeled and approved requirements; outstanding items have owners.
• No secrets or real voices required for early development.
• ADR-001 through ADR-006 have status Proposed or Accepted.
• Critical threats have an associated control and test.

Risks

• Ambiguous identity/roles.
• Unresolved licensing or voice consent.
• MCP tools without idempotency or risk classification.

Phase 1 — Local vertical slice without write tools

Deliverables

• Web client + WebSocket, Relay, Orchestrator and Inference Gateway.
• Streaming STT → LLM → TTS with an authorized test voice.
• State machine, IDs, cancellation and structured logs.
• Deterministic STT/LLM/TTS simulators for CI.

Dependencies

• Loadable models and compatible runtime.
• Service certificates and an isolated environment.

Acceptance criteria

• Reproducible normal conversation.
• Reconnection does not duplicate turns.
• Canceling a generation prevents late chunks from reaching playout.
• The client-requested voice does not replace the effective voice.
• First latency and resource baseline recorded.

Risks

• GPU contention.
• Endpointing latency.
• Runtime instability or FP16 incompatibility.

Phase 2 — Barge-in, backpressure and robust sessions

Deliverables

• Priority control queue and output.stop_and_flush.
• Playout ACK/cursor; history truncation.
• Buffer limits, shedding, deadlines and circuit breakers.
• Tests with noise, echo, interjections and degraded networks.

Dependencies

• Per-stage instrumentation.
• Dataset of representative conversations and acoustic conditions.

Acceptance criteria

• No stale-generation audio after the approved interruption limit.
• No unbounded memory growth with a slow client.
• No duplicate response after reconnection/cancellation.
• False barge-in measured and within the threshold defined by the business.

Risks

• Echo triggers VAD.
• Inaccurate playout cursor.
• Races between cancellation, TTS and network delivery.

Phase 3 — Read-only MCP gateway

Deliverables

• StableAdapter_2025_11_25.
• Allowlisted discovery, schema validation and read-only tools/call .
• Policy Engine, delegated identity, limits, auditing and idempotency.
• Simulated MCP server and contract tests.

Dependencies

• Customer MCP server and authorization schema.
• Tool and data classification.

Acceptance criteria

• A tool outside the allowlist is rejected before reaching downstream.
• Invalid arguments and results do not reach the LLM/systems.
• Every tool call correlates with user, session, turn and policy.
• Timeouts do not cause unsafe retries.

Risks

• Prompt injection in results.
• Overprivileged credentials.
• Unstable downstream contracts.

Phase 4 — Writes, confirmation and voice governance

Deliverables

• Confirmation flows bound to hashes and idempotency keys.
• Action ledger and unknown status.
• Voice Vault, approval workflow, revocation and secure cache.
• Abuse tests and incident runbooks.

Dependencies

• Legal approval of voices/licenses.
• Idempotency semantics in enterprise systems.

Acceptance criteria

• No sensitive action executes without the required policy/confirmation.
• Repeated events do not repeat side effects.
• Revoking a voice blocks new synthesis and clears caches.
• Cross-tenant access fails and is audited.

Risks

• Non-idempotent actions.
• Incomplete consent.
• Voice exfiltration or misuse.

Phase 5 — Preproduction, SLOs and resilience

Deliverables

• Load, soak, chaos and recovery tests.
• Dashboards, alerts, ratified SLOs and runbooks.
• Capacity model and admission limits.
• DR/backup for configuration, auditing and Voice Vault.
• Security review and go-live checklist.

Dependencies

• Representative traffic/profiles.
• Production-equivalent environment.
• Defined on-call model and ownership.

Acceptance criteria

• Experience targets ratified or renegotiated with evidence.
• Demonstrated recovery from critical failures.
• No open critical vulnerabilities.
• Operations can detect, diagnose and mitigate runbook scenarios.

Risks

• Single GB10 as a SPOF.
• Insufficient capacity under a real traffic mix.
• Differences between test and field networks.

Phase 6 — Progressive rollout and evolution

Deliverables

• Canary by user/tenant and automatic rollback.
• Post-pilot review of quality, latency, security and task completion.
• WebRTC roadmap if WebSocket does not meet targets.
• CandidateAdapter_2026_07_28 only if the specification has been ratified.

Acceptance criteria

• Canary without SLO/security regression.
• Validated support and operational escalation.
• MCP migration passes the conformance suite and promotion ADR.

15. Testing strategy

15.1 Pyramid

Unit tests

• state machine, sanitization, policy, error mapping, deduplication, expiration and redaction.

Contract tests

• Event JSON Schema, backward compatibility, MCP adapters, profile and identity APIs.

Integration tests

• Real STT/LLM/TTS; simulated and real MCP; Voice Vault; KMS; stores.

End-to-end

• audio to audio with measured playout, direct paths, tools, confirmation, cancellation and reconnection.

Load and soak

• concurrent sessions, mixed turns, long audio, slow clients, slow tools, connection churn and warm/cold models.

Chaos

• dead process, GPU OOM, slow store, network loss, late response, expired certificate, partial dependency failure and restart during a tool call.

Security

• authn/authz, tenant isolation, replay, prompt injection, schema bombs, SSRF, egress, secrets, DLP, voice abuse and supply chain.

15.2 Minimum scenario matrix

15.3 Test oracles and reproducibility

• Versioned test audio with consent and labels.
• Deterministic fakes for races and errors.
• Recorded generation seeds/parameters where applicable.
• Golden transcripts and schemas with defined acoustic tolerances.
• Capture monotonic timestamps at every boundary.
• Report percentiles and conditions, not only averages.
• A/B comparison of VAD, chunking and voice configurations.

16. Decision matrix and ADRs

Each ADR must include context, decision, alternatives, consequences, risks, status and review condition. Reversible decisions may start as Proposed; decisions affecting security, data, licenses or external compatibility require explicit approval.

17. Outstanding items and open questions

Product and experience

• Which channels are included: browser only, mobile, telephony or all?
• Which languages, accents, vocabulary and acoustic conditions take priority?
• What behavior is expected when voice fails: text, an alternative voice or closure?
• Which actions require verbal confirmation, visual confirmation or stronger authentication?

Identity and authorization

• Actual contract for /api/v1/auth/user-by-phone and authentication mechanism.
• Roles, tenants, scopes and data-isolation model.
• Token exchange/delegation to MCP and downstream systems.

MCP

• Inventory of servers, tools, schemas, side effects, idempotency and downstream SLOs.
• Stable version actually supported by the customer's server.
• Discovery cache/refresh strategy.
• Future ratification of 2026-07-28 and SDK/conformance-suite availability.

Inference and capacity

• Exact runtime for LLaMA, memory allocation and KV cache policy.
• Sustainable STT + LLM + TTS concurrency on GB10.
• Effects of context length, agent prompts and tool results on TTFT.
• Cold start, model loading, fragmentation and OOM recovery.
• Parakeet quality on real audio and domain content.

Voice and legal considerations

• Owner and consent evidence for every voice.
• Applicable licensing for weights, code, redistribution and commercial use.
• Disclosure, retention, localization and revocation-right rules.
• Fidelity/security metric and acceptance threshold.

Operations

• Environments, topology, certificates, secrets, KMS, stores and backups.
• RTO/RPO and on-call model.
• Logging, sampling, redaction and retention policy.
• Spare-node capacity or alternative service.

18. Consolidated risks

Nonnumeric likelihoods reflect missing data; update them after testing and organizational review.

19. Pilot readiness criteria

The solution is ready for a pilot only when:

1. requirements and critical ADRs are approved;
2. the stable MCP path is tested and the future candidate remains isolated;
3. the end-to-end benchmark demonstrates capacity for the pilot load;
4. barge-in and backpressure pass acoustic and network tests;
5. sensitive tools have policy, confirmation and idempotency;
6. the voice is authorized, versioned and revocable;
7. security has no open critical findings;
8. dashboards, alerts and runbooks enable service operation;
9. provisional SLOs were ratified or replaced with evidence;
10. tested application, profile, model and configuration rollback exists.

20. Sources

1. [1] OpenAI — Realtime and audio — https://developers.openai.com/api/docs/guides/realtime
2. [2] Microsoft Azure — Realtime audio — https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/realtime-audio
3. [3] LiveKit — Turns overview — https://docs.livekit.io/agents/logic/turns/
4. [4] Model Context Protocol — 2026-07-28 Release Candidate — https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/
5. [5] Model Context Protocol — Transport specification 2025-11-25 — https://modelcontextprotocol.io/specification/2025-11-25/basic/transports
6. [6] Model Context Protocol — Architecture overview — https://modelcontextprotocol.io/docs/learn/architecture
7. [7] Qwen — Qwen3-TTS 12Hz 1.7B Base model card — https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
8. [8] QwenLM — Qwen3-TTS repository — https://github.com/QwenLM/Qwen3-TTS
9. [9] Telnyx — Inference Benchmark: TTFT vs E2E Latency — https://telnyx.com/resources/inference-benchmark-ttft-vs-e2e
10. [10] Arun Baby — Real-Time Agent Pipelines — https://www.arunbaby.com/ai-agents/0016-real-time-agent-pipelines/
11. [11] NVIDIA — Parakeet TDT 0.6B v3 model card — https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3

Appendix A — Implementation checklist

• [ ] Requirements [REQ], recommendations [REC] and assumptions [BENCH] reviewed.
• [ ] Critical ADRs approved.
• [ ] Models and hashes pinned; SBOM and licenses reviewed.
• [ ] Identity, roles, scopes and tenants defined.
• [ ] Event schemas v1 published and contract-tested.
• [ ] Stable MCP adapter tested against 2025-11-25.
• [ ] Candidate MCP adapter disabled by default.
• [ ] Voice Vault, consent, rotation and revocation operational.
• [ ] Admission control and buffer limits configured.
• [ ] Barge-in measured through actual playout.
• [ ] Tool confirmation and idempotency verified.
• [ ] Telemetry redacted and correlated.
• [ ] SLOs ratified through benchmarking.
• [ ] Runbooks and rollback rehearsed.
• [ ] Security review and go-live completed.