docs: consolidate English documentation under root docs
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
# Documentation
|
||||
|
||||
All maintained documentation is in English. Spanish examples represent actual
|
||||
user input or spoken output. Historical proposals are identified explicitly and
|
||||
do not define the current implementation.
|
||||
|
||||
## Browser client
|
||||
|
||||
- [Usage and configuration](client/usage.md)
|
||||
- [Architecture and source map](client/architecture.md)
|
||||
- [Reasoning modes](client/reasoning.md)
|
||||
- [Context injection](client/context-injection.md)
|
||||
- [Tool calling](client/tool-calling.md)
|
||||
- [Design language](client/design.md)
|
||||
- [Glossary](client/glossary.md)
|
||||
- [Troubleshooting](client/troubleshooting.md)
|
||||
- [Change history](../s2s-client/CHANGELOG.md)
|
||||
|
||||
## Speech backend
|
||||
|
||||
- [Language model backends](speech/llm.md)
|
||||
- [Speech-to-text backends](speech/stt.md)
|
||||
- [Text-to-speech backends](speech/tts.md)
|
||||
- [Realtime API, tools, and interruption handling](speech/realtime-api.md)
|
||||
|
||||
## Decisions, proposals, and examples
|
||||
|
||||
- [Example files and usage](examples/README.md)
|
||||
- [ADR 0001: Docker hosting and search proxy](adr/0001-docker-space-with-search-proxy.md)
|
||||
- [Historical Vocero architecture proposal](proposals/vocero-architecture.html)
|
||||
- [Agent-profile example](examples/agent-profile.json): an illustrative profile
|
||||
with agent instructions and tool definitions; this is not the current
|
||||
`/api/config` response or an automatically loaded runtime configuration.
|
||||
- [Reference audio](examples/reference-audio.wav): an optional voice-cloning
|
||||
example described in the [TTS guide](speech/tts.md).
|
||||
@@ -0,0 +1,33 @@
|
||||
# Convert the static Space into a Docker app with a search proxy
|
||||
|
||||
Status: accepted. This records the original hosting decision; the current route
|
||||
is `/api/search`, and the current container also includes the root documentation.
|
||||
See [client usage](../client/usage.md) for the root-context Docker build.
|
||||
|
||||
To give the model a web search tool, the executor (which runs in the browser of a
|
||||
public Space) needs a search key. A `sdk: static` Space serves files as-is with no
|
||||
runtime process, so it cannot hold a secret the browser uses without exposing it.
|
||||
We convert the Space from `sdk: static` to `sdk: docker`: a single container runs a
|
||||
small server (FastAPI + uvicorn) that both serves the existing front-end *unchanged*
|
||||
and exposes a same-origin `/api/search` proxy holding `SERPER_API_KEY` server-side. The
|
||||
whole app lives in that one container; the s2s speech-to-speech backend stays the
|
||||
separate load-balanced service it already is.
|
||||
|
||||
## Considered options
|
||||
|
||||
- **Stay static, user-supplied key only** — no owner key; search works only if each
|
||||
user pastes their own. Rejected as the default because it leaves the deployed demo
|
||||
with no working search.
|
||||
- **Separate proxy service** — same secrecy, but splits the app across two deploys.
|
||||
Rejected: the app must live in one container.
|
||||
- **Key baked into client JS at build time** — would be readable in the served
|
||||
bundle. Rejected: defeats the point.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Deployment is no longer static: there is a Dockerfile and a server process; the
|
||||
README front-matter changes from `sdk: static` to `sdk: docker`.
|
||||
- The client calls `/api/search` same-origin; the server reads the key from env. A user
|
||||
may still supply their own key as a fallback, sent per-request to the proxy.
|
||||
- The front-end, audio pipeline, and s2s handshake are untouched — only the hosting
|
||||
shape and the new route are added.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Client architecture
|
||||
|
||||
The [browser client](../../s2s-client) combines a voice UI with a FastAPI server.
|
||||
The speech pipeline and optional Node-RED reasoning service run separately.
|
||||
Runtime configuration comes from environment variables and the root `.env`.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Browser[Browser UI]
|
||||
Server[Client FastAPI server]
|
||||
Speech[Realtime speech backend]
|
||||
Reasoning[Node-RED or HTTP reasoning provider]
|
||||
Tools[ApiPilot backend]
|
||||
Browser -->|HTTP configuration and proxies| Server
|
||||
Browser <-->|WebSocket via local proxy| Server
|
||||
Server <-->|Realtime events and audio| Speech
|
||||
Browser <-->|Optional direct connection| Speech
|
||||
Server -->|Optional transcript reasoning| Reasoning
|
||||
Server -->|Tool discovery and execution| Tools
|
||||
```
|
||||
|
||||
## Connection modes
|
||||
|
||||
`SPEECH_TO_SPEECH_URL` pins a direct browser connection. Otherwise,
|
||||
`S2S_PROXY_TARGET` exposes the same-origin `/v1/realtime` proxy. Either setting
|
||||
disables load-balancer mode. With neither set, `LOAD_BALANCER_URL` enables the
|
||||
session/queue proxy, or the user can enter a direct URL in settings. See
|
||||
[usage](usage.md) for the configuration matrix.
|
||||
|
||||
The local realtime proxy relays audio and events in both directions. It also
|
||||
patches selected `session.update` instructions, handles ApiPilot marker-based
|
||||
tool execution, filters internal text, and can suppress speculative/tool-marker
|
||||
audio. It is not an unconditional byte-for-byte tunnel.
|
||||
|
||||
## Source map
|
||||
|
||||
| Source | Responsibility |
|
||||
|---|---|
|
||||
| [server.py](../../s2s-client/server.py) | Configuration, static serving, WebSocket relay, context loading, ApiPilot, reasoning, search, queue and usage routes |
|
||||
| [main.js](../../s2s-client/main.js) | UI state, settings, transcripts, caller/report displays, custom RAG context, and external reasoning |
|
||||
| [index.html](../../s2s-client/index.html) | Orb and light-theme voice controls, inline chat, dialogs, profile and reports sidebar |
|
||||
| [style.css](../../s2s-client/style.css) | Dark aurora/glass theme, light theme, responsive layout and motion |
|
||||
| [ui/chat.js](../../s2s-client/ui/chat.js) | Transcript history, tool results, export and autoscroll |
|
||||
| [ui/account.js](../../s2s-client/ui/account.js) | Account controls, queue and usage dialogs |
|
||||
| [ui/dom.js](../../s2s-client/ui/dom.js) | DOM, escaping and error-display helpers |
|
||||
| [ws/s2s-ws-client.js](../../s2s-client/ws/s2s-ws-client.js) | Realtime events, microphone/playback, session updates, tool-call events and ApiPilot prompt catalog |
|
||||
| [ws/codec.js](../../s2s-client/ws/codec.js) | PCM/base64 conversion and transcript extraction |
|
||||
| [ws/orb-visualizer.js](../../s2s-client/ws/orb-visualizer.js) | Audio-reactive orb state |
|
||||
| [auth.py](../../s2s-client/auth.py) | Hugging Face OAuth and caller identity/tier |
|
||||
| [limiter.py](../../s2s-client/limiter.py) | SQLite daily talk-time reservations and reconciliation |
|
||||
| [flows.json](../../s2s-reasoning-engine/flows.json) | Node-RED reasoning and assistant-response logging |
|
||||
|
||||
## Server routes
|
||||
|
||||
| Route | Purpose |
|
||||
|---|---|
|
||||
| `GET /api/config` | Connection settings, primary prompt, reasoning/theme/start modes, and feature flags |
|
||||
| `WS /v1/realtime` | Local realtime proxy to `S2S_PROXY_TARGET` |
|
||||
| `POST /api/reasoning` | Node-RED or OpenAI-compatible HTTP reasoning; disabled in `PROXY` mode |
|
||||
| `GET /api/user/phone` | Configured caller profile lookup |
|
||||
| `GET /api/user/reports` | Configured scheduled report lookup, subject to report-injection controls |
|
||||
| `GET /api/apipilot/tools` | Tool discovery with optional authorization and cache refresh |
|
||||
| `POST /api/apipilot/execute` | Validated tool execution returning a `success/data/error` envelope |
|
||||
| `POST /api/search` | Serper web search proxy |
|
||||
| `/api/session`, `/api/queue/*` | Load-balancer session and waiting-queue proxies |
|
||||
| `/api/me`, `/api/session/heartbeat`, `/api/session/end` | Identity, reservation extension and usage reconciliation |
|
||||
|
||||
The root serves the UI. `/docs` remains the FastAPI API explorer, while project
|
||||
documentation files are served in the `/docs/` subdirectories. Markdown remains
|
||||
source text, and directories without `index.html` are not browsable pages.
|
||||
Static HTML, JS and CSS
|
||||
responses receive `Cache-Control: no-store` during development.
|
||||
|
||||
## Audio path
|
||||
|
||||
1. The browser captures microphone audio with echo cancellation, noise
|
||||
suppression and automatic gain-control settings.
|
||||
2. [mic-capture.js](../../s2s-client/worklets/mic-capture.js) resamples to 16 kHz
|
||||
mono PCM16. The browser sends base64 `input_audio_buffer.append` frames.
|
||||
3. The backend processes VAD → STT → LLM → TTS and emits realtime events.
|
||||
4. Assistant PCM16 audio at 24 kHz is decoded and sent to
|
||||
[audio-playback.js](../../s2s-client/worklets/audio-playback.js), which buffers,
|
||||
resamples and fades playback edges. Input speech can clear playback for barge-in.
|
||||
|
||||
Audio worklets are loaded through `AudioWorklet.addModule`; keep their processor
|
||||
registration structure intact when editing them. The speech backend's thread
|
||||
and cancellation model is documented in [realtime API](../speech/realtime-api.md).
|
||||
|
||||
## Reasoning and context
|
||||
|
||||
The browser triggers external reasoning on completed user transcripts only in
|
||||
`OPENAI` or `NODE_RED` mode. Node-RED `log_only=true` responses leave native speech
|
||||
output running; an answer-producing response uses the external-answer path.
|
||||
The server can independently forward assistant text to Node-RED for paired
|
||||
conversation logs. These HTTP operations supplement the realtime audio path.
|
||||
|
||||
Read [reasoning](reasoning.md), [context injection](context-injection.md), and
|
||||
[tool calling](tool-calling.md) for the conditions and limitations of each path.
|
||||
@@ -0,0 +1,132 @@
|
||||
# Context injection
|
||||
|
||||
Context reaches the assistant through browser-built instructions, server-patched
|
||||
realtime session instructions, and HTTP reasoning context. The implementation is
|
||||
in [server.py](../../s2s-client/server.py),
|
||||
[main.js](../../s2s-client/main.js), and the
|
||||
[WebSocket client](../../s2s-client/ws/s2s-ws-client.js).
|
||||
|
||||
## Independent controls
|
||||
|
||||
`S2S_INJECT_PROFILE` defaults to false and gates server loading of caller,
|
||||
episode, report and prior-conversation context. It also controls whether the
|
||||
browser appends caller profile text to its realtime instructions.
|
||||
|
||||
It is not a master switch for every context source. Operator-supplied RAG text
|
||||
and configured ApiPilot discovery work independently. `S2S_INJECT_TODAY`
|
||||
defaults to true; date and voice-output guards are built on paths that invoke
|
||||
instruction augmentation or HTTP context assembly. A WebSocket session with no
|
||||
profile/episode/report/summary or ApiPilot block skips proxy augmentation.
|
||||
Direct browser connections bypass server-side WebSocket injection entirely.
|
||||
|
||||
| Setting | Default | Effect |
|
||||
|---|---|---|
|
||||
| `S2S_INJECT_PROFILE` | false | Enable server-resolved caller, episode and history context |
|
||||
| `S2S_INJECT_REPORTS` | false | Allow scheduled-report fetch/injection; also requires the profile gate |
|
||||
| `S2S_REPORTS_INJECT_MODE` | `minimal` | `minimal`: labels/dates; `full`: scripts; `off`/`none`: no report prompt block |
|
||||
| `S2S_REPORTS_QUICK_HINT_MAX` | 5 | Keep at most this many newest reports, clamped to 1–50 |
|
||||
| `S2S_INJECT_TODAY` | true | Add the current server-local date guard when its context builder runs |
|
||||
| `S2S_DEBUG_PROMPT` | false | Log complete prompt/context text instead of only summaries |
|
||||
|
||||
## Context sources
|
||||
|
||||
| Block | Trigger and source | Size/behavior |
|
||||
|---|---|---|
|
||||
| Podcast episode | `episode_id` page/query/body value and profile gate | Up to 24,000 characters of serialized episode content |
|
||||
| Caller profile | `phoneNumber` and `VOCERO_USER_PHONE_AUTH_URL` | Up to 1,600 serialized characters |
|
||||
| Scheduled reports | `S2S_REPORT_LIST`, profile gate and report gate | Newest N labels/dates by default; full mode includes scripts with a 12,000-character block limit |
|
||||
| Latest conversation | `S2S_LATEST_SUMMARY_TEXT`, or `S2S_LATEST_SUMMARY` with a phone number | Up to 1,600 characters; static text overrides HTTP fetch |
|
||||
| Custom RAG | Operator applies text through the RAG dialog | Up to 12,000 characters; appended in browser instructions and HTTP `user_context` |
|
||||
| ApiPilot discovery | Configured `S2S_APIPILOT_TOOLS_URL` | Tool names/descriptions/auth requirements and call-marker rules, independent of the profile gate |
|
||||
|
||||
### Episode
|
||||
|
||||
The current episode fetch uses the deployment-specific URL:
|
||||
|
||||
```text
|
||||
https://aquachile.azurianlab.com/api/v1/podcast/episodes/{episode_id}
|
||||
```
|
||||
|
||||
It follows redirects with a 15-second timeout, accepts JSON or plain text,
|
||||
unwraps a `result` field, and includes arbitrary episode fields. The prompt
|
||||
labels it as the primary reference for the user's current episode. The endpoint
|
||||
is hardcoded in the fetch helper rather than selected through an environment
|
||||
variable.
|
||||
|
||||
### Caller profile and login
|
||||
|
||||
For caller-specific context, the proxy can first post
|
||||
`{"phoneNumber":"+56912345678"}` to `S2S_LOGIN_BY_PHONE`, extract a bearer token,
|
||||
and forward it to subsequent profile/history requests. Profile retrieval uses
|
||||
`GET VOCERO_USER_PHONE_AUTH_URL?phoneNumber=...` with a 10-second timeout and
|
||||
accepts an object, optionally wrapped in `result`.
|
||||
|
||||
Encode the leading plus sign in page links, for example
|
||||
`?phoneNumber=%2B56912345678`. Caller information is also fetched through
|
||||
`/api/user/phone` for the UI. An available profile can update live browser
|
||||
instructions when profile injection is enabled.
|
||||
|
||||
### Scheduled reports
|
||||
|
||||
`S2S_INJECT_REPORTS=false` skips report fetching and prompt content, including
|
||||
the table of contents. When enabled, the report endpoint may be called with or
|
||||
without a phone number. The resolver keeps dictionary entries, reads
|
||||
`sourceScheduleLabel` or `sourceSchedule.label`, reads `date` or
|
||||
`sourceSchedule.date`, and optionally keeps `podcastScript` in full mode.
|
||||
|
||||
Entries are sorted newest first using ISO/date/timestamp handling and an
|
||||
ISO-like string fallback. Empty/unparseable dates sort last. The newest-item
|
||||
limit applies before constructing the prompt block. `minimal` exposes only the
|
||||
labels and dates; `full` explicitly opts into script content.
|
||||
|
||||
### Latest conversation summary
|
||||
|
||||
`S2S_LATEST_SUMMARY_TEXT` takes priority over `S2S_LATEST_SUMMARY` and needs no
|
||||
phone number or network request. Literal `\n` sequences in the environment value
|
||||
become newlines. Otherwise the proxy fetches
|
||||
`S2S_LATEST_SUMMARY?phoneNumber=...` and accepts text or JSON with an optional
|
||||
`result` wrapper. Both sources pass through the same cap and history wrapper.
|
||||
|
||||
### Custom RAG
|
||||
|
||||
The operator can paste reference material and explicitly apply it to the
|
||||
session. The browser appends it to instructions even with profile injection
|
||||
disabled, includes it in HTTP `user_context`, and updates an open realtime
|
||||
session. This context does not require an external retrieval endpoint.
|
||||
|
||||
## Application timing and order
|
||||
|
||||
At local WebSocket connect, the server resolves enabled context and stores it in
|
||||
`session_ctx`. Each client `session.update` then passes through
|
||||
`_patch_client_session_update_with_profile()`. Other client frame types are not
|
||||
modified by this helper. The broader relay still performs ApiPilot handling and
|
||||
text/audio filtering.
|
||||
|
||||
The current augmentation function **keeps base instructions first**, then
|
||||
appends surviving voice-output guards, date context, episode, profile, reports,
|
||||
and summary blocks. ApiPilot discovery is appended separately. If a block type
|
||||
is already present in the base, deduplication may omit its server candidate.
|
||||
This describes string assembly, not a guarantee of model attention or priority.
|
||||
|
||||
Every HTTP reasoning request rebuilds enabled context. Its `user_context`
|
||||
assembly orders voice guard, date, episode, profile, reports, summary, ApiPilot
|
||||
and browser-provided context, then deduplicates recognized types and strips
|
||||
internal markers. Server instruction augmentation remains separate, so avoid
|
||||
assuming the WebSocket and HTTP prompts have identical ordering.
|
||||
|
||||
Context markers support internal deduplication and diagnostics; ordinary
|
||||
profile/report/episode/history sentinels are stripped before final augmentation.
|
||||
ApiPilot call/catalog markers have their own handling. Treat the source helpers
|
||||
as the reference for the exact final payload.
|
||||
|
||||
## Failure behavior and inspection
|
||||
|
||||
Optional context-fetch errors are logged and the affected context is skipped.
|
||||
The speech conversation normally continues without that reference material.
|
||||
This does not mean all API calls ignore errors: missing reasoning configuration,
|
||||
invalid tool arguments and tool authorization can return explicit errors.
|
||||
|
||||
Inspect `/api/config` for active gates, the actual outbound `session.update` in
|
||||
browser DevTools, and `[INJECT-*]`/`[DEBUG-PROMPT]` server logs. Full prompt
|
||||
logging includes caller data and conversation text. See
|
||||
[troubleshooting](troubleshooting.md) for a step-by-step diagnostic flow.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Design language
|
||||
|
||||
This guide describes the current browser UI. Use
|
||||
[style.css](../../s2s-client/style.css) as the token/selector reference and
|
||||
[index.html](../../s2s-client/index.html) for the theme-specific structure.
|
||||
Earlier documentation described a monochrome-only dark UI; the current design
|
||||
also uses aurora gradients, glass surfaces, and a separate light theme.
|
||||
|
||||
## Themes and tokens
|
||||
|
||||
`S2S_THEME=white` selects the white/blue interface. Empty or `dark` selects the
|
||||
dark aurora interface. The browser sets `html[data-theme]` after configuration
|
||||
loads; an initial visibility guard prevents the wrong theme flashing on first
|
||||
paint. Extend the existing theme selectors when adding components.
|
||||
|
||||
| Token group | Dark theme values/purpose |
|
||||
|---|---|
|
||||
| Canvas | `--bg: #07080d`; translucent `--bg-elev` and `--bg-elev-2` surfaces |
|
||||
| Glass | `--glass` and `--glass-strong` for translucent floating elements |
|
||||
| Text | `--text`, `--text-dim`, `--text-faint` |
|
||||
| Borders | `--border` and `--border-strong` hairlines |
|
||||
| Brand | `--brand: #8b5cf6`, `--brand-2: #ec4899`, `--brand-3: #22d3ee` |
|
||||
| Radius | `--radius-sm: 10px`, `--radius-md: 16px`, `--radius-lg: 26px`, `--radius-xl: 34px` |
|
||||
| Shadows | `--shadow-soft`, `--shadow-glow-brand`, `--shadow-glass` |
|
||||
|
||||
Prefer existing variables and `color-mix` over new unrelated colors. The dark
|
||||
theme's orb is the strongest visual focus, while gradients and restrained role
|
||||
tints also appear in the surrounding interface. The white theme changes
|
||||
surfaces, typography contrast, controls and chat presentation through its own
|
||||
selector overrides rather than a second stylesheet.
|
||||
|
||||
## Voice state
|
||||
|
||||
| State | Dark-theme color |
|
||||
|---|---|
|
||||
| Listening/user speaking | Cyan (`--listening`) |
|
||||
| Assistant speaking | Violet (`--speaking`) |
|
||||
| Processing | Amber (`--processing`) |
|
||||
| Ready/authenticated | Green (`--success`) |
|
||||
| Error | Red (`--error`) |
|
||||
|
||||
`--voice-user`, `--voice-assistant` and `--voice-tool` connect transcript roles
|
||||
to the same state vocabulary. When adding a state, define its existing control,
|
||||
caption and transcript behavior together. Light-theme controls can use their
|
||||
blue palette while retaining recognizable state feedback.
|
||||
|
||||
## Typography and copy
|
||||
|
||||
Use Inter (`--font-sans`) for human prose, buttons, settings and transcript
|
||||
bodies. Geist Mono (`--font-mono`) is reserved for machine status, role labels,
|
||||
tool identifiers and diagnostic JSON. Font stacks have system fallbacks.
|
||||
|
||||
Visible labels, accessibility text, permission instructions and recovery
|
||||
messages default to Spanish. Source identifiers, comments, diagnostic logs and
|
||||
internal model instructions use English. Exact model/voice/protocol identifiers
|
||||
and proper names are preserved. Spanish wake phrases are user-input literals.
|
||||
|
||||
Use sentence case and concrete verbs. Empty states should explain the next
|
||||
action; errors should say what failed and how to recover. Keep internal system
|
||||
details out of ordinary user flows.
|
||||
|
||||
## Layout and components
|
||||
|
||||
The page uses a column layout: header, voice stage, always-visible inline
|
||||
conversation, and sticky footer. The conversation is no longer the old
|
||||
toggleable right overlay. Preserve space below the latest message so the sticky
|
||||
footer does not obscure it; chat autoscroll retries after layout/reflow changes.
|
||||
|
||||
In the dark theme, the reports sidebar is fixed on the left at desktop widths
|
||||
of at least 1024px. On smaller screens it becomes an off-canvas panel with a
|
||||
toggle and backdrop. The white theme hides the sidebar, its toggle, and hint.
|
||||
The user avatar opens a profile dialog; profile fields collapse from two
|
||||
columns on narrow screens. Some controls are intentionally gated by `?debug=true`.
|
||||
|
||||
Use the existing button, icon-button, modal, switch and chat-message classes.
|
||||
The dark theme uses glass bubbles with role tints, borders and restrained glow;
|
||||
the light theme uses its white/blue message and avatar treatment. Keep required
|
||||
chat DOM elements intact; optional historical overlay controls may be absent.
|
||||
|
||||
## Motion and accessibility
|
||||
|
||||
Microphone RMS and assistant audio levels drive the voice visualization.
|
||||
Breathing, glow and decorative gradients should remain subordinate to speech
|
||||
feedback. `body.booting` suppresses initial transitions; new animation should
|
||||
respect that state and `prefers-reduced-motion`.
|
||||
|
||||
For changes, verify both themes, narrow mobile layouts down to 360px, keyboard
|
||||
focus, dialog dismissal, and readable contrast. Prefer touch targets of at least
|
||||
44px and respect safe-area padding. A CSS guideline is not proof that every
|
||||
existing control already meets the target.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Context glossary
|
||||
|
||||
These terms explain the product and its conversation flow. Implementation
|
||||
details belong in [architecture](architecture.md) and the source. Model/provider
|
||||
names shown in a deployment are configuration examples, not definitions.
|
||||
|
||||
| Term | Meaning |
|
||||
|---|---|
|
||||
| Speech-to-speech client | The browser interface used to start a voice conversation, display transcripts, and play the assistant's reply |
|
||||
| Pipeline | The ordered backend path: VAD → STT → LLM → TTS |
|
||||
| VAD | Voice activity detection; determines when a user speech segment starts and stops |
|
||||
| STT | Speech to text; transcribes an audio utterance |
|
||||
| LLM | Language model; generates response text and optional tool calls |
|
||||
| VLM | Vision-language model; a language model with image support when that capability is configured |
|
||||
| TTS | Text to speech; synthesizes response audio with the configured voice/model |
|
||||
| Realtime backend | The service that processes audio and exposes the `/v1/realtime` WebSocket protocol |
|
||||
| Client server/proxy | The local FastAPI service serving the page and optional HTTP/WebSocket bridges |
|
||||
| Builder | A person/team credited with creating the interface or deployment, distinct from a model's authors |
|
||||
| Powered by | Hosting or inference infrastructure credited by the deployment |
|
||||
| Tool | A callable operation used on demand; distinct from the pipeline stages that process ordinary turns |
|
||||
| Native tool declaration | A function name, description and JSON Schema sent in `session.tools` |
|
||||
| Tool executor | Code that receives a tool call, executes it and returns the result to the model |
|
||||
| Web search | The browser-supported native tool whose request goes through the server's `/api/search` proxy |
|
||||
| Search proxy | Same-origin server route holding a deployment search key and calling the external provider |
|
||||
| ApiPilot | The dynamic-tool discovery/execution bridge, with both HTTP endpoints and model-text call markers |
|
||||
| Call marker | Internal structured text such as `<<<APIPILOT_CALL:...>>>` used by the ApiPilot bridge; separate from native function-call events |
|
||||
| Tools panel | Controls for available browser tools and search-key configuration |
|
||||
| Settings | Connection, voice, audio and instruction controls persisted in browser storage |
|
||||
| Identity block | The header's product/user identity, separate from configuration dialogs |
|
||||
| About panel | Introduction and project/pipeline credits |
|
||||
| Caller profile | Information looked up using the page's `phoneNumber`, used for identity display and optionally model context |
|
||||
| RAG context | Reference material explicitly applied by the operator for the current session |
|
||||
| Episode context | Content loaded for the page's `episode_id`, treated as the current episode reference |
|
||||
| Queue | Users waiting for an available compute slot; waiting is not yet an active conversation |
|
||||
| Ticket | A held queue position returned by the load balancer |
|
||||
| Position | The displayed place in the queue; distinct from a time estimate |
|
||||
| Claim | The transition from a waiting ticket to an allocated realtime session and usage reservation |
|
||||
| At capacity | A full waiting queue that cannot accept another ticket |
|
||||
| Barge-in | User speech interrupts an active assistant response, coordinating playback clearing and backend cancellation |
|
||||
| Log-only reasoning | External transcript processing that observes/logs the turn without replacing native speech output |
|
||||
@@ -0,0 +1,106 @@
|
||||
# Reasoning modes
|
||||
|
||||
Reasoning determines the assistant's response text. The realtime backend always
|
||||
provides the speech channel; optional HTTP reasoning can observe a transcript or
|
||||
produce a separate answer. Set `S2S_REASONING_TYPE` to select the mode.
|
||||
|
||||
| Mode | Response owner | HTTP hook |
|
||||
|---|---|---|
|
||||
| `PROXY` (default) | Realtime backend | Browser does not call `/api/reasoning`; that endpoint returns 409 |
|
||||
| `NODE_RED` | Realtime backend for `log_only=true`; external answer otherwise | Server forwards to `S2S_REASONING_NODE_RED_URL/api/reasoning` |
|
||||
| `OPENAI` | OpenAI-compatible Chat Completions response | Server calls `OPENAI_BASE_URL/chat/completions` |
|
||||
|
||||
The mode is normalized case-insensitively. `NODERED` and `NODE-RED` resolve to
|
||||
`NODE_RED`; unknown values resolve to `PROXY`.
|
||||
|
||||
## Shared instruction assembly
|
||||
|
||||
In [main.js](../../s2s-client/main.js), `effectiveReasoningInstructions()` starts
|
||||
with `pinnedPrompt || settings.instructions`. `/api/config` pins
|
||||
`S2S_PRIMARY_PROMPT` when non-empty; otherwise the settings prompt or
|
||||
`DEFAULT_INSTRUCTIONS` is used. The current server does not fetch a remote agent
|
||||
profile to choose this base prompt.
|
||||
|
||||
The browser adds caller context when `injectProfile` is enabled, custom RAG text
|
||||
when the operator has applied it, and an available ApiPilot catalog block.
|
||||
`effectiveInstructions()` adds `OPENAI_RENDER_HINT` only in `OPENAI` mode.
|
||||
The WebSocket client packages these instructions in `session.update.session`;
|
||||
the local proxy may then add server-resolved context. Details belong in
|
||||
[context injection](context-injection.md).
|
||||
|
||||
## Native realtime reasoning (`PROXY`)
|
||||
|
||||
The realtime backend receives session instructions and enabled native tools,
|
||||
then processes each voice turn through STT, model generation and TTS.
|
||||
Native function-call events are executed by the browser. ApiPilot also has a
|
||||
separate marker-based path handled by the local proxy. These are distinct
|
||||
protocols, described in [tool calling](tool-calling.md).
|
||||
|
||||
Using a direct browser URL bypasses local WebSocket proxy processing. Browser
|
||||
instructions still apply, but server-side session context and marker execution
|
||||
require the local `/v1/realtime` route.
|
||||
|
||||
## Node-RED reasoning
|
||||
|
||||
Configure a Node-RED base URL reachable by the UI server:
|
||||
|
||||
```env
|
||||
S2S_REASONING_TYPE=NODE_RED
|
||||
S2S_REASONING_NODE_RED_URL=http://localhost:1881
|
||||
```
|
||||
|
||||
`maybeRunOpenAiReasoning()` handles both external modes despite its historical
|
||||
function name. It runs after a complete user transcript and posts a body with
|
||||
`transcript`, `instructions`, `language`, `user_context`, `phoneNumber`, and
|
||||
`episode_id`. The server builds context and forwards the reasoning fields to
|
||||
Node-RED with a 30-second timeout.
|
||||
|
||||
The checked-in Node-RED flow exposes `POST /api/reasoning` and
|
||||
`POST /api/log/system-response`. Its reasoning response uses `log_only=true` and
|
||||
an empty answer. The browser returns early in that case, preserving the native
|
||||
backend's response. Assistant-text logging is forwarded separately by the relay
|
||||
and is not an audio transport.
|
||||
|
||||
An answer-producing flow must return a non-empty `answer` and omit `log_only` or
|
||||
set it to `false`. That activates the external-answer behavior below.
|
||||
|
||||
## OpenAI-compatible HTTP reasoning
|
||||
|
||||
```env
|
||||
S2S_REASONING_TYPE=OPENAI
|
||||
OPENAI_API_KEY=YOUR_API_KEY
|
||||
OPENAI_BASE_URL=https://api.openai.com/v1
|
||||
OPENAI_REASONING_MODEL=gpt-4.1-mini
|
||||
```
|
||||
|
||||
`OPENAI_ORGANIZATION` and `OPENAI_PROJECT` are optional headers. The model value
|
||||
is a runtime default, not a recommendation about provider availability.
|
||||
|
||||
The server combines instructions and context into model messages, requests a
|
||||
final answer, and returns `answer` with available usage metadata. ApiPilot can
|
||||
append discovery instructions, execute a detected call marker, and request a
|
||||
second response based on the tool result. The endpoint does not forward the
|
||||
browser's native `session.tools` catalog as OpenAI Chat Completions `tools`.
|
||||
|
||||
For a non-log-only answer, the browser cancels/suppresses the active native
|
||||
response and appends the external assistant text. It uses browser
|
||||
`SpeechSynthesis` when available. Its compatibility fallback sends
|
||||
`[VOCERO_RENDER_REPLY]` plus the answer to the realtime backend and requests a
|
||||
response. The render hint asks the model to read only the answer after that
|
||||
marker. This instruction is model behavior, not a guarantee of exact playback.
|
||||
|
||||
External reasoning adds an HTTP operation to each completed transcript and has
|
||||
separate provider token usage. It is not the speech pipeline's `responses-api`
|
||||
or `chat-completions` LLM backend setting; those configure the model inside the
|
||||
realtime backend instead.
|
||||
|
||||
## Where to change behavior
|
||||
|
||||
| Goal | Source |
|
||||
|---|---|
|
||||
| Change the base agent prompt | `S2S_PRIMARY_PROMPT` or browser settings when unpinned |
|
||||
| Change browser context assembly | `effectiveReasoningInstructions()` in `main.js` |
|
||||
| Change HTTP provider messages | `reasoning()` in `server.py` |
|
||||
| Change external-answer rendering | `maybeRunOpenAiReasoning()` in `main.js` |
|
||||
| Change Node-RED processing | `s2s-reasoning-engine/flows.json` |
|
||||
| Change native model/tool generation | `speech_to_speech/LLM` and realtime handlers |
|
||||
@@ -0,0 +1,137 @@
|
||||
# Tool calling
|
||||
|
||||
The application has two tool mechanisms: native realtime function calling and
|
||||
ApiPilot calls detected from model text. A spoken promise to fetch something is
|
||||
not evidence that either mechanism executed a tool.
|
||||
|
||||
## Native realtime function calls
|
||||
|
||||
The browser sends tool declarations inside the `session` object:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "session.update",
|
||||
"session": {
|
||||
"type": "realtime",
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "web_search",
|
||||
"description": "Search the web for information.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"query": {"type": "string"}},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep configuration inside `session`, not at the event root. The backend has a
|
||||
legacy flat-payload compatibility path, but new clients should use the nested
|
||||
shape. The current browser's `activeToolDefs()` declares `web_search` when
|
||||
enabled and a deploy/user search key is available. It does not load the old
|
||||
remote agent-profile MCP catalog into native session tools.
|
||||
|
||||
For a native call, the backend emits:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "response.function_call_arguments.done",
|
||||
"call_id": "call_example",
|
||||
"name": "web_search",
|
||||
"arguments": "{\"query\":\"example query\"}"
|
||||
}
|
||||
```
|
||||
|
||||
`arguments` is a JSON-encoded string. The
|
||||
[WebSocket client](../../s2s-client/ws/s2s-ws-client.js) parses it and dispatches
|
||||
`toolcall`; [main.js](../../s2s-client/main.js) runs the supported executor.
|
||||
Web search uses `POST /api/search`, keeping the deployment's Serper key on the
|
||||
server. Tool output returns to the model as:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "function_call_output",
|
||||
"call_id": "call_example",
|
||||
"output": "Tool result text"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The client requests follow-up generation with `response.create` when the
|
||||
result should be spoken. A `function_call_output` item alone does not start
|
||||
generation. See the [realtime API guide](../speech/realtime-api.md) for local
|
||||
model `<code>` extraction and native remote-provider tool calls.
|
||||
|
||||
## ApiPilot discovery and execution
|
||||
|
||||
`S2S_APIPILOT_TOOLS_URL` enables the current dynamic-tool bridge. The backend
|
||||
discovers tools through its `/tools` route and executes through `/execute`.
|
||||
`S2S_APIPILOT_TOOLS_EXECUTE_URL` can override the execution target. The same-origin UI
|
||||
server exposes:
|
||||
|
||||
- `GET /api/apipilot/tools?include_unauthorized=true`: discovery envelope with
|
||||
`data.tools`, `data.count`, and `data.include_unauthorized`.
|
||||
- `POST /api/apipilot/execute`: body `{"name":"tool_name","arguments":{}}` and
|
||||
a `success/data/error` envelope.
|
||||
|
||||
Tool definitions contain `name`, `description`, JSON Schema `parameters`, and
|
||||
`requires_auth`. Execution looks up the current discovery cache, refreshes once
|
||||
if the tool is missing, and applies an authorization gate and argument
|
||||
validation before forwarding. Caller `Authorization` and
|
||||
`x-demo-session` can be forwarded; configured service credentials and
|
||||
login-by-phone provide additional server paths. Display `error.message` to the
|
||||
user; keep `error.code`, `error.status_code`, and `error.raw` for diagnostics.
|
||||
|
||||
The discovery cache defaults to 300 seconds server-side
|
||||
(`S2S_APIPILOT_TTL_SEC`) and can be refreshed with `refresh=true`. The browser
|
||||
also keeps a per-user catalog cache. ApiPilot configuration is independent of
|
||||
`S2S_INJECT_PROFILE`.
|
||||
|
||||
## Marker-based two-pass reasoning
|
||||
|
||||
The browser and proxy add an ApiPilot catalog as **instruction text**, including
|
||||
tool names, descriptions, authentication requirements and required parameters.
|
||||
The model is instructed to emit a call marker when it chooses a matching tool:
|
||||
|
||||
```text
|
||||
<<<APIPILOT_CALL:NAME=tool_name,ARGUMENTS={"parameter":"value"}>>>
|
||||
```
|
||||
|
||||
Tool names are exact identifiers and must not be translated. The local realtime
|
||||
proxy accumulates assistant text, detects the marker, validates and executes
|
||||
the ApiPilot call, then injects the result as context and requests a new
|
||||
response. The HTTP reasoning route has analogous discovery/execution and
|
||||
second-answer handling. Internal marker text and its speculative audio are
|
||||
filtered from user output where those handlers run.
|
||||
|
||||
This marker is not `response.function_call_arguments.done`, and ApiPilot prompt
|
||||
discovery is not the same as declaring a native `session.tools` array. Direct
|
||||
browser connections bypass the local relay's marker executor even if the
|
||||
browser includes catalog instructions.
|
||||
|
||||
## Retired documentation assumptions
|
||||
|
||||
The former `HOWITWORKS.md` described `/api/mcp/tool`, remote
|
||||
`S2S_AGENT_PROFILE` loading, and Spanish/Italian-to-English client ranking.
|
||||
Those APIs/rankers are absent from the current client/server path. The retained
|
||||
agent-profile JSON is an illustrative historical/example shape. Add or modify
|
||||
current ApiPilot tool definitions in the provider; do not restore obsolete
|
||||
endpoints solely to match older documentation.
|
||||
|
||||
## Verify a tool turn
|
||||
|
||||
1. Confirm `/api/config` reports the intended feature and connection mode.
|
||||
2. Inspect `session.update`: native `tools` and ApiPilot instruction text are
|
||||
separate evidence.
|
||||
3. For native calls, find `response.function_call_arguments.done`, then the
|
||||
browser `toolcall` and executor request.
|
||||
4. For ApiPilot, find the discovery, call-marker detection and execution logs.
|
||||
5. Confirm the tool result reaches the model and follow-up text/audio reaches
|
||||
the browser. A spoken acknowledgement alone does not complete the flow.
|
||||
@@ -0,0 +1,115 @@
|
||||
# Troubleshooting
|
||||
|
||||
Diagnose configuration, protocol events, execution and playback separately.
|
||||
Use [architecture](architecture.md), [tool calling](tool-calling.md) and
|
||||
[context injection](context-injection.md) to identify which path is active.
|
||||
|
||||
## Check configuration and connection first
|
||||
|
||||
Inspect the server configuration without printing secret environment files:
|
||||
|
||||
```bash
|
||||
curl -fsS http://localhost:8420/api/config | jq
|
||||
```
|
||||
|
||||
Confirm `s2sUrl`, `lb`, `reasoningType`, `injectProfile`, `apipilot`, `theme`
|
||||
and `startSessionMode`. A direct WebSocket URL bypasses the local relay's
|
||||
server-side context injection and marker execution. For a local proxy, confirm
|
||||
the server can resolve and reach `S2S_PROXY_TARGET`.
|
||||
|
||||
In browser DevTools, inspect the WebSocket handshake and `session.created`,
|
||||
`session.update` and error events. Check that instructions and native tools are
|
||||
inside `session`. A cached browser module can disagree with current server
|
||||
code; inspect served asset versions and reload with cache disabled when needed.
|
||||
|
||||
## The assistant promises a tool but nothing happens
|
||||
|
||||
Follow the actual execution mechanism:
|
||||
|
||||
1. For a native tool, confirm it appears in the outbound `session.tools` array.
|
||||
2. Find `response.function_call_arguments.done` with a tool name, `call_id`, and
|
||||
JSON-string arguments.
|
||||
3. Confirm the browser dispatched `toolcall` and the executor sent its request.
|
||||
4. Confirm `function_call_output` and any required `response.create` were sent.
|
||||
5. For ApiPilot, inspect discovery, marker detection, authorization/schema
|
||||
validation, execution, and second-pass result injection instead.
|
||||
|
||||
A spoken sentence such as “Voy a buscar el reporte” does not satisfy step 2.
|
||||
If the function-call event is absent, inspect model/backend tool generation
|
||||
before debugging a browser executor that has not been invoked.
|
||||
|
||||
Earlier local debug notes recorded a reproduction where tools and required
|
||||
tool policy were sent but the model produced only spoken text. Those notes
|
||||
refer to the retired MCP ranking path; they do not establish a current ApiPilot
|
||||
failure. Both local investigation notes remain marked `OPEN`. Their useful
|
||||
lesson is to collect evidence at each boundary and reproduce against current
|
||||
code before claiming a root cause is resolved.
|
||||
|
||||
## The assistant misses caller or report context
|
||||
|
||||
- Verify the page has an encoded `phoneNumber` or the intended `episode_id`.
|
||||
- Inspect `/api/user/phone` and context-fetch logs for HTTP/JSON failures.
|
||||
- Confirm profile injection is enabled; a visible profile alone does not prove
|
||||
realtime instructions contain it.
|
||||
- Reports need `S2S_INJECT_REPORTS=true`; `minimal` sends only labels and dates.
|
||||
A historical script will not be in the prompt in that mode.
|
||||
- Static summary text overrides the HTTP summary endpoint.
|
||||
- Inspect the final `session.update` rather than only its browser base prompt.
|
||||
The proxy can append and deduplicate context after the browser sends it.
|
||||
|
||||
`S2S_DEBUG_PROMPT=true` enables complete prompt logs, including caller information
|
||||
and conversation text. Use summaries when full payloads are unnecessary. The
|
||||
browser's application functions are module-scoped; do not assume names such as
|
||||
`effectiveInstructions()` are available directly on `window` in DevTools.
|
||||
|
||||
## Short gaps or clipped words around tool results
|
||||
|
||||
The retained audio investigation found very low local serialization/render
|
||||
times for one large tool result, while still observing audio interruptions.
|
||||
That evidence ruled out a UI-render bottleneck for that reproduction; it does
|
||||
not rule out rendering costs for all future payloads.
|
||||
|
||||
Useful checks are:
|
||||
|
||||
- Measure tool-result size, prompt length, serialization time and UI render time.
|
||||
- Inspect whether result injection overlaps an active or pending response.
|
||||
- Check whether a speculative response starts before its `response.created`
|
||||
reaches the browser. Cancellation based only on an existing response ID can
|
||||
miss that window.
|
||||
- Inspect deferred text/response queues and whether they flush after
|
||||
`response.done` rather than interrupting the active turn.
|
||||
- Inspect playback prebuffering and queue underflow, especially when the first
|
||||
words of every sentence are affected.
|
||||
- Confirm oversized tool results are summarized or capped on the relevant
|
||||
path rather than assuming every tool payload gets the same treatment.
|
||||
|
||||
The historical investigation introduced a 90ms playback prebuffer and reduced
|
||||
MCP prompt reinjection. The old 150ms MCP fallback timer is historical, not a
|
||||
current ApiPilot configuration knob. Verify the live worklet and current relay
|
||||
before changing audio timing based on those observations.
|
||||
|
||||
## External reasoning and startup
|
||||
|
||||
`/api/reasoning` returning 409 is expected in `PROXY` mode. In `NODE_RED`, check
|
||||
that `S2S_REASONING_NODE_RED_URL` is reachable from the server and whether the
|
||||
response is `log_only=true`. An empty answer is valid only in log-only mode.
|
||||
In `OPENAI`, check the configured endpoint/model/key and provider logs; HTTP
|
||||
reasoning does not use the native session's tool array.
|
||||
|
||||
For `TALK`/`TALKFIRST`, the two-second warmup does not override browser
|
||||
microphone/audio permissions. If automatic startup fails, use the unlocker
|
||||
gesture and inspect the permission error rather than increasing the timer
|
||||
without evidence.
|
||||
|
||||
## Read-only source checks
|
||||
|
||||
From the repository root, useful syntax checks are:
|
||||
|
||||
```bash
|
||||
node --check s2s-client/main.js
|
||||
node --check s2s-client/ws/s2s-ws-client.js
|
||||
python -m py_compile s2s-client/server.py
|
||||
```
|
||||
|
||||
Syntax checks do not validate microphone access, network services, model output,
|
||||
or audio quality. Collect a fresh browser/backend reproduction for those paths.
|
||||
@@ -0,0 +1,153 @@
|
||||
# Client usage and configuration
|
||||
|
||||
The browser voice client connects to an existing OpenAI Realtime-compatible
|
||||
speech backend. The UI server can also proxy WebSocket traffic, search,
|
||||
reasoning, caller context, ApiPilot tools and load-balancer requests.
|
||||
|
||||
## Local startup
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime ./start_ui.sh
|
||||
```
|
||||
|
||||
The launcher loads the root `.env`, creates `.venv` if needed, installs the
|
||||
client requirements when the dependency check fails, and runs uvicorn on port
|
||||
8420. `UI_PORT` overrides the UI port. Because `.env` is sourced by the script,
|
||||
values in that file can replace variables supplied on the command line.
|
||||
|
||||
To install and start the UI manually:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r s2s-client/requirements.txt
|
||||
SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime \
|
||||
.venv/bin/python -m uvicorn --app-dir s2s-client server:app --port 8420
|
||||
```
|
||||
|
||||
Open <http://localhost:8420/>, tap the orb/microphone control, grant microphone
|
||||
access, and speak. Microphone access requires HTTPS or a local browser origin;
|
||||
use HTTPS when accessing the UI from another machine.
|
||||
|
||||
The speech pipeline has separate model/runtime dependencies and no complete
|
||||
dependency manifest in this checkout. In an already prepared speech
|
||||
environment, a local realtime server can be started with:
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--mode realtime \
|
||||
--stt parakeet-tdt \
|
||||
--llm_backend transformers \
|
||||
--tts kokoro \
|
||||
--model_name Qwen/Qwen3-4B-Instruct-2507 \
|
||||
--llm_device cuda \
|
||||
--llm_torch_dtype float16 \
|
||||
--enable_live_transcription
|
||||
```
|
||||
|
||||
Select backends and devices supported by the host; the example model is not a
|
||||
fixed deployment requirement. The default realtime endpoint is
|
||||
`ws://localhost:8765/v1/realtime`, configured through `--ws_host` and `--ws_port`.
|
||||
`--num_pipelines` controls the number of isolated concurrent realtime sessions.
|
||||
See [LLM](../speech/llm.md), [STT](../speech/stt.md), and [TTS](../speech/tts.md).
|
||||
|
||||
## Connection modes
|
||||
|
||||
| Configuration | Browser connection | Settings URL | Load-balancer features |
|
||||
|---|---|---|---|
|
||||
| `SPEECH_TO_SPEECH_URL` set | Direct to pinned URL | Locked | Disabled |
|
||||
| `S2S_PROXY_TARGET` set, no direct URL | Same-origin `/v1/realtime` proxy | Locked | Disabled |
|
||||
| `LOAD_BALANCER_URL` set, neither above set | Per-session compute URL from `/api/session` | Hidden | Queue enabled; metering requires `SPACE_ID` too |
|
||||
| None set | User-provided realtime URL | Editable | Disabled |
|
||||
|
||||
Direct URLs may be full `ws(s)://.../v1/realtime` addresses or host values that
|
||||
the browser normalizes. A direct URL must be reachable **from the browser**.
|
||||
`S2S_PROXY_TARGET` must be reachable **from the UI server**. HTTPS deployments
|
||||
need a secure browser WebSocket URL.
|
||||
|
||||
The launcher supplies `ws://ai-server:12345/v1/realtime` as a proxy-target
|
||||
default when no value is configured. That hostname must resolve from the UI
|
||||
server; it is not a universally available local service.
|
||||
|
||||
## Docker
|
||||
|
||||
Use the project root as the build context:
|
||||
|
||||
```bash
|
||||
docker build -f s2s-client/Dockerfile -t s2s-client .
|
||||
docker run --rm -p 7860:7860 \
|
||||
-e SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime s2s-client
|
||||
```
|
||||
|
||||
For the proxy path, replace the direct setting with a server-reachable
|
||||
`S2S_PROXY_TARGET`. If the upstream is another container, attach the UI to the
|
||||
appropriate existing network and use that service's hostname. An upstream
|
||||
bound only to host loopback is not reachable via an arbitrary container name.
|
||||
|
||||
The image includes `s2s-client` and root `docs`. `/docs` is still the FastAPI API
|
||||
explorer; documentation lives under `/docs/client/`, `/docs/speech/`,
|
||||
`/docs/adr/`, `/docs/examples/`, and `/docs/proposals/`. Legacy `/project/` and
|
||||
`/public/docs.html` redirect to the historical proposal. Markdown is served as a
|
||||
source file rather than rendered through a separate documentation site. Request
|
||||
file paths such as `/docs/README.md`; directories without `index.html` are not
|
||||
browsable pages.
|
||||
|
||||
## Settings and tools
|
||||
|
||||
Voice, editable system instructions and connection settings are kept in browser
|
||||
`localStorage` under `s2s.ws.*` keys. A non-empty `S2S_PRIMARY_PROMPT` pins the
|
||||
instruction field. Restart applies connection/voice changes to a new session.
|
||||
Model voice names are runtime identifiers and stay unchanged across UI
|
||||
translations.
|
||||
|
||||
Web search needs `SERPER_API_KEY` on the server or a user-supplied key in the
|
||||
tools panel. The server proxies requests through `/api/search`; its own key is
|
||||
not returned to the browser. Dynamic ApiPilot tools are configured through
|
||||
`S2S_APIPILOT_TOOLS_URL`; see [tool calling](tool-calling.md).
|
||||
|
||||
Operator RAG text is applied through the RAG dialog and updates live
|
||||
instructions. Caller links may include `phoneNumber=%2B...`; episode links may
|
||||
include `episode_id=...`. See [context injection](context-injection.md) for
|
||||
enabled data sources and limits.
|
||||
|
||||
## Theme and session startup
|
||||
|
||||
`S2S_THEME=white` selects the light interface. `dark` or an empty value selects
|
||||
the dark aurora theme. Theme-specific sections remain hidden until configuration
|
||||
resolves, with a fallback to avoid an invisible page after config failure.
|
||||
|
||||
`S2S_START_SESSION_MODE` controls initial microphone/session startup:
|
||||
|
||||
| Value | Behavior |
|
||||
|---|---|
|
||||
| Empty/default | User taps to start |
|
||||
| `TALK` | After a 2-second warmup, show an audio unlocker; a user gesture starts the session and greeting |
|
||||
| `TALKFIRST` | After a 2-second warmup, attempt automatic start and inject the Spanish wake phrase; microphone failure falls back to the unlocker |
|
||||
|
||||
The warmup is a timer, not proof that all upstream context requests have
|
||||
finished. Browser permission and audio policies can still require a gesture.
|
||||
The greeting uses the same `hola vocero` input as wake-word handling.
|
||||
|
||||
## Queue and daily usage
|
||||
|
||||
Load-balancer mode proxies session acquisition and queue polling. Waiting does
|
||||
not reserve conversation time; a daily budget is reserved when a compute slot
|
||||
is granted. The server enables metering only with both `LOAD_BALANCER_URL` and
|
||||
`SPACE_ID` and with neither pinned direct/proxy mode active.
|
||||
|
||||
| Environment setting | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `LIMIT_ANON_SEC` | 300 | Anonymous seconds per UTC day |
|
||||
| `LIMIT_FREE_SEC` | 600 | Signed-in non-PRO seconds per UTC day |
|
||||
| `UNLIMITED_ORGS` | Adds to built-in organizations | Additional unlimited organization members |
|
||||
| `USAGE_HASH_SECRET` | Random if unset | Identity-key hashing and anonymous cookie signing |
|
||||
|
||||
PRO members are unlimited. Built-in unlimited organizations are `cerebras`,
|
||||
`HuggingFaceM4`, `smolagents`, and `pollen-robotics`; organization matching is
|
||||
case-insensitive. See [auth.py](../../s2s-client/auth.py) and
|
||||
[limiter.py](../../s2s-client/limiter.py) for the current enforcement behavior.
|
||||
|
||||
For external transcript processing, configure the selected
|
||||
[reasoning mode](reasoning.md). For connection, context and audio diagnostics,
|
||||
use [troubleshooting](troubleshooting.md).
|
||||
@@ -0,0 +1,16 @@
|
||||
# Examples
|
||||
|
||||
- [agent-profile.json](agent-profile.json) is an illustrative agent-profile
|
||||
response with a dynamic tool catalog. Its model instructions are written in
|
||||
English and request Spanish speech through both instructions and `language`.
|
||||
Tool names and schemas illustrate the original API response; this file is not
|
||||
loaded by the application and is not a live API contract.
|
||||
- [reference-audio.wav](reference-audio.wav) is the existing optional
|
||||
voice-cloning sample, moved out of the implementation directory. CustomVoice
|
||||
models do not require it. When testing a compatible Base/voice-cloning model,
|
||||
supply its path explicitly with `--qwen3_tts_ref_audio
|
||||
docs/examples/reference-audio.wav` from the project root and use the matching
|
||||
transcription required by that model. See [the TTS guide](../speech/tts.md).
|
||||
|
||||
These examples contain no deployment credentials and are not default runtime
|
||||
configuration. Use an authorized reference voice for your own deployment.
|
||||
@@ -0,0 +1,572 @@
|
||||
{
|
||||
"agent": {
|
||||
"name": "Aquachile Voice Agent",
|
||||
"description": "Aquachile voice assistant for support, operational information, and real-time conversational guidance.",
|
||||
"model": "gpt-realtime",
|
||||
"voice": "alloy",
|
||||
"language": "es",
|
||||
"enabled": true,
|
||||
"interruptible": true,
|
||||
"tools": [
|
||||
{
|
||||
"name": "describe_copernicus_marine_catalogue",
|
||||
"description": "Describe the Copernicus Marine catalogue with optional contains, dataset and product filters.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"contains": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"dataset_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"product_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"show_all_versions": {
|
||||
"default": false,
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_api_info",
|
||||
"description": "Return basic AquaPortal API and MCP runtime metadata.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_copernicus_marine_status",
|
||||
"description": "Return Copernicus Marine toolbox installation and credential status.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_copernicus_marine_subset",
|
||||
"description": "Open and summarize a Copernicus Marine dataset subset, defaulting to an allowed zone when bounds are omitted.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"dataset_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"zone_name": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"variables": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"dataset_version": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"dataset_part": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"minimum_longitude": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"maximum_longitude": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"minimum_latitude": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"maximum_latitude": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"minimum_depth": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"maximum_depth": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"start_datetime": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"end_datetime": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"coordinates_selection_method": {
|
||||
"default": "nearest",
|
||||
"type": "string"
|
||||
},
|
||||
"max_values_per_variable": {
|
||||
"default": 8,
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"dataset_id"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_public_day_report",
|
||||
"description": "Return the public day report payload for a business role.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"role": {
|
||||
"default": "CEO",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_report_detail",
|
||||
"description": "Return one report with linked metrics and insights.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"report_id": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"report_id"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_today_podcast",
|
||||
"description": "Return today's AquaPortal podcast metadata and URLs.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "get_user_by_phone",
|
||||
"description": "Resolve an AquaPortal user from an E.164 or WhatsApp phone number.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"phone_number": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"phone_number"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_business_roles",
|
||||
"description": "List configured business role definitions.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_insights",
|
||||
"description": "List insights with optional center, priority, role and date filters.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"center": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"priority": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"role": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"report_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"start_date": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"end_date": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_metrics",
|
||||
"description": "List extracted metrics with optional filters.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"center": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"jaula": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"metric": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"report_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"start_date": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
},
|
||||
"end_date": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_recent_reports",
|
||||
"description": "List the most recent uploaded reports stored by AquaPortal.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"limit": {
|
||||
"default": 5,
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_recent_whatsapp_deliveries",
|
||||
"description": "List recent WhatsApp delivery records, optionally filtered by recipient.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"limit": {
|
||||
"default": 10,
|
||||
"type": "integer"
|
||||
},
|
||||
"recipient": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_users",
|
||||
"description": "List AquaPortal users.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "list_whatsapp_mappings",
|
||||
"description": "List active and inactive WhatsApp user mappings.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
},
|
||||
{
|
||||
"name": "ping",
|
||||
"description": "Echo a message to verify AquaPortal MCP connectivity.",
|
||||
"enabled": true,
|
||||
"parameters": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"message": {
|
||||
"default": "hello",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"tags": []
|
||||
}
|
||||
],
|
||||
"workflowId": "aquachile-spanish-support-flow",
|
||||
"status": "ready",
|
||||
"tags": [
|
||||
"aquachile",
|
||||
"spanish",
|
||||
"support",
|
||||
"operations",
|
||||
"voice"
|
||||
],
|
||||
"instructions": "You are Aquachile's official voice assistant. Always speak Spanish in a professional, clear, friendly, and confident tone. Keep replies brief and useful. Ask one question per turn when clarification is needed. Do not invent data, prices, dates, availability, or policies. Use the available tools first when you need verifiable information. If an action requires user confirmation, ask before proceeding. Summarize the action or conclusion at the end of each reply. If the user changes the subject, adapt the context while staying clear. If you lack sufficient information, say so explicitly and suggest the most useful next step. Never reply in Italian or English unless the user explicitly asks.",
|
||||
"turnDetection": {
|
||||
"type": "server_vad"
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,162 @@
|
||||
# Language model backends
|
||||
|
||||
## Available LLM backends (`--llm_backend`)
|
||||
|
||||
Runtime-supported values in [s2s_pipeline.py](../../speech_to_speech/s2s_pipeline.py):
|
||||
|
||||
- `transformers` → `language_model.py` (Transformers backend)
|
||||
- `mlx-lm` → `language_model.py` (MLX backend)
|
||||
- `responses-api` → `responses_api_language_model.py`
|
||||
- `chat-completions` → `chat_completions_language_model.py`
|
||||
|
||||
Run the commands below from the repository root in an environment with the speech
|
||||
pipeline dependencies installed. This checkout has no package manifest or complete
|
||||
speech dependency installer; the UI requirements alone do not install the pipeline.
|
||||
|
||||
## Usage
|
||||
|
||||
### 1) Transformers (`--llm_backend transformers`)
|
||||
|
||||
- Handler: `LanguageModelHandler`
|
||||
- Typical use: local GPU/CPU inference using Hugging Face Transformers
|
||||
- Backend-specific args prefix: `--llm_*`
|
||||
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--llm_backend transformers \
|
||||
--model_name Qwen/Qwen3-4B-Instruct-2507 \
|
||||
--llm_device cuda \
|
||||
--llm_torch_dtype float16 \
|
||||
--llm_gen_max_new_tokens 128
|
||||
```
|
||||
|
||||
Common options:
|
||||
- `--llm_gen_min_new_tokens`
|
||||
- `--llm_gen_temperature`
|
||||
- `--llm_gen_do_sample`
|
||||
- `--chat_size`
|
||||
- `--init_chat_prompt`
|
||||
|
||||
### 2) MLX-LM (`--llm_backend mlx-lm`)
|
||||
|
||||
- Handler: `LanguageModelHandler`
|
||||
- Typical use: Apple Silicon local inference
|
||||
- Backend-specific args prefix: same as Transformers (`--llm_*`)
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--llm_backend mlx-lm \
|
||||
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
|
||||
--llm_device mps \
|
||||
--llm_gen_max_new_tokens 128
|
||||
```
|
||||
|
||||
Common options:
|
||||
- `--llm_gen_temperature`
|
||||
- `--llm_gen_do_sample`
|
||||
- `--chat_size`
|
||||
- `--init_chat_prompt`
|
||||
|
||||
### 3) OpenAI-compatible API (`--llm_backend responses-api`)
|
||||
|
||||
- Handler: `ResponsesApiModelHandler`
|
||||
- Typical use: remote model serving via OpenAI-compatible endpoints
|
||||
- Backend-specific args prefix: `--responses_api_*`
|
||||
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--llm_backend responses-api \
|
||||
--model_name gpt-5.4-mini \
|
||||
--responses_api_api_key YOUR_API_KEY \
|
||||
--responses_api_base_url https://api.example.com/v1 \
|
||||
--responses_api_stream true
|
||||
```
|
||||
|
||||
Common options:
|
||||
- `--chat_size`
|
||||
- `--init_chat_prompt`
|
||||
- `--user_role`
|
||||
|
||||
### 4) Chat Completions (`--llm_backend chat-completions`)
|
||||
|
||||
- Handler: `ChatCompletionsLanguageModelHandler`
|
||||
- Uses the provider's `/chat/completions` API rather than `/responses`
|
||||
- Inherits the same `--responses_api_base_url`, `--responses_api_api_key`,
|
||||
`--responses_api_stream`, and `--responses_api_disable_thinking` flags
|
||||
- Adds `--responses_api_reasoning_effort` for providers that support that setting
|
||||
|
||||
The current handler removes `tools` and `tool_choice` from outgoing provider
|
||||
requests as a compatibility workaround. Native tool calling therefore differs
|
||||
from the `responses-api` backend; see the [realtime guide](realtime-api.md).
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--llm_backend chat-completions \
|
||||
--model_name YOUR_MODEL \
|
||||
--responses_api_base_url https://api.example.com/v1 \
|
||||
--responses_api_api_key YOUR_API_KEY
|
||||
```
|
||||
|
||||
The default LLM backend is `responses-api`. Model names in this guide are examples;
|
||||
select a model available in your environment or from your provider.
|
||||
|
||||
## LLM Behavior
|
||||
|
||||
When STT is set to language auto-detection (`--language auto`), LLM handlers can receive `(text, language_code)` and prepend a language control instruction like:
|
||||
|
||||
- `Please reply to my message in <language>.`
|
||||
|
||||
This helps the assistant respond in the detected language. The behavior is opt-in via `--enable_lang_prompt` (shared across all backends); it defaults to `False`.
|
||||
|
||||
## Setup
|
||||
|
||||
### CUDA setup
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--llm_backend transformers \
|
||||
--model_name microsoft/Phi-3-mini-4k-instruct
|
||||
```
|
||||
|
||||
### Local Mac setup
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--local_mac_optimal_settings \
|
||||
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16
|
||||
```
|
||||
|
||||
`--local_mac_optimal_settings` already sets `--llm_backend mlx-lm` and will default the model to `mlx-community/Qwen3-4B-Instruct-2507-bf16` if not overridden.
|
||||
|
||||
### Realtime (OpenAI-compatible) setup
|
||||
|
||||
Run the server in realtime mode, then connect with the [browser client](../client/usage.md):
|
||||
|
||||
```bash
|
||||
# 1. Start the pipeline in realtime mode
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--mode realtime \
|
||||
--llm_backend mlx-lm \
|
||||
--model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
|
||||
--ws_host 0.0.0.0 \
|
||||
--ws_port 8765
|
||||
|
||||
# 2. Point the browser client at ws://localhost:8765/v1/realtime.
|
||||
# See the browser-client usage guide for startup commands.
|
||||
```
|
||||
|
||||
For realtime mode on Apple Silicon, set the desired MLX/STT/TTS/device options
|
||||
explicitly. The current `--local_mac_optimal_settings` helper forces
|
||||
`mode="local"`, so combining it with `--mode realtime` does not preserve realtime
|
||||
mode.
|
||||
|
||||
### Remote API setup
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--llm_backend responses-api \
|
||||
--model_name gpt-5.4-mini \
|
||||
--responses_api_api_key YOUR_API_KEY
|
||||
```
|
||||
@@ -0,0 +1,230 @@
|
||||
# Realtime API
|
||||
|
||||
## High-level architecture
|
||||
|
||||
The server runs as a handler thread inside the existing pipeline. A FastAPI/uvicorn instance serves the WebSocket endpoint at `/v1/realtime`, while the same queue-driven handler chain (VAD, STT, LLM, TTS) processes audio in parallel threads.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph client [Client]
|
||||
WS["WebSocket /v1/realtime"]
|
||||
end
|
||||
|
||||
subgraph server [Realtime Server]
|
||||
Router["WebSocket Router"]
|
||||
Service["RealtimeService"]
|
||||
Config["RuntimeConfig"]
|
||||
end
|
||||
|
||||
subgraph pipeline [Pipeline Threads]
|
||||
VAD["VAD"]
|
||||
STT["STT"]
|
||||
TN["TranscriptionNotifier"]
|
||||
LLM["LLM"]
|
||||
Proc["LMOutputProcessor"]
|
||||
TTS["TTS"]
|
||||
end
|
||||
|
||||
WS -- "client events (JSON)" --> Router
|
||||
Router -- "parse / dispatch" --> Service
|
||||
Service -- "PCM chunks" --> VAD
|
||||
Service -- "session config" --> Config
|
||||
Config -. "read by" .-> VAD & LLM & TTS
|
||||
VAD -- "speech segments" --> STT
|
||||
STT -- "transcript" --> TN
|
||||
TN -- "transcript + notify" --> LLM
|
||||
LLM -- "text + tools" --> Proc
|
||||
Proc -- "clean text" --> TTS
|
||||
TTS -- "PCM audio" --> Router
|
||||
Proc -- "assistant_text / tools" --> Router
|
||||
VAD -- "speech_started/stopped" --> Router
|
||||
TN -- "transcription events" --> Router
|
||||
Router -- "server events (JSON)" --> WS
|
||||
```
|
||||
|
||||
**Key flow:**
|
||||
|
||||
1. **Inbound audio**: Client sends `input_audio_buffer.append` with base64 PCM. `RealtimeService` decodes, resamples to 16 kHz, splits into 512-sample chunks, and puts them on the `recv_audio_chunks_queue` for VAD.
|
||||
2. **Speech detection**: VAD detects speech boundaries and emits `speech_started` / `speech_stopped` events on the `text_output_queue`. Full utterance audio goes to STT.
|
||||
3. **Transcription**: STT output passes through `TranscriptionNotifier`, which taps the transcript for `transcription.delta` / `transcription.completed` events before forwarding to the LLM.
|
||||
4. **Generation**: The LLM generates text (and optional tool calls). `LMOutputProcessor` splits the output: clean text goes to TTS, and `assistant_text` + tool call dicts go to the `text_output_queue`.
|
||||
5. **Outbound audio**: TTS writes PCM chunks to `send_audio_chunks_queue`. The router's async `_send_loop` drains both queues, encoding PCM as `response.output_audio.delta` events and translating internal messages into protocol events.
|
||||
6. **Session config**: `session.update` events deep-merge into `RuntimeConfig`, which is a shared Pydantic model read by VAD (turn detection thresholds), LLM (instructions, tools), and TTS (voice) at processing time.
|
||||
|
||||
---
|
||||
|
||||
## Supported OpenAI Realtime Events
|
||||
|
||||
### Client -> Server
|
||||
|
||||
| Event | Description |
|
||||
|---|---|
|
||||
| `input_audio_buffer.append` | Stream base64 PCM audio. Decoded, resampled to 16 kHz, and chunked for the VAD. |
|
||||
| `session.update` | Deep-merge session config (instructions, tools, voice, turn detection, audio format). |
|
||||
| `conversation.item.create` | Inject `input_text` or `function_call_output` into the LLM context without triggering generation. |
|
||||
| `response.create` | Trigger LLM generation. Supports per-response `instructions` and `tool_choice` overrides. |
|
||||
| `response.cancel` | Cancel the in-progress response and re-enable listening. |
|
||||
|
||||
### Server -> Client
|
||||
|
||||
| Event | Description |
|
||||
|---|---|
|
||||
| `session.created` | Sent on connection with current session config. |
|
||||
| `error` | Protocol errors (`session_limit_reached`, `unknown_or_invalid_event`, `invalid_session_type`, `conversation_already_has_active_response`, etc.) |
|
||||
| `input_audio_buffer.speech_started` | VAD detected user speech. |
|
||||
| `input_audio_buffer.speech_stopped` | End of user speech segment. |
|
||||
| `conversation.item.created` | Acknowledges injected `input_text` from `conversation.item.create`. |
|
||||
| `conversation.item.input_audio_transcription.delta` | Streaming partial transcript (when live transcription is enabled). |
|
||||
| `conversation.item.input_audio_transcription.completed` | Final transcript for the user turn (with duration usage). |
|
||||
| `response.created` | Emitted on the first outbound audio chunk (response is `in_progress`). |
|
||||
| `response.output_audio.delta` | Base64 PCM audio chunk from TTS. |
|
||||
| `response.output_audio.done` | Audio stream complete for the current output item. |
|
||||
| `response.output_audio_transcript.done` | Full assistant text transcript for the turn. |
|
||||
| `response.function_call_arguments.done` | Tool call with `call_id`, `name`, and JSON `arguments`. |
|
||||
| `response.done` | Response finished (`completed`, `cancelled` with reason `turn_detected` or `client_cancelled`). |
|
||||
|
||||
---
|
||||
|
||||
## Tool Calling Design
|
||||
|
||||
Tool calling works through two distinct paths depending on the LLM backend, but both converge to the same wire protocol for the client.
|
||||
|
||||
### Local LLM path (`LanguageModelHandler` -- transformers / mlx-lm)
|
||||
|
||||
Tools defined in `session.update` are converted to `FunctionTool` objects. Each tool's JSON Schema `parameters` are turned into a Python `inspect.Signature` (via `signature_from_schema`), and `to_code_prompt()` renders a human-readable `def name(...): """docstring"""` block.
|
||||
|
||||
These tool signatures are injected into the system prompt using a Jinja2 template (`tool_prompt.py`) that instructs the model to wrap tool calls in `<code>...</code>` delimiters:
|
||||
|
||||
```
|
||||
<code>
|
||||
function_name(arg_name_1=value1, arg_name_2='string_value')
|
||||
</code>
|
||||
```
|
||||
|
||||
After generation, `_extract_tools` uses a regex to find `<code>` blocks, then `extract_function_calls_from_text` parses each `name(kwargs)` call and validates it against registered tools. Valid calls become `ResponseFunctionToolCall` dicts with generated `call_id`s.
|
||||
|
||||
### Remote API paths
|
||||
|
||||
`ResponsesApiModelHandler` passes tools natively to `client.responses.create`.
|
||||
The API returns structured `function_call` items. Per-response `tool_choice`
|
||||
overrides are read from `response.create`.
|
||||
|
||||
`ChatCompletionsLanguageModelHandler` also contains conversion for Chat
|
||||
Completions tool-call results, but its current `_request()` removes `tools` and
|
||||
`tool_choice` before the HTTP call to avoid a provider rejection. That backend
|
||||
does not currently declare native tools to its provider. Do not assume its
|
||||
result parsing implies working native tool selection.
|
||||
|
||||
### Common output path
|
||||
|
||||
Both handlers yield `(text, language_code, tools)` tuples. `LMOutputProcessor` forwards clean text to TTS and puts `{"type": "assistant_text", "text": ..., "tools": [...]}` on the `text_output_queue`. The router's `_send_loop` translates these into:
|
||||
- `response.output_audio_transcript.done` for the text
|
||||
- `response.function_call_arguments.done` for each tool call
|
||||
|
||||
### Tool result flow
|
||||
|
||||
1. Client executes the tool and sends `conversation.item.create` with `type: "function_call_output"` and `output: "<result>"`
|
||||
2. `RealtimeService` appends the tool output to the chat context and emits `conversation.item.created`; this does not trigger generation.
|
||||
3. If the tool result needs to be spoken to the user, such as camera/search/data results, the client sends `response.create` to trigger follow-up generation.
|
||||
4. For fire-and-forget robot actions such as dance, emotion, head movement, stop, or idle tools, the client can stop after `conversation.item.created`; the assistant should already have spoken the natural lead-in before the tool call.
|
||||
|
||||
---
|
||||
|
||||
## Interruption Handling
|
||||
|
||||
Barge-in (user speaks while the assistant is playing audio) is handled cooperatively between the VAD, the `_send_loop`, and the LLM/TTS handlers via a shared `CancelScope` object (`cancel_scope.py`).
|
||||
|
||||
### CancelScope design
|
||||
|
||||
`CancelScope` replaces the old two-signal pattern (`cancel_response` Event + `discard_stale_output` boolean) with a single object that manages:
|
||||
|
||||
- **Generation counter** (`cancel_scope.generation`): pipeline threads (LLM, TTS) capture the current generation at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token. When `cancel()` is called, the generation increments and all prior generations become stale -- no timing games required.
|
||||
- **Discard flag** (`cancel_scope.discarding`): set by `cancel()`, checked by the async `_send_loop` to drop output from superseded generations that arrives between `cancel()` and `response_done()`. Cleared by `response_done(generation)` (only when the sentinel's generation matches the discarded or current one -- sentinels from unrelated older generations are ignored), by `new_response()` on an explicit `response.create`, or by `reset()` on session claim/release.
|
||||
|
||||
Pipeline output is **generation-tagged**: `AudioOutput` chunks and `AssistantTextEvent`s carry a `cancel_generation` field stamped by the handler that produced them. The send loop's `_generation_is_discardable` drops an item if its generation is stale, or if `discarding` is set and the item is not from the current generation. Output from the *current* generation always passes through, so a fresh response is never swallowed by a lingering discard window (e.g. a superseded speculative turn whose TTS never emitted a `__RESPONSE_DONE__` sentinel).
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant VAD
|
||||
participant SendLoop as _send_loop
|
||||
participant LLM
|
||||
participant TTS
|
||||
participant Client
|
||||
|
||||
Note over TTS,Client: Response active or pending (in_response / response_pending)
|
||||
User->>VAD: speaks
|
||||
VAD->>SendLoop: speech_started on text_output_queue
|
||||
SendLoop->>Client: input_audio_buffer.speech_started
|
||||
SendLoop->>Client: response.done (status=cancelled, reason=turn_detected)
|
||||
SendLoop->>SendLoop: cancel_scope.cancel() (gen++ & discarding=True)
|
||||
SendLoop->>SendLoop: flush output_queue + text_output_queue
|
||||
SendLoop->>SendLoop: response_playing.clear()
|
||||
LLM->>LLM: is_stale(gen) → True, aborts generation
|
||||
TTS->>TTS: is_stale(gen) → True, aborts generation
|
||||
TTS->>SendLoop: __RESPONSE_DONE__ (tagged with gen)
|
||||
SendLoop->>SendLoop: cancel_scope.response_done(gen) (discarding=False)
|
||||
Note over VAD,Client: Pipeline is now processing the new user utterance
|
||||
```
|
||||
|
||||
**Step by step:**
|
||||
|
||||
1. **VAD detects speech**: puts a `SpeechStartedEvent` on `text_output_queue`.
|
||||
2. **`_send_loop` processes text events first** (priority over audio): translates `speech_started` into protocol events. If an active response was in progress, `RealtimeService.dispatch_pipeline_event` emits `response.output_audio.done` + `response.done` with `status="cancelled"` and `reason="turn_detected"`.
|
||||
3. **Cancel + queue flush**: if a response is active (`in_response`) *or* pending (`response_pending` -- accepted `response.create`, no audio yet), and interrupts are enabled (see step 4), the send loop calls `cancel_scope.cancel()` (increments generation, enables discard), clears `response_pending`, drains `output_queue` (preserving `__RESPONSE_DONE__` sentinels) and `text_output_queue` (preserving user-side events: `speech_stopped`, partial/completed transcriptions, token usage), then clears `response_playing`.
|
||||
4. **Interrupt gating**: the cancel only fires if the `SpeechStartedEvent.interrupt_response` flag is set *and* the session config allows it (`turn_detection.interrupt_response`, read via `RuntimeConfig.interrupt_response_enabled`, default true). When disabled, user speech during a response is transcribed but the response keeps playing.
|
||||
5. **LLM/TTS cancellation**: handlers capture `gen = cancel_scope.generation` at the start of each response and check `cancel_scope.is_stale(gen)` on every streaming token, aborting early when stale.
|
||||
6. **Discard guard**: while `cancel_scope.discarding` is True, the send loop drops audio chunks and assistant text whose `cancel_generation` is not current (see `_generation_is_discardable` above). The guard clears when a `__RESPONSE_DONE__` with a matching generation arrives (via `cancel_scope.response_done(gen)`), or when an explicit `response.create` starts a new response (`cancel_scope.new_response()`).
|
||||
7. **Client-initiated cancel**: `response.cancel` calls `cancel_scope.cancel()` (only if a response was active), flushes both queues with the same preservation rules, triggers `finish_response(status="cancelled", reason="client_cancelled")`, re-enables `should_listen`, and clears `response_playing`.
|
||||
8. **Spurious cancel safety**: if no response is active, `cancel_scope.cancel()` is not called, preventing the discard guard from being set without a `__RESPONSE_DONE__` to clear it.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Run these examples from the repository root in an environment with the speech
|
||||
dependencies installed. See [LLM](llm.md), [STT](stt.md), and [TTS](tts.md) for
|
||||
backend-specific options; this checkout does not include a package manifest.
|
||||
|
||||
### Local LLM with Transformers
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--mode realtime \
|
||||
--stt parakeet-tdt \
|
||||
--llm_backend transformers \
|
||||
--tts kokoro \
|
||||
--model_name "Qwen/Qwen3-4B-Instruct-2507" \
|
||||
--llm_device mps \
|
||||
--llm_torch_dtype float16 \
|
||||
--enable_live_transcription
|
||||
```
|
||||
|
||||
### Local LLM with MLX-LM
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--mode realtime \
|
||||
--stt parakeet-tdt \
|
||||
--llm_backend mlx-lm \
|
||||
--tts kokoro \
|
||||
--model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" \
|
||||
--llm_device mps \
|
||||
--llm_torch_dtype float16 \
|
||||
--enable_live_transcription
|
||||
```
|
||||
|
||||
### Remote LLM with OpenAI-compatible API
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--mode realtime \
|
||||
--stt parakeet-tdt \
|
||||
--llm_backend responses-api \
|
||||
--tts kokoro \
|
||||
--model_name "openai/gpt-oss-20b:groq" \
|
||||
--responses_api_base_url "https://router.huggingface.co/v1" \
|
||||
--responses_api_api_key "$HF_TOKEN" \
|
||||
--responses_api_stream \
|
||||
--enable_live_transcription
|
||||
```
|
||||
@@ -0,0 +1,166 @@
|
||||
# Speech-to-text backends
|
||||
|
||||
This document summarizes the speech-to-text implementations in
|
||||
[speech_to_speech/STT](../../speech_to_speech/STT), including language support,
|
||||
language abbreviations, and pipeline options. Run examples from the repository
|
||||
root in an environment with the speech dependencies installed.
|
||||
|
||||
## Available STT Modes (`--stt`)
|
||||
|
||||
- `whisper` → `STT/whisper_stt_handler.py`
|
||||
- `whisper-mlx` → `STT/lightning_whisper_mlx_handler.py`
|
||||
- `mlx-audio-whisper` → `STT/mlx_audio_whisper_handler.py`
|
||||
- `faster-whisper` → `STT/faster_whisper_handler.py`
|
||||
- `parakeet-tdt` → `STT/parakeet_tdt_handler.py`
|
||||
- `paraformer` → `STT/paraformer_handler.py`
|
||||
|
||||
## Language Support by Handler
|
||||
|
||||
### 1) Whisper (`--stt whisper`)
|
||||
|
||||
- Handler: `WhisperSTTHandler`
|
||||
- Language input flag: `--language` (from shared Whisper args)
|
||||
- Supports fixed language (e.g. `en`) or `auto`
|
||||
- Internal supported language list:
|
||||
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
|
||||
- Behavior:
|
||||
- Detects language from token output
|
||||
- If detected language is outside the supported list, it falls back to the previous language
|
||||
|
||||
### 2) Lightning Whisper MLX (`--stt whisper-mlx`)
|
||||
|
||||
- Handler: `LightningWhisperSTTHandler`
|
||||
- Uses same shared `--language` argument as Whisper
|
||||
- Internal supported language list:
|
||||
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
|
||||
- Behavior:
|
||||
- If `--language auto`, model auto-detects each utterance
|
||||
- If detected language is unsupported, falls back to last supported language
|
||||
|
||||
### 3) MLX Audio Whisper (`--stt mlx-audio-whisper`)
|
||||
|
||||
- Handler: `MLXAudioWhisperSTTHandler`
|
||||
- Model flag: `--mlx_audio_whisper_model_name`
|
||||
- Language still comes from shared `--language` flag (wired in pipeline)
|
||||
- Internal supported language list:
|
||||
- `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
|
||||
- Behavior:
|
||||
- Uses fixed language unless `--language auto`
|
||||
- Falls back to last known supported language when needed
|
||||
|
||||
### 4) Faster-Whisper (`--stt faster-whisper`)
|
||||
|
||||
- Handler: `FasterWhisperSTTHandler`
|
||||
- Language flag: `--faster_whisper_stt_gen_language`
|
||||
- Default language: `en`
|
||||
- Note:
|
||||
- This handler passes generation kwargs directly to `faster_whisper.WhisperModel.transcribe(...)`
|
||||
- Effective language coverage depends on selected Faster-Whisper/OpenAI Whisper model
|
||||
|
||||
### 5) Parakeet TDT (`--stt parakeet-tdt`)
|
||||
|
||||
- Handler: `ParakeetTDTSTTHandler`
|
||||
- Language flag: `--parakeet_tdt_language` (optional)
|
||||
- Supports auto language detection when language not specified
|
||||
- Declared supported language list (25 European languages):
|
||||
- `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `pl`, `ru`, `uk`, `cs`, `sk`, `hu`, `ro`, `bg`, `hr`, `sl`, `sr`, `da`, `no`, `sv`, `fi`, `et`, `lv`, `lt`
|
||||
- Backend behavior:
|
||||
- On macOS/MPS: MLX (`mlx-community/parakeet-tdt-0.6b-v3`)
|
||||
- On CUDA/CPU: nano-parakeet (`nvidia/parakeet-tdt-0.6b-v3`)
|
||||
|
||||
### 6) Paraformer (`--stt paraformer`)
|
||||
|
||||
- Handler: `ParaformerSTTHandler`
|
||||
- Model flag: `--paraformer_stt_model_name`
|
||||
- Default model: `paraformer-zh`
|
||||
- No dedicated language flag in current args class
|
||||
- Practical support:
|
||||
- Depends on selected FunASR model checkpoint
|
||||
- Default setup is Chinese-oriented (`zh`)
|
||||
|
||||
## Language Abbreviations (ISO-style codes seen in STT handlers)
|
||||
|
||||
| Code | Language |
|
||||
|---|---|
|
||||
| `en` | English |
|
||||
| `fr` | French |
|
||||
| `es` | Spanish |
|
||||
| `zh` | Chinese |
|
||||
| `ja` | Japanese |
|
||||
| `ko` | Korean |
|
||||
| `hi` | Hindi |
|
||||
| `de` | German |
|
||||
| `pt` | Portuguese |
|
||||
| `pl` | Polish |
|
||||
| `it` | Italian |
|
||||
| `nl` | Dutch |
|
||||
| `ru` | Russian |
|
||||
| `uk` | Ukrainian |
|
||||
| `cs` | Czech |
|
||||
| `sk` | Slovak |
|
||||
| `hu` | Hungarian |
|
||||
| `ro` | Romanian |
|
||||
| `bg` | Bulgarian |
|
||||
| `hr` | Croatian |
|
||||
| `sl` | Slovenian |
|
||||
| `sr` | Serbian |
|
||||
| `da` | Danish |
|
||||
| `no` | Norwegian |
|
||||
| `sv` | Swedish |
|
||||
| `fi` | Finnish |
|
||||
| `et` | Estonian |
|
||||
| `lv` | Latvian |
|
||||
| `lt` | Lithuanian |
|
||||
| `auto` | Per-utterance automatic language detection |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Whisper (Transformers)
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt whisper --language es
|
||||
python -m speech_to_speech.s2s_pipeline --stt whisper --language auto
|
||||
```
|
||||
|
||||
### Whisper MLX (LightningWhisperMLX)
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt whisper-mlx --language auto --device mps
|
||||
```
|
||||
|
||||
### MLX Audio Whisper
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt mlx-audio-whisper \
|
||||
--mlx_audio_whisper_model_name mlx-community/whisper-large-v3-turbo \
|
||||
--language auto
|
||||
```
|
||||
|
||||
### Faster-Whisper
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt faster-whisper \
|
||||
--faster_whisper_stt_model_name large-v3 \
|
||||
--faster_whisper_stt_gen_language en
|
||||
```
|
||||
|
||||
### Parakeet TDT
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt --parakeet_tdt_device auto
|
||||
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt --parakeet_tdt_language es
|
||||
```
|
||||
|
||||
With live transcription (MLX or CUDA/nano-parakeet backend):
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt \
|
||||
--enable_live_transcription \
|
||||
--live_transcription_update_interval 0.25
|
||||
```
|
||||
|
||||
### Paraformer
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline --stt paraformer --paraformer_stt_model_name paraformer-zh
|
||||
```
|
||||
@@ -0,0 +1,151 @@
|
||||
# Text-to-speech backends
|
||||
|
||||
## Available TTS modes (`--tts`)
|
||||
|
||||
Runtime-supported values in [s2s_pipeline.py](../../speech_to_speech/s2s_pipeline.py):
|
||||
|
||||
- `chatTTS` → `chatTTS_handler.py`
|
||||
- `facebookMMS` → `facebookmms_handler.py`
|
||||
- `pocket` → `pocket_tts_handler.py`
|
||||
- `kokoro` → `kokoro_handler.py`
|
||||
- `qwen3` → `qwen3_tts_handler.py`
|
||||
|
||||
Run examples from the repository root in an environment with the speech
|
||||
dependencies installed. The UI requirements do not install TTS dependencies.
|
||||
|
||||
## Usage
|
||||
|
||||
### 1) ChatTTS (`--tts chatTTS`)
|
||||
|
||||
Primary args prefix: `--chat_tts_*`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts chatTTS \
|
||||
--chat_tts_device cuda \
|
||||
--chat_tts_stream true \
|
||||
--chat_tts_chunk_size 512
|
||||
```
|
||||
|
||||
### 2) Facebook MMS (`--tts facebookMMS`)
|
||||
|
||||
Primary args prefix: `--facebook_mms_*` plus `--tts_language`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts facebookMMS \
|
||||
--facebook_mms_device cuda \
|
||||
--tts_language en
|
||||
```
|
||||
|
||||
This handler maps STT language codes (e.g. `en`, `fr`, `es`) to MMS model suffixes (e.g. `eng`, `fra`, `spa`) and reloads the model on language changes.
|
||||
|
||||
### 3) Pocket TTS (`--tts pocket`)
|
||||
|
||||
Primary args prefix: `--pocket_tts_*`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts pocket \
|
||||
--pocket_tts_voice jean \
|
||||
--pocket_tts_device cpu \
|
||||
--pocket_tts_sample_rate 16000
|
||||
```
|
||||
|
||||
Available preset voices include:
|
||||
`alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`, `azelma`.
|
||||
|
||||
### 4) Kokoro (`--tts kokoro`)
|
||||
|
||||
Primary args prefix: `--kokoro_*`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts kokoro \
|
||||
--kokoro_device auto \
|
||||
--kokoro_voice bm_fable \
|
||||
--kokoro_lang_code b
|
||||
```
|
||||
|
||||
Behavior:
|
||||
- Uses MLX backend on Apple Silicon (`mlx-community/Kokoro-82M-bf16`)
|
||||
- Uses native kokoro pipeline otherwise (`hexgrad/Kokoro-82M`)
|
||||
- Can auto-switch voice/language based on STT language code mapping
|
||||
|
||||
### 5) Qwen3-TTS (`--tts qwen3`)
|
||||
|
||||
Primary args prefix: `--qwen3_tts_*`
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts qwen3 \
|
||||
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--qwen3_tts_device cuda \
|
||||
--qwen3_tts_backend ggml \
|
||||
--qwen3_tts_speaker Aiden \
|
||||
--qwen3_tts_language auto \
|
||||
--qwen3_tts_non_streaming_mode True
|
||||
```
|
||||
|
||||
Behavior:
|
||||
- Uses `faster-qwen3-tts` on non-macOS platforms, defaulting to the GGML backend. Pass `--qwen3_tts_backend torch` to use the CUDA-graphs backend instead.
|
||||
- Uses `mlx-audio` on Apple Silicon and auto-maps `Qwen/...` model IDs to `mlx-community/...`, defaulting to the `6bit` MLX variant unless the model name already pins a suffix.
|
||||
- Supports MLX quantization overrides on Apple Silicon via `--qwen3_tts_mlx_quantization bf16|4bit|6bit|8bit`.
|
||||
- Keeps the existing voice-clone/custom-voice/voice-design handler flow intact.
|
||||
- Defaults to the CustomVoice model with speaker `Aiden`, so no reference audio is required. Voice-clone/base models can still use `--qwen3_tts_ref_audio`.
|
||||
|
||||
Install notes for Linux GGML:
|
||||
- The default PyPI `qwentts-cpp-python` wheel targets CUDA 12.8.
|
||||
- If that wheel does not match your CUDA runtime, install one of the Hugging Face wheelhouse builds before installing `speech-to-speech`.
|
||||
|
||||
```bash
|
||||
pip install "qwentts-cpp-python==0.3.1+cu130" \
|
||||
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130
|
||||
pip install speech-to-speech
|
||||
```
|
||||
|
||||
Available wheelhouse directories include `cu124`, `cu128`, `cu130`, and `cpu`.
|
||||
|
||||
Example for Apple Silicon using the default 6-bit MLX variant:
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts qwen3 \
|
||||
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--qwen3_tts_speaker Aiden
|
||||
```
|
||||
|
||||
You can override the default and select `bf16`, `4bit`, or `8bit` explicitly:
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--tts qwen3 \
|
||||
--qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--qwen3_tts_mlx_quantization 4bit \
|
||||
--qwen3_tts_speaker Aiden
|
||||
```
|
||||
|
||||
For voice-cloning examples, an optional reference clip is retained at
|
||||
[docs/examples/reference-audio.wav](../examples/reference-audio.wav). Supply an
|
||||
appropriate base/voice-cloning model and `--qwen3_tts_ref_audio` explicitly;
|
||||
CustomVoice mode does not need this clip. The repository does not include a TTS
|
||||
benchmark script.
|
||||
|
||||
## Setup
|
||||
|
||||
### Low-latency GPU setup
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--stt whisper \
|
||||
--tts pocket
|
||||
```
|
||||
|
||||
### Apple Silicon setup
|
||||
|
||||
```bash
|
||||
python -m speech_to_speech.s2s_pipeline \
|
||||
--local_mac_optimal_settings
|
||||
```
|
||||
|
||||
`--tts pocket` and `--tts kokoro` are also valid options on macOS.
|
||||
Reference in New Issue
Block a user