vocero-s2s/docs/PROJECT_OVERVIEW.md

# speech-to-speech — Project Overview

> **Package**: `speech-to-speech`
> **Version**: `0.2.11`
> **Author**: Hugging Face
> **License**: Apache-2.0
> **Python**: 3.10 – 3.12
> **Tagline**: Low-latency end-to-end Speech-to-Speech pipeline for building realtime voice agents.

---

## 1. What is `speech-to-speech`?

`speech-to-speech` is a fully modular, **low-latency audio pipeline** that turns a user's spoken input into a spoken reply by chaining four AI stages together:
**VAD → STT → LLM → TTS**.

Out of the box it exposes an **OpenAI-compatible Realtime WebSocket endpoint** (`ws://host:port/v1/realtime`),
so any browser or SDK that knows how to speak the OpenAI Realtime Protocol can connect and start having natural voice conversations in seconds — without writing a single line of server code.

The pipeline is designed to be **backend-agnostic at every stage**: you can plug different STT / LLM / TTS models
depending on the available hardware (NVIDIA CUDA, Apple Silicon MLX, or CPU-only), the language(s) of your users,
and the latency / quality trade-off you need.

A built-in, optional **RAG (Retrieval Augmented Generation) server-side subsystem** lets the LLM answers be grounded
in your own private knowledge base (Markdown / TXT / JSONL documents), while staying 100 % transparent for the client.
Dynamic knowledge base updates are exposed over the same HTTP server via a 11-endpoint REST API.

---

## 2. Key Features

| Feature | Description |
|---|---|
| **Real-time voice protocol** | Native OpenAI `v1/realtime` WebSocket support: `session.update`, `response.create`, function calling, audio deltas, interruptions. Drop-in compatible with the official Realtime SDK and browser playgrounds. |
| **4 execution modes** | `local` (mic → speakers), `socket` (TCP IPC), `websocket` (raw audio WS), `realtime` (OpenAI protocol — default). |
| **6 pluggable STT backends** | Whisper / Whisper-MLX / MLX-Audio-Whisper / Faster-Whisper / Parakeet TDT (default) / Paraformer. |
| **4 pluggable LLM backends** | `transformers` (local) · `mlx-lm` (Apple Silicon) · `responses-api` (OpenAI tool-calling aware endpoint) · `chat-completions` (any OpenAI-compatible `/v1/chat/completions` server). |
| **5 pluggable TTS backends** | ChatTTS · Facebook MMS · Pocket (tiny CPU) · Kokoro · Qwen3-TTS (default, custom voice 1.7B). |
| **Multi-hardware ready** | NVIDIA CUDA (Linux), Apple Silicon MLX + MPS (macOS), CPU fallback for every stage. |
| **Pool of N isolated pipelines** | `--num_pipelines N` runs N independent sessions in parallel (each one with its own VAD / STT / LLM / TTS handlers and conversation state). Ideal for small multi-concurrent deployments. |
| **Built-in VAD & interruption** | Voice-activity detection with configurable threshold and streaming silence → STT finalization; the LLM can be interrupted mid-speech by the user and cancelled cleanly through a `CancelScope`. |
| **Automatic language detection** | Via `lingua-language-detector` for the assistant reply language. |
| **Live partial transcription** | Parakeet-TDT emits live partial transcriptions every 500 ms so clients can display "user is speaking …" text. |
| **Full tool-calling support** | With `responses-api` backend: streamed function argument delivery, parallel tool calls, custom voices over TTS — everything compliant with the OpenAI Responses API surface. |
| **Optional RAG server-side injection** | Transparent retrieval hook on every LLM turn. Sentence-Transformers embeddings, NPZ-index persistence, configurable top-k / cosine threshold, system or user-message injection. Multilingual by default (Spanish / Italian / English out of the box). |
| **Dynamic Knowledge Base REST API** | 11 endpoints to list, search, upsert, update, remove, and reload KB contents at runtime — including cross-restart persistence through `kb/_dynamic.jsonl`. |
| **Structured CLI / JSON configuration** | Every parameter is a `@dataclass` HfArgumentParser argument with sane defaults, or a full JSON config file you can pass in as single argument. |
| **PyPI-ready packaging & release flow** | Configured `uv build` + `twine check` + GitHub Actions publish workflow (tag `vX.Y.Z` triggers upload). |

---

## 3. Architecture at a glance

A single conversation turn in **realtime** mode looks like this:

```
                         ┌─────────────────────────────────────────────────────────┐
                         │                    uvicorn + FastAPI                    │
                         │  ┌───────────────────────────────────────────────────┐  │
   Mic / Browser  ──► WS │  │   RealtimeService (session + routing + events)    │  │
   (audio in +       │   │  └───────────────────────────────────────────────────┘  │
    events)          │   │                                                           │
                     │   │  Pipeline pool  ──►  PipelineUnit #1  ──►  PipelineUnit #N│
                     │   └─────────────────────────────────────────────────────────┘
                     │                      │
                     ▼                      ▼
              ┌──────────────────────────────────────────────┐
              │     Single pipeline unit (one per user)       │
              │                                                │
              │   1. VAD            ──►   speech start / end   │
              │   2. STT (5 flavours) ──►   last user text     │
              │        │                                     │
              │        ▼                                     │
              │   ┌────────────────────────┐                  │
              │   │  RAG SEARCH HOOK        │ ◄─── kb/_index.npz
              │   │  (inject only if ≥ N)   │      + md/txt/jsonl
              │   └────────────────────────┘      + _dynamic.jsonl
              │        │                                     │
              │        ▼                                     │
              │   3. LLM (4 flavours) ──►  text reply + tools│
              │        │                                     │
              │        ▼                                     │
              │   4. TTS (5 flavours) ──►  streamed PCM audio│
              └──────────────────────────────────────────────┘
                                                      │
                                                      ▼
                                              ◄── WS audio / delta events
```

Key invariants:

- **Each connection = one `PipelineUnit`** (allocated atomically; queues and state are isolated; if N units are busy, the `N+1`-th connection is rejected).
- **The LLM stage never sees raw audio**. The pipeline only forwards text buffers to the LLM handler, so model vendors and backend switches remain transparent.
- **The RAG stage is a pure side-effect on the text buffer**: it reads the most recent user text, runs an embedding + cosine top-k + threshold filter, and prepends matching knowledge as an extra system (or user) message — with a clear log line so you can always see exactly what was injected.
- **TTS streams** (all modern backends): audio deltas are emitted as they are synthesised, so the first byte of a reply reaches the speaker well before the LLM has finished generating text.

---

## 4. Execution modes (`--mode`)

Configurable via [ModuleArguments.mode](file:///home/azurian/speech-to-speech/src/speech_to_speech/arguments_classes/module_arguments.py#L11-L16)
(default: `realtime`).

| Mode | Input | Output | Best for |
|---|---|---|---|
| `local` | Local microphone via `sounddevice` / `miniaudio` | Local speakers | Desktop demos, quick prototyping on laptop. |
| `socket` | Raw PCM chunks on TCP socket | Raw PCM on TCP socket | Legacy / embedded setups, custom transport. |
| `websocket` | Raw PCM over WS endpoint | Raw PCM over WS | Minimal custom frontend that just pushes audio. |
| `realtime` | OpenAI Realtime Protocol WS (`/v1/realtime`) | Same protocol + RAG REST API on the same HTTP server | **Production & SDK integration.** All clients that support the Realtime API (OpenAI SDK, web playgrounds, Swift / Kotlin / JS wrappers) connect here. |

---

## 5. Supported backends

### 5.1 STT — Speech to Text

| `--stt` name | Model family | Default model | Device highlights |
|---|---|---|---|
| `whisper` | HuggingFace Transformers Whisper | `distil-whisper/distil-large-v3` | CUDA / CPU |
| `whisper-mlx` | MLX Whisper | `mlx-community/whisper-large-v3-turbo` | Apple Silicon (macOS) |
| `mlx-audio-whisper` | Apple `mlx-audio` | `mlx-community/whisper-large-v3-turbo` | Apple Silicon |
| `faster-whisper` | CTranslate2 Quantized Whisper | `tiny.en` | Very low latency CPU |
| `parakeet-tdt` | **(default)** HuggingFace Parakeet TDT | `parakeet-tdt-1.1b` | Streaming-friendly + **live partial transcription** on CUDA |
| `paraformer` | Alibaba Paraformer | `paraformer-zh` | Chinese-only deployments |

### 5.2 LLM — Language Model

| `--llm_backend` | Description | Default model |
|---|---|---|
| `transformers` | Local, process-bound HF Transformers generation | `Qwen/Qwen3-4B-Instruct-2507` |
| `mlx-lm` | Apple Silicon local 4-bit / 8-bit quantised inference | `mlx-community/...` |
| `responses-api` | **(default)** OpenAI Responses-API compatible remote endpoint (full tool-calling surface + streamed function args) | `gpt-5.4-mini` |
| `chat-completions` | Any OpenAI-compatible `/v1/chat/completions` remote — plug in vLLM, TGI, Ollama, llama.cpp server, SGLang, TabbyAPI, etc. | (auto-detects via `base_url + /v1/models`) |

### 5.3 TTS — Text to Speech

| `--tts` name | Backend | Default model | Strengths |
|---|---|---|---|
| `chatTTS` | ChatTTS (optional group `chattts`) | — | Highly conversational, English + Chinese, very expressive prosody |
| `facebookMMS` | Facebook MMS (optional group `facebook-mms`) | `facebook/mms-tts-eng` | Ultra-lightweight, covers 1,100+ languages |
| `pocket` | PocketTTS (small CPU) | — | Zero-dependency on-device, great for embedded / low-RAM |
| `kokoro` | Kokoro TTS (optional group `kokoro`) | — | State of the art English & Japanese neural TTS |
| `qwen3` | **(default)** Qwen3-TTS via `faster-qwen3-tts` GGML | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Multilingual, supports reference voice cloning, 12 Hz token rate → ultra-low latency. |

All TTS backends are reachable through a common streaming interface so the rest of the pipeline is backend-agnostic.

---

## 6. RAG Server-Side (optional add-on)

The RAG subsystem is fully documented in:

- 🌍 English / Spanish logic: [RAG_SERVER_SIDE.md](file:///home/azurian/speech-to-speech/docs/RAG_SERVER_SIDE.md)
- 🇪🇸 Spanish translation: [RAG_SERVER_SIDE.es.md](file:///home/azurian/speech-to-speech/docs/RAG_SERVER_SIDE.es.md)

At a glance:

```
$ ./start_pipeline_rag.sh                         # starts pipeline + RAG REST API
RAG: Inizializzazione modello embedding=paraphrase-multilingual-MiniLM-L12-v2 device=cuda su kb_path=/home/.../kb
RAG: Índice cargado desde disco: 9 chunk (shape=(9, 384)).
RAG: Activo. kb=... top_k=3 umbral=0.250 inject_as=system idioma=es
RAG HTTP API montata su prefix='/v1/rag'
```

Highlights:

- **Zero client changes** — injection happens server-side in the LLM chat buffer before every turn.
- **Pluggable embedding models** with a default multilingual model (`paraphrase-multilingual-MiniLM-L12-v2`, 384 dims) and recommended alternatives for Spanish-only and very-large-KB scenarios.
- **NPZ index persistence** — rebuilds only if content changed. Supports `--rag_force_rebuild`.
- **Three source formats**: Markdown / TXT with automatic recursive chunking (size + overlap configurable), or JSONL for hand-crafted chunks.
- **Configurable retrieval behaviour**: `--rag_top_k`, `--rag_threshold`, `--rag_inject_as (system/user)`, `--rag_language (es/it/en)`.
- **Thread-safe dynamic REST API** (11 endpoints): `/status`, `/sources`, `/chunks`, `/chunks/list`, `/chunks/update`, `/upsert/document`, `/search`, `/add/document`, `/add/chunks`, `/remove`, `/reload`.
- **Cross-restart persistence**: API calls with `persist=true` are appended atomically to `kb/_dynamic.jsonl` and automatically re-indexed on the next startup.
- **Idempotent CRUD pattern**: `/upsert/document` handles create / replace / delete (`text=""`) atomically so clients only need one URL.
- **Fails gracefully**: any retrieval exception is logged but never breaks the conversation turn.

---

## 7. Quick start

### 7.1 Install (PyPI-style editable install)

```bash
git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

# Base package (includes default parakeet-tdt STT + qwen3 TTS)
uv pip install -e .

# Optional: add the RAG subsystem for server-side KB retrieval
uv pip install -e ".[rag]"

# Optional: select which TTS / STT extras you want to bring along
uv pip install -e ".[rag,chattts,kokoro,faster-whisper]"
```

### 7.2 Launch in Realtime mode (default) using an external LLM

This is the most common configuration: LLM runs on a remote OpenAI-compatible server
(e.g. `http://127.0.0.1:8001/v1` with vLLM or TGI), STT + TTS run locally on CUDA / MLX.

Create a small shell wrapper (see [start_pipeline.sh](file:///home/azurian/speech-to-speech/start_pipeline.sh) for the full template):

```bash
#!/bin/bash
set -euo pipefail

LLM_BASE_URL="${LLM_BASE_URL:-http://127.0.0.1:8001/v1}"
LLM_MODEL="${LLM_MODEL:-Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4}"
LLM_API_KEY="${LLM_API_KEY:-placeholder}"
TTS_MODEL="${TTS_MODEL:-Qwen/Qwen3-TTS-12Hz-1.7B-Base}"
STT_MODEL="${STT_MODEL:-distil-whisper/distil-large-v3}"

cd /home/azurian/speech-to-speech

uv run speech-to-speech \
  --mode realtime \
  --ws_host 0.0.0.0 \
  --ws_port 12345 \
  --num_pipelines 2 \
  \
  --llm_backend chat-completions \
  --chat_completions_handler_base_url "$LLM_BASE_URL" \
  --chat_completions_handler_model_name "$LLM_MODEL" \
  --chat_completions_handler_api_key "$LLM_API_KEY" \
  \
  --stt whisper \
  --whisper_stt_model_name "$STT_MODEL" \
  \
  --tts qwen3 \
  --qwen3_tts_model_name "$TTS_MODEL"
```

Run it, then connect any OpenAI Realtime client to:

```
ws://<host>:12345/v1/realtime
```

### 7.3 Same pipeline + RAG enabled

Append these flags (see §6 for the full reference and the helper script [start_pipeline_rag.sh](file:///home/azurian/speech-to-speech/start_pipeline_rag.sh)):

```bash
  --rag_enabled \
  --rag_kb_path ./kb \
  --rag_top_k 3 \
  --rag_threshold 0.25 \
  --rag_language es \
  --rag_inject_as system
```

The RAG REST API appears immediately on the same HTTP server:

```bash
curl http://127.0.0.1:12345/v1/rag/status | jq
```

---

## 8. Project layout (highlights)

```
speech-to-speech/
├── pyproject.toml                                     ← package metadata + optional deps groups
├── LICENSE                                            ← Apache-2.0
├── start_pipeline.sh                                  ← reference launch script
├── start_pipeline_rag.sh                              ← launch script + RAG enabled
│
├── src/speech_to_speech/
│   ├── s2s_pipeline.py                                ← entry point (main), pipeline builder, realtime pool
│   ├── baseHandler.py
│   ├── chat.py
│   ├── pipeline/                                      ← queue types, CancelScope, handler types
│   │
│   ├── arguments_classes/                             ← @dataclass HfArgumentParser args (one per handler)
│   │   ├── module_arguments.py                        ← mode / stt / tts / llm_backend / num_pipelines
│   │   ├── rag_arguments.py                           ← 12 RAG-specific flags
│   │   └── ... (Whisper, Qwen3, ChatTTS, VAD, …)
│   │
│   ├── STT/                                           ← six STT handlers
│   ├── TTS/                                           ← five TTS handlers
│   ├── LLM/
│   │   ├── base_openai_compatible_language_model.py   ← RAG injection hook + full tool-calling
│   │   ├── chat_completions_language_model.py
│   │   ├── responses_api_language_model.py
│   │   └── language_model.py                          ← local transformers / mlx-lm handlers
│   │
│   ├── RAG/
│   │   ├── retriever.py                               ← singleton, embeddings, NPZ, search, full CRUD
│   │   └── router.py                                  ← 11 FastAPI /v1/rag endpoints
│   │
│   └── api/openai_realtime/
│       ├── websocket_router.py                        ← FastAPI + Realtime + conditional RAG mount
│       ├── service.py                                 ← Realtime event loop + session/handler routing
│       └── ...
│
├── kb/
│   ├── 01_faq_producto.md                             ← example Spanish FAQ (included in scaffold)
│   ├── 02_politicas_internas.md                       ← example Spanish policies
│   ├── README.md                                      ← format guide
│   └── _dynamic.jsonl                                 ← API-added chunks (auto-generated)
│
└── docs/
    ├── RAG_SERVER_SIDE.md                             ← Italian/English in-depth RAG guide (600+ lines)
    └── RAG_SERVER_SIDE.es.md                          ← Spanish translation
```

---

## 9. Versioning and publishing

The repository ships with a full PyPI release pipeline in
`.github/workflows/publish.yml`:

1. Bump `version` in [pyproject.toml](file:///home/azurian/speech-to-speech/pyproject.toml#L7)
   and `__version__` in [src/speech_to_speech/__init__.py](file:///home/azurian/speech-to-speech/src/speech_to_speech/__init__.py).
2. Merge a release PR containing only those two changes.
3. Tag and push:
   ```bash
   git checkout main && git pull origin main
   git tag -a vX.Y.Z -m "Release vX.Y.Z"
   git push origin vX.Y.Z
   ```
4. The workflow runs `uv build` + `twine check --strict` + PyPI upload automatically.

See [AGENTS.md](file:///home/azurian/speech-to-speech/AGENTS.md) for the repository-level release rules.

---

## 10. Where to go next

- Start with the launch scripts: [start_pipeline.sh](file:///home/azurian/speech-to-speech/start_pipeline.sh) (base) and [start_pipeline_rag.sh](file:///home/azurian/speech-to-speech/start_pipeline_rag.sh) (RAG enabled).
- Dive into the RAG subsystem: [RAG_SERVER_SIDE.md](file:///home/azurian/speech-to-speech/docs/RAG_SERVER_SIDE.md) / [RAG_SERVER_SIDE.es.md](file:///home/azurian/speech-to-speech/docs/RAG_SERVER_SIDE.es.md).
- Tune the per-stage handlers by reading the argument classes in [arguments_classes](file:///home/azurian/speech-to-speech/src/speech_to_speech/arguments_classes) — every flag has inline help.
- Build your own Realtime client against [service.py](file:///home/azurian/speech-to-speech/src/speech_to_speech/api/openai_realtime/service.py) and [websocket_router.py](file:///home/azurian/speech-to-speech/src/speech_to_speech/api/openai_realtime/websocket_router.py).