vocero-client/docs/speech/llm.md

# Language model backends

## Available LLM backends (`--llm_backend`)

Runtime-supported values in [s2s_pipeline.py](../../speech_to_speech/s2s_pipeline.py):

- `transformers` → `language_model.py` (Transformers backend)
- `mlx-lm` → `language_model.py` (MLX backend)
- `responses-api` → `responses_api_language_model.py`
- `chat-completions` → `chat_completions_language_model.py`

Run the commands below from the repository root in an environment with the speech
pipeline dependencies installed. This checkout has no package manifest or complete
speech dependency installer; the UI requirements alone do not install the pipeline.

## Usage

### 1) Transformers (`--llm_backend transformers`)

- Handler: `LanguageModelHandler`
- Typical use: local GPU/CPU inference using Hugging Face Transformers
- Backend-specific args prefix: `--llm_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`

```bash
python -m speech_to_speech.s2s_pipeline \
  --llm_backend transformers \
  --model_name Qwen/Qwen3-4B-Instruct-2507 \
  --llm_device cuda \
  --llm_torch_dtype float16 \
  --llm_gen_max_new_tokens 128
```

Common options:
- `--llm_gen_min_new_tokens`
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`

### 2) MLX-LM (`--llm_backend mlx-lm`)

- Handler: `LanguageModelHandler`
- Typical use: Apple Silicon local inference
- Backend-specific args prefix: same as Transformers (`--llm_*`)

```bash
python -m speech_to_speech.s2s_pipeline \
  --llm_backend mlx-lm \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
  --llm_device mps \
  --llm_gen_max_new_tokens 128
```

Common options:
- `--llm_gen_temperature`
- `--llm_gen_do_sample`
- `--chat_size`
- `--init_chat_prompt`

### 3) OpenAI-compatible API (`--llm_backend responses-api`)

- Handler: `ResponsesApiModelHandler`
- Typical use: remote model serving via OpenAI-compatible endpoints
- Backend-specific args prefix: `--responses_api_*`
- Shared args (from base): `--model_name`, `--chat_size`, `--init_chat_prompt`, `--enable_lang_prompt`

```bash
python -m speech_to_speech.s2s_pipeline \
  --llm_backend responses-api \
  --model_name gpt-5.4-mini \
  --responses_api_api_key YOUR_API_KEY \
  --responses_api_base_url https://api.example.com/v1 \
  --responses_api_stream true
```

Common options:
- `--chat_size`
- `--init_chat_prompt`
- `--user_role`

### 4) Chat Completions (`--llm_backend chat-completions`)

- Handler: `ChatCompletionsLanguageModelHandler`
- Uses the provider's `/chat/completions` API rather than `/responses`
- Inherits the same `--responses_api_base_url`, `--responses_api_api_key`,
  `--responses_api_stream`, and `--responses_api_disable_thinking` flags
- Adds `--responses_api_reasoning_effort` for providers that support that setting

The current handler removes `tools` and `tool_choice` from outgoing provider
requests as a compatibility workaround. Native tool calling therefore differs
from the `responses-api` backend; see the [realtime guide](realtime-api.md).

```bash
python -m speech_to_speech.s2s_pipeline \
  --llm_backend chat-completions \
  --model_name YOUR_MODEL \
  --responses_api_base_url https://api.example.com/v1 \
  --responses_api_api_key YOUR_API_KEY
```

The default LLM backend is `responses-api`. Model names in this guide are examples;
select a model available in your environment or from your provider.

## LLM Behavior

When STT is set to language auto-detection (`--language auto`), LLM handlers can receive `(text, language_code)` and prepend a language control instruction like:

- `Please reply to my message in <language>.`

This helps the assistant respond in the detected language. The behavior is opt-in via `--enable_lang_prompt` (shared across all backends); it defaults to `False`.

## Setup

### CUDA setup

```bash
python -m speech_to_speech.s2s_pipeline \
  --llm_backend transformers \
  --model_name microsoft/Phi-3-mini-4k-instruct
```

### Local Mac setup

```bash
python -m speech_to_speech.s2s_pipeline \
  --local_mac_optimal_settings \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16
```

`--local_mac_optimal_settings` already sets `--llm_backend mlx-lm` and will default the model to `mlx-community/Qwen3-4B-Instruct-2507-bf16` if not overridden.

### Realtime (OpenAI-compatible) setup

Run the server in realtime mode, then connect with the [browser client](../client/usage.md):

```bash
# 1. Start the pipeline in realtime mode
python -m speech_to_speech.s2s_pipeline \
  --mode realtime \
  --llm_backend mlx-lm \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
  --ws_host 0.0.0.0 \
  --ws_port 8765

# 2. Point the browser client at ws://localhost:8765/v1/realtime.
# See the browser-client usage guide for startup commands.
```

For realtime mode on Apple Silicon, set the desired MLX/STT/TTS/device options
explicitly. The current `--local_mac_optimal_settings` helper forces
`mode="local"`, so combining it with `--mode realtime` does not preserve realtime
mode.

### Remote API setup

```bash
python -m speech_to_speech.s2s_pipeline \
  --llm_backend responses-api \
  --model_name gpt-5.4-mini \
  --responses_api_api_key YOUR_API_KEY
```