vocero-client/docs/speech/llm.md

Language model backends

Available LLM backends (--llm_backend)

Runtime-supported values in s2s_pipeline.py:

• transformers → language_model.py (Transformers backend)
• mlx-lm → language_model.py (MLX backend)
• responses-api → responses_api_language_model.py
• chat-completions → chat_completions_language_model.py

Run the commands below from the repository root in an environment with the speech pipeline dependencies installed. This checkout has no package manifest or complete speech dependency installer; the UI requirements alone do not install the pipeline.

Usage

1) Transformers (--llm_backend transformers)

• Handler: LanguageModelHandler
• Typical use: local GPU/CPU inference using Hugging Face Transformers
• Backend-specific args prefix: --llm_*
• Shared args (from base): --model_name, --chat_size, --init_chat_prompt, --enable_lang_prompt

python -m speech_to_speech.s2s_pipeline \
  --llm_backend transformers \
  --model_name Qwen/Qwen3-4B-Instruct-2507 \
  --llm_device cuda \
  --llm_torch_dtype float16 \
  --llm_gen_max_new_tokens 128

Common options:

• --llm_gen_min_new_tokens
• --llm_gen_temperature
• --llm_gen_do_sample
• --chat_size
• --init_chat_prompt

2) MLX-LM (--llm_backend mlx-lm)

• Handler: LanguageModelHandler
• Typical use: Apple Silicon local inference
• Backend-specific args prefix: same as Transformers (--llm_*)

python -m speech_to_speech.s2s_pipeline \
  --llm_backend mlx-lm \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
  --llm_device mps \
  --llm_gen_max_new_tokens 128

Common options:

• --llm_gen_temperature
• --llm_gen_do_sample
• --chat_size
• --init_chat_prompt

3) OpenAI-compatible API (--llm_backend responses-api)

• Handler: ResponsesApiModelHandler
• Typical use: remote model serving via OpenAI-compatible endpoints
• Backend-specific args prefix: --responses_api_*
• Shared args (from base): --model_name, --chat_size, --init_chat_prompt, --enable_lang_prompt

python -m speech_to_speech.s2s_pipeline \
  --llm_backend responses-api \
  --model_name gpt-5.4-mini \
  --responses_api_api_key YOUR_API_KEY \
  --responses_api_base_url https://api.example.com/v1 \
  --responses_api_stream true

Common options:

• --chat_size
• --init_chat_prompt
• --user_role

4) Chat Completions (--llm_backend chat-completions)

• Handler: ChatCompletionsLanguageModelHandler
• Uses the provider's /chat/completions API rather than /responses
• Inherits the same --responses_api_base_url, --responses_api_api_key, --responses_api_stream, and --responses_api_disable_thinking flags
• Adds --responses_api_reasoning_effort for providers that support that setting

The current handler removes tools and tool_choice from outgoing provider requests as a compatibility workaround. Native tool calling therefore differs from the responses-api backend; see the realtime guide.

python -m speech_to_speech.s2s_pipeline \
  --llm_backend chat-completions \
  --model_name YOUR_MODEL \
  --responses_api_base_url https://api.example.com/v1 \
  --responses_api_api_key YOUR_API_KEY

The default LLM backend is responses-api. Model names in this guide are examples; select a model available in your environment or from your provider.

LLM Behavior

When STT is set to language auto-detection (--language auto), LLM handlers can receive (text, language_code) and prepend a language control instruction like:

• Please reply to my message in <language>.

This helps the assistant respond in the detected language. The behavior is opt-in via --enable_lang_prompt (shared across all backends); it defaults to False.

Setup

CUDA setup

python -m speech_to_speech.s2s_pipeline \
  --llm_backend transformers \
  --model_name microsoft/Phi-3-mini-4k-instruct

Local Mac setup

python -m speech_to_speech.s2s_pipeline \
  --local_mac_optimal_settings \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16

--local_mac_optimal_settings already sets --llm_backend mlx-lm and will default the model to mlx-community/Qwen3-4B-Instruct-2507-bf16 if not overridden.

Realtime (OpenAI-compatible) setup

Run the server in realtime mode, then connect with the browser client:

# 1. Start the pipeline in realtime mode
python -m speech_to_speech.s2s_pipeline \
  --mode realtime \
  --llm_backend mlx-lm \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
  --ws_host 0.0.0.0 \
  --ws_port 8765

# 2. Point the browser client at ws://localhost:8765/v1/realtime.
# See the browser-client usage guide for startup commands.

For realtime mode on Apple Silicon, set the desired MLX/STT/TTS/device options explicitly. The current --local_mac_optimal_settings helper forces mode="local", so combining it with --mode realtime does not preserve realtime mode.

Remote API setup

python -m speech_to_speech.s2s_pipeline \
  --llm_backend responses-api \
  --model_name gpt-5.4-mini \
  --responses_api_api_key YOUR_API_KEY