vocero-client/docs/speech/stt.md

Speech-to-text backends

This document summarizes the speech-to-text implementations in speech_to_speech/STT, including language support, language abbreviations, and pipeline options. Run examples from the repository root in an environment with the speech dependencies installed.

Available STT Modes (--stt)

• whisper → STT/whisper_stt_handler.py
• whisper-mlx → STT/lightning_whisper_mlx_handler.py
• mlx-audio-whisper → STT/mlx_audio_whisper_handler.py
• faster-whisper → STT/faster_whisper_handler.py
• parakeet-tdt → STT/parakeet_tdt_handler.py
• paraformer → STT/paraformer_handler.py

Language Support by Handler

1) Whisper (--stt whisper)

• Handler: WhisperSTTHandler
• Language input flag: --language (from shared Whisper args)
• Supports fixed language (e.g. en) or auto
• Internal supported language list:
  • en, fr, es, zh, ja, ko, hi, de, pt, pl, it, nl
• Behavior:
  • Detects language from token output
  • If detected language is outside the supported list, it falls back to the previous language

2) Lightning Whisper MLX (--stt whisper-mlx)

• Handler: LightningWhisperSTTHandler
• Uses same shared --language argument as Whisper
• Internal supported language list:
  • en, fr, es, zh, ja, ko, hi, de, pt, pl, it, nl
• Behavior:
  • If --language auto, model auto-detects each utterance
  • If detected language is unsupported, falls back to last supported language

3) MLX Audio Whisper (--stt mlx-audio-whisper)

• Handler: MLXAudioWhisperSTTHandler
• Model flag: --mlx_audio_whisper_model_name
• Language still comes from shared --language flag (wired in pipeline)
• Internal supported language list:
  • en, fr, es, zh, ja, ko, hi, de, pt, pl, it, nl
• Behavior:
  • Uses fixed language unless --language auto
  • Falls back to last known supported language when needed

4) Faster-Whisper (--stt faster-whisper)

• Handler: FasterWhisperSTTHandler
• Language flag: --faster_whisper_stt_gen_language
• Default language: en
• Note:
  • This handler passes generation kwargs directly to faster_whisper.WhisperModel.transcribe(...)
  • Effective language coverage depends on selected Faster-Whisper/OpenAI Whisper model

5) Parakeet TDT (--stt parakeet-tdt)

• Handler: ParakeetTDTSTTHandler
• Language flag: --parakeet_tdt_language (optional)
• Supports auto language detection when language not specified
• Declared supported language list (25 European languages):
  • en, de, fr, es, it, pt, nl, pl, ru, uk, cs, sk, hu, ro, bg, hr, sl, sr, da, no, sv, fi, et, lv, lt
• Backend behavior:
  • On macOS/MPS: MLX (mlx-community/parakeet-tdt-0.6b-v3)
  • On CUDA/CPU: nano-parakeet (nvidia/parakeet-tdt-0.6b-v3)

6) Paraformer (--stt paraformer)

• Handler: ParaformerSTTHandler
• Model flag: --paraformer_stt_model_name
• Default model: paraformer-zh
• No dedicated language flag in current args class
• Practical support:
  • Depends on selected FunASR model checkpoint
  • Default setup is Chinese-oriented (zh)

Language Abbreviations (ISO-style codes seen in STT handlers)

Code	Language
en	English
fr	French
es	Spanish
zh	Chinese
ja	Japanese
ko	Korean
hi	Hindi
de	German
pt	Portuguese
pl	Polish
it	Italian
nl	Dutch
ru	Russian
uk	Ukrainian
cs	Czech
sk	Slovak
hu	Hungarian
ro	Romanian
bg	Bulgarian
hr	Croatian
sl	Slovenian
sr	Serbian
da	Danish
no	Norwegian
sv	Swedish
fi	Finnish
et	Estonian
lv	Latvian
lt	Lithuanian
auto	Per-utterance automatic language detection

Usage Examples

Whisper (Transformers)

python -m speech_to_speech.s2s_pipeline --stt whisper --language es
python -m speech_to_speech.s2s_pipeline --stt whisper --language auto

Whisper MLX (LightningWhisperMLX)

python -m speech_to_speech.s2s_pipeline --stt whisper-mlx --language auto --device mps

MLX Audio Whisper

python -m speech_to_speech.s2s_pipeline --stt mlx-audio-whisper \
  --mlx_audio_whisper_model_name mlx-community/whisper-large-v3-turbo \
  --language auto

Faster-Whisper

python -m speech_to_speech.s2s_pipeline --stt faster-whisper \
  --faster_whisper_stt_model_name large-v3 \
  --faster_whisper_stt_gen_language en

Parakeet TDT

python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt --parakeet_tdt_device auto
python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt --parakeet_tdt_language es

With live transcription (MLX or CUDA/nano-parakeet backend):

python -m speech_to_speech.s2s_pipeline --stt parakeet-tdt \
  --enable_live_transcription \
  --live_transcription_update_interval 0.25

Paraformer

python -m speech_to_speech.s2s_pipeline --stt paraformer --paraformer_stt_model_name paraformer-zh