fixes tool callong

This commit is contained in:
valenti
2026-10-21 10:44:49 +01:00
parent 79f927d968
commit 1cafe3dd3c
14 changed files with 2308 additions and 61 deletions
+92 -4
View File
@@ -69,6 +69,14 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
- **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to
`TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's
start-up latency behind the previous chunk's audio.
- **OpenAPI / Swagger UI** — the REST endpoints are documented by an `utoipa`
OpenAPI document rendered as a Swagger UI at `/docs` (raw JSON at
`/api-docs/openapi.json`). See
[API documentation](#api-documentation-swagger-ui).
- **File transcription endpoint** — `POST /api/transcribe` decodes an uploaded
audio file with `ffmpeg` and returns the Parakeet transcript of the whole take
(reusing the live session's STT model). See
[Transcription endpoint](#transcription-endpoint).
- **Double jitter buffer** — server pre-buffer + browser playback buffer absorb
uneven TTS chunk arrival, eliminating drop-outs.
- **Silero VAD (ONNX)** — server-side speech segmentation and fast barge-in.
@@ -80,7 +88,9 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
chain-of-thought (`reasoning_content` / ` thinking`) is stripped from both the
audio and the chat box.
- **Tool calling** — the model can call external HTTP tools described in a JSON
file (`LLM_TOOLS_FILE`) before answering. Calls are parsed client-side from
file (`LLM_TOOLS_FILE`) before answering, or installed at runtime with
`POST /api/tools` (in-memory override; without it the file definitions are
used). Calls are parsed client-side from
either the structured `tool_calls` array or the model's native `<tool_call>`
format (a round that streams back empty is re-asked without streaming),
executed over HTTP and fed back to the model; the tool-call block is never
@@ -129,6 +139,7 @@ cargo run --release
# http://localhost:8787/ playground (TTS / audio test)
# http://localhost:8787/mivocero speech-to-speech agent with chat box
# http://localhost:8787/customvoice custom-voice builder (record / upload a reference)
# http://localhost:8787/docs Swagger UI (REST API documentation)
```
Environment variables can also be `export`ed directly; a `.env` file is loaded
@@ -259,7 +270,7 @@ See [`.env.example`](.env.example) for a copy-paste template.
| `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. |
| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). |
| `LLM_TOOLS_ENABLED` | *(auto)* | Master switch for tool calling. Default: enabled when `LLM_TOOLS_FILE` is a non-empty path. |
| `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method`, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
| `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method` — any HTTP verb, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
| `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops. |
| `LLM_TOOL_TIMEOUT_SECS` | `15` | Timeout of a single tool HTTP request. |
| `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | Tool responses are truncated to this many characters before being handed back to the model. |
@@ -1023,12 +1034,36 @@ be exposed without touching the code:
```
- `url` may contain `{arg}` placeholders substituted from the call arguments.
- `GET` sends the arguments that were **not** consumed by a placeholder as query
parameters; `POST` (the default) sends them as a JSON body.
- `method` takes **any** HTTP verb, case-insensitive (`GET`, `POST`, `PUT`,
`PATCH`, `DELETE`, `HEAD`, `OPTIONS`, `TRACE`, custom verbs); default `POST`.
The arguments left after the URL placeholders are sent as **query parameters**
for `GET`/`HEAD`/`OPTIONS`/`TRACE`/`DELETE` and as a **JSON body** for every
other verb — so a full REST surface is covered: list/look up (`GET`), add
(`POST`), update (`PUT`/`PATCH`), remove (`DELETE`).
- `headers` are optional and sent verbatim (API keys, etc.).
See [`tools.example.json`](tools.example.json) for a ready-to-use file.
### Runtime definitions (`GET` / `POST /api/tools`)
The file is read once at startup, so `POST /api/tools` installs a different set
without a restart — the body is **the same JSON array** the file holds:
```bash
curl -X POST localhost:8787/api/tools -H 'Content-Type: application/json' \
-d '[{"name":"get_weather","description":"…","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]},"url":"https://wttr.in/{city}?format=3","method":"GET"}]'
```
- The override lives **in memory only**: it replaces the `LLM_TOOLS_FILE`
definitions from the next turn on, and a restart brings the file ones back.
When the endpoint is never called, the file definitions stay in use.
- `GET /api/tools` reports the active definitions and their origin
(`source`: `file`, `request` or `none`).
- `400` on an empty/invalid body or on a JSON array with no tool; `503` when the
LLM is disabled. See [`src/tools_api.rs`](src/tools_api.rs).
See [TOOL_CALLING.md](TOOL_CALLING.md) for the full guide.
### Client-side parsing (and the streaming fallback)
A deployment may or may not run a server-side tool parser. When it does, the
@@ -1475,8 +1510,11 @@ audio-agent-engine/
├── vad/ # Silero VAD v5 + onset/offset state machine
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
├── tools.rs # configurable HTTP tool calling (native <tool_call> parser + executor)
├── tools_api.rs # GET/POST /api/tools — runtime tool definitions (in-memory override)
├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
├── customvoice.rs # custom-voice builder endpoints (record → WAV reference → set_reference)
├── transcribe.rs # POST /api/transcribe — Parakeet transcription of an uploaded audio file
├── apidoc.rs # OpenAPI document (utoipa) rendered by the Swagger UI at /docs
├── audit.rs # per-session JSONL audit log (config snapshot, events, summary)
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
├── ws/ # handler, session (per-connection audio loop)
@@ -1493,8 +1531,58 @@ audio-agent-engine/
| `GET /api/customvoice/phrases` | Reading phrases per language + the default language (JSON). |
| `POST /api/customvoice?lang=<code>[&text=<transcript>]` | Upload a recording or an audio file (with its transcript); transcode it to a WAV reference and install it as the active voice. |
| `POST /api/customvoice/clear` | Restore the reference configured at startup. |
| `POST /api/transcribe` | Upload a raw audio file (any ffmpeg-readable container); returns the Parakeet transcript of the whole file (`text`, `duration_ms`, `tokens`). |
| `GET /api/tools` | The active LLM tool definitions and their origin (`source`: `file` / `request` / `none`). |
| `POST /api/tools` | Install LLM tool definitions at runtime (same JSON array as `LLM_TOOLS_FILE`) — in-memory, no restart ([Tool calling](#tool-calling)). |
| `GET /health` | Liveness probe (`ok`). |
| `GET /ws` | WebSocket upgrade → per-session audio loop. |
| `GET /docs` | Swagger UI for the REST surface ([API documentation](#api-documentation-swagger-ui)). |
| `GET /api-docs/openapi.json` | The raw OpenAPI 3 document the Swagger UI renders. |
### API documentation (Swagger UI)
The REST endpoints are documented with an **OpenAPI 3** document derived from
the handlers ([`utoipa`](https://docs.rs/utoipa)) and browsable in a **Swagger
UI** served by the engine itself ([`utoipa-swagger-ui`](https://docs.rs/utoipa-swagger-ui)):
- **UI** — `GET /docs` (→ `/docs/`) renders the interactive Swagger UI.
- **JSON** — `GET /api-docs/openapi.json` serves the raw OpenAPI document.
The document is built in [`src/apidoc.rs`](src/apidoc.rs) (the `ApiDoc`
`#[derive(OpenApi)]`, the response schemas) with per-handler
`#[utoipa::path]` annotations on the endpoints. Only the JSON/REST endpoints are
covered — the WebSocket endpoint (`/ws`) and the HTML client pages are streaming
/ static assets and are intentionally left out.
The `utoipa-swagger-ui` build script embeds the UI assets from a zip at compile
time; the `zip` crate is therefore pinned to the `2.2.x` line in `Cargo.toml`
(the 2.3+ API no longer matches the 8.1 build script).
### Transcription endpoint
`POST /api/transcribe` ([`src/transcribe.rs`](src/transcribe.rs)) turns an
uploaded audio file into text with the same Parakeet TDT model the live session
uses:
```bash
curl -s -X POST http://localhost:8787/api/transcribe \
-H 'Content-Type: application/octet-stream' \
--data-binary @clip.wav
# {"ok":true,"text":"…","duration_ms":14918,"tokens":83}
```
- **Input** — the raw audio bytes in the request body (any container `ffmpeg`
can read: wav, mp3, m4a, ogg, flac…); no multipart, no `lang` parameter —
Parakeet TDT v3 is multilingual and the language is auto-detected.
- **Processing** — `ffmpeg` decodes to mono 16 kHz `s16le`, and the engine's
**already-loaded** STT model ([`AppState::stt`](src/ws/handler.rs)) runs one
inference over the whole file (`spawn_blocking`: the inference is CPU-bound
and holds the ONNX session lock). No second ONNX environment is created.
- **Output** — a single transcript for the entire file (the audio is **not**
segmented): `{ "ok": true, "text": …, "duration_ms": …, "tokens": … }`.
- **Errors** — `400` empty body, `503` STT model not loaded, `502` ffmpeg
decode failed, `500` temp-file / inference failure. Uploads are capped at
32 MB (same as `/api/customvoice`).
---