fixes tool callong
This commit is contained in:
@@ -69,6 +69,14 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
|
||||
- **Pipelined TTS** — reply chunks are synthesized ahead of playback (up to
|
||||
`TTS_PIPELINE_DEPTH` at once) and replayed in order, hiding each request's
|
||||
start-up latency behind the previous chunk's audio.
|
||||
- **OpenAPI / Swagger UI** — the REST endpoints are documented by an `utoipa`
|
||||
OpenAPI document rendered as a Swagger UI at `/docs` (raw JSON at
|
||||
`/api-docs/openapi.json`). See
|
||||
[API documentation](#api-documentation-swagger-ui).
|
||||
- **File transcription endpoint** — `POST /api/transcribe` decodes an uploaded
|
||||
audio file with `ffmpeg` and returns the Parakeet transcript of the whole take
|
||||
(reusing the live session's STT model). See
|
||||
[Transcription endpoint](#transcription-endpoint).
|
||||
- **Double jitter buffer** — server pre-buffer + browser playback buffer absorb
|
||||
uneven TTS chunk arrival, eliminating drop-outs.
|
||||
- **Silero VAD (ONNX)** — server-side speech segmentation and fast barge-in.
|
||||
@@ -80,7 +88,9 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
|
||||
chain-of-thought (`reasoning_content` / ` thinking`) is stripped from both the
|
||||
audio and the chat box.
|
||||
- **Tool calling** — the model can call external HTTP tools described in a JSON
|
||||
file (`LLM_TOOLS_FILE`) before answering. Calls are parsed client-side from
|
||||
file (`LLM_TOOLS_FILE`) before answering, or installed at runtime with
|
||||
`POST /api/tools` (in-memory override; without it the file definitions are
|
||||
used). Calls are parsed client-side from
|
||||
either the structured `tool_calls` array or the model's native `<tool_call>`
|
||||
format (a round that streams back empty is re-asked without streaming),
|
||||
executed over HTTP and fed back to the model; the tool-call block is never
|
||||
@@ -129,6 +139,7 @@ cargo run --release
|
||||
# http://localhost:8787/ playground (TTS / audio test)
|
||||
# http://localhost:8787/mivocero speech-to-speech agent with chat box
|
||||
# http://localhost:8787/customvoice custom-voice builder (record / upload a reference)
|
||||
# http://localhost:8787/docs Swagger UI (REST API documentation)
|
||||
```
|
||||
|
||||
Environment variables can also be `export`ed directly; a `.env` file is loaded
|
||||
@@ -259,7 +270,7 @@ See [`.env.example`](.env.example) for a copy-paste template.
|
||||
| `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. |
|
||||
| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). |
|
||||
| `LLM_TOOLS_ENABLED` | *(auto)* | Master switch for tool calling. Default: enabled when `LLM_TOOLS_FILE` is a non-empty path. |
|
||||
| `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method`, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
|
||||
| `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method` — any HTTP verb, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
|
||||
| `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops. |
|
||||
| `LLM_TOOL_TIMEOUT_SECS` | `15` | Timeout of a single tool HTTP request. |
|
||||
| `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | Tool responses are truncated to this many characters before being handed back to the model. |
|
||||
@@ -1023,12 +1034,36 @@ be exposed without touching the code:
|
||||
```
|
||||
|
||||
- `url` may contain `{arg}` placeholders substituted from the call arguments.
|
||||
- `GET` sends the arguments that were **not** consumed by a placeholder as query
|
||||
parameters; `POST` (the default) sends them as a JSON body.
|
||||
- `method` takes **any** HTTP verb, case-insensitive (`GET`, `POST`, `PUT`,
|
||||
`PATCH`, `DELETE`, `HEAD`, `OPTIONS`, `TRACE`, custom verbs); default `POST`.
|
||||
The arguments left after the URL placeholders are sent as **query parameters**
|
||||
for `GET`/`HEAD`/`OPTIONS`/`TRACE`/`DELETE` and as a **JSON body** for every
|
||||
other verb — so a full REST surface is covered: list/look up (`GET`), add
|
||||
(`POST`), update (`PUT`/`PATCH`), remove (`DELETE`).
|
||||
- `headers` are optional and sent verbatim (API keys, etc.).
|
||||
|
||||
See [`tools.example.json`](tools.example.json) for a ready-to-use file.
|
||||
|
||||
### Runtime definitions (`GET` / `POST /api/tools`)
|
||||
|
||||
The file is read once at startup, so `POST /api/tools` installs a different set
|
||||
without a restart — the body is **the same JSON array** the file holds:
|
||||
|
||||
```bash
|
||||
curl -X POST localhost:8787/api/tools -H 'Content-Type: application/json' \
|
||||
-d '[{"name":"get_weather","description":"…","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]},"url":"https://wttr.in/{city}?format=3","method":"GET"}]'
|
||||
```
|
||||
|
||||
- The override lives **in memory only**: it replaces the `LLM_TOOLS_FILE`
|
||||
definitions from the next turn on, and a restart brings the file ones back.
|
||||
When the endpoint is never called, the file definitions stay in use.
|
||||
- `GET /api/tools` reports the active definitions and their origin
|
||||
(`source`: `file`, `request` or `none`).
|
||||
- `400` on an empty/invalid body or on a JSON array with no tool; `503` when the
|
||||
LLM is disabled. See [`src/tools_api.rs`](src/tools_api.rs).
|
||||
|
||||
See [TOOL_CALLING.md](TOOL_CALLING.md) for the full guide.
|
||||
|
||||
### Client-side parsing (and the streaming fallback)
|
||||
|
||||
A deployment may or may not run a server-side tool parser. When it does, the
|
||||
@@ -1475,8 +1510,11 @@ audio-agent-engine/
|
||||
├── vad/ # Silero VAD v5 + onset/offset state machine
|
||||
├── llm.rs # OpenAI-compatible chat-completions client (streaming + think stripping)
|
||||
├── tools.rs # configurable HTTP tool calling (native <tool_call> parser + executor)
|
||||
├── tools_api.rs # GET/POST /api/tools — runtime tool definitions (in-memory override)
|
||||
├── media.rs # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
|
||||
├── customvoice.rs # custom-voice builder endpoints (record → WAV reference → set_reference)
|
||||
├── transcribe.rs # POST /api/transcribe — Parakeet transcription of an uploaded audio file
|
||||
├── apidoc.rs # OpenAPI document (utoipa) rendered by the Swagger UI at /docs
|
||||
├── audit.rs # per-session JSONL audit log (config snapshot, events, summary)
|
||||
├── orchestrator/ # turn_manager (EOU/STT → LLM → filler → TTS) + backchannel
|
||||
├── ws/ # handler, session (per-connection audio loop)
|
||||
@@ -1493,8 +1531,58 @@ audio-agent-engine/
|
||||
| `GET /api/customvoice/phrases` | Reading phrases per language + the default language (JSON). |
|
||||
| `POST /api/customvoice?lang=<code>[&text=<transcript>]` | Upload a recording or an audio file (with its transcript); transcode it to a WAV reference and install it as the active voice. |
|
||||
| `POST /api/customvoice/clear` | Restore the reference configured at startup. |
|
||||
| `POST /api/transcribe` | Upload a raw audio file (any ffmpeg-readable container); returns the Parakeet transcript of the whole file (`text`, `duration_ms`, `tokens`). |
|
||||
| `GET /api/tools` | The active LLM tool definitions and their origin (`source`: `file` / `request` / `none`). |
|
||||
| `POST /api/tools` | Install LLM tool definitions at runtime (same JSON array as `LLM_TOOLS_FILE`) — in-memory, no restart ([Tool calling](#tool-calling)). |
|
||||
| `GET /health` | Liveness probe (`ok`). |
|
||||
| `GET /ws` | WebSocket upgrade → per-session audio loop. |
|
||||
| `GET /docs` | Swagger UI for the REST surface ([API documentation](#api-documentation-swagger-ui)). |
|
||||
| `GET /api-docs/openapi.json` | The raw OpenAPI 3 document the Swagger UI renders. |
|
||||
|
||||
### API documentation (Swagger UI)
|
||||
|
||||
The REST endpoints are documented with an **OpenAPI 3** document derived from
|
||||
the handlers ([`utoipa`](https://docs.rs/utoipa)) and browsable in a **Swagger
|
||||
UI** served by the engine itself ([`utoipa-swagger-ui`](https://docs.rs/utoipa-swagger-ui)):
|
||||
|
||||
- **UI** — `GET /docs` (→ `/docs/`) renders the interactive Swagger UI.
|
||||
- **JSON** — `GET /api-docs/openapi.json` serves the raw OpenAPI document.
|
||||
|
||||
The document is built in [`src/apidoc.rs`](src/apidoc.rs) (the `ApiDoc`
|
||||
`#[derive(OpenApi)]`, the response schemas) with per-handler
|
||||
`#[utoipa::path]` annotations on the endpoints. Only the JSON/REST endpoints are
|
||||
covered — the WebSocket endpoint (`/ws`) and the HTML client pages are streaming
|
||||
/ static assets and are intentionally left out.
|
||||
|
||||
The `utoipa-swagger-ui` build script embeds the UI assets from a zip at compile
|
||||
time; the `zip` crate is therefore pinned to the `2.2.x` line in `Cargo.toml`
|
||||
(the 2.3+ API no longer matches the 8.1 build script).
|
||||
|
||||
### Transcription endpoint
|
||||
|
||||
`POST /api/transcribe` ([`src/transcribe.rs`](src/transcribe.rs)) turns an
|
||||
uploaded audio file into text with the same Parakeet TDT model the live session
|
||||
uses:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8787/api/transcribe \
|
||||
-H 'Content-Type: application/octet-stream' \
|
||||
--data-binary @clip.wav
|
||||
# {"ok":true,"text":"…","duration_ms":14918,"tokens":83}
|
||||
```
|
||||
|
||||
- **Input** — the raw audio bytes in the request body (any container `ffmpeg`
|
||||
can read: wav, mp3, m4a, ogg, flac…); no multipart, no `lang` parameter —
|
||||
Parakeet TDT v3 is multilingual and the language is auto-detected.
|
||||
- **Processing** — `ffmpeg` decodes to mono 16 kHz `s16le`, and the engine's
|
||||
**already-loaded** STT model ([`AppState::stt`](src/ws/handler.rs)) runs one
|
||||
inference over the whole file (`spawn_blocking`: the inference is CPU-bound
|
||||
and holds the ONNX session lock). No second ONNX environment is created.
|
||||
- **Output** — a single transcript for the entire file (the audio is **not**
|
||||
segmented): `{ "ok": true, "text": …, "duration_ms": …, "tokens": … }`.
|
||||
- **Errors** — `400` empty body, `503` STT model not loaded, `502` ffmpeg
|
||||
decode failed, `500` temp-file / inference failure. Uploads are capped at
|
||||
32 MB (same as `/api/customvoice`).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user