piperust/TOOL_CALLING.md

# Tool calling — Operating guide

Code references: [`src/tools.rs`](src/tools.rs), [`src/llm.rs`](src/llm.rs),
[`src/orchestrator/turn_manager.rs`](src/orchestrator/turn_manager.rs),
[`src/tools_api.rs`](src/tools_api.rs), [`tools.example.json`](tools.example.json).

---

## 1. What it does

The engine's LLM can **call external HTTP services** before answering. A "tool" is
not code: it is the **JSON description of an HTTP request** (URL, method, headers,
argument schema). Adding a tool means writing a few lines in a JSON file —
**no Rust changes**.

Per user turn:

1. The engine attaches the OpenAI-shaped `tools` array to **every** completion
   request, with `tool_choice: "auto"`.
2. If needed, the model emits the call. The production LLM server may run without a
   tool parser, so the call comes back **as text** in the Qwen native format
   (`<tool_call><function=…><parameter=…>…`), which the engine **parses
   client-side**.
3. The engine executes the HTTP request, truncates the result and hands it back to
   the model as a `tool` role message.
4. The model produces the final answer, which is synthesised to speech.

The `<tool_call>` block is **always stripped** from both the spoken and the
displayed text: the user never hears or sees it.

## 2. Enabling it and runtime override

| Variable | Default | Meaning |
|---|---|---|
| `LLM_TOOLS_ENABLED` | auto | Master switch. When unset it is `true` if `LLM_TOOLS_FILE` is a non-empty path. |
| `LLM_TOOLS_FILE` | *(empty)* | Path to the JSON definitions file. Empty or missing ⇒ tool calling **disabled**. |
| `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool rounds per turn (call → execute → re-prompt). Bounds a runaway loop. |
| `LLM_TOOL_TIMEOUT_SECS` | `15` | Timeout of **one** tool HTTP request. |
| `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | The tool result is truncated to this many characters before going back to the model. |

```dotenv
LLM_TOOLS_ENABLED=true
LLM_TOOLS_FILE=tools.example.json
LLM_TOOL_MAX_ITERATIONS=4
LLM_TOOL_TIMEOUT_SECS=15
LLM_TOOL_RESULT_MAX_CHARS=2000
```

Note: the tools file is read **once at startup** (`ToolRegistry::load`). After
editing the JSON, **restart the service** for the new definitions to take effect.

### Runtime override — `POST /api/tools`

To change the definitions **without a restart**, POST the same JSON array to
`POST /api/tools`:

```bash
curl -X POST localhost:8787/api/tools -H 'Content-Type: application/json' \
  -d '[{"name":"get_weather","description":"Current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]},"url":"https://wttr.in/{city}?format=3","method":"GET"}]'
```

- The definitions you send **replace** the file ones from the **next turn** on
  (no restart). If the endpoint is **never called**, the engine keeps using the
  `LLM_TOOLS_FILE` definitions — that is the default, and a restart restores it.
- The override lives **in memory only**: it does not write the file and does not
  survive a process restart.
- `GET /api/tools` reports the active definitions and their origin: `source` is
  `file` (the `.env` default), `request` (installed through the endpoint) or
  `none`.
- Responses: `200` with `{ok,count,source,tools}`; `400` when the body is empty,
  is not UTF-8, is not a valid JSON array, or holds no tool definition; `503`
  when the LLM is disabled.
- The body uses **the same format as the file**: an array that works here also
  works as an `LLM_TOOLS_FILE`.

```bash
curl -s localhost:8787/api/tools
# {"ok":true,"count":2,"source":"file","tools":[ … ]}
```

Implementation: [`src/tools_api.rs`](src/tools_api.rs). Both endpoints also show
up in the Swagger UI (`/docs`, tag `tools`).

## 3. Defining a tool

The file is a JSON array. Each object has these fields:

| Field | Required | Description |
|---|---|---|
| `name` | yes | Name of the function the model calls. Must match the `name` in the definition. |
| `description` | recommended | **This is what the model reads to decide whether and when to call the tool**: make it explicit and usage-oriented ("call this tool when the user asks…"). |
| `parameters` | no | JSON Schema for the arguments. When absent it defaults to `{"type":"object","properties":{}}`. |
| `url` | yes | Target URL. May contain `{arg}` placeholders. |
| `method` | no | **Any** HTTP verb, case-insensitive: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`, `TRACE`, and custom verbs (e.g. `PURGE`). Default `POST`. |
| `headers` | no | Headers sent verbatim (API keys, `Content-Type`, …). Values may contain `{arg}` placeholders. |

### Placeholder substitution

In `url` **and** `headers`, every `{argument_name}` is replaced with the matching
call argument (strings unquoted, numbers/booleans rendered as text).

### Where the arguments go (by verb)

Arguments **already consumed** by a `{arg}` placeholder in the URL are *not* sent
again: they are already in the path. The remaining ones follow the verb:

| Verb | Remaining arguments |
|---|---|
| `GET`, `HEAD`, `OPTIONS`, `TRACE`, `DELETE` | **query parameters** (no body) |
| `POST`, `PUT`, `PATCH` and custom verbs | **JSON body** (`{}` when nothing is left) |

## 4. Examples

**GET with a URL placeholder** (`city` goes into the path, not the query):

```json
{
  "name": "get_weather",
  "description": "Current weather and temperature for a city. Call this whenever the user asks about the weather, the temperature or the forecast for a specific place.",
  "parameters": {
    "type": "object",
    "properties": { "city": { "type": "string", "description": "City name, e.g. 'Paris' or 'Roma'" } },
    "required": ["city"]
  },
  "url": "https://wttr.in/{city}?format=3",
  "method": "GET"
}
```

**POST with an authenticated header** (secret taken from the arguments):

```json
{
  "name": "create_note",
  "description": "Save a note into the internal management system.",
  "parameters": {
    "type": "object",
    "properties": {
      "token": { "type": "string", "description": "User session token" },
      "text":  { "type": "string", "description": "Note content" }
    },
    "required": ["token", "text"]
  },
  "url": "https://internal.example.local/api/notes",
  "method": "POST",
  "headers": { "Authorization": "Bearer {token}" }
}
```

**Plain POST (webhook)**:

```json
{
  "name": "send_webhook",
  "description": "POST a JSON payload to the configured webhook endpoint (notification / home-automation hook).",
  "parameters": {
    "type": "object",
    "properties": { "message": { "type": "string" } },
    "required": ["message"]
  },
  "url": "https://httpbin.org/post",
  "method": "POST",
  "headers": { "X-Tool": "audio-agent-engine" }
}
```

**A full REST surface** — one tool per verb on the same endpoint
(`list` → `GET`, `detail` → `GET` with `{id}`, `add` → `POST`,
`update` → `PATCH`, `replace` → `PUT`, `delete` → `DELETE`, `exists?` → `HEAD`):

```json
[
  {
    "name": "list_items",
    "description": "List the items, with optional filters.",
    "parameters": {
      "type": "object",
      "properties": {
        "limit":  { "type": "integer" },
        "search": { "type": "string" }
      }
    },
    "url": "https://api.example.local/api/items",
    "method": "GET"
  },
  {
    "name": "get_item",
    "description": "Details of one item by id.",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "GET"
  },
  {
    "name": "add_item",
    "description": "Create a new item.",
    "parameters": {
      "type": "object",
      "properties": { "name": { "type": "string" }, "qty": { "type": "integer" } },
      "required": ["name"]
    },
    "url": "https://api.example.local/api/items",
    "method": "POST"
  },
  {
    "name": "update_item",
    "description": "Partially update an item (only the fields sent).",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" }, "qty": { "type": "integer" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "PATCH"
  },
  {
    "name": "replace_item",
    "description": "Replace an item entirely.",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" }, "name": { "type": "string" } },
      "required": ["id", "name"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "PUT"
  },
  {
    "name": "delete_item",
    "description": "Delete an item by id.",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "DELETE"
  },
  {
    "name": "item_exists",
    "description": "Check whether an item exists (status only, no body).",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "HEAD"
  }
]
```

Resulting requests: `GET /api/items?limit=10`, `GET /api/items/5`,
`POST /api/items` + body `{"name":"…"}`, `PATCH /api/items/5` + body `{"qty":2}`,
`PUT /api/items/5` + body `{"name":"…"}`, `DELETE /api/items/5`,
`HEAD /api/items/5`.

Start from [`tools.example.json`](tools.example.json): copy it, adapt it and point
`LLM_TOOLS_FILE` at your file.

## 5. Multi-round loop and streaming fallback

- Each round produces **one** completion; the calls found are executed and their
  results are handed back for the next round (standard OpenAI round-trip: an
  `assistant` message with the call, then one `tool` message per result).
- The loop stops when there are no more calls, or at `LLM_TOOL_MAX_ITERATIONS`.
- **Streaming fallback**: some OpenAI-compatible servers (vLLM with a
  `--tool-call-parser` that does not match the model) **swallow** the call on the
  SSE path: the round ends empty, with no HTTP error. When a round comes back empty
  and tools are configured, the engine **re-asks once without streaming**
  (`chat_reply`) and takes the call from there. If even that yields no call but
  holds text, that text is spoken. The fallback only triggers on empty rounds, so
  it costs nothing on normal turns.

## 6. Errors and truncation

Tool execution **never fails** in a blocking way: an error becomes text the model
can read and recover from.

| Situation | Text handed to the model |
|---|---|
| Unknown tool | `Error: unknown tool '<name>'` |
| Invalid HTTP verb in the definition | `Error: invalid HTTP method '<verb>' for tool '<name>'` |
| Non-2xx HTTP status | `Error: HTTP <status> — <truncated body>` |
| Network error / timeout | `Error: tool request failed: <detail>` |
| Result longer than the limit | Truncated to `LLM_TOOL_RESULT_MAX_CHARS`, suffixed with `… [truncated]` |

## 7. Important note on the server-side parser

If the LLM server runs **with** a tool parser, the flag must match the model's
native format (`--enable-auto-tool-choice --tool-call-parser …`). A mismatch is
**silent**: an `hermes` parser (expecting JSON inside `<tool_call>`) against a Qwen
model that emits XML makes the parse throw, the call is dropped and, in streaming,
the reply comes back empty (vLLM logs: `Error in extracting tool call from
response`). The correct parser for this model is `qwen3_coder`. The engine is
robust either way: it handles **both** formats, the structured
`message.tool_calls[]` array **and** the native `<tool_call>` text, plus the
fallback described in section 5.

Typical mismatch symptom: the agent goes silent, or "narrates" the call without
executing it.

## 8. End-to-end verification

With the service running on `ws://localhost:8787/ws`, send user text that must
trigger the tool and check the events:

```python
import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://localhost:8787/ws") as ws:
        await ws.send(json.dumps({"type": "user_text", "text": "What's the weather in Paris?"}))
        while True:
            msg = json.loads(await ws.recv())
            if msg.get("type") == "tool_call":
                print("TOOL:", msg["name"], msg["arguments"])
            if msg.get("type") == "assistant_end":
                print("turn finished")
                break

asyncio.run(main())
```

Expected WebSocket events, in order: `tool_call` (with `name` and `arguments`) then
`assistant_end`. The `<tool_call>` block must not appear in any `assistant_delta`.

## 9. Limits and best practices

- Calls are not streamed to the client: the agent's answer arrives **after** the
  tool execution. For slow tools, raise `LLM_TOOL_TIMEOUT_SECS` only as much as
  needed, or reduce server-side work.
- The `description` drives tool usage: the default system prompt does not mention
  tools, the model decides solely from the descriptions.
- Keep results **short**: beyond `LLM_TOOL_RESULT_MAX_CHARS` they are truncated and
  the model may lose useful information.
- Do not put secrets in cleartext in the `url`: use `headers` with placeholders, or
  an internal endpoint.
- The tools file is read **once**: restart the service after editing it.