piperust/TOOL_CALLING.md

Tool calling — Operating guide

Code references: src/tools.rs, src/llm.rs, src/orchestrator/turn_manager.rs, src/tools_api.rs, tools.example.json.

---

1. What it does

The engine's LLM can call external HTTP services before answering. A "tool" is not code: it is the JSON description of an HTTP request (URL, method, headers, argument schema). Adding a tool means writing a few lines in a JSON file — no Rust changes.

Per user turn:

1. The engine attaches the OpenAI-shaped tools array to every completion request, with tool_choice: "auto".
2. If needed, the model emits the call. The production LLM server may run without a tool parser, so the call comes back as text in the Qwen native format (<tool_call><function=…><parameter=…>…), which the engine parses client-side.
3. The engine executes the HTTP request, truncates the result and hands it back to the model as a tool role message.
4. The model produces the final answer, which is synthesised to speech.

The <tool_call> block is always stripped from both the spoken and the displayed text: the user never hears or sees it.

2. Enabling it and runtime override

Variable	Default	Meaning
LLM_TOOLS_ENABLED	auto	Master switch. When unset it is true if LLM_TOOLS_FILE is a non-empty path.
LLM_TOOLS_FILE	(empty)	Path to the JSON definitions file. Empty or missing ⇒ tool calling disabled.
LLM_TOOL_MAX_ITERATIONS	4	Maximum tool rounds per turn (call → execute → re-prompt). Bounds a runaway loop.
LLM_TOOL_TIMEOUT_SECS	15	Timeout of one tool HTTP request.
LLM_TOOL_RESULT_MAX_CHARS	2000	The tool result is truncated to this many characters before going back to the model.

LLM_TOOLS_ENABLED=true
LLM_TOOLS_FILE=tools.example.json
LLM_TOOL_MAX_ITERATIONS=4
LLM_TOOL_TIMEOUT_SECS=15
LLM_TOOL_RESULT_MAX_CHARS=2000

Note: the tools file is read once at startup (ToolRegistry::load). After editing the JSON, restart the service for the new definitions to take effect.

Runtime override — POST /api/tools

To change the definitions without a restart, POST the same JSON array to POST /api/tools:

curl -X POST localhost:8787/api/tools -H 'Content-Type: application/json' \
  -d '[{"name":"get_weather","description":"Current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]},"url":"https://wttr.in/{city}?format=3","method":"GET"}]'

• The definitions you send replace the file ones from the next turn on (no restart). If the endpoint is never called, the engine keeps using the LLM_TOOLS_FILE definitions — that is the default, and a restart restores it.
• The override lives in memory only: it does not write the file and does not survive a process restart.
• GET /api/tools reports the active definitions and their origin: source is file (the .env default), request (installed through the endpoint) or none.
• Responses: 200 with {ok,count,source,tools}; 400 when the body is empty, is not UTF-8, is not a valid JSON array, or holds no tool definition; 503 when the LLM is disabled.
• The body uses the same format as the file: an array that works here also works as an LLM_TOOLS_FILE.

curl -s localhost:8787/api/tools
# {"ok":true,"count":2,"source":"file","tools":[ … ]}

Implementation: src/tools_api.rs. Both endpoints also show up in the Swagger UI (/docs, tag tools).

3. Defining a tool

The file is a JSON array. Each object has these fields:

Field	Required	Description
name	yes	Name of the function the model calls. Must match the name in the definition.
description	recommended	This is what the model reads to decide whether and when to call the tool: make it explicit and usage-oriented ("call this tool when the user asks…").
parameters	no	JSON Schema for the arguments. When absent it defaults to {"type":"object","properties":{}}.
url	yes	Target URL. May contain {arg} placeholders.
method	no	Any HTTP verb, case-insensitive: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE, and custom verbs (e.g. PURGE). Default POST.
headers	no	Headers sent verbatim (API keys, Content-Type, …). Values may contain {arg} placeholders.

Placeholder substitution

In url and headers, every {argument_name} is replaced with the matching call argument (strings unquoted, numbers/booleans rendered as text).

Where the arguments go (by verb)

Arguments already consumed by a {arg} placeholder in the URL are not sent again: they are already in the path. The remaining ones follow the verb:

Verb	Remaining arguments
GET, HEAD, OPTIONS, TRACE, DELETE	query parameters (no body)
POST, PUT, PATCH and custom verbs	JSON body ({} when nothing is left)

4. Examples

GET with a URL placeholder (city goes into the path, not the query):

{
  "name": "get_weather",
  "description": "Current weather and temperature for a city. Call this whenever the user asks about the weather, the temperature or the forecast for a specific place.",
  "parameters": {
    "type": "object",
    "properties": { "city": { "type": "string", "description": "City name, e.g. 'Paris' or 'Roma'" } },
    "required": ["city"]
  },
  "url": "https://wttr.in/{city}?format=3",
  "method": "GET"
}

POST with an authenticated header (secret taken from the arguments):

{
  "name": "create_note",
  "description": "Save a note into the internal management system.",
  "parameters": {
    "type": "object",
    "properties": {
      "token": { "type": "string", "description": "User session token" },
      "text":  { "type": "string", "description": "Note content" }
    },
    "required": ["token", "text"]
  },
  "url": "https://internal.example.local/api/notes",
  "method": "POST",
  "headers": { "Authorization": "Bearer {token}" }
}

Plain POST (webhook):

{
  "name": "send_webhook",
  "description": "POST a JSON payload to the configured webhook endpoint (notification / home-automation hook).",
  "parameters": {
    "type": "object",
    "properties": { "message": { "type": "string" } },
    "required": ["message"]
  },
  "url": "https://httpbin.org/post",
  "method": "POST",
  "headers": { "X-Tool": "audio-agent-engine" }
}

A full REST surface — one tool per verb on the same endpoint (list → GET, detail → GET with {id}, add → POST, update → PATCH, replace → PUT, delete → DELETE, exists? → HEAD):

[
  {
    "name": "list_items",
    "description": "List the items, with optional filters.",
    "parameters": {
      "type": "object",
      "properties": {
        "limit":  { "type": "integer" },
        "search": { "type": "string" }
      }
    },
    "url": "https://api.example.local/api/items",
    "method": "GET"
  },
  {
    "name": "get_item",
    "description": "Details of one item by id.",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "GET"
  },
  {
    "name": "add_item",
    "description": "Create a new item.",
    "parameters": {
      "type": "object",
      "properties": { "name": { "type": "string" }, "qty": { "type": "integer" } },
      "required": ["name"]
    },
    "url": "https://api.example.local/api/items",
    "method": "POST"
  },
  {
    "name": "update_item",
    "description": "Partially update an item (only the fields sent).",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" }, "qty": { "type": "integer" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "PATCH"
  },
  {
    "name": "replace_item",
    "description": "Replace an item entirely.",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" }, "name": { "type": "string" } },
      "required": ["id", "name"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "PUT"
  },
  {
    "name": "delete_item",
    "description": "Delete an item by id.",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "DELETE"
  },
  {
    "name": "item_exists",
    "description": "Check whether an item exists (status only, no body).",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    },
    "url": "https://api.example.local/api/items/{id}",
    "method": "HEAD"
  }
]

Resulting requests: GET /api/items?limit=10, GET /api/items/5, POST /api/items + body {"name":"…"}, PATCH /api/items/5 + body {"qty":2}, PUT /api/items/5 + body {"name":"…"}, DELETE /api/items/5, HEAD /api/items/5.

Start from tools.example.json: copy it, adapt it and point LLM_TOOLS_FILE at your file.

5. Multi-round loop and streaming fallback

• Each round produces one completion; the calls found are executed and their results are handed back for the next round (standard OpenAI round-trip: an assistant message with the call, then one tool message per result).
• The loop stops when there are no more calls, or at LLM_TOOL_MAX_ITERATIONS.
• Streaming fallback: some OpenAI-compatible servers (vLLM with a --tool-call-parser that does not match the model) swallow the call on the SSE path: the round ends empty, with no HTTP error. When a round comes back empty and tools are configured, the engine re-asks once without streaming (chat_reply) and takes the call from there. If even that yields no call but holds text, that text is spoken. The fallback only triggers on empty rounds, so it costs nothing on normal turns.

6. Errors and truncation

Tool execution never fails in a blocking way: an error becomes text the model can read and recover from.

Situation	Text handed to the model
Unknown tool	Error: unknown tool '<name>'
Invalid HTTP verb in the definition	Error: invalid HTTP method '<verb>' for tool '<name>'
Non-2xx HTTP status	Error: HTTP <status> — <truncated body>
Network error / timeout	Error: tool request failed: <detail>
Result longer than the limit	Truncated to LLM_TOOL_RESULT_MAX_CHARS, suffixed with … [truncated]

7. Important note on the server-side parser

If the LLM server runs with a tool parser, the flag must match the model's native format (--enable-auto-tool-choice --tool-call-parser …). A mismatch is silent: an hermes parser (expecting JSON inside <tool_call>) against a Qwen model that emits XML makes the parse throw, the call is dropped and, in streaming, the reply comes back empty (vLLM logs: Error in extracting tool call from response). The correct parser for this model is qwen3_coder. The engine is robust either way: it handles both formats, the structured message.tool_calls[] array and the native <tool_call> text, plus the fallback described in section 5.

Typical mismatch symptom: the agent goes silent, or "narrates" the call without executing it.

8. End-to-end verification

With the service running on ws://localhost:8787/ws, send user text that must trigger the tool and check the events:

import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://localhost:8787/ws") as ws:
        await ws.send(json.dumps({"type": "user_text", "text": "What's the weather in Paris?"}))
        while True:
            msg = json.loads(await ws.recv())
            if msg.get("type") == "tool_call":
                print("TOOL:", msg["name"], msg["arguments"])
            if msg.get("type") == "assistant_end":
                print("turn finished")
                break

asyncio.run(main())

Expected WebSocket events, in order: tool_call (with name and arguments) then assistant_end. The <tool_call> block must not appear in any assistant_delta.

9. Limits and best practices

• Calls are not streamed to the client: the agent's answer arrives after the tool execution. For slow tools, raise LLM_TOOL_TIMEOUT_SECS only as much as needed, or reduce server-side work.
• The description drives tool usage: the default system prompt does not mention tools, the model decides solely from the descriptions.
• Keep results short: beyond LLM_TOOL_RESULT_MAX_CHARS they are truncated and the model may lose useful information.
• Do not put secrets in cleartext in the url: use headers with placeholders, or an internal endpoint.
• The tools file is read once: restart the service after editing it.