client sdk

2026-10-21 17:56:04 +01:00
parent 1cafe3dd3c
commit d6bc2437c9
237 changed files with 5227 additions and 1019 deletions
@@ -30,6 +30,9 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
 ## Features

 - **Full-duplex WebSocket audio** — 44.1 kHz / 16-bit mono PCM, 20 ms frames.
+- **JavaScript client SDK** — a dependency-free, single-file ESM module (browser
+  + Node) wrapping the WebSocket session and the REST surface, so the client can
+  live on another server. See [Client SDK (JavaScript)](#client-sdk-javascript).
 - **Pluggable TTS engines** — switch engines (Fish Audio, Kokoro, …) via `.env`
  without changing orchestration code. See [TTS Plugin System](#tts-plugin-system).
 - **Latency filler** — a preloaded filler is played on end-of-utterance *only if
@@ -95,6 +98,11 @@ Browser mic ──► WebSocket PCM ──► STT (Parakeet/ONNX) + VAD/EOU
  format (a round that streams back empty is re-asked without streaming),
  executed over HTTP and fed back to the model; the tool-call block is never
  spoken. See [Tool calling](#tool-calling).
+- **Per-session system prompt** — each WebSocket session gets a server-assigned
+  id (delivered to the client in a `{"type":"session","id":"…"}` message) and the
+  LLM system prompt can be read and overridden **for that session** — or
+  process-wide — at runtime through `GET`/`POST /api/prompt`, without a restart.
+  See [Runtime system prompt](#runtime-system-prompt-get--post-apiprompt).
 - **Embedded playground** — a single-page HTML client served at `/`.
 - **Mivocero speech-to-speech client** — an example agent client at `/mivocero`
  with a microphone, a live chat box (user transcript + streaming assistant
@@ -223,7 +231,7 @@ See [`.env.example`](.env.example) for a copy-paste template.
 | `VAD_SEGMENT_STT` | `true` | Feed the STT engine with VAD-segmented utterances instead of the worker's RMS segmentation. |
 | `VAD_BARGE_IN` | `true` | Interrupt the agent when the user starts speaking over it. |
 | `VAD_DEBUG` | `false` | Log the speech probability of every analysis window. |
-| `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/fillers/es/`). |
+| `FILLERS_DIR` | `assets/fillers` | Directory of `*.raw` filler PCM files (16-bit LE mono, 44.1 kHz), played round-robin, at most once per turn. Clips live in a per-language, per-voice-profile subdirectory `<dir>/<lang>/<voice_profile>` (e.g. `assets/fillers/es/maricel/`). |
 | `FILLER_ENABLED` | `true` | Master switch for the end-of-turn filler. `false` => no filler is ever queued. |
 | `FILLER_DELAY_MS` | `2500` | How long (ms) to wait for the response audio to start before playing the filler. If the reply starts earlier, the filler is dropped instead of talking over the beginning of the answer. |
 | `FILLER_GAIN_IDLE` | `1.0` | Filler gain when TTS is idle. |
@@ -231,11 +239,12 @@ See [`.env.example`](.env.example) for a copy-paste template.
 | `START_FILLER_ENABLED` | `true` | Master switch for the **start filler** — a turn-start acknowledgment queued on its own mixer channel when the reply audio is late, independently of the delayed filler above. `false` => no cue is ever queued. |
 | `START_FILLER_GO_DELAY` | `1800` | How long (ms) the agent waits for the **first audio chunk of the reply** to reach the mixer before queueing the start filler. If audio arrives within this window the cue is dropped instead of talking over the beginning of the answer. The gate is latched on chunk *arrival* (`AudioMixer::tts_audio_arrived`), not on playback start, so it does not wait for the `TTS_PREBUFFER_MS` jitter buffer. Same idea as `FILLER_DELAY_MS`, but shorter — the start filler only covers the gap before the reply begins, not the whole generation. If the cue does fire and the reply then starts, it is faded out (80 ms) and cut so it does not overlap the answer. |
 | `START_FILLER_TEXTS` | `[calm] Un momento...` | Phrases spoken by the start filler, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and a **random** one is played when the reply is late, so the acknowledgment varies. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, `[sigh]`, …) are passed through to the TTS. |
-| `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/start_filler/es/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. |
+| `START_FILLER_DIR` | `assets/start_filler` | Directory holding the start-filler clips (`sf01.raw`, `sf02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language, per-voice-profile subdirectory `<dir>/<lang>/<voice_profile>` (e.g. `assets/start_filler/es/maricel/`). Kept out of `FILLERS_DIR` so the clips are not part of the round-robin filler bank. |
+| `ASSET_VOICE_PROFILE` | *(auto)* | Optional explicit cue-asset profile name. When unset, the engine derives it from the active voice setup (prefer `TTS_VOICE` when it is not `default`, otherwise fall back to the reference-audio file stem, then to `default`). |
 | `TTS_PREBUFFER_MS` | `550` | Synthesized audio (ms) buffered on the TTS channel before the reply starts playing. Lower = the agent speaks sooner; higher = fewer drop-outs on uneven TTS chunk arrival. With the deployment's vocoder windows (first `185.8` ms, then `371.6` ms), any value in `(186, 557]` ms primes on the *same* second window, so `550` takes the whole cushion at no extra TTFA (~`0.73 s`). Below the first window (~`186` ms) playback starts on that window (TTFA ≈ server TTFB, ~`0.38 s`) but leaves almost no margin and exposes chunk seams. Only awaited at the start of a reply; mid-reply underruns resume as soon as any audio is queued again. |
 | `TTS_PIPELINE_DEPTH` | `3` | How many TTS requests may be in flight at once. Chunks are synthesized ahead of playback and replayed in order, so each request's start-up latency hides behind the previous chunk's audio. `1` restores strictly serial synthesis. |
 | `BACKCHANNEL_ENABLED` | `true` | Master switch for during-utterance acknowledgments. |
-| `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of the generated `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz; `bc01.raw`, `bc02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language subdirectory `<dir>/<lang>` (e.g. `assets/backchannel/es/`). |
+| `BACKCHANNEL_DIR` | `assets/backchannel` | Directory of the generated `*.raw` backchannel PCM clips (16-bit LE mono, 44.1 kHz; `bc01.raw`, `bc02.raw`, …, plus a `texts.txt` list index). Clips live in a per-language, per-voice-profile subdirectory `<dir>/<lang>/<voice_profile>` (e.g. `assets/backchannel/es/maricel/`). |
 | `BACKCHANNEL_TEXTS` | `ack::[calm] Mm-hmm...` … | Phrases spoken by the backchannel, **one per line** (a `\|` separator also works). One clip is synthesized per phrase and one is played per cue. Editing the list re-synthesizes the whole bank at the next restart (no manual deletion); an unchanged list is skipped to keep startup fast, and a failed synthesis keeps the previous clip. Drama-3 style tags (`[calm]`, …) are passed through to the TTS. A line may carry an optional **`category::`** prefix (e.g. `agree::[agreeing] Claro...`) used to progress through the turn; a line without it belongs to the default `ack` category. |
 | `BACKCHANNEL_GAIN` | `0.35` | Mix gain applied to the backchannel channel (0..1). |
 | `BACKCHANNEL_MIN_SPEECH_MS` | `1500` | Confirmed speech required before the first cue may be considered. |
@@ -246,7 +255,7 @@ See [`.env.example`](.env.example) for a copy-paste template.
 | `BACKCHANNEL_PROB` | `0.65` | Probability (0..1) of cueing on a qualifying micro-pause — keeps the agent from acknowledging every pause. |
 | `BACKCHANNEL_VOLUME_JITTER` | `0.2` | Random per-cue volume scaling (± fraction), so cues don't sit at a fixed level. |
 | `BACKCHANNEL_MAX_PER_TURN` | `3` | Maximum backchannels emitted per user turn. |
-| `LANG_DEFAULT` | `es` | Base language of the **filler, start-filler and backchannel** banks — its clips live in the `<dir>/<lang>` subdirectory (e.g. `assets/backchannel/es/`). Other languages are generated on demand into their own per-language subdirectories. See [Multilingual clip banks](#multilingual-clip-banks). |
+| `LANG_DEFAULT` | `es` | Base language of the **filler, start-filler and backchannel** banks — its clips live in the `<dir>/<lang>/<voice_profile>` subdirectory (e.g. `assets/backchannel/es/maricel/`). Other languages are generated on demand into their own language/voice-profile subdirectories. See [Multilingual clip banks](#multilingual-clip-banks). |
 | `LANG_DETECT_ENABLED` | `true` | Master switch for LLM-based language detection. Requires the LLM (`LLM_ENABLED=true`). |
 | `LANG_MIN_CHARS` | `12` | Minimum transcript length (chars) before a sample is sent to the detector. |
 | `LANG_REDETECT_EVERY` | `5` | The first user turn is always checked; afterwards the language is re-checked once every N turns. |
@@ -268,7 +277,7 @@ See [`.env.example`](.env.example) for a copy-paste template.
 | `LLM_ENABLE_THINKING` | *(unset)* | `false` sends `chat_template_kwargs: {"enable_thinking": false}` — turns Qwen3-style thinking off on vLLM/SGLang so the model answers directly. Unset omits the vendor-specific field entirely (safe for strict OpenAI-compatible servers). |
 | `LLM_STRIP_REASONING` | `true` | Drop the chain-of-thought from the reply before it reaches the chat box / TTS: ignores `reasoning_content` deltas and removes inline think blocks. `false` passes the raw output through. |
 | `LLM_DIRECT_ANSWERS` | `true` | Append an "answer directly, never narrate your reasoning" directive to the system prompt. |
-| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). |
+| `LLM_SYSTEM_PROMPT` | *(built-in)* | Overrides the system prompt. The default asks for short, spoken-friendly replies (no markdown / bullet points). Can also be overridden at runtime per session via `GET`/`POST /api/prompt` ([Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)). |
 | `LLM_TOOLS_ENABLED` | *(auto)* | Master switch for tool calling. Default: enabled when `LLM_TOOLS_FILE` is a non-empty path. |
 | `LLM_TOOLS_FILE` | *(empty)* | JSON file with the tool definitions (`name`, `description`, `parameters` JSON schema, `url`, `method` — any HTTP verb, `headers`). See [Tool calling](#tool-calling) and [`tools.example.json`](tools.example.json). |
 | `LLM_TOOL_MAX_ITERATIONS` | `4` | Maximum tool-call rounds per turn (call → execute → re-prompt). Bounds runaway loops. |
@@ -276,6 +285,9 @@ See [`.env.example`](.env.example) for a copy-paste template.
 | `LLM_TOOL_RESULT_MAX_CHARS` | `2000` | Tool responses are truncated to this many characters before being handed back to the model. |
 | `TLS_CERT` | *(empty)* | PEM certificate chain. Set together with `TLS_KEY` to serve HTTPS/WSS. |
 | `TLS_KEY` | *(empty)* | PEM private key matching `TLS_CERT`. |
+| `CORS_ALLOW_ORIGIN` | *(empty)* | Comma-separated browser origins allowed to call the REST surface cross-origin (single `*` = any). Empty adds no CORS layer (default behaviour). Needed when the [client SDK](#client-sdk-javascript) is served from another origin. Does not affect `/ws` (a WebSocket handshake is not a CORS request). |
+| `AUTH_API_BASE` | `https://aquachile.azurianlab.com` | Base URL of the auth API proxied by `POST /api/auth/phone-login` ([Phone login](#phone-login-post-apiauthphone-login)). |
+| `AUTH_PHONE_LOGIN_PATH` | `/api/v1/auth/phone-login` | Login path on `AUTH_API_BASE`. |
 | `SESSION_AUDIT_ENABLED` | `true` | Write a structured JSONL audit file per WebSocket session (config snapshot, inbound control messages, every outbound event, cues, language switches, barge-ins, end-of-session summary). Best-effort — a write failure never affects the session. See [Session audit log](#session-audit-log). |
 | `SESSION_AUDIT_DIR` | `logs/sessions` | Directory holding the per-session audit files (`<id>.jsonl`). Relative to the process working directory; git-ignored. |

@@ -882,20 +894,26 @@ of any other language.

 ### Layout

-Every language — **including the base `LANG_DEFAULT`** — lives in its own
-subdirectory named after its ISO 639-1 code under `FILLERS_DIR` /
-`START_FILLER_DIR` / `BACKCHANNEL_DIR`:
+Every language — **including the base `LANG_DEFAULT`** — keeps a separate bank
+for each cue-asset voice profile under `FILLERS_DIR` / `START_FILLER_DIR` /
+`BACKCHANNEL_DIR`:

 ```text
-assets/backchannel/es/     <- base language (LANG_DEFAULT), pre-synthesized
-assets/fillers/es/
-assets/start_filler/es/
-assets/backchannel/en/     <- English, generated on demand
-assets/backchannel/it/     <- Italian, generated on demand
-assets/fillers/en/
-assets/start_filler/en/
+assets/backchannel/es/maricel/     <- base language (LANG_DEFAULT), profile maricel
+assets/fillers/es/maricel/
+assets/start_filler/es/maricel/
+assets/backchannel/es/solis/       <- same language, different voice profile
+assets/backchannel/en/maricel/     <- English, generated on demand
+assets/fillers/en/maricel/
+assets/start_filler/en/maricel/
 ```

+On startup, if a legacy single-voice bank still exists directly under
+`<dir>/<lang>/` (for example `assets/backchannel/es/bc01.raw`), the engine
+copies those flat files into the active `<dir>/<lang>/<voice_profile>/`
+directory when the target files are missing. The legacy files are left in
+place, so the migration is non-destructive.
+
 Because the refresh pass (`remove_stale_clips`) only prunes `*.raw` files inside
 the directory it is given, scoping each language — base included — to its own
 subdirectory is what guarantees a regeneration never deletes another language's
@@ -997,7 +1015,9 @@ chat box  ──────────────┘
  overlaps each request's start-up latency with the previous chunk's playback,
  so raising the pre-buffer is no longer needed to avoid inter-chunk gaps.
 - **History** — a rolling window of the last `LLM_HISTORY_MESSAGES` user/assistant
-  messages is kept per connection and prepended with `LLM_SYSTEM_PROMPT`.
+  messages is kept per connection and prepended with the effective system prompt
+  (the session override if any, else `LLM_SYSTEM_PROMPT`, else the built-in
+  default — see [Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)).
 - **Barge-in** — an interrupt (client JSON, VAD onset, or disconnect) cancels
  both the in-flight HTTP request ([`CancellationToken`] in the `reqwest`
  `tokio::select!`, including while waiting for the next SSE chunk) and the TTS
@@ -1007,6 +1027,51 @@ chat box  ──────────────┘
 - **Failure handling** — request errors are reported to the client as
  `{"type":"error","text":…}` and the turn ends cleanly.

+### Runtime system prompt (`GET` / `POST /api/prompt`)
+
+The system prompt is normally the process-wide default (the built-in one, or
+`LLM_SYSTEM_PROMPT`). Because it is injected identically into every session, two
+clients cannot run with different instructions. `GET`/`POST /api/prompt` adds a
+**runtime, in-memory** override — per session or process-wide.
+
+Each WebSocket session is assigned a server id (e.g. `s-1729170000000-0001`, the
+same id used for its audit log) and the id is pushed to the client right after
+connect, before the `cues` message:
+
+```json
+{"type":"session","id":"s-1729170000000-0001"}
+```
+
+That id is the key of the REST surface:
+
+```bash
+# Effective prompt for a session (+ where it comes from)
+curl 'localhost:8787/api/prompt?session_id=s-1729170000000-0001'
+
+# Override it for that session
+curl -X POST localhost:8787/api/prompt -H 'Content-Type: application/json' \
+  -d '{"session_id":"s-1729170000000-0001","prompt":"You are a terse assistant. One sentence max."}'
+
+# Clear the override (falls back to the configured default)
+curl -X POST localhost:8787/api/prompt -H 'Content-Type: application/json' \
+  -d '{"session_id":"s-1729170000000-0001","prompt":null}'
+```
+
+- **Scope** — with a `session_id` the override applies to that session only; omit
+  it (`null`) to set a process-wide fallback used by every session without its
+  own override.
+- **Precedence** — session override → process-wide override → `LLM_SYSTEM_PROMPT`
+  → built-in default. The response's `source` field (`session` / `global` /
+  `env` / `default` / `none`) reports which one won, and `has_override` whether a
+  runtime override is in effect.
+- **In-memory only** — a restart restores the configured default, and a session
+  override is dropped when that session disconnects. An unknown/disconnected
+  `session_id` returns `404`.
+- `prompt: null` (or `""`) clears the override.
+- The response also carries `direct_answers_applied` (whether the
+  `LLM_DIRECT_ANSWERS` directive is currently appended).
+- See [`src/prompt.rs`](src/prompt.rs) and [`src/prompt_api.rs`](src/prompt_api.rs).
+
 ---

 ## Tool calling
@@ -1419,6 +1484,7 @@ Connect to `ws://<host>:8787/ws`.
 | **Text — `{"type":"audio_source","state":"…","url":"…"}`** | External audio channel state change: `loading` (decoding is being set up / YouTube URL resolving), `playing` (PCM is streaming), `stopped`, or `error` (with a `text` message). |
 | **Text — `{"type":"cue","kind":"…","name":"…","category":"…"}`** | A latency-mask / backchannel cue was actually emitted into the mixer: `kind` is `filler` (end-of-turn), `start_filler` (turn start) or `backchannel` (during user speech); `name` is the clip name and `category` (backchannel only) its category. Fired only when the cue really plays — a skipped cue sends nothing. Drives the demo's cue indicators. |
 | **Text — `{"type":"cues","lang":"…","config":{…}}`** | Sent once at session start: the effective policy of the three cues for the active language — the filler parameters (`delay_ms`, gains, clip count), the start-filler `go_delay_ms` and the full backchannel configuration (gain, min-speech, interval, jitter, pause window, probability, volume jitter, per-turn cap, categories). Purely informational; drives the demo's **Cue methods** panel so it shows the live values instead of hard-coded ones. |
+| **Text — `{"type":"session","id":"…"}`** | Sent once right after connect: the server-assigned session id (also the audit-log file stem). It is the key used by `GET`/`POST /api/prompt` to read/override the system prompt of this session ([Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)). |
 | **Text — `{"type":"barge_in","source":"vad","prob":…}`** | The agent was interrupted. Sent for the server-side VAD barge-in (the browser has no other way to know, since the onset is detected in the session task); the client-side `interrupt` does not echo one. Drives the demo's barge-in indicator. |

 ---
@@ -1491,6 +1557,7 @@ audio-agent-engine/
 ├── certs/                    # self-signed TLS cert+key (generated)
 ├── UPDATES.md                # changelog (English, newest-first)
 ├── tools.example.json        # tool definitions for LLM tool calling (copy/edit)
+├── sdk/                      # dependency-free JavaScript client SDK (+ .d.ts, served at /sdk/)
 ├── assets/fillers/<lang>/    # *.raw filler PCM (16-bit LE mono, 44.1 kHz; round-robin), one dir per language
 ├── assets/start_filler/<lang>/  # sf01.raw … turn-start start-filler clips + texts.txt index
 ├── assets/backchannel/<lang>/   # bc01.raw … backchannel clips + texts.txt index
@@ -1511,6 +1578,9 @@ audio-agent-engine/
    ├── llm.rs                # OpenAI-compatible chat-completions client (streaming + think stripping)
    ├── tools.rs              # configurable HTTP tool calling (native <tool_call> parser + executor)
    ├── tools_api.rs          # GET/POST /api/tools — runtime tool definitions (in-memory override)
+    ├── prompt.rs             # per-session system-prompt registry (overrides + resolution)
+    ├── prompt_api.rs         # GET/POST /api/prompt — read/override the LLM system prompt per session
+    ├── auth_api.rs           # POST /api/auth/phone-login — server-side proxy to the AquaChile auth API
    ├── media.rs              # external audio source (URL / radio / YouTube) via ffmpeg + yt-dlp
    ├── customvoice.rs        # custom-voice builder endpoints (record → WAV reference → set_reference)
    ├── transcribe.rs         # POST /api/transcribe — Parakeet transcription of an uploaded audio file
@@ -1534,7 +1604,12 @@ audio-agent-engine/
 | `POST /api/transcribe` | Upload a raw audio file (any ffmpeg-readable container); returns the Parakeet transcript of the whole file (`text`, `duration_ms`, `tokens`). |
 | `GET /api/tools` | The active LLM tool definitions and their origin (`source`: `file` / `request` / `none`). |
 | `POST /api/tools` | Install LLM tool definitions at runtime (same JSON array as `LLM_TOOLS_FILE`) — in-memory, no restart ([Tool calling](#tool-calling)). |
+| `GET /api/prompt` | The effective LLM system prompt, with its origin (`source`: `session` / `global` / `env` / `default` / `none`). Optional `?session_id=…`; `404` for an unknown session ([Runtime system prompt](#runtime-system-prompt-get--post-apiprompt)). |
+| `POST /api/prompt` | Override the LLM system prompt at runtime — for one session (`{"session_id":"…","prompt":"…"}`) or process-wide (`prompt` only); `prompt: null` clears it. In-memory, no restart. |
+| `POST /api/auth/phone-login` | Authenticate a phone number (`{"phoneNumber":"+44…","session_id":"s-…"}`) through a **server-side proxy** to the AquaChile auth API (`AUTH_API_BASE` + `AUTH_PHONE_LOGIN_PATH`) and return its payload (`sessionToken` + `user`). When `session_id` is provided, the engine stores a sanitized user profile as hidden LLM context for that live session ([Phone login](#phone-login-post-apiauthphone-login)). |
 | `GET /health` | Liveness probe (`ok`). |
+| `GET /sdk/audio-agent-client.js` | The dependency-free JavaScript client SDK ([Client SDK (JavaScript)](#client-sdk-javascript), source in [`sdk/`](sdk)). |
+| `GET /sdk/demo.html` | Demo client built on the SDK (source in [`sdk/demo.html`](sdk/demo.html)). |
 | `GET /ws` | WebSocket upgrade → per-session audio loop. |
 | `GET /docs` | Swagger UI for the REST surface ([API documentation](#api-documentation-swagger-ui)). |
 | `GET /api-docs/openapi.json` | The raw OpenAPI 3 document the Swagger UI renders. |
@@ -1586,6 +1661,110 @@ curl -s -X POST http://localhost:8787/api/transcribe \

 ---

+## Phone login (`POST /api/auth/phone-login`)
+
+The SDK demo can authenticate the user by phone number, read from the page URL
+(`?phoneNumber=+44…`), at the start of the session. The call is made **through
+the engine**, not directly from the browser:
+
+```bash
+curl -X POST localhost:8787/api/auth/phone-login \
+  -H 'Content-Type: application/json' \
+  -d '{"phoneNumber":"+447432477433"}'
+```
+
+The engine forwards the same JSON to the upstream AquaChile auth API
+(`AUTH_API_BASE` + `AUTH_PHONE_LOGIN_PATH`) and returns its status and body
+unchanged — on success a payload like
+`{"sessionToken":"…","user":{"id":…,"name":…,"email":…,"permissions":[…]}}`.
+
+If the request also carries a live WebSocket `session_id`, the engine extracts a
+sanitized user profile (`name`, `email`, `businessRoles`, `permissionSet`,
+`centers`, `permissions`) and injects it into that session's LLM context as a
+hidden system block. The SDK demo does this automatically right after connect.
+
+The proxy exists because the upstream API rejects browser calls from the client
+origin: its CORS preflight answers `400 Disallowed CORS origin` and its response
+carries no `Access-Control-Allow-Origin`, so a direct `fetch` from the demo is
+blocked. Routing the call through the engine makes it same-origin for the
+browser. Errors: `400` for an empty/invalid body or a missing `phoneNumber`,
+`502` when the upstream is unreachable. See [`src/auth_api.rs`](src/auth_api.rs).
+
+---
+
+## Client SDK (JavaScript)
+
+[`sdk/audio-agent-client.js`](sdk/audio-agent-client.js) is a **dependency-free,
+single-file ESM module** (with [`sdk/audio-agent-client.d.ts`](sdk/audio-agent-client.d.ts))
+for writing clients against the engine. It is deliberately decoupled from the
+server: copy the file to any host, no build step, no npm install. The engine also
+serves it at `GET /sdk/audio-agent-client.js`.
+
+A ready-made **[demo client](sdk/demo.html)** built on the SDK is served at
+`GET /sdk/demo.html` — open `http://localhost:8787/sdk/demo.html` and press
+*Connect*.
+
+It covers:
+
+- **Realtime session** — `AudioAgentClient`: connects to `/ws`, sends PCM
+  (`sendAudio`) and typed controls (`sendUserText`, `interrupt`,
+  `playAudioSource`, `setReference`, `speak`, …), and emits one event per server
+  message (`assistant_delta`, `cue`, `barge_in`, `tool_call`, `audio_source`,
+  `error`, …) plus a local `audio` event carrying the decoded PCM frames.
+- **REST** — `AudioAgentRest`: `health`, `getTools` / `setTools`,
+  `getCustomVoicePhrases` / `setCustomVoice` / `clearCustomVoice`, `transcribe`,
+  `openapi`; failures throw an error carrying `status` and the parsed `body`.
+- **Browser audio helpers** — `MicCapture` (mic → 882-sample `Int16Array` frames,
+  via an inline AudioWorklet), `PcmPlayer` (jitter-buffered playback,
+  `flush()` for barge-in) and `AudioSession`, which wires the three together.
+- **PCM utilities** — `int16ArrayFromBytes`, `float32ToInt16`, `int16ToFloat32`,
+  `toInt16`, plus the `AUDIO` constants (44.1 kHz, 20 ms = 882 samples).
+
+### Browser
+
+```html
+<script type="module">
+  import { AudioSession } from 'http://localhost:8787/sdk/audio-agent-client.js';
+
+  const session = new AudioSession({ baseUrl: 'http://localhost:8787' });
+  session.on('user', (e) => console.log('you:', e.text));
+  session.on('assistant_delta', (e) => (log.textContent += e.text));
+  session.on('error', (e) => console.error(e));
+  await session.start();              // connect + microphone + playback
+  session.sendUserText('hello');      // text turn
+  session.interrupt();                // barge-in
+</script>
+```
+
+Serve the page from a **secure context** (`https://` or `localhost`) or the
+browser will refuse microphone access. Cross-origin pages must have CORS enabled
+on the engine (`CORS_ALLOW_ORIGIN=https://your.app`); `/ws` needs no CORS.
+
+### Node
+
+The core (client + REST + PCM helpers) is browser-agnostic. Node 22+ has a global
+`WebSocket`; on older versions pass one in (e.g. the `ws` package). The audio
+helpers (`MicCapture`, `PcmPlayer`, `AudioSession`) require the Web Audio API and
+are browser-only.
+
+```js
+import { AudioAgentClient, AudioAgentRest, toInt16 } from './sdk/audio-agent-client.js';
+import WebSocket from 'ws';
+
+const client = new AudioAgentClient({ baseUrl: 'http://localhost:8787', WebSocket });
+client.on('assistant_delta', (e) => process.stdout.write(e.text));
+await client.connect();
+client.sendUserText("what's the weather in Rome?");
+client.sendAudio(toInt16(new Float32Array(882)));   // PCM 16-bit LE, 44.1 kHz
+
+const rest = new AudioAgentRest('http://localhost:8787');
+console.log(await rest.getTools());
+```
+
+See [`sdk/README.md`](sdk/README.md) for the full reference.
+
+---
+
 ## Development

 ```bash