# audio-agent-client — JavaScript SDK

Client SDK for the `audio-agent-engine` engine. A single ESM file, zero
dependencies, no build step: copy `audio-agent-client.js` wherever you want
(another server, a CDN, your own project) and import it. The engine also serves
it at `GET /sdk/audio-agent-client.js`.

Files: [`audio-agent-client.js`](audio-agent-client.js) ·
[`audio-agent-client.d.ts`](audio-agent-client.d.ts) ·
[`demo.html`](demo.html).

---

## Demo

[`demo.html`](demo.html) is a complete client built on the SDK:
connect/disconnect, session state, streaming agent replies with word-level
*karaoke* highlighting, chat bubbles, inline chips for `cue` / `tool_call` /
`audio_source`, a barge-in counter, a microphone toggle with a level meter, a
text-turn input, an external audio source (play/stop/volume), a **System prompt**
panel (shows the server-assigned session id and lets you reload / override /
clear the session's LLM prompt over REST), automatic phone-number authentication
(read from `?phoneNumber=…` in the page URL at session start, via the engine's
`/api/auth/phone-login` proxy; when a live session id is present the engine also
stores a sanitized user profile as hidden LLM context), a default runtime tool
set installed via `POST /api/tools` and a raw event log. A tool definition may
carry an extra `audio: true` flag (the backend ignores it): when the model calls
such a tool, the demo rebuilds the call's URL, puts it in the external audio
source field and starts playback after a fixed post-`assistant_end` delay, so
media results (e.g. `get_today_podcast_mp3`) do not overlap the agent's spoken
reply while still autoplaying reliably. When the model emits a
`<chat_markdown>...</chat_markdown>` block, the demo renders that document as
formatted Markdown in chat while keeping it out of TTS.
The engine serves it at `GET /sdk/demo.html`; you can also copy it elsewhere (it
imports `./audio-agent-client.js` relatively). Opened from `file://` it works but
without a microphone: it warns via a banner (a *secure context* is required).

## Why

The client is **decoupled** from the engine: the engine stays put, the UI can
live on another origin/server. The SDK wraps the protocol (WebSocket + REST) and
the PCM audio so you don't reimplement framing, sampling and events.

## Install

Nothing to install. Copy the file and import it, or load it from the engine:

```js
import { AudioSession, AudioAgentRest } from 'http://localhost:8787/sdk/audio-agent-client.js';
```

If the client is on **another origin**, enable CORS on the engine:

```dotenv
# engine .env
CORS_ALLOW_ORIGIN=https://your-app.example      # or * for any origin
```

The WebSocket (`/ws`) needs **no** CORS (the handshake is not a CORS request).
For the microphone the browser requires a *secure context*: `https://` or
`localhost`.

## Quick start (browser)

```js
import { AudioSession } from './audio-agent-client.js';

const session = new AudioSession({ baseUrl: 'https://engine.example.com:8787' });

session.on('user', (e) => console.log('user:', e.text));
session.on('assistant_delta', (e) => (bubble.textContent += e.text));
session.on('error', (e) => console.error(e));

await session.start();          // connect + microphone + playback
session.sendUserText('hello');  // text turn (LLM → TTS)
session.interrupt();            // barge-in
await session.stop();
```

## API

### `new AudioAgentClient(options)`

| Option | Default | Description |
|---|---|---|
| `url` | — | Full `ws://` / `wss://` URL. Wins over `baseUrl`. |
| `baseUrl` | — | Engine origin (`http://host:port`); `/ws` is appended. |
| `path` | `/ws` | Path appended to `baseUrl`. |
| `autoReconnect` | `false` | Reconnect after an unexpected close. |
| `reconnectMinDelayMs` / `reconnectMaxDelayMs` | `500` / `8000` | Exponential backoff. |
| `WebSocket` | global | WebSocket implementation (Node < 22: pass the `ws` one). |

Methods: `connect()`, `close()`, `sendAudio(chunk)`, `sendUserText(text)`,
`interrupt()`, `playAudioSource(url)`, `stopAudioSource()`,
`setAudioSourceVolume(gain)`, `setReference(path, text?)`, `clearReference()`,
`speak(text)`, `sendText(text)`, `sendControl(obj)`.
Properties: `state` (`idle`/`connecting`/`open`/`closed`), `connected`.

`sendAudio` accepts `Int16Array`, `Float32Array` (normalized -1..1),
`ArrayBuffer`, a view, or `number[]`. The engine expects **16-bit LE mono PCM at
44.1 kHz**; sending **882-sample** frames (20 ms) keeps the cadence.

Events: `open`, `close`, `state`, `reconnecting`, plus one event per server
`type` — `session`, `cues`, `user`, `assistant_delta`, `assistant_markdown`,
`assistant_end`, `cue`, `barge_in`, `tool_call`, `audio_source`, `error` — and:
- `audio` → `{ samples, buffer, sampleRate }` with the decoded PCM frames;
- `event` → any raw JSON message;
- `error` → either a transport error `{ error, event, fatal }` or the server
  event `{ type: 'error', text }`.

The `session` event (`{ type: 'session', id }`) is sent once at connect time and
carries the server-assigned session id. It is the key used by the REST prompt
endpoints, so capture it to read or override the system prompt for your session.

Every `on()` returns an unsubscribe function.

### `new AudioAgentRest(baseUrl, options?)`

| Method | Endpoint |
|---|---|
| `health()` | `GET /health` |
| `getTools()` | `GET /api/tools` |
| `setTools(tools)` | `POST /api/tools` |
| `getPrompt(sessionId?)` | `GET /api/prompt` |
| `setPrompt(prompt, sessionId?)` | `POST /api/prompt` |
| `phoneLogin(phoneNumber, sessionId?)` | `POST /api/auth/phone-login` |
| `getCustomVoicePhrases()` | `GET /api/customvoice/phrases` |
| `setCustomVoice(lang, audio, text?)` | `POST /api/customvoice` |
| `clearCustomVoice()` | `POST /api/customvoice/clear` |
| `transcribe(audio)` | `POST /api/transcribe` |
| `openapi()` | `GET /api-docs/openapi.json` |

On a non-2xx response the thrown error carries `status` and `body`.

### Audio helpers (browser only)

- **`MicCapture`** — microphone → `frame` events (`Int16Array`, 882 samples).
  `await start()`, `stop()`.
- **`PcmPlayer`** — jitter-buffered playback (`bufferMs`, default 150).
  `push(chunk)`, `flush()` (barge-in), `setGain()`, `close()`.
- **`AudioSession`** — wires client + mic + player and re-emits client events.
  `await start({ microphone = true })`, `stop()`, `sendUserText()`, `interrupt()`.

### PCM utilities

`int16ArrayFromBytes(buffer)`, `float32ToInt16(floats)`,
`int16ToFloat32(ints)`, `toInt16(chunk)` and the `AUDIO` constant
(`sampleRate: 44100`, `frameSamples: 882`, `frameBytes: 1764`).

## Common pitfalls

- **Microphone denied** → the page is not a *secure context*: use `https://` or
  `localhost`.
- **`Failed to fetch` on REST from another origin** → set `CORS_ALLOW_ORIGIN` on
  the engine.
- **No playback** → the `AudioContext` starts suspended until a user gesture:
  call `start()` / `player.resume()` from a click.
- **Node: `no WebSocket implementation`** → Node < 22: pass the `ws` `WebSocket`.