159 lines
6.9 KiB
Markdown
159 lines
6.9 KiB
Markdown
# audio-agent-client — JavaScript SDK
|
|
|
|
Client SDK for the `audio-agent-engine` engine. A single ESM file, zero
|
|
dependencies, no build step: copy `audio-agent-client.js` wherever you want
|
|
(another server, a CDN, your own project) and import it. The engine also serves
|
|
it at `GET /sdk/audio-agent-client.js`.
|
|
|
|
Files: [`audio-agent-client.js`](audio-agent-client.js) ·
|
|
[`audio-agent-client.d.ts`](audio-agent-client.d.ts) ·
|
|
[`demo.html`](demo.html).
|
|
|
|
---
|
|
|
|
## Demo
|
|
|
|
[`demo.html`](demo.html) is a complete client built on the SDK:
|
|
connect/disconnect, session state, streaming agent replies with word-level
|
|
*karaoke* highlighting, chat bubbles, inline chips for `cue` / `tool_call` /
|
|
`audio_source`, a barge-in counter, a microphone toggle with a level meter, a
|
|
text-turn input, an external audio source (play/stop/volume), a **System prompt**
|
|
panel (shows the server-assigned session id and lets you reload / override /
|
|
clear the session's LLM prompt over REST), automatic phone-number authentication
|
|
(read from `?phoneNumber=…` in the page URL at session start, via the engine's
|
|
`/api/auth/phone-login` proxy; when a live session id is present the engine also
|
|
stores a sanitized user profile as hidden LLM context), a default runtime tool
|
|
set installed via `POST /api/tools` and a raw event log. A tool definition may
|
|
carry an extra `audio: true` flag (the backend ignores it): when the model calls
|
|
such a tool, the demo rebuilds the call's URL, puts it in the external audio
|
|
source field and starts playback after a fixed post-`assistant_end` delay, so
|
|
media results (e.g. `get_today_podcast_mp3`) do not overlap the agent's spoken
|
|
reply while still autoplaying reliably. When the model emits a
|
|
`<chat_markdown>...</chat_markdown>` block, the demo renders that document as
|
|
formatted Markdown in chat while keeping it out of TTS.
|
|
The engine serves it at `GET /sdk/demo.html`; you can also copy it elsewhere (it
|
|
imports `./audio-agent-client.js` relatively). Opened from `file://` it works but
|
|
without a microphone: it warns via a banner (a *secure context* is required).
|
|
|
|
## Why
|
|
|
|
The client is **decoupled** from the engine: the engine stays put, the UI can
|
|
live on another origin/server. The SDK wraps the protocol (WebSocket + REST) and
|
|
the PCM audio so you don't reimplement framing, sampling and events.
|
|
|
|
## Install
|
|
|
|
Nothing to install. Copy the file and import it, or load it from the engine:
|
|
|
|
```js
|
|
import { AudioSession, AudioAgentRest } from 'http://localhost:8787/sdk/audio-agent-client.js';
|
|
```
|
|
|
|
If the client is on **another origin**, enable CORS on the engine:
|
|
|
|
```dotenv
|
|
# engine .env
|
|
CORS_ALLOW_ORIGIN=https://your-app.example # or * for any origin
|
|
```
|
|
|
|
The WebSocket (`/ws`) needs **no** CORS (the handshake is not a CORS request).
|
|
For the microphone the browser requires a *secure context*: `https://` or
|
|
`localhost`.
|
|
|
|
## Quick start (browser)
|
|
|
|
```js
|
|
import { AudioSession } from './audio-agent-client.js';
|
|
|
|
const session = new AudioSession({ baseUrl: 'https://engine.example.com:8787' });
|
|
|
|
session.on('user', (e) => console.log('user:', e.text));
|
|
session.on('assistant_delta', (e) => (bubble.textContent += e.text));
|
|
session.on('error', (e) => console.error(e));
|
|
|
|
await session.start(); // connect + microphone + playback
|
|
session.sendUserText('hello'); // text turn (LLM → TTS)
|
|
session.interrupt(); // barge-in
|
|
await session.stop();
|
|
```
|
|
|
|
## API
|
|
|
|
### `new AudioAgentClient(options)`
|
|
|
|
| Option | Default | Description |
|
|
|---|---|---|
|
|
| `url` | — | Full `ws://` / `wss://` URL. Wins over `baseUrl`. |
|
|
| `baseUrl` | — | Engine origin (`http://host:port`); `/ws` is appended. |
|
|
| `path` | `/ws` | Path appended to `baseUrl`. |
|
|
| `autoReconnect` | `false` | Reconnect after an unexpected close. |
|
|
| `reconnectMinDelayMs` / `reconnectMaxDelayMs` | `500` / `8000` | Exponential backoff. |
|
|
| `WebSocket` | global | WebSocket implementation (Node < 22: pass the `ws` one). |
|
|
|
|
Methods: `connect()`, `close()`, `sendAudio(chunk)`, `sendUserText(text)`,
|
|
`interrupt()`, `playAudioSource(url)`, `stopAudioSource()`,
|
|
`setAudioSourceVolume(gain)`, `setReference(path, text?)`, `clearReference()`,
|
|
`speak(text)`, `sendText(text)`, `sendControl(obj)`.
|
|
Properties: `state` (`idle`/`connecting`/`open`/`closed`), `connected`.
|
|
|
|
`sendAudio` accepts `Int16Array`, `Float32Array` (normalized -1..1),
|
|
`ArrayBuffer`, a view, or `number[]`. The engine expects **16-bit LE mono PCM at
|
|
44.1 kHz**; sending **882-sample** frames (20 ms) keeps the cadence.
|
|
|
|
Events: `open`, `close`, `state`, `reconnecting`, plus one event per server
|
|
`type` — `session`, `cues`, `user`, `assistant_delta`, `assistant_markdown`,
|
|
`assistant_end`, `cue`, `barge_in`, `tool_call`, `audio_source`, `error` — and:
|
|
- `audio` → `{ samples, buffer, sampleRate }` with the decoded PCM frames;
|
|
- `event` → any raw JSON message;
|
|
- `error` → either a transport error `{ error, event, fatal }` or the server
|
|
event `{ type: 'error', text }`.
|
|
|
|
The `session` event (`{ type: 'session', id }`) is sent once at connect time and
|
|
carries the server-assigned session id. It is the key used by the REST prompt
|
|
endpoints, so capture it to read or override the system prompt for your session.
|
|
|
|
Every `on()` returns an unsubscribe function.
|
|
|
|
### `new AudioAgentRest(baseUrl, options?)`
|
|
|
|
| Method | Endpoint |
|
|
|---|---|
|
|
| `health()` | `GET /health` |
|
|
| `getTools()` | `GET /api/tools` |
|
|
| `setTools(tools)` | `POST /api/tools` |
|
|
| `getPrompt(sessionId?)` | `GET /api/prompt` |
|
|
| `setPrompt(prompt, sessionId?)` | `POST /api/prompt` |
|
|
| `phoneLogin(phoneNumber, sessionId?)` | `POST /api/auth/phone-login` |
|
|
| `getCustomVoicePhrases()` | `GET /api/customvoice/phrases` |
|
|
| `setCustomVoice(lang, audio, text?)` | `POST /api/customvoice` |
|
|
| `clearCustomVoice()` | `POST /api/customvoice/clear` |
|
|
| `transcribe(audio)` | `POST /api/transcribe` |
|
|
| `openapi()` | `GET /api-docs/openapi.json` |
|
|
|
|
On a non-2xx response the thrown error carries `status` and `body`.
|
|
|
|
### Audio helpers (browser only)
|
|
|
|
- **`MicCapture`** — microphone → `frame` events (`Int16Array`, 882 samples).
|
|
`await start()`, `stop()`.
|
|
- **`PcmPlayer`** — jitter-buffered playback (`bufferMs`, default 150).
|
|
`push(chunk)`, `flush()` (barge-in), `setGain()`, `close()`.
|
|
- **`AudioSession`** — wires client + mic + player and re-emits client events.
|
|
`await start({ microphone = true })`, `stop()`, `sendUserText()`, `interrupt()`.
|
|
|
|
### PCM utilities
|
|
|
|
`int16ArrayFromBytes(buffer)`, `float32ToInt16(floats)`,
|
|
`int16ToFloat32(ints)`, `toInt16(chunk)` and the `AUDIO` constant
|
|
(`sampleRate: 44100`, `frameSamples: 882`, `frameBytes: 1764`).
|
|
|
|
## Common pitfalls
|
|
|
|
- **Microphone denied** → the page is not a *secure context*: use `https://` or
|
|
`localhost`.
|
|
- **`Failed to fetch` on REST from another origin** → set `CORS_ALLOW_ORIGIN` on
|
|
the engine.
|
|
- **No playback** → the `AudioContext` starts suspended until a user gesture:
|
|
call `start()` / `player.resume()` from a click.
|
|
- **Node: `no WebSocket implementation`** → Node < 22: pass the `ws` `WebSocket`.
|