audio-agent-client — JavaScript SDK

Client SDK for the audio-agent-engine engine. A single ESM file, zero dependencies, no build step: copy audio-agent-client.js wherever you want (another server, a CDN, your own project) and import it. The engine also serves it at GET /sdk/audio-agent-client.js.

Files: audio-agent-client.js · audio-agent-client.d.ts · demo.html.

Demo

demo.html is a complete client built on the SDK: connect/disconnect, session state, streaming agent replies with word-level karaoke highlighting, chat bubbles, inline chips for cue / tool_call / audio_source, a barge-in counter, a microphone toggle with a level meter, a text-turn input, an external audio source (play/stop/volume), a System prompt panel (shows the server-assigned session id and lets you reload / override / clear the session's LLM prompt over REST), automatic phone-number authentication (read from ?phoneNumber=… in the page URL at session start, via the engine's /api/auth/phone-login proxy; when a live session id is present the engine also stores a sanitized user profile as hidden LLM context), a default runtime tool set installed via POST /api/tools and a raw event log. A tool definition may carry an extra audio: true flag (the backend ignores it): when the model calls such a tool, the demo rebuilds the call's URL, puts it in the external audio source field and starts playback after a fixed post-assistant_end delay, so media results (e.g. get_today_podcast_mp3) do not overlap the agent's spoken reply while still autoplaying reliably. When the model emits a <chat_markdown>...</chat_markdown> block, the demo renders that document as formatted Markdown in chat while keeping it out of TTS. The engine serves it at GET /sdk/demo.html; you can also copy it elsewhere (it imports ./audio-agent-client.js relatively). Opened from file:// it works but without a microphone: it warns via a banner (a secure context is required).

Why

The client is decoupled from the engine: the engine stays put, the UI can live on another origin/server. The SDK wraps the protocol (WebSocket + REST) and the PCM audio so you don't reimplement framing, sampling and events.

Install

Nothing to install. Copy the file and import it, or load it from the engine:

import { AudioSession, AudioAgentRest } from 'http://localhost:8787/sdk/audio-agent-client.js';

If the client is on another origin, enable CORS on the engine:

# engine .env
CORS_ALLOW_ORIGIN=https://your-app.example      # or * for any origin

The WebSocket (/ws) needs no CORS (the handshake is not a CORS request). For the microphone the browser requires a secure context: https:// or localhost.

Quick start (browser)

import { AudioSession } from './audio-agent-client.js';

const session = new AudioSession({ baseUrl: 'https://engine.example.com:8787' });

session.on('user', (e) => console.log('user:', e.text));
session.on('assistant_delta', (e) => (bubble.textContent += e.text));
session.on('error', (e) => console.error(e));

await session.start();          // connect + microphone + playback
session.sendUserText('hello');  // text turn (LLM → TTS)
session.interrupt();            // barge-in
await session.stop();

API

`new AudioAgentClient(options)`

Option	Default	Description
`url`	—	Full `ws://` / `wss://` URL. Wins over `baseUrl`.
`baseUrl`	—	Engine origin (`http://host:port`); `/ws` is appended.
`path`	`/ws`	Path appended to `baseUrl`.
`autoReconnect`	`false`	Reconnect after an unexpected close.
`reconnectMinDelayMs` / `reconnectMaxDelayMs`	`500` / `8000`	Exponential backoff.
`WebSocket`	global	WebSocket implementation (Node < 22: pass the `ws` one).

Methods: connect(), close(), sendAudio(chunk), sendUserText(text), interrupt(), playAudioSource(url), stopAudioSource(), setAudioSourceVolume(gain), setReference(path, text?), clearReference(), speak(text), sendText(text), sendControl(obj). Properties: state (idle/connecting/open/closed), connected.

sendAudio accepts Int16Array, Float32Array (normalized -1..1), ArrayBuffer, a view, or number[]. The engine expects 16-bit LE mono PCM at 44.1 kHz; sending 882-sample frames (20 ms) keeps the cadence.

Events: open, close, state, reconnecting, plus one event per server type — session, cues, user, assistant_delta, assistant_markdown, assistant_end, cue, barge_in, tool_call, audio_source, error — and:

audio → { samples, buffer, sampleRate } with the decoded PCM frames;
event → any raw JSON message;
error → either a transport error { error, event, fatal } or the server event { type: 'error', text }.

The session event ({ type: 'session', id }) is sent once at connect time and carries the server-assigned session id. It is the key used by the REST prompt endpoints, so capture it to read or override the system prompt for your session.

Every on() returns an unsubscribe function.

`new AudioAgentRest(baseUrl, options?)`

Method	Endpoint
`health()`	`GET /health`
`getTools()`	`GET /api/tools`
`setTools(tools)`	`POST /api/tools`
`getPrompt(sessionId?)`	`GET /api/prompt`
`setPrompt(prompt, sessionId?)`	`POST /api/prompt`
`phoneLogin(phoneNumber, sessionId?)`	`POST /api/auth/phone-login`
`getCustomVoicePhrases()`	`GET /api/customvoice/phrases`
`setCustomVoice(lang, audio, text?)`	`POST /api/customvoice`
`clearCustomVoice()`	`POST /api/customvoice/clear`
`transcribe(audio)`	`POST /api/transcribe`
`openapi()`	`GET /api-docs/openapi.json`

On a non-2xx response the thrown error carries status and body.

Audio helpers (browser only)

MicCapture — microphone → frame events (Int16Array, 882 samples). await start(), stop().
PcmPlayer — jitter-buffered playback (bufferMs, default 150). push(chunk), flush() (barge-in), setGain(), close().
AudioSession — wires client + mic + player and re-emits client events. await start({ microphone = true }), stop(), sendUserText(), interrupt().

PCM utilities

int16ArrayFromBytes(buffer), float32ToInt16(floats), int16ToFloat32(ints), toInt16(chunk) and the AUDIO constant (sampleRate: 44100, frameSamples: 882, frameBytes: 1764).

Common pitfalls

Microphone denied → the page is not a secure context: use https:// or localhost.
Failed to fetch on REST from another origin → set CORS_ALLOW_ORIGIN on the engine.
No playback → the AudioContext starts suspended until a user gesture: call start() / player.resume() from a click.
Node: no WebSocket implementation → Node < 22: pass the ws WebSocket.

6.9 KiB Raw Blame History