client sdk
This commit is contained in:
valenti
+158
@@ -0,0 +1,158 @@
|
||||
# audio-agent-client — JavaScript SDK
|
||||
|
||||
Client SDK for the `audio-agent-engine` engine. A single ESM file, zero
|
||||
dependencies, no build step: copy `audio-agent-client.js` wherever you want
|
||||
(another server, a CDN, your own project) and import it. The engine also serves
|
||||
it at `GET /sdk/audio-agent-client.js`.
|
||||
|
||||
Files: [`audio-agent-client.js`](audio-agent-client.js) ·
|
||||
[`audio-agent-client.d.ts`](audio-agent-client.d.ts) ·
|
||||
[`demo.html`](demo.html).
|
||||
|
||||
---
|
||||
|
||||
## Demo
|
||||
|
||||
[`demo.html`](demo.html) is a complete client built on the SDK:
|
||||
connect/disconnect, session state, streaming agent replies with word-level
|
||||
*karaoke* highlighting, chat bubbles, inline chips for `cue` / `tool_call` /
|
||||
`audio_source`, a barge-in counter, a microphone toggle with a level meter, a
|
||||
text-turn input, an external audio source (play/stop/volume), a **System prompt**
|
||||
panel (shows the server-assigned session id and lets you reload / override /
|
||||
clear the session's LLM prompt over REST), automatic phone-number authentication
|
||||
(read from `?phoneNumber=…` in the page URL at session start, via the engine's
|
||||
`/api/auth/phone-login` proxy; when a live session id is present the engine also
|
||||
stores a sanitized user profile as hidden LLM context), a default runtime tool
|
||||
set installed via `POST /api/tools` and a raw event log. A tool definition may
|
||||
carry an extra `audio: true` flag (the backend ignores it): when the model calls
|
||||
such a tool, the demo rebuilds the call's URL, puts it in the external audio
|
||||
source field and starts playback after a fixed post-`assistant_end` delay, so
|
||||
media results (e.g. `get_today_podcast_mp3`) do not overlap the agent's spoken
|
||||
reply while still autoplaying reliably. When the model emits a
|
||||
`<chat_markdown>...</chat_markdown>` block, the demo renders that document as
|
||||
formatted Markdown in chat while keeping it out of TTS.
|
||||
The engine serves it at `GET /sdk/demo.html`; you can also copy it elsewhere (it
|
||||
imports `./audio-agent-client.js` relatively). Opened from `file://` it works but
|
||||
without a microphone: it warns via a banner (a *secure context* is required).
|
||||
|
||||
## Why
|
||||
|
||||
The client is **decoupled** from the engine: the engine stays put, the UI can
|
||||
live on another origin/server. The SDK wraps the protocol (WebSocket + REST) and
|
||||
the PCM audio so you don't reimplement framing, sampling and events.
|
||||
|
||||
## Install
|
||||
|
||||
Nothing to install. Copy the file and import it, or load it from the engine:
|
||||
|
||||
```js
|
||||
import { AudioSession, AudioAgentRest } from 'http://localhost:8787/sdk/audio-agent-client.js';
|
||||
```
|
||||
|
||||
If the client is on **another origin**, enable CORS on the engine:
|
||||
|
||||
```dotenv
|
||||
# engine .env
|
||||
CORS_ALLOW_ORIGIN=https://your-app.example # or * for any origin
|
||||
```
|
||||
|
||||
The WebSocket (`/ws`) needs **no** CORS (the handshake is not a CORS request).
|
||||
For the microphone the browser requires a *secure context*: `https://` or
|
||||
`localhost`.
|
||||
|
||||
## Quick start (browser)
|
||||
|
||||
```js
|
||||
import { AudioSession } from './audio-agent-client.js';
|
||||
|
||||
const session = new AudioSession({ baseUrl: 'https://engine.example.com:8787' });
|
||||
|
||||
session.on('user', (e) => console.log('user:', e.text));
|
||||
session.on('assistant_delta', (e) => (bubble.textContent += e.text));
|
||||
session.on('error', (e) => console.error(e));
|
||||
|
||||
await session.start(); // connect + microphone + playback
|
||||
session.sendUserText('hello'); // text turn (LLM → TTS)
|
||||
session.interrupt(); // barge-in
|
||||
await session.stop();
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
### `new AudioAgentClient(options)`
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `url` | — | Full `ws://` / `wss://` URL. Wins over `baseUrl`. |
|
||||
| `baseUrl` | — | Engine origin (`http://host:port`); `/ws` is appended. |
|
||||
| `path` | `/ws` | Path appended to `baseUrl`. |
|
||||
| `autoReconnect` | `false` | Reconnect after an unexpected close. |
|
||||
| `reconnectMinDelayMs` / `reconnectMaxDelayMs` | `500` / `8000` | Exponential backoff. |
|
||||
| `WebSocket` | global | WebSocket implementation (Node < 22: pass the `ws` one). |
|
||||
|
||||
Methods: `connect()`, `close()`, `sendAudio(chunk)`, `sendUserText(text)`,
|
||||
`interrupt()`, `playAudioSource(url)`, `stopAudioSource()`,
|
||||
`setAudioSourceVolume(gain)`, `setReference(path, text?)`, `clearReference()`,
|
||||
`speak(text)`, `sendText(text)`, `sendControl(obj)`.
|
||||
Properties: `state` (`idle`/`connecting`/`open`/`closed`), `connected`.
|
||||
|
||||
`sendAudio` accepts `Int16Array`, `Float32Array` (normalized -1..1),
|
||||
`ArrayBuffer`, a view, or `number[]`. The engine expects **16-bit LE mono PCM at
|
||||
44.1 kHz**; sending **882-sample** frames (20 ms) keeps the cadence.
|
||||
|
||||
Events: `open`, `close`, `state`, `reconnecting`, plus one event per server
|
||||
`type` — `session`, `cues`, `user`, `assistant_delta`, `assistant_markdown`,
|
||||
`assistant_end`, `cue`, `barge_in`, `tool_call`, `audio_source`, `error` — and:
|
||||
- `audio` → `{ samples, buffer, sampleRate }` with the decoded PCM frames;
|
||||
- `event` → any raw JSON message;
|
||||
- `error` → either a transport error `{ error, event, fatal }` or the server
|
||||
event `{ type: 'error', text }`.
|
||||
|
||||
The `session` event (`{ type: 'session', id }`) is sent once at connect time and
|
||||
carries the server-assigned session id. It is the key used by the REST prompt
|
||||
endpoints, so capture it to read or override the system prompt for your session.
|
||||
|
||||
Every `on()` returns an unsubscribe function.
|
||||
|
||||
### `new AudioAgentRest(baseUrl, options?)`
|
||||
|
||||
| Method | Endpoint |
|
||||
|---|---|
|
||||
| `health()` | `GET /health` |
|
||||
| `getTools()` | `GET /api/tools` |
|
||||
| `setTools(tools)` | `POST /api/tools` |
|
||||
| `getPrompt(sessionId?)` | `GET /api/prompt` |
|
||||
| `setPrompt(prompt, sessionId?)` | `POST /api/prompt` |
|
||||
| `phoneLogin(phoneNumber, sessionId?)` | `POST /api/auth/phone-login` |
|
||||
| `getCustomVoicePhrases()` | `GET /api/customvoice/phrases` |
|
||||
| `setCustomVoice(lang, audio, text?)` | `POST /api/customvoice` |
|
||||
| `clearCustomVoice()` | `POST /api/customvoice/clear` |
|
||||
| `transcribe(audio)` | `POST /api/transcribe` |
|
||||
| `openapi()` | `GET /api-docs/openapi.json` |
|
||||
|
||||
On a non-2xx response the thrown error carries `status` and `body`.
|
||||
|
||||
### Audio helpers (browser only)
|
||||
|
||||
- **`MicCapture`** — microphone → `frame` events (`Int16Array`, 882 samples).
|
||||
`await start()`, `stop()`.
|
||||
- **`PcmPlayer`** — jitter-buffered playback (`bufferMs`, default 150).
|
||||
`push(chunk)`, `flush()` (barge-in), `setGain()`, `close()`.
|
||||
- **`AudioSession`** — wires client + mic + player and re-emits client events.
|
||||
`await start({ microphone = true })`, `stop()`, `sendUserText()`, `interrupt()`.
|
||||
|
||||
### PCM utilities
|
||||
|
||||
`int16ArrayFromBytes(buffer)`, `float32ToInt16(floats)`,
|
||||
`int16ToFloat32(ints)`, `toInt16(chunk)` and the `AUDIO` constant
|
||||
(`sampleRate: 44100`, `frameSamples: 882`, `frameBytes: 1764`).
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
- **Microphone denied** → the page is not a *secure context*: use `https://` or
|
||||
`localhost`.
|
||||
- **`Failed to fetch` on REST from another origin** → set `CORS_ALLOW_ORIGIN` on
|
||||
the engine.
|
||||
- **No playback** → the `AudioContext` starts suspended until a user gesture:
|
||||
call `start()` / `player.resume()` from a click.
|
||||
- **Node: `no WebSocket implementation`** → Node < 22: pass the `ws` `WebSocket`.
|
||||
Vendored
+336
@@ -0,0 +1,336 @@
|
||||
/**
|
||||
* Type declarations for `audio-agent-client.js`.
|
||||
*
|
||||
* The SDK is plain ESM with no build step; this file exists so TypeScript
|
||||
* consumers get autocompletion and type checking. Keep it in sync with the JS.
|
||||
*/
|
||||
|
||||
export interface AudioFormat {
|
||||
readonly sampleRate: 44100;
|
||||
readonly frameSamples: 882;
|
||||
readonly frameBytes: 1764;
|
||||
readonly sttSampleRate: 16000;
|
||||
}
|
||||
|
||||
export const AUDIO: AudioFormat;
|
||||
|
||||
export const CLIENT_EVENTS: readonly string[];
|
||||
|
||||
/** Union accepted by every audio-sending helper. */
|
||||
export type AudioChunk =
|
||||
| Int16Array
|
||||
| Float32Array
|
||||
| ArrayBuffer
|
||||
| ArrayBufferView
|
||||
| number[];
|
||||
|
||||
export function int16ArrayFromBytes(buffer: ArrayBuffer | ArrayBufferView): Int16Array;
|
||||
export function float32ToInt16(floats: Float32Array | number[]): Int16Array;
|
||||
export function int16ToFloat32(ints: Int16Array): Float32Array;
|
||||
export function toInt16(chunk: AudioChunk): Int16Array;
|
||||
|
||||
/** Any object with a `type` discriminator, as emitted by the engine. */
|
||||
export interface ServerEvent {
|
||||
type: string;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/** Server-assigned session id, sent once at connect time. */
|
||||
export interface SessionEvent extends ServerEvent {
|
||||
type: 'session';
|
||||
id: string;
|
||||
}
|
||||
export interface CuesEvent extends ServerEvent {
|
||||
type: 'cues';
|
||||
lang: string;
|
||||
config: Record<string, unknown>;
|
||||
}
|
||||
export interface UserEvent extends ServerEvent {
|
||||
type: 'user';
|
||||
text: string;
|
||||
}
|
||||
export interface AssistantDeltaEvent extends ServerEvent {
|
||||
type: 'assistant_delta';
|
||||
text: string;
|
||||
}
|
||||
export interface AssistantMarkdownEvent extends ServerEvent {
|
||||
type: 'assistant_markdown';
|
||||
text: string;
|
||||
}
|
||||
export interface AssistantEndEvent extends ServerEvent {
|
||||
type: 'assistant_end';
|
||||
}
|
||||
export interface CueEvent extends ServerEvent {
|
||||
type: 'cue';
|
||||
kind: 'filler' | 'start_filler' | 'backchannel';
|
||||
name: string;
|
||||
category?: string;
|
||||
pause_ms?: number;
|
||||
}
|
||||
export interface BargeInEvent extends ServerEvent {
|
||||
type: 'barge_in';
|
||||
source: string;
|
||||
prob?: number;
|
||||
}
|
||||
export interface ToolCallEvent extends ServerEvent {
|
||||
type: 'tool_call';
|
||||
name: string;
|
||||
arguments: unknown;
|
||||
}
|
||||
export interface AudioSourceEvent extends ServerEvent {
|
||||
type: 'audio_source';
|
||||
state: 'loading' | 'playing' | 'stopped' | 'error';
|
||||
url: string;
|
||||
text?: string;
|
||||
}
|
||||
export interface ErrorEvent extends ServerEvent {
|
||||
type: 'error';
|
||||
text: string;
|
||||
}
|
||||
|
||||
/** Payload of the local `audio` event: one decoded PCM frame. */
|
||||
export interface AudioFrame {
|
||||
samples: Int16Array;
|
||||
buffer: ArrayBuffer;
|
||||
sampleRate: number;
|
||||
}
|
||||
|
||||
/** Payload of the local `close` event. */
|
||||
export interface CloseEvent {
|
||||
code: number;
|
||||
reason: string;
|
||||
wasClean: boolean;
|
||||
}
|
||||
|
||||
/** Payload for connection-level failures (as opposed to a server `error` event). */
|
||||
export interface TransportError {
|
||||
error: Error;
|
||||
event?: unknown;
|
||||
fatal: boolean;
|
||||
}
|
||||
|
||||
export interface AudioAgentClientOptions {
|
||||
/** Full `ws://`/`wss://` URL; wins over `baseUrl`. */
|
||||
url?: string;
|
||||
/** Engine origin, e.g. `http://localhost:8787`; `/ws` is appended. */
|
||||
baseUrl?: string;
|
||||
/** Path appended to `baseUrl` (default `/ws`). */
|
||||
path?: string;
|
||||
/** Reconnect automatically after an unexpected close (default `false`). */
|
||||
autoReconnect?: boolean;
|
||||
reconnectMinDelayMs?: number;
|
||||
reconnectMaxDelayMs?: number;
|
||||
/** WebSocket implementation to use (required on Node < 22, e.g. the `ws` package). */
|
||||
WebSocket?: typeof WebSocket;
|
||||
}
|
||||
|
||||
export type ConnectionState = 'idle' | 'connecting' | 'open' | 'closed';
|
||||
|
||||
export class AudioAgentClient {
|
||||
constructor(options?: AudioAgentClientOptions);
|
||||
readonly url: string;
|
||||
autoReconnect: boolean;
|
||||
reconnectMinDelayMs: number;
|
||||
reconnectMaxDelayMs: number;
|
||||
readonly state: ConnectionState;
|
||||
readonly connected: boolean;
|
||||
|
||||
connect(): Promise<this>;
|
||||
close(code?: number, reason?: string): void;
|
||||
|
||||
sendAudio(chunk: AudioChunk): boolean;
|
||||
sendText(text: string): boolean;
|
||||
sendControl(message: Record<string, unknown>): boolean;
|
||||
sendUserText(text: string): boolean;
|
||||
interrupt(): boolean;
|
||||
playAudioSource(url: string): boolean;
|
||||
stopAudioSource(): boolean;
|
||||
setAudioSourceVolume(gain: number): boolean;
|
||||
setReference(path: string, text?: string): boolean;
|
||||
clearReference(): boolean;
|
||||
speak(text: string): boolean;
|
||||
|
||||
on(type: 'audio', handler: (frame: AudioFrame) => void): () => void;
|
||||
on(type: 'open', handler: (info: { url: string }) => void): () => void;
|
||||
on(type: 'close', handler: (info: CloseEvent) => void): () => void;
|
||||
on(type: 'state', handler: (state: ConnectionState) => void): () => void;
|
||||
on(
|
||||
type: 'reconnecting',
|
||||
handler: (info: { attempt: number; delay: number }) => void,
|
||||
): () => void;
|
||||
on(type: 'error', handler: (payload: TransportError | ErrorEvent) => void): () => void;
|
||||
on(type: 'event' | 'unknown', handler: (event: ServerEvent) => void): () => void;
|
||||
on(type: string, handler: (payload: any) => void): () => void;
|
||||
|
||||
once(type: string, handler: (payload: any) => void): () => void;
|
||||
off(type: string, handler: (payload: any) => void): void;
|
||||
emit(type: string, payload?: unknown): void;
|
||||
}
|
||||
|
||||
export interface AudioAgentRestOptions {
|
||||
fetch?: typeof fetch;
|
||||
headers?: Record<string, string>;
|
||||
}
|
||||
|
||||
export interface ToolsResponse {
|
||||
ok: boolean;
|
||||
count: number;
|
||||
source: 'file' | 'request' | 'none';
|
||||
tools: object[];
|
||||
}
|
||||
|
||||
export interface PhrasesResponse {
|
||||
default: string;
|
||||
languages: Array<{ code: string; label: string; phrase: string }>;
|
||||
}
|
||||
|
||||
export interface SetVoiceResponse {
|
||||
ok: boolean;
|
||||
lang: string;
|
||||
audio: string;
|
||||
phrase: string;
|
||||
}
|
||||
|
||||
export interface TranscribeResponse {
|
||||
ok: boolean;
|
||||
text: string;
|
||||
duration_ms: number;
|
||||
tokens: number;
|
||||
}
|
||||
|
||||
/** Where the effective system prompt comes from (decreasing precedence). */
|
||||
export type PromptSource = 'session' | 'global' | 'env' | 'default' | 'none';
|
||||
|
||||
/** Response of `GET`/`POST /api/prompt`. */
|
||||
export interface PromptResponse {
|
||||
ok: boolean;
|
||||
session_id: string | null;
|
||||
source: PromptSource;
|
||||
prompt: string;
|
||||
has_override: boolean;
|
||||
direct_answers_applied: boolean;
|
||||
}
|
||||
|
||||
/** Response of `POST /api/auth/phone-login` (upstream AquaChile payload). */
|
||||
export interface PhoneLoginResponse {
|
||||
sessionToken?: string;
|
||||
user?: Record<string, any>;
|
||||
[key: string]: any;
|
||||
}
|
||||
|
||||
/** Error thrown by {@link AudioAgentRest}, carrying the HTTP status and body. */
|
||||
export interface RestError extends Error {
|
||||
status: number;
|
||||
body: unknown;
|
||||
}
|
||||
|
||||
export class AudioAgentRest {
|
||||
constructor(baseUrl: string, options?: AudioAgentRestOptions);
|
||||
readonly baseUrl: string;
|
||||
|
||||
health(): Promise<string>;
|
||||
getTools(): Promise<ToolsResponse>;
|
||||
setTools(tools: object[]): Promise<ToolsResponse>;
|
||||
getPrompt(sessionId?: string): Promise<PromptResponse>;
|
||||
setPrompt(prompt: string | null, sessionId?: string | null): Promise<PromptResponse>;
|
||||
phoneLogin(phoneNumber: string, sessionId?: string | null): Promise<PhoneLoginResponse>;
|
||||
getCustomVoicePhrases(): Promise<PhrasesResponse>;
|
||||
setCustomVoice(
|
||||
lang: string,
|
||||
audio: Blob | ArrayBuffer | ArrayBufferView,
|
||||
text?: string,
|
||||
): Promise<SetVoiceResponse>;
|
||||
clearCustomVoice(): Promise<{ ok: boolean }>;
|
||||
transcribe(audio: Blob | ArrayBuffer | ArrayBufferView): Promise<TranscribeResponse>;
|
||||
openapi(): Promise<object>;
|
||||
}
|
||||
|
||||
export interface MicCaptureOptions {
|
||||
sampleRate?: number;
|
||||
frameSamples?: number;
|
||||
constraints?: MediaStreamConstraints;
|
||||
}
|
||||
|
||||
export class MicCapture {
|
||||
constructor(options?: MicCaptureOptions);
|
||||
readonly sampleRate: number;
|
||||
readonly frameSamples: number;
|
||||
readonly contextSampleRate: number | null;
|
||||
start(): Promise<void>;
|
||||
stop(): void;
|
||||
on(type: 'frame', handler: (samples: Int16Array) => void): () => void;
|
||||
on(type: 'start', handler: (info: { sampleRate: number }) => void): () => void;
|
||||
on(type: 'stop', handler: () => void): () => void;
|
||||
on(type: string, handler: (payload: any) => void): () => void;
|
||||
once(type: string, handler: (payload: any) => void): () => void;
|
||||
off(type: string, handler: (payload: any) => void): void;
|
||||
}
|
||||
|
||||
export interface PcmPlayerOptions {
|
||||
sampleRate?: number;
|
||||
bufferMs?: number;
|
||||
gain?: number;
|
||||
}
|
||||
|
||||
export class PcmPlayer {
|
||||
constructor(options?: PcmPlayerOptions);
|
||||
readonly sampleRate: number;
|
||||
bufferMs: number;
|
||||
gain: number;
|
||||
resume(): Promise<void>;
|
||||
push(chunk: AudioChunk): void;
|
||||
setGain(value: number): void;
|
||||
flush(): void;
|
||||
close(): void;
|
||||
isIdle(): boolean;
|
||||
on(type: 'chunk', handler: (info: { startAt: number; samples: number }) => void): () => void;
|
||||
on(type: 'flush', handler: () => void): () => void;
|
||||
on(type: 'idle', handler: () => void): () => void;
|
||||
on(type: string, handler: (payload: any) => void): () => void;
|
||||
once(type: string, handler: (payload: any) => void): () => void;
|
||||
off(type: string, handler: (payload: any) => void): void;
|
||||
}
|
||||
|
||||
export interface AudioSessionOptions extends AudioAgentClientOptions {
|
||||
client?: AudioAgentClient;
|
||||
player?: PcmPlayer;
|
||||
/** Playback jitter buffer in ms (default 150). */
|
||||
bufferMs?: number;
|
||||
/** Stream the microphone automatically (default `true`). */
|
||||
forwardMic?: boolean;
|
||||
}
|
||||
|
||||
export class AudioSession {
|
||||
constructor(options?: AudioSessionOptions);
|
||||
readonly client: AudioAgentClient;
|
||||
readonly player: PcmPlayer;
|
||||
readonly mic: MicCapture | null;
|
||||
forwardMic: boolean;
|
||||
|
||||
start(options?: { microphone?: boolean }): Promise<void>;
|
||||
stop(): Promise<void>;
|
||||
sendUserText(text: string): boolean;
|
||||
interrupt(): boolean;
|
||||
|
||||
on(type: 'audio' | 'session' | 'assistant_delta' | 'assistant_markdown' | 'cues' | 'user' | 'cue' | 'barge_in'
|
||||
| 'tool_call' | 'audio_source' | 'error' | 'assistant_end' | 'open' | 'close'
|
||||
| 'state' | 'reconnecting' | string, handler: (payload: any) => void): () => void;
|
||||
once(type: string, handler: (payload: any) => void): () => void;
|
||||
off(type: string, handler: (payload: any) => void): void;
|
||||
emit(type: string, payload?: unknown): void;
|
||||
}
|
||||
|
||||
declare const _default: {
|
||||
AUDIO: AudioFormat;
|
||||
CLIENT_EVENTS: readonly string[];
|
||||
int16ArrayFromBytes: typeof int16ArrayFromBytes;
|
||||
float32ToInt16: typeof float32ToInt16;
|
||||
int16ToFloat32: typeof int16ToFloat32;
|
||||
toInt16: typeof toInt16;
|
||||
AudioAgentClient: typeof AudioAgentClient;
|
||||
AudioAgentRest: typeof AudioAgentRest;
|
||||
MicCapture: typeof MicCapture;
|
||||
PcmPlayer: typeof PcmPlayer;
|
||||
AudioSession: typeof AudioSession;
|
||||
};
|
||||
export default _default;
|
||||
File diff suppressed because it is too large
Load Diff
+976
@@ -0,0 +1,976 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>audio-agent — SDK demo</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0b0f1a;
|
||||
--panel: #141b2d;
|
||||
--accent: #4f8cff;
|
||||
--accent-dim: #2a4a8a;
|
||||
--ok: #3ddc97;
|
||||
--warn: #ffb454;
|
||||
--err: #ff5470;
|
||||
--text: #e6edf7;
|
||||
--muted: #8a94a6;
|
||||
--border: #243049;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { height: 100%; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
padding: 18px;
|
||||
}
|
||||
.app {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
width: 100%;
|
||||
max-width: 900px;
|
||||
height: calc(100vh - 36px);
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 14px;
|
||||
overflow: hidden;
|
||||
}
|
||||
header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
header h1 { font-size: 15px; margin: 0; white-space: nowrap; }
|
||||
header h1 span { color: var(--muted); font-weight: 400; }
|
||||
.status { display: inline-flex; align-items: center; gap: 7px; font-size: 12px; color: var(--muted); }
|
||||
.dot { width: 9px; height: 9px; border-radius: 50%; background: var(--muted); flex: none; }
|
||||
.dot.on { background: var(--ok); box-shadow: 0 0 8px var(--ok); }
|
||||
.dot.err { background: var(--err); box-shadow: 0 0 8px var(--err); }
|
||||
.dot.warn { background: var(--warn); box-shadow: 0 0 8px var(--warn); }
|
||||
/* Authenticated user name, shown in the header after phone-login. */
|
||||
#userStatus {
|
||||
display: none;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
color: var(--text);
|
||||
background: var(--accent-dim);
|
||||
border: 1px solid var(--accent);
|
||||
border-radius: 999px;
|
||||
padding: 3px 11px;
|
||||
max-width: 220px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
#userStatus.show { display: inline-flex; }
|
||||
#userStatus::before { content: "\1F464"; font-size: 12px; line-height: 1; }
|
||||
#userStatus.bad { background: transparent; border-color: var(--err); color: var(--err); }
|
||||
#userStatus.bad::before { content: "\26A0"; }
|
||||
input[type="text"], input[type="url"], input[type="number"], textarea {
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
color: var(--text);
|
||||
padding: 8px 10px;
|
||||
font-size: 13px;
|
||||
font-family: inherit;
|
||||
outline: none;
|
||||
min-width: 0;
|
||||
}
|
||||
input:focus, textarea:focus { border-color: var(--accent); }
|
||||
textarea { width: 100%; resize: vertical; min-height: 68px; line-height: 1.4; }
|
||||
#promptPanel { padding: 0 16px 12px; display: flex; flex-direction: column; gap: 8px; }
|
||||
#promptPanel .meta { font-size: 11.5px; color: var(--muted); }
|
||||
#promptPanel .meta b { color: var(--text); font-weight: 600; }
|
||||
button {
|
||||
background: var(--accent-dim);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--accent);
|
||||
border-radius: 8px;
|
||||
padding: 8px 13px;
|
||||
font-size: 13px;
|
||||
font-family: inherit;
|
||||
cursor: pointer;
|
||||
transition: all .15s;
|
||||
white-space: nowrap;
|
||||
}
|
||||
button:hover:not(:disabled) { background: var(--accent); }
|
||||
button:disabled { opacity: .4; cursor: not-allowed; }
|
||||
button.ghost { background: transparent; border-color: var(--border); }
|
||||
button.ghost:hover:not(:disabled) { background: var(--border); }
|
||||
button.danger { border-color: var(--err); background: rgba(255,84,112,.15); }
|
||||
button.danger:hover:not(:disabled) { background: var(--err); }
|
||||
button.active { background: var(--ok); border-color: var(--ok); color: #06170f; }
|
||||
|
||||
.banner {
|
||||
display: none;
|
||||
margin: 10px 16px 0;
|
||||
padding: 9px 12px;
|
||||
font-size: 12.5px;
|
||||
line-height: 1.4;
|
||||
border: 1px solid var(--warn);
|
||||
border-radius: 8px;
|
||||
background: rgba(255,180,84,.08);
|
||||
color: var(--warn);
|
||||
}
|
||||
.banner.show { display: block; }
|
||||
|
||||
#chat {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 16px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
}
|
||||
.hint { color: var(--muted); font-size: 13px; text-align: center; margin: auto; }
|
||||
.msg { display: flex; flex-direction: column; gap: 4px; max-width: 82%; }
|
||||
.msg .who { font-size: 10.5px; text-transform: uppercase; letter-spacing: .5px; color: var(--muted); }
|
||||
.msg .bubble {
|
||||
padding: 9px 12px;
|
||||
border-radius: 12px;
|
||||
font-size: 14px;
|
||||
line-height: 1.45;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
.msg.user { align-self: flex-end; align-items: flex-end; }
|
||||
.msg.user .bubble { background: var(--accent-dim); border: 1px solid var(--accent); }
|
||||
.msg.agent { align-self: flex-start; }
|
||||
.msg.agent .bubble { background: var(--bg); border: 1px solid var(--border); }
|
||||
.msg.error .bubble { border-color: var(--err); color: var(--err); }
|
||||
.msg .kw { color: var(--muted); transition: color .18s ease; }
|
||||
.msg .kw.done { color: var(--text); }
|
||||
.msg .kw.now { color: var(--accent); text-shadow: 0 0 8px rgba(79,140,255,.55); }
|
||||
.msg .bubble.markdown { white-space: normal; overflow-x: auto; }
|
||||
.msg .bubble.markdown > :first-child { margin-top: 0; }
|
||||
.msg .bubble.markdown > :last-child { margin-bottom: 0; }
|
||||
.msg .bubble.markdown h1,
|
||||
.msg .bubble.markdown h2,
|
||||
.msg .bubble.markdown h3,
|
||||
.msg .bubble.markdown h4 { margin: 0 0 10px; line-height: 1.25; }
|
||||
.msg .bubble.markdown p,
|
||||
.msg .bubble.markdown ul,
|
||||
.msg .bubble.markdown ol,
|
||||
.msg .bubble.markdown blockquote,
|
||||
.msg .bubble.markdown pre,
|
||||
.msg .bubble.markdown table { margin: 0 0 10px; }
|
||||
.msg .bubble.markdown ul,
|
||||
.msg .bubble.markdown ol { padding-left: 20px; }
|
||||
.msg .bubble.markdown li + li { margin-top: 4px; }
|
||||
.msg .bubble.markdown code {
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
font-size: 12.5px;
|
||||
background: rgba(79,140,255,.10);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
padding: 1px 5px;
|
||||
}
|
||||
.msg .bubble.markdown pre {
|
||||
background: #0a1222;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
padding: 10px 12px;
|
||||
overflow-x: auto;
|
||||
}
|
||||
.msg .bubble.markdown pre code {
|
||||
background: transparent;
|
||||
border: 0;
|
||||
padding: 0;
|
||||
font-size: 12px;
|
||||
}
|
||||
.msg .bubble.markdown blockquote {
|
||||
border-left: 3px solid var(--accent);
|
||||
padding-left: 10px;
|
||||
color: var(--muted);
|
||||
}
|
||||
.msg .bubble.markdown table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 13px;
|
||||
}
|
||||
.msg .bubble.markdown th,
|
||||
.msg .bubble.markdown td {
|
||||
border: 1px solid var(--border);
|
||||
padding: 6px 8px;
|
||||
text-align: left;
|
||||
vertical-align: top;
|
||||
}
|
||||
.msg .bubble.markdown th { color: var(--text); background: rgba(79,140,255,.10); }
|
||||
.msg .bubble.markdown a { color: #8db6ff; }
|
||||
|
||||
/* Inline chips for cue / tool_call / audio_source events. */
|
||||
.chips { display: flex; flex-wrap: wrap; gap: 6px; align-self: flex-start; }
|
||||
.chip {
|
||||
font-size: 11.5px;
|
||||
color: var(--muted);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 999px;
|
||||
padding: 3px 10px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.chip b { color: var(--text); font-weight: 600; }
|
||||
.chip.cue { border-color: var(--accent); color: var(--accent); }
|
||||
.chip.tool { border-color: var(--ok); color: var(--ok); }
|
||||
.chip.flash { animation: flash .5s ease; }
|
||||
@keyframes flash { from { background: rgba(255,84,112,.35); } to { background: transparent; } }
|
||||
|
||||
footer { border-top: 1px solid var(--border); padding: 10px 16px; display: flex; flex-direction: column; gap: 9px; }
|
||||
.row { display: flex; align-items: center; gap: 8px; }
|
||||
.row.grow input[type="text"] { flex: 1; }
|
||||
.meters { display: flex; align-items: center; gap: 12px; font-size: 12px; color: var(--muted); flex-wrap: wrap; }
|
||||
.bargein { display: inline-flex; align-items: center; gap: 6px; border: 1px solid var(--border); border-radius: 999px; padding: 4px 10px; transition: border-color .2s, color .2s, box-shadow .2s; }
|
||||
.bargein.active { border-color: var(--err); color: var(--err); box-shadow: 0 0 12px -2px var(--err); }
|
||||
.bargein b { color: var(--text); font-variant-numeric: tabular-nums; }
|
||||
.mic { display: inline-flex; align-items: center; gap: 6px; border: 1px solid var(--border); border-radius: 999px; padding: 4px 10px; }
|
||||
.mic .bar { display: inline-flex; gap: 2px; align-items: flex-end; height: 12px; }
|
||||
.mic .bar i { width: 3px; background: var(--muted); border-radius: 1px; height: 3px; transition: height .08s; }
|
||||
.mic.on .bar i { background: var(--ok); }
|
||||
|
||||
details { border-top: 1px solid var(--border); }
|
||||
details summary { cursor: pointer; padding: 8px 16px; font-size: 12px; color: var(--muted); user-select: none; }
|
||||
#log { max-height: 190px; overflow-y: auto; padding: 0 16px 12px; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 11.5px; color: var(--muted); }
|
||||
#log div { padding: 1px 0; white-space: pre-wrap; word-break: break-all; }
|
||||
#log .l-user { color: var(--text); }
|
||||
#log .l-cue { color: var(--accent); }
|
||||
#log .l-tool { color: var(--ok); }
|
||||
#log .l-err { color: var(--err); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
<header>
|
||||
<h1>audio-agent <span>· SDK demo</span></h1>
|
||||
<span class="status"><i class="dot" id="dot"></i><span id="state">disconnected</span></span>
|
||||
<span class="status" id="userStatus"></span>
|
||||
<span style="flex:1"></span>
|
||||
<input type="text" id="baseUrl" style="width: 230px" spellcheck="false" />
|
||||
<button id="connectBtn">Connect</button>
|
||||
</header>
|
||||
|
||||
<div class="banner" id="insecure">
|
||||
The microphone requires a <b>secure context</b>: open this page over <code>https://</code> or
|
||||
<code>localhost</code>, not from <code>file://</code>.
|
||||
</div>
|
||||
|
||||
<div id="chat"><div class="hint" id="hint">Connect to start talking to the agent.</div></div>
|
||||
|
||||
<footer>
|
||||
<div class="row grow">
|
||||
<input type="text" id="textInput" placeholder="Type a message (text turn)…" autocomplete="off" />
|
||||
<button id="sendBtn" disabled>Send</button>
|
||||
</div>
|
||||
|
||||
<div class="row grow">
|
||||
<input type="url" id="sourceUrl" placeholder="External audio source URL (radio / stream)…" style="flex:1" spellcheck="false" />
|
||||
<button id="sourcePlay" class="ghost" disabled>Play</button>
|
||||
<button id="sourceStop" class="ghost" disabled>Stop</button>
|
||||
<input type="range" id="sourceVol" min="0" max="100" value="70" title="External source volume" disabled style="width:90px" />
|
||||
</div>
|
||||
|
||||
<div class="meters">
|
||||
<button id="micBtn" disabled>Microphone: OFF</button>
|
||||
<span class="mic" id="micMeter"><span>mic</span><span class="bar"><i></i><i></i><i></i><i></i><i></i></span></span>
|
||||
<button id="interruptBtn" class="danger" disabled>Interrupt</button>
|
||||
<span class="bargein" id="bargein"><span>barge-in</span><b id="bargeCount">0</b></span>
|
||||
<span style="flex:1"></span>
|
||||
<label title="Local playback buffer" style="display:inline-flex;align-items:center;gap:6px">
|
||||
buffer <input type="number" id="bufferMs" value="150" min="20" max="1000" step="10" style="width:70px" /> ms
|
||||
</label>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
<details id="promptDetails">
|
||||
<summary>System prompt (per session)</summary>
|
||||
<div id="promptPanel">
|
||||
<div class="meta">
|
||||
session <b id="sessionId">—</b> · source <b id="promptSource">—</b> ·
|
||||
override <b id="promptOverride">—</b>
|
||||
</div>
|
||||
<textarea id="promptText" spellcheck="false"
|
||||
placeholder="Connect to load the session's system prompt…" disabled></textarea>
|
||||
<div class="row">
|
||||
<button id="promptLoad" class="ghost" disabled>Reload</button>
|
||||
<button id="promptApply" disabled>Apply override</button>
|
||||
<button id="promptClear" class="ghost" disabled>Clear override</button>
|
||||
<span style="flex:1"></span>
|
||||
</div>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Event log (raw SDK events)</summary>
|
||||
<div id="log"></div>
|
||||
</details>
|
||||
</div>
|
||||
|
||||
<script type="module">
|
||||
import { AudioSession, AudioAgentRest, MicCapture, AUDIO } from './audio-agent-client.js';
|
||||
|
||||
/* System prompt applied automatically to every new session (per-session
|
||||
override via POST /api/prompt). Edit it freely, or clear the override from the
|
||||
"System prompt" panel below. */
|
||||
const SESSION_PROMPT = `You are Vocero, a friendly, proactive AI voice assistant for AquaChile, helping users understand operational data, KPIs, reports and alerts.
|
||||
Always speak exclusively in Spanish, regardless of the user's language. Use natural, professional Chilean Spanish.
|
||||
Keep spoken answers short, usually one to three sentences, without markdown, lists or unnecessary details.
|
||||
Prioritize relevant information, interpret verified data and suggest actions when appropriate. Never invent facts or figures.
|
||||
Maintain conversation context, especially when initiated from an alert or report. Provide deeper explanations only when requested.
|
||||
Use the available tools immediately on the first relevant user request instead of replying generically.
|
||||
If the user asks for today's podcast, the daily podcast, the MP3, the audio file, or asks to play or listen to the podcast, call get_today_podcast_mp3 right away.
|
||||
If the user asks for today's report, the day report, the daily briefing, alerts, or operational highlights for a date, call get_public_day_report right away.
|
||||
When a tool can provide the requested data or media, call it before answering. Do not ask the user to repeat the request just to trigger a tool.
|
||||
If the user explicitly asks for a Markdown document, a report in Markdown, or a long structured written document, first give a very short spoken introduction outside the block, then put the full Markdown document inside <chat_markdown>...</chat_markdown>.
|
||||
Only the Markdown document itself may appear inside <chat_markdown>...</chat_markdown>. Do not repeat the full document outside that block.
|
||||
Answer directly without revealing reasoning or internal processes.`;
|
||||
|
||||
/* Runtime tools installed by the demo through `POST /api/tools`. The backend
|
||||
executes them server-side; the browser only installs the definitions and
|
||||
observes `tool_call` events over WebSocket. */
|
||||
const DEMO_TOOLS = [
|
||||
{
|
||||
name: 'get_public_day_report',
|
||||
description: 'Use this immediately when the user asks for a day report, daily briefing, alerts, operational highlights, or a report for a specific date. Returns the AquaChile public day briefing ordered by severity and recency.',
|
||||
parameters: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
limit: {
|
||||
type: 'integer',
|
||||
description: 'Maximum number of insights to return after severity and recency ordering.',
|
||||
minimum: 1,
|
||||
maximum: 10,
|
||||
default: 3,
|
||||
},
|
||||
date: {
|
||||
type: 'string',
|
||||
description: 'Exact report date in YYYY-MM-DD format. Omit it to use today or the latest earlier processed report date.',
|
||||
},
|
||||
},
|
||||
},
|
||||
url: 'https://aquachile.azurianlab.com/api/v1/day-report',
|
||||
method: 'GET',
|
||||
headers: {
|
||||
accept: '*/*',
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'get_today_podcast_mp3',
|
||||
description: 'Use this immediately when the user asks to listen to, play, open, or get today\'s AquaChile podcast or its MP3/audio file. Returns the MP3 file of the latest published daily AquaChile podcast episode.',
|
||||
parameters: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
download: {
|
||||
type: 'boolean',
|
||||
description: 'When true (default) the MP3 is returned as a download attachment. When false it is served inline for playback.',
|
||||
default: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
url: 'https://aquachile.azurianlab.com/api/v1/podcast/today/mp3',
|
||||
method: 'GET',
|
||||
headers: {
|
||||
accept: '*/*',
|
||||
},
|
||||
// `audio: true` marks a tool whose result is a media file. The backend
|
||||
// ignores this extra field; the demo uses it to route the call's URL to the
|
||||
// engine's external audio source and start playback.
|
||||
audio: true,
|
||||
},
|
||||
];
|
||||
|
||||
/* Verbs whose arguments travel as query parameters (same rule the backend
|
||||
applies), so the demo can rebuild the exact URL a tool call used. */
|
||||
const QUERY_VERBS = new Set(['GET', 'HEAD', 'OPTIONS', 'TRACE', 'DELETE']);
|
||||
|
||||
/** Effective URL of a tool call, mirroring the backend's argument placement. */
|
||||
function toolUrl(def, args = {}) {
|
||||
const url = new URL(def.url);
|
||||
if (QUERY_VERBS.has((def.method || 'POST').toUpperCase())) {
|
||||
for (const [k, v] of Object.entries(args || {})) {
|
||||
if (v !== undefined && v !== null) url.searchParams.set(k, String(v));
|
||||
}
|
||||
}
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
/* Phone-login: the page reads `phoneNumber` from its URL at session start and
|
||||
authenticates through the engine's server-side proxy (`POST
|
||||
/api/auth/phone-login`), which forwards to the AquaChile auth API. The proxy
|
||||
exists because that API blocks browser (CORS) calls from this origin. */
|
||||
|
||||
/* ------------------------------------------------------------------ helpers */
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const chat = $('chat');
|
||||
const hint = $('hint');
|
||||
const logEl = $('log');
|
||||
|
||||
function log(kind, text) {
|
||||
const line = document.createElement('div');
|
||||
line.className = 'l-' + kind;
|
||||
line.textContent = `${new Date().toLocaleTimeString()} ${text}`;
|
||||
logEl.appendChild(line);
|
||||
logEl.scrollTop = logEl.scrollHeight;
|
||||
}
|
||||
|
||||
function setState(text, cls) {
|
||||
$('state').textContent = text;
|
||||
$('dot').className = 'dot' + (cls ? ' ' + cls : '');
|
||||
}
|
||||
|
||||
function escapeHtml(s) {
|
||||
return s.replace(/[&<>"]/g, (c) => ({ '&': '&', '<': '<', '>': '>', '"': '"' }[c]));
|
||||
}
|
||||
|
||||
/** Append a bubble to the chat (scrolls to the bottom). */
|
||||
function addBubble(who, cls) {
|
||||
hint.remove();
|
||||
const wrap = document.createElement('div');
|
||||
wrap.className = 'msg ' + cls;
|
||||
const label = document.createElement('div');
|
||||
label.className = 'who';
|
||||
label.textContent = who;
|
||||
const bubble = document.createElement('div');
|
||||
bubble.className = 'bubble';
|
||||
wrap.append(label, bubble);
|
||||
chat.appendChild(wrap);
|
||||
chat.scrollTop = chat.scrollHeight;
|
||||
return bubble;
|
||||
}
|
||||
|
||||
function addChip(cls, html) {
|
||||
hint.remove();
|
||||
let chips = chat.lastElementChild;
|
||||
if (!chips || !chips.classList.contains('chips')) {
|
||||
chips = document.createElement('div');
|
||||
chips.className = 'chips';
|
||||
chat.appendChild(chips);
|
||||
}
|
||||
const chip = document.createElement('span');
|
||||
chip.className = 'chip ' + cls;
|
||||
chip.innerHTML = html;
|
||||
chips.appendChild(chip);
|
||||
chat.scrollTop = chat.scrollHeight;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------ karaoke render */
|
||||
// The engine streams the reply in `assistant_delta` chunks; the words of the
|
||||
// newest chunk are highlighted until the next one arrives.
|
||||
function renderKaraoke(bubble, fullText, latest) {
|
||||
const words = fullText.split(/(\s+)/);
|
||||
const latestWords = new Set(latest.trim().split(/\s+/).filter(Boolean));
|
||||
bubble.innerHTML = words
|
||||
.map((w) => {
|
||||
if (!w.trim()) return escapeHtml(w);
|
||||
const cls = latestWords.has(w) ? 'now' : 'done';
|
||||
return `<span class="kw ${cls}">${escapeHtml(w)}</span>`;
|
||||
})
|
||||
.join('');
|
||||
}
|
||||
|
||||
function renderInlineMarkdown(text) {
|
||||
let html = escapeHtml(text);
|
||||
html = html.replace(/`([^`]+)`/g, '<code>$1</code>');
|
||||
html = html.replace(/\*\*([^*]+)\*\*/g, '<strong>$1</strong>');
|
||||
html = html.replace(/\*([^*]+)\*/g, '<em>$1</em>');
|
||||
html = html.replace(/\[([^\]]+)\]\((https?:\/\/[^\s)]+)\)/g, '<a href="$2" target="_blank" rel="noreferrer noopener">$1</a>');
|
||||
return html;
|
||||
}
|
||||
|
||||
function markdownTableCells(line) {
|
||||
return line.trim().replace(/^\||\|$/g, '').split('|').map((cell) => cell.trim());
|
||||
}
|
||||
|
||||
function renderMarkdown(text) {
|
||||
const lines = String(text || '').replace(/\r\n?/g, '\n').split('\n');
|
||||
const html = [];
|
||||
let i = 0;
|
||||
|
||||
while (i < lines.length) {
|
||||
const line = lines[i];
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
const fence = line.match(/^```(\w+)?\s*$/);
|
||||
if (fence) {
|
||||
const lang = fence[1] ? ` class="language-${escapeHtml(fence[1])}"` : '';
|
||||
const body = [];
|
||||
i++;
|
||||
while (i < lines.length && !/^```/.test(lines[i])) {
|
||||
body.push(lines[i]);
|
||||
i++;
|
||||
}
|
||||
if (i < lines.length) i++;
|
||||
html.push(`<pre><code${lang}>${escapeHtml(body.join('\n'))}</code></pre>`);
|
||||
continue;
|
||||
}
|
||||
const heading = line.match(/^(#{1,4})\s+(.*)$/);
|
||||
if (heading) {
|
||||
const level = heading[1].length;
|
||||
html.push(`<h${level}>${renderInlineMarkdown(heading[2].trim())}</h${level}>`);
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
if (trimmed.includes('|') && i + 1 < lines.length && /^[\s|:-]+$/.test(lines[i + 1].trim())) {
|
||||
const head = markdownTableCells(line);
|
||||
const rows = [];
|
||||
i += 2;
|
||||
while (i < lines.length && lines[i].includes('|') && lines[i].trim()) {
|
||||
rows.push(markdownTableCells(lines[i]));
|
||||
i++;
|
||||
}
|
||||
const thead = `<thead><tr>${head.map((cell) => `<th>${renderInlineMarkdown(cell)}</th>`).join('')}</tr></thead>`;
|
||||
const tbody = rows.length
|
||||
? `<tbody>${rows.map((row) => `<tr>${row.map((cell) => `<td>${renderInlineMarkdown(cell)}</td>`).join('')}</tr>`).join('')}</tbody>`
|
||||
: '';
|
||||
html.push(`<table>${thead}${tbody}</table>`);
|
||||
continue;
|
||||
}
|
||||
if (/^\s*>\s?/.test(line)) {
|
||||
const quote = [];
|
||||
while (i < lines.length && /^\s*>\s?/.test(lines[i])) {
|
||||
quote.push(lines[i].replace(/^\s*>\s?/, ''));
|
||||
i++;
|
||||
}
|
||||
html.push(`<blockquote>${quote.map((part) => renderInlineMarkdown(part)).join('<br>')}</blockquote>`);
|
||||
continue;
|
||||
}
|
||||
if (/^\s*[-*+]\s+/.test(line)) {
|
||||
const items = [];
|
||||
while (i < lines.length && /^\s*[-*+]\s+/.test(lines[i])) {
|
||||
items.push(lines[i].replace(/^\s*[-*+]\s+/, ''));
|
||||
i++;
|
||||
}
|
||||
html.push(`<ul>${items.map((item) => `<li>${renderInlineMarkdown(item)}</li>`).join('')}</ul>`);
|
||||
continue;
|
||||
}
|
||||
if (/^\s*\d+\.\s+/.test(line)) {
|
||||
const items = [];
|
||||
while (i < lines.length && /^\s*\d+\.\s+/.test(lines[i])) {
|
||||
items.push(lines[i].replace(/^\s*\d+\.\s+/, ''));
|
||||
i++;
|
||||
}
|
||||
html.push(`<ol>${items.map((item) => `<li>${renderInlineMarkdown(item)}</li>`).join('')}</ol>`);
|
||||
continue;
|
||||
}
|
||||
const para = [];
|
||||
while (
|
||||
i < lines.length &&
|
||||
lines[i].trim() &&
|
||||
!/^```/.test(lines[i]) &&
|
||||
!/^(#{1,4})\s+/.test(lines[i]) &&
|
||||
!/^\s*>\s?/.test(lines[i]) &&
|
||||
!/^\s*[-*+]\s+/.test(lines[i]) &&
|
||||
!/^\s*\d+\.\s+/.test(lines[i]) &&
|
||||
!(lines[i].includes('|') && i + 1 < lines.length && /^[\s|:-]+$/.test(lines[i + 1].trim()))
|
||||
) {
|
||||
para.push(lines[i].trim());
|
||||
i++;
|
||||
}
|
||||
html.push(`<p>${renderInlineMarkdown(para.join(' '))}</p>`);
|
||||
}
|
||||
|
||||
return html.join('') || `<p>${renderInlineMarkdown(text)}</p>`;
|
||||
}
|
||||
|
||||
function addMarkdownBubble(text) {
|
||||
const bubble = addBubble('agent', 'agent');
|
||||
bubble.classList.add('markdown');
|
||||
bubble.innerHTML = renderMarkdown(text);
|
||||
return bubble;
|
||||
}
|
||||
|
||||
/* -------------------------------------------------------------------- state */
|
||||
let session = null;
|
||||
let rest = null;
|
||||
let sessionId = null;
|
||||
let userData = null;
|
||||
let mic = null;
|
||||
let micAnalyser = null;
|
||||
let micRaf = 0;
|
||||
let agentBubble = null;
|
||||
let agentText = '';
|
||||
let bargeCount = 0;
|
||||
let pendingExternalSourceUrl = '';
|
||||
let agentTextEnded = false;
|
||||
let pendingExternalSourceTimer = 0;
|
||||
|
||||
const AGENT_AUDIO_AUTOPLAY_DELAY_MS = 8000;
|
||||
|
||||
const secure = window.isSecureContext ||
|
||||
['localhost', '127.0.0.1', '[::1]'].includes(location.hostname);
|
||||
if (!secure) $('insecure').classList.add('show');
|
||||
if (location.protocol.startsWith('http')) $('baseUrl').value = location.origin;
|
||||
else $('baseUrl').value = 'http://localhost:8787';
|
||||
|
||||
/* --------------------------------------------------------------- mic meter */
|
||||
async function micMeterOn(stream) {
|
||||
micAnalyser = null;
|
||||
const ctx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
const src = ctx.createMediaStreamSource(stream);
|
||||
const analyser = ctx.createAnalyser();
|
||||
analyser.fftSize = 256;
|
||||
src.connect(analyser);
|
||||
const data = new Uint8Array(analyser.frequencyBinCount);
|
||||
const bars = [...$('micMeter').querySelectorAll('.bar i')];
|
||||
const tick = () => {
|
||||
analyser.getByteFrequencyData(data);
|
||||
for (let i = 0; i < bars.length; i++) {
|
||||
const v = data[i * 3] / 255;
|
||||
bars[i].style.height = Math.max(3, v * 12) + 'px';
|
||||
}
|
||||
micRaf = requestAnimationFrame(tick);
|
||||
};
|
||||
tick();
|
||||
micAnalyser = ctx;
|
||||
}
|
||||
|
||||
function micMeterOff() {
|
||||
cancelAnimationFrame(micRaf);
|
||||
if (micAnalyser) micAnalyser.close().catch(() => {});
|
||||
micAnalyser = null;
|
||||
for (const b of $('micMeter').querySelectorAll('.bar i')) b.style.height = '3px';
|
||||
$('micMeter').classList.remove('on');
|
||||
}
|
||||
|
||||
async function micStart() {
|
||||
if (mic) return;
|
||||
mic = new MicCapture({ sampleRate: AUDIO.sampleRate });
|
||||
mic.on('frame', (samples) => session.client.sendAudio(samples));
|
||||
await mic.start();
|
||||
$('micMeter').classList.add('on');
|
||||
// The SDK does not expose the MediaStream, so re-open one just for the meter.
|
||||
navigator.mediaDevices
|
||||
.getUserMedia({ audio: true })
|
||||
.then((stream) => micMeterOn(stream))
|
||||
.catch(() => {});
|
||||
$('micBtn').textContent = 'Microphone: ON';
|
||||
$('micBtn').classList.add('active');
|
||||
log('user', 'mic → ON');
|
||||
}
|
||||
|
||||
function micStop() {
|
||||
if (!mic) return;
|
||||
mic.stop();
|
||||
mic = null;
|
||||
micMeterOff();
|
||||
$('micBtn').textContent = 'Microphone: OFF';
|
||||
$('micBtn').classList.remove('active');
|
||||
log('user', 'mic → OFF');
|
||||
}
|
||||
|
||||
function clearPendingExternalSourceAutoplay() {
|
||||
if (pendingExternalSourceTimer) clearTimeout(pendingExternalSourceTimer);
|
||||
pendingExternalSourceTimer = 0;
|
||||
}
|
||||
|
||||
function schedulePendingExternalSourceAutoplay() {
|
||||
if (!session || !pendingExternalSourceUrl || !agentTextEnded) return;
|
||||
clearPendingExternalSourceAutoplay();
|
||||
const url = pendingExternalSourceUrl;
|
||||
pendingExternalSourceTimer = setTimeout(() => {
|
||||
pendingExternalSourceTimer = 0;
|
||||
if (!session) return;
|
||||
if (!pendingExternalSourceUrl || pendingExternalSourceUrl !== url) return;
|
||||
if (!agentTextEnded) return;
|
||||
pendingExternalSourceUrl = '';
|
||||
agentTextEnded = false;
|
||||
playExternalSource(url);
|
||||
}, AGENT_AUDIO_AUTOPLAY_DELAY_MS);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------ connect / wire */
|
||||
function wire(s) {
|
||||
s.on('state', (st) => {
|
||||
if (st === 'connecting') setState('connecting…', 'warn');
|
||||
else if (st === 'open') setState('connected', 'on');
|
||||
else if (st === 'closed') setState('disconnected', '');
|
||||
});
|
||||
s.on('reconnecting', (e) => setState(`reconnecting… (${e.attempt + 1})`, 'warn'));
|
||||
|
||||
s.on('session', async (e) => {
|
||||
sessionId = e.id;
|
||||
log('user', `session id = ${e.id}`);
|
||||
// Apply the demo's system prompt for the whole session, right at startup.
|
||||
try {
|
||||
showPrompt(await rest.setPrompt(SESSION_PROMPT, sessionId));
|
||||
log('tool', `session prompt applied (session ${sessionId})`);
|
||||
} catch (err) {
|
||||
log('err', `prompt apply failed: ${err.message}`);
|
||||
refreshPrompt();
|
||||
}
|
||||
// Authenticate the user identified by ?phoneNumber=… in the page URL.
|
||||
await authenticateUser();
|
||||
});
|
||||
|
||||
s.on('cues', (e) => {
|
||||
log('cue', `cues lang=${e.lang} ${JSON.stringify(e.config ?? {})}`);
|
||||
});
|
||||
|
||||
s.on('user', (e) => {
|
||||
clearPendingExternalSourceAutoplay();
|
||||
agentTextEnded = false;
|
||||
pendingExternalSourceUrl = '';
|
||||
log('user', `user: ${e.text}`);
|
||||
});
|
||||
|
||||
s.on('assistant_delta', (e) => {
|
||||
agentTextEnded = false;
|
||||
if (!agentBubble) agentBubble = addBubble('agent', 'agent');
|
||||
agentText += e.text;
|
||||
renderKaraoke(agentBubble, agentText, e.text);
|
||||
chat.scrollTop = chat.scrollHeight;
|
||||
});
|
||||
|
||||
s.on('assistant_markdown', (e) => {
|
||||
addMarkdownBubble(e.text ?? '');
|
||||
log('tool', `assistant_markdown (${String(e.text ?? '').length} char)`);
|
||||
});
|
||||
|
||||
s.on('assistant_end', () => {
|
||||
if (agentBubble) renderKaraoke(agentBubble, agentText, '');
|
||||
log('user', `assistant_end (${agentText.length} char)`);
|
||||
agentBubble = null;
|
||||
agentText = '';
|
||||
agentTextEnded = true;
|
||||
schedulePendingExternalSourceAutoplay();
|
||||
});
|
||||
|
||||
s.on('cue', (e) => {
|
||||
log('cue', `cue ${e.kind} ${e.name ?? ''} ${e.category ?? ''}`);
|
||||
addChip('cue flash', `cue <b>${escapeHtml(e.kind)}</b>${e.name ? ' · ' + escapeHtml(e.name) : ''}`);
|
||||
});
|
||||
|
||||
s.on('barge_in', (e) => {
|
||||
bargeCount++;
|
||||
$('bargeCount').textContent = bargeCount;
|
||||
$('bargein').classList.add('active');
|
||||
setTimeout(() => $('bargein').classList.remove('active'), 700);
|
||||
clearPendingExternalSourceAutoplay();
|
||||
agentTextEnded = false;
|
||||
pendingExternalSourceUrl = '';
|
||||
log('cue', `barge_in source=${e.source ?? '?'} prob=${e.prob ?? '-'}`);
|
||||
});
|
||||
|
||||
s.on('tool_call', (e) => {
|
||||
log('tool', `tool_call ${e.name} ${JSON.stringify(e.arguments ?? {})}`);
|
||||
addChip('tool', `tool <b>${escapeHtml(e.name)}</b>`);
|
||||
// A tool flagged `audio` returns a media file: queue its URL and start
|
||||
// playback only after the agent finishes speaking for this turn.
|
||||
const def = DEMO_TOOLS.find((t) => t.name === e.name);
|
||||
if (def?.audio) {
|
||||
try {
|
||||
pendingExternalSourceUrl = toolUrl(def, e.arguments);
|
||||
$('sourceUrl').value = pendingExternalSourceUrl;
|
||||
addChip('tool', `audio <b>queued</b>`);
|
||||
if (agentTextEnded) schedulePendingExternalSourceAutoplay();
|
||||
} catch (err) {
|
||||
log('err', `audio tool playback failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
s.on('audio_source', (e) => {
|
||||
log('cue', `audio_source ${e.state} ${e.url ?? ''}`);
|
||||
if (e.state === 'error') addChip('chip flash', `source <b>error</b> ${escapeHtml(e.text ?? '')}`);
|
||||
});
|
||||
|
||||
s.on('error', (e) => {
|
||||
if (e.text) {
|
||||
log('err', `server error: ${e.text}`);
|
||||
addBubble('error', 'error').textContent = e.text;
|
||||
} else {
|
||||
log('err', `socket error${e.fatal ? ' (fatal)' : ''}: ${e.error?.message ?? ''}`);
|
||||
}
|
||||
});
|
||||
|
||||
s.on('audio', () => {}); // frames are played by the session; logged nowhere (too noisy)
|
||||
}
|
||||
|
||||
async function connect() {
|
||||
const baseUrl = $('baseUrl').value.trim();
|
||||
rest = new AudioAgentRest(baseUrl);
|
||||
session = new AudioSession({
|
||||
baseUrl,
|
||||
autoReconnect: true,
|
||||
forwardMic: false, // the mic is toggled separately below
|
||||
bufferMs: Number($('bufferMs').value) || 150,
|
||||
});
|
||||
wire(session);
|
||||
try {
|
||||
await session.start({ microphone: false });
|
||||
} catch (err) {
|
||||
log('err', `connect failed: ${err.message}`);
|
||||
setState('error', 'err');
|
||||
return;
|
||||
}
|
||||
$('connectBtn').textContent = 'Disconnect';
|
||||
for (const id of ['sendBtn', 'micBtn', 'interruptBtn', 'sourcePlay', 'sourceStop', 'sourceVol',
|
||||
'promptLoad', 'promptApply', 'promptClear', 'promptText']) {
|
||||
$(id).disabled = false;
|
||||
}
|
||||
try {
|
||||
const tools = await rest.setTools(DEMO_TOOLS);
|
||||
log('tool', `demo tools installed (${tools.count})`);
|
||||
} catch (err) {
|
||||
log('err', `tool install failed: ${err.message}`);
|
||||
}
|
||||
log('user', `connected → ${session.client.url}`);
|
||||
}
|
||||
|
||||
function disconnect() {
|
||||
micStop();
|
||||
if (session) session.stop();
|
||||
session = null;
|
||||
rest = null;
|
||||
sessionId = null;
|
||||
userData = null;
|
||||
clearPendingExternalSourceAutoplay();
|
||||
pendingExternalSourceUrl = '';
|
||||
agentTextEnded = false;
|
||||
setUser('');
|
||||
$('sessionId').textContent = '—';
|
||||
$('promptSource').textContent = '—';
|
||||
$('promptOverride').textContent = '—';
|
||||
$('promptText').value = '';
|
||||
$('connectBtn').textContent = 'Connect';
|
||||
for (const id of ['sendBtn', 'micBtn', 'interruptBtn', 'sourcePlay', 'sourceStop', 'sourceVol',
|
||||
'promptLoad', 'promptApply', 'promptClear', 'promptText']) {
|
||||
$(id).disabled = true;
|
||||
}
|
||||
setState('disconnected', '');
|
||||
}
|
||||
|
||||
/* ----------------------------------------------------------- system prompt */
|
||||
function showPrompt(res) {
|
||||
$('sessionId').textContent = res.session_id ?? '—';
|
||||
$('promptSource').textContent = res.source;
|
||||
$('promptOverride').textContent = res.has_override ? 'yes' : 'no';
|
||||
$('promptText').value = res.prompt;
|
||||
}
|
||||
|
||||
async function refreshPrompt() {
|
||||
if (!rest) return;
|
||||
try {
|
||||
showPrompt(await rest.getPrompt(sessionId));
|
||||
} catch (err) {
|
||||
log('err', `prompt load failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/* --------------------------------------------------------------- phone auth */
|
||||
/** Read `phoneNumber` from the page URL. Kept raw (not via URLSearchParams) so
|
||||
* a leading `+` in `?phoneNumber=+44…` survives instead of becoming a space. */
|
||||
function phoneFromUrl() {
|
||||
const m = location.search.match(/[?&](?:phoneNumber|phone)=([^&]*)/i);
|
||||
return m ? decodeURIComponent(m[1]) : '';
|
||||
}
|
||||
|
||||
/** Best-effort human label out of the login payload (unknown schema). */
|
||||
function userLabel(u) {
|
||||
if (!u || typeof u !== 'object') return '';
|
||||
const cand = [
|
||||
u.name, u.fullName, u.displayName, u.email, u.phoneNumber,
|
||||
u.user?.name, u.user?.fullName, u.user?.email,
|
||||
];
|
||||
return cand.find((v) => typeof v === 'string' && v) || '';
|
||||
}
|
||||
|
||||
/** Show (or clear) the authenticated user's name in the header. */
|
||||
function setUser(label, ok = true) {
|
||||
const el = $('userStatus');
|
||||
el.textContent = label || '';
|
||||
el.classList.toggle('show', !!label);
|
||||
el.classList.toggle('bad', !ok);
|
||||
el.title = label ? `${label} · session ${sessionId ?? '—'}` : '';
|
||||
}
|
||||
|
||||
/** Authenticate the user identified in the URL, at session start. */
|
||||
async function authenticateUser() {
|
||||
const phoneNumber = phoneFromUrl();
|
||||
if (!phoneNumber) {
|
||||
log('user', 'no phoneNumber in URL — skipping phone-login');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
userData = await rest.phoneLogin(phoneNumber, sessionId);
|
||||
const label = userLabel(userData) || phoneNumber;
|
||||
setUser(label);
|
||||
addChip('tool', `user <b>${escapeHtml(label)}</b>`);
|
||||
log('tool', `phone-login ok (${phoneNumber}) → ${JSON.stringify(userData).slice(0, 300)}`);
|
||||
} catch (err) {
|
||||
setUser('auth failed', false);
|
||||
log('err', `phone-login failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/* -------------------------------------------------------------------- events */
|
||||
$('connectBtn').onclick = () => (session ? disconnect() : connect());
|
||||
|
||||
$('sendBtn').onclick = () => {
|
||||
const text = $('textInput').value.trim();
|
||||
if (!text || !session) return;
|
||||
session.sendUserText(text);
|
||||
$('textInput').value = '';
|
||||
};
|
||||
$('textInput').onkeydown = (e) => {
|
||||
if (e.key === 'Enter') $('sendBtn').click();
|
||||
};
|
||||
|
||||
$('micBtn').onclick = async () => {
|
||||
if (!session) return;
|
||||
try {
|
||||
mic ? micStop() : await micStart();
|
||||
} catch (err) {
|
||||
log('err', `mic error: ${err.message}`);
|
||||
}
|
||||
};
|
||||
|
||||
$('interruptBtn').onclick = () => session && session.interrupt();
|
||||
|
||||
/** Route a URL to the engine's external audio source channel and start it. */
|
||||
function playExternalSource(url) {
|
||||
if (!session || !url) return;
|
||||
$('sourceUrl').value = url;
|
||||
session.client.playAudioSource(url);
|
||||
log('cue', `audio_source play ${url}`);
|
||||
}
|
||||
|
||||
$('sourcePlay').onclick = () => playExternalSource($('sourceUrl').value.trim());
|
||||
$('sourceStop').onclick = () => session && session.client.stopAudioSource();
|
||||
$('sourceVol').oninput = (e) => session && session.client.setAudioSourceVolume(e.target.value / 100);
|
||||
|
||||
$('promptLoad').onclick = () => refreshPrompt();
|
||||
$('promptApply').onclick = async () => {
|
||||
if (!rest) return;
|
||||
const prompt = $('promptText').value;
|
||||
if (!prompt.trim()) return;
|
||||
try {
|
||||
showPrompt(await rest.setPrompt(prompt, sessionId));
|
||||
log('tool', `prompt override set (session ${sessionId})`);
|
||||
} catch (err) {
|
||||
log('err', `prompt apply failed: ${err.message}`);
|
||||
}
|
||||
};
|
||||
$('promptClear').onclick = async () => {
|
||||
if (!rest) return;
|
||||
try {
|
||||
showPrompt(await rest.setPrompt(null, sessionId));
|
||||
log('tool', `prompt override cleared (session ${sessionId})`);
|
||||
} catch (err) {
|
||||
log('err', `prompt clear failed: ${err.message}`);
|
||||
}
|
||||
};
|
||||
|
||||
setState('disconnected', '');
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user