# Client Events
Source: https://docs.outspeed.com/api-spec/client
These are the events that Realtime servers accept from the client
## Overview
These events are accepted by the Realtime servers via WebRTC datachannel from the client.
Each event has a specific structure and purpose for managing realtime interactions.
## `session.update`
Send this event to update the session's default configuration. The client may send this event at any time to update any field, except for voice. Note that once a session has been initialized with a particular model, it can't be changed to another model using `session.update`.
When the server receives a `session.update`, it will respond with a `session.updated` event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string.
### Properties
Client-generated ID used to identify this event.
The event type, must be `session.update`.
Realtime session object configuration.
```json
{
"event_id": "event_123",
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"instructions": "You are a helpful assistant.",
"voice": "sage",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"input_audio_transcription": {
"model": "whisper-1"
},
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"prefix_padding_ms": 300,
"silence_duration_ms": 500,
"create_response": true
},
"tools": [
{
"type": "function",
"name": "get_weather",
"description": "Get the current weather...",
"parameters": {
"type": "object",
"properties": {
"location": { "type": "string" }
},
"required": ["location"]
}
}
],
"tool_choice": "auto",
"temperature": 0.8,
"max_response_output_tokens": "inf"
}
}
```
## `conversation.item.create`
Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.
### Properties
Client-generated ID used to identify this event.
The event type, must be `conversation.item.create`.
The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation.
The item to add to the conversation.
```json
{
"event_id": "event_345",
"type": "conversation.item.create",
"previous_item_id": null,
"item": {
"id": "msg_001",
"type": "message",
"role": "user",
"content": [
{
"type": "input_text",
"text": "Hello, how are you?"
}
]
}
}
```
## `response.create`
This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically.
### Properties
Client-generated ID used to identify this event.
The event type, must be `response.create`.
Create a new Realtime response with these parameters.
```json
{
"event_id": "event_234",
"type": "response.create",
"response": {
"modalities": ["text", "audio"],
"instructions": "Please assist the user.",
"voice": "sage",
"output_audio_format": "pcm16",
"tools": [
{
"type": "function",
"name": "calculate_sum",
"description": "Calculates the sum of two numbers.",
"parameters": {
"type": "object",
"properties": {
"a": { "type": "number" },
"b": { "type": "number" }
},
"required": ["a", "b"]
}
}
],
"tool_choice": "auto",
"temperature": 0.8,
"max_output_tokens": 1024
}
}
```
# Outspeed API
Source: https://docs.outspeed.com/api-spec/live-api
Embed low-latency voice interaction in your apps with Outspeed API
The Outspeed API enables you to build fast, natural voice experiences by connecting to our hosted speech-to-speech stack.
With Outspeed, you can send and receive both text and audio in real time, leverage voice activity detection, call external functions, and more.
Outspeed API is fully compatible with the OpenAI Realtime API.
## How It Works
Outspeed API is event-driven. After connecting via WebRTC, your app sends [Client Events](/api-spec/client)
and listens for [Server Events](/api-spec/server) to drive the conversation.
Getting started is easy:
* Install the [JS SDK](https://www.npmjs.com/package/@outspeed/client)
* Install the [React SDK](https://www.npmjs.com/package/@outspeed/react)
* Install the [Swift SDK](https://github.com/outspeed-ai/OutspeedSwift)
Each SDK manages the WebRTC connection and event flow for you, so you can focus on building your app.
## Key Capabilities
* **Text & Audio**: Send and receive both text and audio
* **Realtime Responses**: Low-latency replies for natural conversations
* **Function Calling**: Call external tools and services
## API Events
The API is organized around events:
Events your app can send to the Realtime server.
Events your app receives from the Realtime server.
# Server Events
Source: https://docs.outspeed.com/api-spec/server
Events that the Realtime WebRTC server accepts from the client
# Overview
These are events emitted from the Realtime servers and sent via WebRTC datachannel to the client.
## `error`
Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be error.
* **error**: `object` - Details of the error.
```json
{
"event_id": "event_890",
"type": "error",
"error": {
"type": "invalid_request_error",
"code": "invalid_event",
"message": "The 'type' field is missing.",
"param": null,
"event_id": "event_567"
}
}
```
## `session.created`
Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be session.created.
* **session**: `object` - Realtime session object configuration.
```json
{
"event_id": "event_1234",
"type": "session.created",
"session": {
"id": "sess_001",
"object": "realtime.session",
"model": "gpt-4o-realtime-preview-2024-12-17",
"modalities": ["text", "audio"],
"instructions": "...model instructions here...",
"voice": "sage",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"input_audio_transcription": null,
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"prefix_padding_ms": 300,
"silence_duration_ms": 200
},
"tools": [],
"tool_choice": "auto",
"temperature": 0.8,
"max_response_output_tokens": "inf"
}
}
```
## `session.updated`
Returned when a session is updated with a session.update event, unless there is an error.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be session.updated.
* **session**: `object` - Realtime session object configuration.
```json
{
"event_id": "event_5678",
"type": "session.updated",
"session": {
"id": "sess_001",
"object": "realtime.session",
"model": "gpt-4o-realtime-preview-2024-12-17",
"modalities": ["text"],
"instructions": "New instructions",
"voice": "sage",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"input_audio_transcription": {
"model": "whisper-1"
},
"turn_detection": null,
"tools": [],
"tool_choice": "none",
"temperature": 0.7,
"max_response_output_tokens": 200
}
}
```
## `conversation.item.created`
Returned when a conversation item is created. There are several scenarios that produce this event:
* The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function\_call.
* The input audio buffer has been committed, either by the client or the server (in server\_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item.
* The client has sent a conversation.item.create event to add a new Item to the Conversation.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be conversation.item.created.
* **previous\_item\_id**: `string` - The ID of the preceding item in the Conversation context, allows the client to understand the order of the conversation.
* **item**: `object` - The item to add to the conversation.
```json
{
"event_id": "event_1920",
"type": "conversation.item.created",
"previous_item_id": "msg_002",
"item": {
"id": "msg_003",
"object": "realtime.item",
"type": "message",
"status": "completed",
"role": "user",
"content": [
{
"type": "input_audio",
"transcript": "hello how are you",
"audio": "base64encodedaudio=="
}
]
}
}
```
## `input_audio_buffer.committed`
Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item\_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be input\_audio\_buffer.committed.
* **previous\_item\_id**: `string` - The ID of the preceding item after which the new item will be inserted.
* **item\_id**: `string` - The ID of the user message item that will be created.
```json
{
"event_id": "event_1121",
"type": "input_audio_buffer.committed",
"previous_item_id": "msg_001",
"item_id": "msg_002"
}
```
## `input_audio_buffer.speech_started`
Sent by the server when in server\_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user.
The client should expect to receive a input\_audio\_buffer.speech\_stopped event when speech stops. The item\_id property is the ID of the user message item that will be created when speech stops and will also be included in the input\_audio\_buffer.speech\_stopped event (unless the client manually commits the audio buffer during VAD activation).
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be input\_audio\_buffer.speech\_started.
* **audio\_start\_ms**: `integer` - Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix\_padding\_ms configured in the Session.
* **item\_id**: `string` - The ID of the user message item that will be created when speech stops.
```json
{
"event_id": "event_1516",
"type": "input_audio_buffer.speech_started",
"audio_start_ms": 1000,
"item_id": "msg_003"
}
```
## `input_audio_buffer.speech_stopped`
Returned in server\_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be input\_audio\_buffer.speech\_stopped.
* **audio\_end\_ms**: `integer` - Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min\_silence\_duration\_ms configured in the Session.
* **item\_id**: `string` - The ID of the user message item that will be created.
```json
{
"event_id": "event_1718",
"type": "input_audio_buffer.speech_stopped",
"audio_end_ms": 2000,
"item_id": "msg_003"
}
```
## `output_audio_buffer.started`
Returned when the server begins playing audio output to the client. This event is sent at the start of audio playback for a response, allowing clients to synchronize their UI with the audio stream.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be `output_audio_buffer.started`.
* **response\_id**: `string` - The ID of the response that triggered this audio output to start
```json
{
"event_id": "event_1718",
"type": "output_audio_buffer.started",
"response_id": "resp_3876604f-7a35-4c95-ad80-e553ddb9a166"
}
```
## `output_audio_buffer.stopped`
Returned when the output audio buffer stops playing audio. This event is sent by the server when the audio output for a response has been fully generated and transmitted.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be `output_audio_buffer.stopped`.
* **response\_id**: `string` - The ID of the response that triggered this audio output.
```json
{
"event_id": "event_1920",
"type": "output_audio_buffer.stopped",
"response_id": "resp_3876604f-7a35-4c95-ad80-e553ddb9a166"
}
```
## `response.done`
Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be response.done.
* **response**: `object` - The response resource.
```json
{
"event_id": "event_3132",
"type": "response.done",
"response": {
"id": "resp_001",
"object": "realtime.response",
"status": "completed",
"status_details": null,
"output": [
{
"id": "msg_006",
"object": "realtime.item",
"type": "message",
"status": "completed",
"role": "assistant",
"content": [
{
"type": "text",
"text": "Sure, how can I assist you today?"
}
]
}
],
"usage": {
"total_tokens":275,
"input_tokens":127,
"output_tokens":148,
"input_token_details": {
"cached_tokens":384,
"text_tokens":119,
"audio_tokens":8,
"cached_tokens_details": {
"text_tokens": 128,
"audio_tokens": 256
}
},
"output_token_details": {
"text_tokens":36,
"audio_tokens":112
}
}
}
}
```
## `response.audio_transcript.delta`
Returned when the model-generated transcription of audio output is updated.
### Properties
* **event\_id**: `string` - The unique ID of the server event.
* **type**: `string` - The event type, must be response.audio\_transcript.delta.
* **response\_id**: `string` - The ID of the response.
* **item\_id**: `string` - The ID of the item.
* **output\_index**: `integer` - The index of the output item in the response.
* **content\_index**: `integer` - The index of the content part in the item's content array.
* **delta**: `string` - The transcript delta.
```json
{
"event_id": "event_4546",
"type": "response.audio_transcript.delta",
"response_id": "resp_001",
"item_id": "msg_008",
"output_index": 0,
"content_index": 0,
"delta": "Hello, how can I a"
}
```
# Text-to-Speech (HTTP)
Source: https://docs.outspeed.com/api-spec/tts
Generate WAV audio from text using the HTTP TTS endpoint.
## Overview
Generate audio from text using our TTS endpoints. We support both single-voice and multi-voice dialogue generation.
* **Single Voice TTS**: Convert text to speech with one voice
* **Dialogue Generation**: Mix narrator and character voices in the same audio file
## Single Voice TTS
POST `https://api.outspeed.com/v1/tts/`
### Request Body
```json
{
"model": "outspeed-tts-v2",
"voice": "clark",
"text": "Hello, world!",
"stream": false
}
```
* **model**: TTS model to use. Use `outspeed-tts-v2` (`outspeed-tts-v1` is deprecated)
* **voice**: the voice identifier. Find all available voices and their models at [TTS Playground](https://dashboard.outspeed.com/tts)
* **text**: the text to synthesize
* **stream**: set to `true` to stream audio chunks; `false` returns the full WAV
### Response
* Content-Type: `audio/pcm`
* Headers:
* `X-Sample-Rate`: Sample rate (default: 24000)
* `X-Channels`: Number of audio channels (default: 1)
* `X-Bit-Depth`: Bit depth (default: 16)
* Body: Raw PCM audio bytes (little-endian int16), 24kHz, mono. No WAV header is included. Wrap with a WAV header or convert with a tool like ffmpeg.
Authenticate with Authorization: Bearer \.
## Examples (non-streaming)
```bash curl
curl \
-X POST \
'https://api.outspeed.com/v1/tts/' \
-H 'Authorization: Bearer YOUR_OUTSPEED_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"model":"outspeed-tts-v2","voice":"clark","text":"Hello, world!","stream":false}' \
--output tts.pcm
```
```bash javascript
// Node.js 18+
//Use a WAV helper library to write a proper WAV file from PCM:
//npm install wav
import wav from "wav";
const res = await fetch("https://api.outspeed.com/v1/tts/", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_OUTSPEED_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({ model: "outspeed-tts-v2", voice: "clark", text: "Hello, world!", stream: false }),
});
if (!res.ok) throw new Error(`TTS failed: ${res.status}`);
const pcm = Buffer.from(await res.arrayBuffer());
const writer = new wav.FileWriter("tts.wav", { channels: 1, sampleRate: 24000, bitDepth: 16 });
writer.write(pcm);
writer.end();
```
## Examples (streaming)
```bash curl
curl \
-X POST \
'https://api.outspeed.com/v1/tts/' \
-H 'Authorization: Bearer YOUR_OUTSPEED_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"model":"outspeed-tts-v2","voice":"clark","text":"Hello, world!","stream":true}' \
--no-buffer \
--output tts.pcm
```
```bash javascript
// Node.js 18+ (streaming)
//Use a WAV helper library to write a proper WAV file from PCM:
//npm install wav
import { Readable } from "node:stream";
import wav from "wav";
const res = await fetch("https://api.outspeed.com/v1/tts/", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_OUTSPEED_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({ model: "outspeed-tts-v2", voice: "clark", text: "Hello, world!", stream: true }),
});
if (!res.ok || !res.body) throw new Error(`TTS failed: ${res.status}`);
const nodeReadable = Readable.fromWeb(res.body);
const writer = new wav.FileWriter("tts.wav", { channels: 1, sampleRate: 24000, bitDepth: 16 });
await new Promise((resolve, reject) => {
nodeReadable.pipe(writer);
writer.on("finish", resolve);
writer.on("error", reject);
nodeReadable.on("error", reject);
});
```
## Dialogue Generation
Generate audio with multiple voices (narrator + character) using the `outspeed-tts-v2` model.
Try it visually at the [Dialogue Playground](https://dashboard.outspeed.com/tts/dialogue)
### Endpoint
POST `https://api.outspeed.com/v1/tts/dialogue`
### Request Body
```json
{
"model": "outspeed-tts-v2",
"text": "*The old library stood silent.* I pushed open the heavy door. *Inside, dust particles danced in my flashlight beam.*",
"speaker_voice": "9c5c73f4-1cb7-46cf-91d8-24c80b6288f0",
"narrator_voice": "a42c84b2-0e6b-4c9f-a8e7-3f5d1c2e8a9b",
"narrator_delimiter": "*"
}
```
### Parameters
| Parameter | Type | Required | Description |
| -------------------- | ------ | -------- | --------------------------------------------------- |
| `model` | string | Yes | Must be `outspeed-tts-v2` |
| `text` | string | Yes | Dialogue text with delimiters for narrator parts |
| `speaker_voice` | string | Yes | Voice ID for character dialogue |
| `narrator_voice` | string | No | Voice ID for narrator parts (omit to skip narrator) |
| `narrator_delimiter` | string | No | Delimiter: `*`, `(`, `[`, or `{` (default: `*`) |
### Delimiter Usage
* **Text in delimiters** = narrator voice
* **Text outside delimiters** = character voice
Supported delimiters:
* `*` → `*narrator text*`
* `(` → `(narrator text)`
* `[` → `[narrator text]`
* `{` → `{narrator text}`
### Examples
```bash Audiobook Style
curl -X POST https://api.outspeed.com/v1/tts/dialogue \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "outspeed-tts-v2",
"text": "*The old library stood silent.* I pushed open the heavy door. *Inside, dust particles danced in my flashlight beam.*",
"speaker_voice": "",
"narrator_voice": ",
"narrator_delimiter": "*"
}' \
--output dialogue.wav
```
### Voice Selection
Both voices require **voice IDs** (not names). Get voice IDs from:
* [TTS Playground](https://dashboard.outspeed.com/tts) - Browse and copy existing voice IDs
* [Voice Upload](/features/voice-cloning) - Upload custom voices or create [voice clones](/features/voice-cloning)
### Response
Same as single voice TTS:
* Content-Type: `audio/wav`
* Headers: `X-Sample-Rate`, `X-Channels`, `X-Bit-Depth`
* Body: Raw PCM audio bytes
# Auto Reconnection
Source: https://docs.outspeed.com/features/auto-reconnection
Handle connection drops on mobile browsers when app goes to background
## Overview
Mobile browsers have privacy protections that affect voice conversations when your app goes to the background.
**Context is preserved**: When auto-reconnection happens, your conversation context and history are automatically
restored. You don't need to worry about losing the conversation state.
## What Happens on Mobile
### 1. App Goes to Background
When users switch apps or lock their phone:
* **Microphone access is revoked** within seconds (privacy protection)
* User's voice won't reach the AI, even though connection appears active
* WebRTC connection may drop after extended background time
### 2. App Returns to Foreground
When users return to your app:
* SDK automatically attempts to reconnect
* Microphone permissions are restored
* Conversation resumes seamlessly
## Handling Reconnection
### Listen for Connection State Changes
```typescript
const conversation = useConversation({
onStatusChange: (status) => {
// Status values: "connected" | "connecting" | "disconnecting" | "disconnected" | "reconnecting"
if (status === "reconnecting") {
// Show "Reconnecting..." message to user
showReconnectingIndicator();
} else if (status === "connected") {
// Hide loading indicators
hideReconnectingIndicator();
}
},
});
```
### Detect New vs Reconnected Sessions
Use the `session.created` event to know if this is a fresh start or reconnection:
```typescript
conversation.on("session.created", (event) => {
if (event.run_id === 0) {
// This is a brand new session
console.log("New conversation started");
} else {
// This is a reconnection (run_id will be 1, 2, 3, etc.)
console.log(`Reconnected - Run #${event.run_id}`);
// showToast("Connection restored!");
}
});
```
## Best Practices
### User Experience
* **Show clear status**: Use visual indicators for connection states
* **Inform about background**: Educate users that voice won't work in background due to privacy restrictions
* **Seamless return**: Make reconnection feel automatic and smooth
## Summary
1. **Mobile browsers revoke mic access** when app goes to background
2. **SDK auto-reconnects** when app returns to foreground
3. **Use `onStatusChange`** to show connection status to users
4. **Use `session.created` with `run_id`** to detect new vs reconnected sessions
5. **Provide clear visual feedback** so users understand what's happening
# Client Tools
Source: https://docs.outspeed.com/features/client-tools
Create custom tools that extend AI agents with your own functions and APIs
## Overview
Client tools are custom functions that the voice agent can use during the conversation. They are defined and implemented on the client side.
## Simple Example
Here's a complete example with one tool:
### Step 1: Define the Tool
```typescript
const getTimeSchema = {
name: "get_time",
type: "function",
description: "Get the current time",
parameters: {
type: "object",
properties: {},
required: [],
},
};
```
Tool schemas are OpenAI compatible. See the [OpenAI Function Calling guide](https://platform.openai.com/docs/guides/function-calling) for more details on defining functions.
### Step 2: Implement the Function
```typescript
import { type ClientTool } from "@outspeed/client";
// params & context can be skipped here since this tool doesn't use them
const getTime: ClientTool<{}> = (params, context) => {
return new Date().toLocaleTimeString();
};
```
The return value from your function is sent directly to the AI model, which uses it to generate its response to the user.
**Always return a value** - even for action-based tools:
* **Data tools** (weather, calculations): Return the actual data
* **Action tools** (generate image, open browser): Return acknowledgment like "Image generated successfully" or "Browser tab opened"
* **On failure**: Return error description like "Failed to generate image: rate limit error"
This tells the AI whether your tool succeeded or failed. See the [implementation section](#implementation) for more details.
### Step 3: Configure Session
```typescript
const sessionConfig = {
// rest of config...
tools: [getTimeSchema],
};
const conversation = useConversation({
clientTools: {
get_time: getTime,
},
});
```
That's it! When the user asks "What time is it?", the agent will:
1. Call your `getTime()` function
2. Receive the return value (e.g., "2:30:45 PM")
3. Use that information to respond to the user
## Advanced Example with Context
Here's a more complex tool with typed parameters and context usage:
```typescript
import { type ClientTool } from "@outspeed/client";
const setTimer: ClientTool<{ time: number; prompt: string }> = ({ time, prompt }, context) => {
setTimeout(() => {
// show a toast if you want to
// toast.info("Timer completed!");
// we let the model know that the timer is done so that it can respond to the user
context.sendText(
`🔔 TIMER ALERT: The timer you set ${time} seconds ago has finished.
Timer prompt: "${prompt}"
Completed at: ${new Date().toLocaleString()}
This is an automated system notification. Please proceed with any actions related to this timer.`,
);
}, time * 1000);
// we set the timer and let the model know that the timer is set
return "Timer set";
};
```
The `context` parameter provides access to conversation methods like `sendText()` for sending messages back to the AI after your tool completes.
## Tool Schema Format
Tool schemas are OpenAI compatible. See the [OpenAI Function Calling guide](https://platform.openai.com/docs/guides/function-calling) for more details on defining functions.
```typescript
{
name: string, // Unique tool identifier
type: "function", // Always "function" for client tools
description: string, // Clear description for the AI
parameters: {
type: "object",
properties: {
[paramName]: {
type: string, // "string", "number", "boolean", "array", "object"
description: string // Parameter description
}
},
required: string[] // Required parameter names
}
}
```
## Best Practices
### Tool Design
* **Clear descriptions**: Help the AI understand when and how to use each tool
* **Specific parameters**: Define precise parameter types and descriptions
* **Single purpose**: Each tool should do one thing well
* **Predictable naming**: Use descriptive, consistent naming conventions
### Implementation
**Always return meaningful values** - the AI uses your return value to respond to the user:
```typescript
import { type ClientTool } from "@outspeed/client";
// ✅ Good: Return actual data
const getWeather: ClientTool<{ city: string }> = ({ city }, context) => {
return "72°F and sunny in San Francisco";
};
// ✅ Good: Return success confirmation
const sendEmail: ClientTool<{ to: string; subject: string }> = ({ to, subject }, context) => {
// ... send email logic
return `Email sent to ${to}`;
};
// ✅ Good: Return error details
const uploadFile: ClientTool<{ filename: string }> = ({ filename }, context) => {
try {
// ... upload logic
return "File uploaded successfully";
} catch (error) {
return `Upload failed: ${error.message}`;
}
};
// ❌ Bad: Don't return undefined/null
const badTool: ClientTool<{}> = (params, context) => {
// The AI gets nothing to work with
return null;
};
```
**Handle errors gracefully**:
```typescript
const robustTool: ClientTool<{ param: string }> = ({ param }, context) => {
try {
if (!param?.trim()) {
return "Parameter is required";
}
const result = performOperation(param);
// notice that we're returning something that model can use to respond to the user
return result || "Operation completed but no data returned";
} catch (error) {
return `Error: ${error.message}`; // for the model to understand the error
}
};
```
**Use async/await for API calls**:
```typescript
const fetchData: ClientTool<{ query: string }> = async ({ query }, context) => {
try {
const response = await fetch(`/api/search?q=${query}`);
const data = await response.json();
return `Found ${data.results.length} results for "${query}"`;
} catch (error) {
return "Search service unavailable"; // again, for the model to understand what went wrong
}
};
```
### Performance
* **Cache results**: Cache API responses when appropriate
* **Timeout handling**: Set reasonable timeouts for external calls
* **Rate limiting**: Respect API rate limits
* **Graceful degradation**: Provide fallbacks when tools fail
## Error Handling
```typescript
export async function robustToolFunction({ param }: { param: string }) {
try {
// Validate input
if (!param || param.trim() === "") {
return "Parameter is required";
}
// Perform operation
const result = await someApiCall(param);
// Validate result
if (!result) {
return "No data available";
}
return result;
} catch (error) {
console.error("Tool error:", error);
// Return user-friendly error message
if (error instanceof Error) {
return `Error: ${error.message}`;
}
return "An unexpected error occurred";
}
}
```
## Combining with System Tools
You can use client tools alongside system tools:
```typescript
const sessionConfig = {
// rest of config...
tools: [getTimeSchema],
system_tools: [
{ name: "end_call", enabled: true },
{ name: "skip_turn", enabled: true },
],
};
```
# Context Management
Source: https://docs.outspeed.com/features/context-management
Manage conversation context during voice interactions
## Overview
Context updates allow you to dynamically add information to the conversation's context during voice interactions. This helps provide the AI agent with additional context about the user, conversation state, or UI state.
## Automatic Context Updates
During voice conversations, several types of items are automatically added to the context:
* **User speech**: Everything the user says is automatically transcribed and added as message items
* **Agent responses**: All agent responses are added to maintain conversation flow
* **Function calls**: When agents call tools, the function call details are automatically added
* **Function outputs**: Tool results are automatically added by the SDK when tools complete
## Manual Context Updates
Adding context doesn't trigger an agent response.
You can manually add items to the conversation context using the `conversation.item.create` event:
```typescript
import { useConversation } from "@outspeed/react";
// Inside a React component (useConversation is a React hook)
export default function MyComponent() {
const conversation = useConversation({});
const addToContext = async (text: string) => {
const event = {
type: "conversation.item.create",
item: {
type: "message",
role: "user",
content: [
{
type: "input_text",
text,
},
],
},
};
await conversation.send(JSON.stringify(event));
};
// rest of component...
}
```
## Event Notifications
The server sends events for all context operations:
```typescript
// Item added (automatic or manual)
conversation.on("conversation.item.created", (event) => {
console.log("Item added:", event.item);
});
// Item retrieved
conversation.on("conversation.item.retrieved", (event) => {
console.log("Item retrieved:", event.item);
});
// Item deleted
conversation.on("conversation.item.deleted", (event) => {
console.log("Item deleted:", event.item_id);
});
```
## Retrieving Context Items
You can retrieve a specific item from the conversation context by its ID:
```typescript
const retrieveEvent = {
type: "conversation.item.retrieve",
item_id: "item_12345",
};
await conversation.send(JSON.stringify(retrieveEvent));
// Listen for the response
conversation.on("conversation.item.retrieved", (event) => {
console.log("Retrieved item:", event.item);
});
```
## Deleting Context Items
You can remove items from the conversation context:
```typescript
const deleteEvent = {
type: "conversation.item.delete",
item_id: "item_12345",
};
await conversation.send(JSON.stringify(deleteEvent));
// Listen for confirmation
conversation.on("conversation.item.deleted", (event) => {
console.log("Deleted item ID:", event.item_id);
});
```
## Item Structure
Context items follow this structure:
| Field | Type | Description |
| --------- | ----------------- | ---------------------------------------------------------------- |
| `type` | string | Item type: "message", "function\_call", "function\_call\_output" |
| `role` | string | Message role: "user", "assistant", "system" |
| `content` | array | Message content with type and text |
| `status` | string (optional) | Item status: "in\_progress", "completed" |
## Context Item Types
### Message Items
Standard conversation messages from users, assistants, or system
### Function Call Items
Tool invocations with parameters (automatically managed)
### Function Output Items
Results from tool executions (automatically managed)
The context helps the AI agent maintain awareness of the conversation history and make more informed responses based on the full context of the interaction.
# End Call
Source: https://docs.outspeed.com/features/end-session
End Call is a system tool that enables AI agents to gracefully terminate voice conversations when users explicitly indicate they want to end the call.
## Overview
The End Call system tool helps agents handle situations where users are:
* Saying goodbye or farewell
* Clearly indicating they want to end the call
* Using phrases like "bye", "goodbye", "talk to you later", "I'm hanging up"
* Concluding the conversation with clear intent to leave
The agent uses conversation context to distinguish between genuine farewell intentions and brief acknowledgments that don't require ending the session.
## Configuration
Enable `end_call` system tool in your session configuration:
```javascript
const sessionConfig = {
// rest of config...
system_tools: [{ name: "end_call", enabled: true }],
};
```
## Tool parameters
| Parameter | Type | Required | Description |
| ------------------ | ------ | -------- | ---------------------------------------------------------------------------- |
| `farewell_message` | string | Yes | Final message to say before ending the call |
| `reason` | string | Yes | Why the call is ending (e.g., 'user said goodbye', 'conversation concluded') |
## Override tool descriptions
You can override the default tool description and parameter descriptions to customize the agent's behavior:
```javascript
const sessionConfig = {
// rest of config...
system_tools: [
{
name: "end_call",
description: "",
parameter_descriptions: {
farewell_message: "",
reason: "",
},
},
],
};
```
## When end call is used
The agent automatically calls this tool when it detects:
Clear goodbye scenarios like:
* **Direct goodbye**: User says "goodbye", "bye", "farewell"
* **Conversation conclusion**: User says "talk to you later", "see you soon"
* **Explicit end request**: User says "I'm hanging up", "I need to go"
* **Formal conclusion**: User indicates the conversation is finished
### Example situations
```json
// User says goodbye
Agent would call: end_call({
farewell_message: "Goodbye! Have a great day!",
reason: "user said goodbye"
})
// User says "I need to go"
Agent would call: end_call({
farewell_message: "Of course! Take care!",
reason: "user requested to end"
})
// User says "thanks" (should NOT end call)
Agent continues conversation instead of ending
// User says "talk to you later"
Agent would call: end_call({
farewell_message: "Talk to you later! Goodbye!",
reason: "conversation concluded"
})
```
## Default implementation
Here's how the End Call tool is defined on our server:
```python
tool_def = FunctionDefinition(
name="end_call",
description="End the conversation ONLY when the user explicitly says goodbye, farewell, or clearly indicates they want to end the call (e.g., 'bye', 'goodbye', 'talk to you later', 'I'm hanging up'). Do NOT end the call for brief responses like 'thanks', 'okay', 'got it', or simple acknowledgments - just return to listening mode instead.",
parameters=FunctionParameters(
type="object",
properties={
"farewell_message": FunctionProperty(
type="string",
description="Final message to say before ending the call",
),
"reason": FunctionProperty(
type="string",
description="Why the call is ending (e.g., 'user said goodbye', 'conversation concluded', 'user requested to end')",
),
},
required=["farewell_message", "reason"],
),
)
```
## Troubleshooting
If you notice the agent ending sessions when it shouldn't:
1. **Review Context**: Check if the conversation context provides clear goodbye signals
2. **Adjust Instructions**: Refine your system prompt to distinguish between acknowledgments and farewells
3. **Override description**: You can override the tool description to be more or less restrictive
# Input Language
Source: https://docs.outspeed.com/features/input-language
Configure the language users can speak in during voice conversations
## Overview
The `input_language` field in your session configuration provides a **hint** about the primary language users will speak.
However, the model can understand multiple languages simultaneously, regardless of the hint provided.
The AI agent will understand their speech and respond in the language corresponding to the chosen voice.
The response language depends on your chosen voice. See [available voices](/features/voices) for language support. The
`input_language` setting is a hint to optimize recognition, but users can speak in any supported language.
## Supported Languages
Currently supported input language hints:
* **en** - English (default)
* **zh** - Chinese (Mandarin)
* **hi** - Hindi
* **de** - German
* **es** - Spanish
* **fr** - French
* **it** - Italian
* **ja** - Japanese
* **ko** - Korean
* **nl** - Dutch
* **pl** - Polish
* **pt** - Portuguese
* **ru** - Russian
* **sv** - Swedish
* **tr** - Turkish
## Configuration
Add `input_language` to your session configuration as a hint for the primary expected language:
```typescript
const sessionConfig = {
model: "outspeed-v1",
instructions: "You are a helpful assistant.",
voice: "sophie",
input_language: "zh", // Hint: Primary language is Chinese, but users can still switch to English mid-conversation
turn_detection: {
type: "semantic_vad",
},
first_message: "Hello! How can I help you today?",
};
```
## Language Examples
### English (Default)
```typescript
const sessionConfig = {
// rest of config...
input_language: "en", // or omit this field entirely
};
// User speaks: "What's the weather like?"
// Agent responds: "The weather is sunny and 72°F." (English voice responds in English)
```
### Chinese (Mandarin) with Multilingual Support
```typescript
const sessionConfig = {
// rest of config...
input_language: "zh", // Hint: Primary language is Chinese, but users can still switch to English mid-conversation
};
// User speaks: "今天天气怎么样?" (How's the weather today?)
// Agent responds: "The weather is sunny and 72°F." (English voice responds in English)
// User can also mix languages:
// User speaks: "今天 how are you?"
// Agent responds: "Hello! I'm doing well, thank you for asking." (English voice responds in English)
```
### Hindi with Multilingual Support
```typescript
const sessionConfig = {
// rest of config...
voice: "apoorva", // Hindi voice
input_language: "hi", // Hint for Hindi, but users can mix languages
};
// User speaks: "आज मौसम कैसा है?" (How's the weather today?)
// Agent responds: "मौसम धूप है और 72°F है।" (Hindi voice responds in Hindi)
// User can also mix languages:
// User speaks: "हेलो, सब बढ़िया? Everything is fine?"
// Agent responds: "हैलो! हाँ, सब कुछ ठीक है। मैं आपकी कैसे मदद कर सकती हूँ?" (Hindi voice responds in Hindi)
```
## Language Detection
The system automatically detects which languages are spoken in each user input and includes this information in the transcription event:
```json
{
"event_id": "bf03fb6b-1160-411d-b332-f2a97b743d2c",
"type": "conversation.item.input_audio_transcription.completed",
"server_sent": true,
"item_id": "item_5c49066c55c549d29132a467d01e8e3b",
"transcript": "हेलो, सब बढ़िया? Everything is fine?",
"languages": ["hi", "en"],
"content_index": 0
}
```
The `languages` array contains all detected languages in the user's input, allowing you to understand the linguistic composition of each utterance.
You can listen to this event to understand the languages spoken in the user's input.
```typescript
conversation.on("conversation.item.input_audio_transcription.completed", (event) => {
console.log("Languages spoken:", event.languages);
});
```
## Important Notes
* **Hint, not restriction**: `input_language` is a hint to optimize recognition, not a limitation
* **Multilingual support**: Users can speak multiple languages in a single session or even single utterance
* **Voice-dependent output**: The AI agent responds in the language of the chosen voice (see [available voices](/features/voices))
* **Default behavior**: Default input language value is `en` for English
* **Language detection**: Each transcription includes detected languages in the `languages` array
# Pronunciation Rules
Source: https://docs.outspeed.com/features/pronunciation
Customize how the AI pronounces specific words and terms
## Overview
Custom pronunciation rules allow you to control how the AI pronounces specific words, acronyms, and technical terms in its speech output. This is an optional feature.
## Configuration
Add `custom_pronunciation_rules` to your session configuration:
```typescript
const sessionConfig = {
model: "outspeed-v1",
instructions: "You are a helpful assistant.",
voice: "sophie",
custom_pronunciation_rules: [
{
pattern: "Node.js",
pronunciation: "Node JS",
},
{
pattern: "React.js",
pronunciation: "React JS",
},
{
pattern: "API",
pronunciation: "A P I",
},
],
// rest of config...
};
```
## Common Examples
```typescript
custom_pronunciation_rules: [
{ pattern: "AWS", pronunciation: "A W S" },
{ pattern: "JSON", pronunciation: "Jason" },
{ pattern: "API", pronunciation: "A P I" },
{ pattern: "Node.js", pronunciation: "Node JS" },
{ pattern: "React.js", pronunciation: "React JS" },
];
```
# Resume Session
Source: https://docs.outspeed.com/features/resume-session
Continue voice conversations from where they left off
## Overview
Session resumption allows you to continue voice conversations from where they left off, even after page refreshes or application restarts. This provides a seamless user experience for ongoing conversations.
## How It Works
To resume a session, pass the `resume_session` query parameter when requesting an ephemeral key from your backend.
### Backend Implementation
Update your `/token` endpoint to handle session resumption:
```typescript
app.post("/token", async (req, res) => {
const { resume_session } = req.query; // resume_session is the ID of the session to resume
let url = "https://api.outspeed.com/v1/realtime/sessions";
if (resume_session) {
url += `?resume_session=${resume_session}`;
}
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(req.body),
});
const data = await response.json();
res.json(data);
});
```
### Frontend Usage
Request an ephemeral key with the session ID to resume:
```typescript
const getEphemeralKeyFromServer = async (config: SessionConfig, sessionId?: string) => {
let url = `${SERVER_URL}/token`;
if (sessionId) {
url += `?resume_session=${sessionId}`;
}
const tokenResponse = await fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(config),
});
const data = await tokenResponse.json();
return data.client_secret.value;
};
// Resume existing session
const ephemeralKey = await getEphemeralKeyFromServer(sessionConfig, "session_123");
await conversation.startSession(ephemeralKey);
```
## Context Restoration
When a session is resumed, the context is restored and you'll receive a `context.restored` event:
```typescript
conversation.on("context.restored", (event) => {
console.log("Context restored with items:", event.items);
// Handle restored conversation items
event.items.forEach(item => {
console.log(`${item.role}: ${item.content}`);
});
});
```
## Use Cases
* **Page refreshes**: Continue conversations after accidental refreshes
* **Application restarts**: Resume conversations across app sessions
* **Multi-device**: Continue conversations on different devices
# Send Text Messages
Source: https://docs.outspeed.com/features/send-text
Send text messages to the AI agent programmatically
## Overview
You can send text messages to the AI agent programmatically, either by sending events directly or using the convenient `sendText` method.
## Using sendText Method
The React SDK provides a convenient `sendText` method that handles both events for you:
```typescript
import { useConversation } from "@outspeed/react";
// inside a React component
const conversation = useConversation({});
// Send text and get AI response
await conversation.sendText("What's the weather like?");
// Cancel ongoing response and send new text
await conversation.sendText("What's the weather like?", { cancelOngoing: true });
```
`sendText` will generate a response only if the model is currently **not** generating anything.
## Options
The `sendText` method accepts an optional second parameter with these options:
| Option | Type | Default | Description |
| --------------- | ------- | ------- | ------------------------------------------------------------------ |
| `cancelOngoing` | boolean | `false` | Cancel any ongoing response generation before sending the new text |
### Cancel Ongoing Responses
When the AI is generating a response, you can cancel it and send a new message:
```typescript
// This will cancel any ongoing response and send the new text
await conversation.sendText("Actually, tell me about the weather in Tokyo instead", {
cancelOngoing: true,
});
```
## Using Events
To send a text message, you need to send two events:
```typescript
// 1. Add text message to conversation context
const messageEvent = {
type: "conversation.item.create",
item: {
type: "message",
role: "user",
content: [
{
type: "input_text",
text: "What's the weather like?",
},
],
},
};
await conversation.send(JSON.stringify(messageEvent));
// 2. Request AI response
const responseEvent = {
type: "response.create",
};
await conversation.send(JSON.stringify(responseEvent));
```
## Example Usage
```typescript
function ChatInput() {
const [message, setMessage] = useState("");
const conversation = useConversation({});
// add the logic to start & end a conversation session
const handleSubmit = async (e: FormEvent) => {
e.preventDefault();
if (!message.trim()) return;
try {
await conversation.sendText(message);
setMessage(""); // Clear input after sending
} catch (error) {
console.error("Failed to send message:", error);
}
};
return (
);
}
```
## Important Notes
### Concurrent sendText Calls
By default, `sendText` will generate a response only if the model is currently not generating anything.
If you call `sendText` while the AI is responding, you have two options:
**Option 1: Handle the error (default behavior)**
```typescript
const conversation = useConversation({});
conversation.on("error", (event) => {
if (event.error.code === "conversation_already_has_active_response") {
console.log("AI is currently responding, please wait");
// Show user feedback or queue the message
}
});
```
**Option 2: Cancel ongoing response**
```typescript
// This will cancel the current response and send the new message
await conversation.sendText("New message", { cancelOngoing: true });
```
### Error Response
If you don't use `cancelOngoing: true`, simultaneous calls will fail with this error:
```json
{
"event_id": "b5ea58f4-8dfe-4590-959c-0c85ef108b9b",
"type": "error",
"server_sent": true,
"created_at_": 1756734784.1917815,
"error": {
"type": "invalid_request_error",
"code": "conversation_already_has_active_response",
"message": "Conversation already has an active response",
"param": null,
"event_id": null
}
}
```
# Skip Turn
Source: https://docs.outspeed.com/features/skip-turn
Skip Turn is a system tool that enables AI agents to intelligently decide when to skip their speaking turn based on conversation context. The agent uses this tool when users are not directly addressing them or when no response is needed.
## Overview
The Skip Turn system tool helps agents handle situations where users are:
* Singing songs
* Talking to someone else
* Having background conversations
* Talking to themselves
* Saying something unrelated to the conversation
The agent uses conversation context to make intelligent decisions about when to skip turns, creating more natural interaction patterns.
## Configuration
Enable Skip Turn in your session configuration:
```javascript
const sessionConfig = {
// rest of config...
system_tools: [{ name: "skip_turn", enabled: true }],
};
```
## Tool parameters
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | --------------------------------------------------- |
| `reason` | string | Yes | Brief explanation for why the turn is being skipped |
## Override tool descriptions
You can override the default tool description and parameter descriptions to customize the agent's behavior:
```javascript
const sessionConfig = {
// rest of config...
system_tools: [
{
name: "skip_turn",
description: "",
parameter_descriptions: {
reason: "",
},
},
],
};
```
## When skip turn is used
The agent automatically calls this tool when it detects:
### Direct scenarios
* **Singing**: User is singing a song
* **Third-party conversation**: User is talking to someone else in the room
* **Background chatter**: Casual conversation not directed at the agent
* **Self-talk**: User talking to themselves
* **Unrelated content**: User saying something unrelated to the current conversation
### Example situations
```json
// User singing
Agent would call: skip_turn({ reason: "singing song" })
// User talking to someone else
Agent would call: skip_turn({ reason: "talking to someone else" })
// Background conversation
Agent would call: skip_turn({ reason: "background conversation" })
// User talking to themselves
Agent would call: skip_turn({ reason: "not addressing bot" })
```
## Default implementation
Here's how the Skip Turn tool is defined on our server:
```python
tool_def = FunctionDefinition(
name="skip_turn",
description="Call this when the user is not directly addressing you or when no response is needed. Skip if the user is: singing, talking to someone else, having background conversation, talking to themselves, or saying something unrelated to our conversation. Use conversation context to decide. If genuinely unsure whether they're addressing you, ask for clarification instead of skipping.",
parameters=FunctionParameters(
type="object",
properties={
"reason": FunctionProperty(
type="string",
description="Brief reason for skipping this turn (e.g., 'singing song', 'talking to someone else', 'background conversation', 'not addressing bot')",
),
},
required=["reason"],
),
)
```
## Troubleshooting
If you notice the agent skipping turns when it shouldn't:
1. **Review Context**: Check if the conversation context is clear enough
2. **Adjust Instructions**: Refine your system prompt to provide better guidance
3. **Override description**: You can override the tool description as per your need
# Speak
Source: https://docs.outspeed.com/features/speak
Make the AI agent speak specific text directly
## Overview
The `speak` method allows you to make the AI agent speak specific text directly, without any processing. It's basically Text-to-Speech.
## Using speak Method
The React SDK provides a `speak` method on the conversation object:
```typescript
import { useConversation } from "@outspeed/react";
// Inside a React component
const conversation = useConversation({});
// Make the agent speak specific text
await conversation.speak("Hey there, I'm Perry from Outspeed! How can I help you today?");
```
The text will be added to the context as a message item.
## Key Differences
| Method | Purpose | Behavior |
| ------------ | -------------------- | ---------------------------------------------------- |
| `sendText()` | Send user message | AI processes the message and generates a response |
| `speak()` | Direct speech output | AI speaks the exact text provided without processing |
## Example Usage
```typescript
function VoiceAnnouncements() {
const conversation = useConversation({});
const makeAnnouncement = async (text: string) => {
try {
await conversation.speak(text);
} catch (error) {
console.error("Failed to speak:", error);
}
};
return (
);
}
```
## Common Use Cases
* **System announcements**: Status updates or notifications
* **Predefined responses**: Quick replies without AI processing
* **Error messages**: Speaking error states directly
* **Welcome messages**: Greeting users with specific text
* **Instructions**: Providing step-by-step guidance
## Complete Example
```typescript
function CustomerService() {
const [isOnHold, setIsOnHold] = useState(false);
const conversation = useConversation({});
const putOnHold = async () => {
setIsOnHold(true);
await conversation.speak("Please hold while I transfer you to a specialist. This may take a few moments.");
};
const removeFromHold = async () => {
setIsOnHold(false);
await conversation.speak("Thank you for holding. How can I assist you today?");
};
const endCall = async () => {
await conversation.speak("Thank you for contacting us. Have a great day!");
await conversation.endSession();
};
return (
{isOnHold ? (
) : (
)}
);
}
```
# Speak First
Source: https://docs.outspeed.com/features/speak-first
Control whether the agent initiates conversation or waits for user input
## Overview
The `first_message` field allows your AI agent to start the conversation automatically with a predefined greeting, instead of waiting for the user to speak first.
## Basic Usage
Add `first_message` to your session configuration:
```javascript
const sessionConfig = {
// rest of config...
first_message: "Hello! How can I help you today?",
};
```
When the session starts, the agent will immediately speak this message to greet the user.
## Examples
### Professional Assistant
```javascript
first_message: "Good morning! I'm your AI assistant. What can I help you with today?"
```
### Customer Support
```javascript
first_message: "Hi there! Welcome to our support chat. How can I assist you?"
```
### Interview Bot
```javascript
first_message: "Hello! I'm excited to conduct your interview today. Shall we get started?"
```
### Casual Companion
```javascript
first_message: "Hey! Great to see you. What's on your mind?"
```
## Best Practices
* **Keep it brief**: Short, welcoming messages work best
* **Match your brand**: Use language that fits your application's tone
* **Stay consistent**: Use similar greetings across sessions
## Optional Field
The `first_message` field is optional. If you don't include it, the session will start in listening mode, waiting for the user to speak first.
# System Tools
Source: https://docs.outspeed.com/features/system-tools
Pre-built tools that run on Outspeed's servers for conversation management
## Overview
System tools are pre-built tools that run on Outspeed's servers. They handle common conversation management tasks and require no implementation on your part - simply enable them in your session configuration.
System tools help agents manage conversation flow intelligently:
* **Server-side execution**: No client-side implementation needed
* **Automatic decisions**: Agents call tools based on conversation context
* **Override capabilities**: Customize tool behavior with custom descriptions
* **Zero maintenance**: Tools are maintained and updated by Outspeed
## Available System Tools
### Skip turn
Allows agents to intelligently skip their speaking turn when users aren't directly addressing them.
**Use Cases:**
* User is singing or humming
* Background conversations
* User talking to someone else
* Self-talk or thinking aloud
👉 **[Learn about Skip Turn →](skip-turn)**
### End session
Enables agents to gracefully terminate conversations when users explicitly say goodbye.
**Use Cases:**
* User says "goodbye", "bye", "farewell"
* Clear conversation conclusion
* Explicit requests to end the call
👉 **[Learn about End Session →](end-session)**
## Basic sonfiguration
Enable system tools in your session configuration:
```javascript
const sessionConfig = {
// rest of config...
system_tools: [
{ name: "skip_turn", enabled: true },
{ name: "end_call", enabled: true }
],
};
```
## Override tool descriptions
You can customize how system tools behave by overriding their descriptions:
```javascript
const sessionConfig = {
// rest of config...
system_tools: [
{
name: "skip_turn",
enabled: true,
description: "Skip turns more conservatively - only when user is clearly singing or talking to others",
parameter_descriptions: {
reason: "Specific reason for skipping (singing or talking to others only)",
},
},
{
name: "end_call",
enabled: true,
description: "End calls only on very explicit goodbye phrases like 'goodbye' or 'bye'",
parameter_descriptions: {
farewell_message: "Brief, professional farewell message",
reason: "Specific goodbye phrase that triggered the end",
},
}
],
};
```
## Best practices
### When to enable
* **Skip Turn**: Recommended for most applications to handle natural conversation flow
* **End Session**: Essential for applications where users initiate call termination
### Override examples
**More Restrictive Skip Turn:**
```javascript
{
name: "skip_turn",
description: "Only skip when absolutely certain user is not addressing the agent",
parameter_descriptions: {
reason: "Very specific reason with high confidence",
},
}
```
**More Permissive End Session:**
```javascript
{
name: "end_call",
description: "End calls when users show any sign of wanting to conclude",
parameter_descriptions: {
farewell_message: "Warm, understanding farewell",
reason: "Any indication user wants to end conversation",
},
}
```
## Troubleshooting
### Tools not working
1. **Check configuration**: Ensure tools are properly enabled
2. **Verify session**: Confirm tools are included in session config
3. **Review logs**: Check for error messages in session events
### Unexpected behavior
1. **Monitor context**: Review conversation context when tools trigger
2. **Adjust descriptions**: Override tool descriptions to fine-tune behavior
3. **System prompt**: Ensure your system prompt doesn't conflict with tool usage
### Performance impact
* System tools have minimal performance impact
* No client-side processing required
* Server-side execution is optimized and fast
# Tool Use
Source: https://docs.outspeed.com/features/tool-use
Enable AI agents to use external tools and APIs during conversations
## Overview
Tools extend your AI agents with external capabilities, allowing them to interact with APIs, perform actions, and access real-time information during voice conversations.
## Tool Types
Custom functions you define and implement in your application.
Pre-built tools that like `skip-turn` and `end-call` that run on Outspeed's servers.
## Examples
See real implementations of tools in action in our [Examples Repository](https://github.com/outspeed-ai/outspeed-examples). The repository includes working examples with:
* **Weather tool integration** (client tool)
* **Skip turn and end call** (system tools)
* **Multiple frameworks**: React, Next.js, Swift iOS, and more
## Getting Started
1. **Choose your tools**: Decide which client tools you need and which system tools to enable
2. **Implement client tools**: Write functions for your custom tools
3. **Configure session**: Add tools to your session configuration
# Voice Cloning
Source: https://docs.outspeed.com/features/voice-cloning
Create custom AI voices from audio samples
## Overview
Voice cloning allows you to create highly realistic custom AI voices from your own audio samples. This enables a personalized and branded voice experience for your applications.
1. **Upload Audio**: Provide clear audio samples of the voice you want to clone
2. **Generate Voice**: Outspeed's AI processes the samples to create a new voice
3. **Use in Apps**: Integrate your custom voice into your agents and TTS requests
Cloned voices are currently only available for use with our [Text-to-Speech (TTS) API](/api-spec/tts).
## Getting Started
1. **Visit Dashboard**: Go to the [Voice Upload page](https://dashboard.outspeed.com/tts/upload-voice) in your Outspeed Dashboard
2. **Upload Samples**: Follow the instructions to upload your audio files
3. **Create Voice**: Once processed, your new voice will be available to use
4. **Copy Voice ID**: Copy the generated Voice ID for use in your code
## Audio Requirements
* **Length**: Minimum 10 seconds of clear speech
* **Size**: Maximum 3 MB
* **Quality**: High-quality audio (e.g., studio recording, no background noise)
* **Format**: Only WAV files are supported
* **Content**: Natural speech, varied tone and pitch
**Do not upload copyrighted material or impersonate individuals without consent.** Ensure you have the necessary
rights for all uploaded audio.
## Usage
Once you have your custom Voice ID, use it with our [Text-to-Speech (HTTP)](/api-spec/tts):
### In TTS API Requests
```json
{
"model": "outspeed-tts-v2",
"voice": "your-custom-voice-id",
"text": "This is my custom voice."
}
```
### In Dialogue Generation
```json
{
"model": "outspeed-tts-v2",
"text": "*Narrator:* This is my custom voice. Character: Hello!",
"speaker_voice": "your-custom-voice-id",
"narrator_voice": "another-custom-voice-id"
}
```
## Best Practices
* **High-quality audio**: Crucial for best results
* **Consent**: Always obtain consent if cloning a real person's voice
* **Monitor usage**: Track how your custom voices perform in real applications
# Voices
Source: https://docs.outspeed.com/features/voices
Available voices for Outspeed voice agents
## Overview
## Standard Voices
* viktoria `de`
* lena `de`
* thomas `de`
* lukas `de`
* sophie `en`
* savannah `en`
* brooke `en`
* zia `en`
* corinne `en`
* david `en`
* griffin `en`
* carson `en`
* ethan `en`
* daniela `es`
* elena `es`
* pedro `es`
* luis `es`
* camille `fr`
* marie `fr`
* antoine `fr`
* philippe `fr`
* apoorva `hi`
* aarti `hi`
* ishan `hi`
* neeraj `hi`
* liv `it`
* francesca `it`
* lucio `it`
* luca `it`
* sayuri `ja`
* yumiko `ja`
* daisuke `ja`
* akira `ja`
* jihyun `ko`
* mimi `ko`
* jaechul `ko`
* byungtae `ko`
* ana\_paula `pt`
* luana `pt`
* felipe `pt`
* camilo `pt`
* tatiana `ru`
* irina `ru`
* sergei `ru`
* nikolai `ru`
* chen `zh`
* yue `zh`
* liu `zh`
* kai `zh`
* sanne `nl`
* daan `nl`
* lucas\_dutch `nl`
* bram `nl`
* zofia `pl`
* katarzyna `pl`
* tomek `pl`
* wojciech `pl`
* ingrid `sv`
* freja `sv`
* anders `sv`
* cees `sv`
* leyla `tr`
* aylin `tr`
* emre `tr`
* taylan `tr`
## Beta Voices
Beta voices are still being improved.
* clark `en`
* linda `en`
* isabella `en`
* emily `en`
* aria `en`
* jasmine `en`
* kate `en`
* siren `en`
* sienna `en`
* claudia `en`
* dominique `en`
* hannah `en`
* chloe `en`
* chloe-whisper `en`
## Usage
Use these voice names in your session configuration:
#### JavaScript
```javascript
const sessionConfig = {
// rest of config...
voice: "sophie", // Replace with any voice name from the list above
};
```
#### Swift
```swift
let ttsConfig = OutspeedSDK.TTSConfig(voiceId: "sophie")
```
# Authentication
Source: https://docs.outspeed.com/get-started/authentication
Secure your voice agents with authentication and authorization
## Overview
Outspeed provides flexible authentication options to balance ease of development with production security needs.
## Authentication Modes
### Development Mode (No Auth)
Perfect for prototyping and development - no backend required.
1. **Create agent** in the [Outspeed Dashboard](https://dashboard.outspeed.com/agents)
2. **Disable authentication** in agent settings
3. **Use directly** from your client app
```typescript
await conversation.startSession({
agentId: "your-agent-id",
source: "my-app",
});
```
### Production Mode (Auth Enabled)
Recommended for production apps - requires backend for security.
1. **Enable authentication** in agent settings
2. **Set up backend** to [generate ephemeral keys](#backend-implementation)
3. **Pass ephemeral key** when starting sessions
```typescript
const ephemeralKey = await getEphemeralKeyFromServer("your-agent-id");
await conversation.startSession({
agentId: "your-agent-id",
source: "my-app",
ephemeralKey: ephemeralKey,
});
```
## Dashboard Configuration
### Creating an Agent
1. Go to [Outspeed Dashboard](https://dashboard.outspeed.com)
2. Click **Create Agent**
3. Configure your agent:
* **Name**: Give your agent a name
* **Voice**: Choose from available voices
4. Once created, you can configure other settings like system instructions, first message, etc. in the dashboard.
### Authentication Settings
In your agent settings, you can:
* **Toggle Authentication**: Enable/disable auth requirements
* **Set Allowed Hostnames**: Restrict which domains can use your agent
* **View Agent ID**: Copy the ID to use in your code
### Hostname Security
Add allowed hostnames to restrict where your agent can be used:
* `example.com` - Allow only this exact domain
* `app.domain.com` - Allow specific subdomain
We do not support wildcard domains yet.
Sessions will only start if the request origin matches your allowlist. Leave empty to allow any origin.
## Backend Implementation
### Simple Agent-Based Auth
For agents created in the dashboard, you only need the agent ID:
```javascript Express.js
app.use(express.json());
app.post("/token", async (req, res) => {
try {
const { agentId } = req.body;
const response = await fetch(`https://api.outspeed.com/v1/realtime/sessions?agent_id=${agentId}`, {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`,
"Content-Type": "application/json",
},
});
if (!response.ok) {
const error = await response.text();
console.error("Failed to generate ephemeral key:", error);
res.status(response.status).json({ error: "Failed to generate token" });
return;
}
const data = await response.json();
res.json(data);
} catch (error) {
console.error("Failed to generate ephemeral key:", error);
res.status(500).json({ error: "Internal server error" });
}
});
```
```python FastAPI
import os
import httpx
from fastapi import FastAPI, HTTPException
OUTSPEED_API_KEY = os.getenv("OUTSPEED_API_KEY")
app = FastAPI()
@app.post("/token")
async def create_token(request: dict):
try:
agent_id = request.get("agentId")
async with httpx.AsyncClient() as client:
response = await client.post(
f"https://api.outspeed.com/v1/realtime/sessions?agent_id={agent_id}",
headers={
"Authorization": f"Bearer {OUTSPEED_API_KEY}",
"Content-Type": "application/json",
},
)
if not response.is_success:
print("Error generating ephemeral key:", response.text)
raise HTTPException(status_code=response.status_code, detail=response.text)
return response.json()
except Exception as e:
print("Error generating ephemeral key:", e)
raise HTTPException(status_code=500, detail="Internal server error")
```
### Frontend Integration
```typescript
const getEphemeralKeyFromServer = async (agentId: string) => {
const tokenResponse = await fetch("/token", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ agentId }),
});
const data = await tokenResponse.json();
if (!tokenResponse.ok) {
throw new Error("Failed to get ephemeral key");
}
return data.client_secret.value;
};
// Usage
const ephemeralKey = await getEphemeralKeyFromServer("your-agent-id");
await conversation.startSession({
agentId: "your-agent-id",
source: "my-app",
ephemeralKey: ephemeralKey,
});
```
## Legacy: Programmatic Configuration
The new dashboard approach is recommended, but you can still configure agents programmatically for advanced use cases.
### Backend with Session Config
```javascript Express.js
app.use(express.json());
app.post("/token", async (req, res) => {
try {
const response = await fetch("https://api.outspeed.com/v1/realtime/sessions", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(req.body),
});
if (!response.ok) {
const error = await response.text();
console.error("failed to generate ephemeral key:", error);
res.status(response.status).json({ error: "Failed to generate token" });
return;
}
const data = await response.json();
res.json(data);
} catch (error) {
console.error("failed to generate ephemeral key:", error);
res.status(500).json({ error: "Internal server error" });
}
});
```
```python FastAPI
import os
import httpx
from fastapi import FastAPI, HTTPException, Request
OUTSPEED_API_KEY = os.getenv("OUTSPEED_API_KEY")
app = FastAPI()
@app.post("/token")
async def create_token(request: Request):
try:
session_config = await request.json()
async with httpx.AsyncClient() as client:
response = await client.post(
"https://api.outspeed.com/v1/realtime/sessions",
headers={
"Authorization": f"Bearer {OUTSPEED_API_KEY}",
"Content-Type": "application/json",
},
json=session_config,
)
if not response.is_success:
print("error generating ephemeral key:", response.text)
raise HTTPException(status_code=response.status_code, detail=response.text)
return response.json()
except Exception as e:
print("error generating ephemeral key:", e)
raise HTTPException(status_code=500, detail="Internal server error")
```
### Frontend with Session Config
```typescript
import { type SessionConfig } from "@outspeed/client";
const getEphemeralKeyFromServer = async (config: SessionConfig) => {
const tokenResponse = await fetch("/token", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(config),
});
const data = await tokenResponse.json();
if (!tokenResponse.ok) {
throw new Error("Failed to get ephemeral key");
}
return data.client_secret.value;
};
const sessionConfig: SessionConfig = {
model: "outspeed-v1",
instructions: "You are a helpful assistant.",
voice: "sophie",
temperature: 0.5,
turn_detection: {
type: "semantic_vad",
},
first_message: "Hello! How can I help you today?",
};
// Usage
const ephemeralKey = await getEphemeralKeyFromServer(sessionConfig);
await conversation.startSession(ephemeralKey, {
source: "my-app",
});
```
## Environment Setup
### Required Environment Variables
```bash
# Your Outspeed API key from the dashboard
OUTSPEED_API_KEY=your_outspeed_api_key_here
```
### Security Best Practices
**Never expose your API key in client-side code**. Always generate ephemeral tokens on your backend server.
* **API Keys**: Keep server-side only, never in frontend code
* **Ephemeral Keys**: Short-lived tokens for client authentication
* **Hostname Allowlist**: Restrict origins that can use your agent
* **HTTPS**: Always use HTTPS in production
* **Rate Limiting**: Implement rate limiting on your token endpoint
## When to Use Each Mode
### Development Mode (No Auth)
* ✅ **Prototyping** and development
* ✅ **Internal tools** with trusted users
* ✅ **Quick demos** and experiments
* ❌ **Production apps** with external users
### Production Mode (Auth Enabled)
* ✅ **Production applications**
* ✅ **Public-facing** voice agents
* ✅ **Hostname restrictions** for security
## Summary
1. **Start simple**: Use dashboard agents with auth disabled for development
2. **Add security**: Enable auth and set up backend for production
3. **Restrict access**: Use hostname allowlist for additional security
Both approaches give you full control - choose based on your security and deployment needs.
# OpenAI Compatibility
Source: https://docs.outspeed.com/get-started/openai-compatibility
Leverage OpenAI Compatibility with Outspeed Live API
Outspeed Live API offers a seamless integration experience by being compatible with OpenAI's Realtime API. This means you can often use similar integration patterns and even reuse existing code.
You can connect to Outspeed Live API in your applications using [WebRTC](https://webrtc.org/). The process mirrors the one described in [OpenAI's Realtime API WebRTC Guide](https://platform.openai.com/docs/guides/realtime#connect-with-webrtc).
The key difference is the API endpoint:
* **Outspeed Live API:** `api.outspeed.com`
* **OpenAI API:** `api.openai.com`
Here's a summary of the integration steps:
### 1. Obtain an Ephemeral Key
First, your backend needs to obtain an ephemeral key from Outspeed. This process is similar to OpenAI's. You'll make a POST request to the Outspeed sessions endpoint.
**Example: Creating a session and obtaining a key**
```javascript
const outspeedApiHost = "api.outspeed.com"; // Or your model provider's URL
const apiKey = "YOUR_OUTSPEED_API_KEY"; // Replace with your actual API key
async function createRealtimeSession(sessionRequestBody) {
const url = `https://${outspeedApiHost}/v1/realtime/sessions`;
console.log(`👉 Creating session using: ${url}`);
try {
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(sessionRequestBody),
});
if (!response.ok) {
const errorText = await response.text();
console.error("Session creation failed:", response.status, errorText);
// Handle error appropriately in your application
throw new Error(`Failed to create session: ${errorText}`);
}
const sessionData = await response.json();
console.log("🎉 Session created successfully!", sessionData);
return sessionData; // This contains the ephemeral_key
} catch (error) {
console.error("Error during session creation:", error);
throw error;
}
}
```
*Remember to replace `YOUR_OUTSPEED_API_KEY` with your actual Outspeed API key.*
### 2. Establish WebRTC Connection on Frontend
Once you have the ephemeral key, use it on your application's frontend to establish a WebRTC connection with Outspeed Live API.
You can find a client-side code snippet demonstrating this in our [Voice Devtools repository on GitHub](https://github.com/outspeed-ai/voice-devtools/blob/main/src/client/helpers/webrtc.ts#L30).
### 3. Reuse OpenAI Event Handlers
A significant advantage of Outspeed's OpenAI compatibility is that the Live API emits the same events as OpenAI's Realtime API. This allows you to retain your existing event handling logic if you are migrating from or familiar with OpenAI's API.
For more details on the specific events and API specifications, please refer to the [Outspeed Live API documentation](./api-spec/live-api).
# Build with Outspeed
Source: https://docs.outspeed.com/get-started/overview
Add voice interactivity to your app
Embed voice in your mobile or web app
Outspeed helps you add a voice interface to your mobile and web apps. Build an empathetic
therapist, a chatty companion or a professional AI interviewer: possibilities are endless.
## Overview
Get started with Outspeed in minutes.
Explore pre-built templates to jumpstart your voice AI projects.
## Examples
Voice agent for HR interviews and onboarding.
Empathetic voice agent for therapy and support.
Friendly AI companion for conversation.
Two AIs conversing with each other using voice.
{/*
Templates
*/}
Capabilities
Remove unwanted background noise from audio.
Identify and distinguish between speakers.
Detect and express emotions in voice.
# Quickstart
Source: https://docs.outspeed.com/get-started/quickstart
Add a voice agent to your React app in minutes.
## What you'll learn
In this quickstart you'll learn how to:
1. Create an agent in the Outspeed Dashboard
2. Install the Outspeed SDK
3. Build a voice agent that can talk to users
## Step 1: Create Agent in Dashboard
1. Go to the [Outspeed Dashboard](https://dashboard.outspeed.com) and sign in
2. Create a new agent with your desired configuration:
* **Instructions**: "You are a helpful assistant"
* **Voice**: Choose from available voices like `sophie`, `david`, etc.
3. **Important**: Make sure **Authentication** is **disabled** for this quickstart
4. Copy your **Agent ID** - you'll need it in the next step
For production apps, you'll want to enable authentication. See our [Authentication Guide](/get-started/authentication) for details.
## Step 2: Install Outspeed
Install the React SDK using your preferred package manager:
```bash npm
npm install @outspeed/client @outspeed/react
```
```bash pnpm
pnpm add @outspeed/client @outspeed/react
```
```bash yarn
yarn add @outspeed/client @outspeed/react
```
## Step 3: Create React component
Here's a complete working example:
```tsx
import { useConversation } from "@outspeed/react";
import { useState } from "react";
// Replace with your Agent ID from the dashboard
const AGENT_ID = "your-agent-id-here";
export default function App() {
const [sessionCreated, setSessionCreated] = useState(false);
const conversation = useConversation({
onDisconnect: () => {
console.log("Disconnected! cleaning up...");
setSessionCreated(false);
},
});
const startSession = async () => {
try {
conversation.on("session.created", () => {
setSessionCreated(true);
});
await conversation.startSession({
agentId: AGENT_ID,
source: "quickstart"
});
} catch (error) {
console.error("Error starting session", error);
}
};
const endSession = async () => {
try {
await conversation.endSession();
} catch (error) {
console.error("Error ending session", error);
} finally {
setSessionCreated(false);
}
};
if (sessionCreated) {
return (
Voice session active!
Start talking to your AI assistant
);
}
return (
Voice AI Assistant
Click the button to start talking
);
}
```
Want to customize your agent? Edit the configuration in the [Outspeed Dashboard](https://dashboard.outspeed.com) - changes apply instantly without code updates!
## That's it!
You now have a working voice agent that can:
* Start and end voice sessions with zero backend setup
* Talk to users with AI-powered responses
* Handle voice activity detection automatically
* Be customized entirely through the dashboard
## More Examples
Want to see complete implementations with different frameworks? Check out our [Examples Repository](https://github.com/outspeed-ai/outspeed-examples) which includes:
* **React + Vite + FastAPI** - Full-featured voice assistant with weather tools
* **React + Vite + Cloudflare Functions** - Serverless voice assistant
* **Create React App + FastAPI** - Simple React setup with Python backend
* **Next.js** - complete Next.js voice assistant (also includes AI-to-AI chat)
* **Swift iOS** - Native iOS voice bot with SwiftUI
## Next Steps
* **Add Authentication**: For production apps, enable authentication in your agent settings and follow our [Authentication Guide](/get-started/authentication)
* **Add Tools**: Give your agent superpowers with [client tools](/features/client-tools) and [system tools](/features/system-tools)
* **Customize Behavior**: Explore [voices](/features/voices), [context management](/features/context-management), and more features
# Templates
Source: https://docs.outspeed.com/get-started/templates
Ready-to-use templates for building voice applications with Outspeed
***
Get started quickly with our pre-built templates for different frameworks and platforms.
| Template | Framework | Platform | Link |
| ---------------- | -------------------- | -------- | --------------------------------------------------------------------------------------------- |
| Next.js | Next.js | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/next-js-basic) |
| Vite + FastAPI | React, Vite, FastAPI | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/react-vite-fastapi) |
| Cloudflare Pages | React, Vite | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/react-vite-cloudflare) |
| Create React App | React, FastAPI | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/create-react-app-fastapi) |
| Mobile App (iOS) | Swift | Mobile | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/swift-ios-basic) |
## Need Help?
If you need assistance with any template or want to request a new one:
* Visit our [Quickstart Guide](/get-started/quickstart) for step-by-step instructions
* Join our community for support and discussions
* Raise an issue on [GitHub](https://github.com/outspeed-ai/outspeed-examples/issues)
# Migrate from ElevenLabs SwiftSDK
Source: https://docs.outspeed.com/iOS/outspeed-swift-elevenlabs
How to migrate from ElevenLabs Swift SDK to Outspeed Swift SDK.
# ElevenLabs Swift Compatibility
OutspeedSDK is fully compatible with Elevenlabs Swift SDK specifications (with exception of some features.)
If you are migrating from ElevenLabsSDK, follow these steps:
## 1. Replace Imports and Configs
Replace all occurrences of `ElevenLabsSDK` with `OutspeedSDK`. For example:
```swift
import ElevenLabsSDK
let config = ElevenLabsSDK.SessionConfig( agentId : "testagent")
```
becomes
```swift
import OutspeedSDK
let config = OutspeedSDK.SessionConfig( agentId : "testagent")
```
## 2. Add Your Outspeed API Key
Update your `startSession` call to include your Outspeed API key:
```swift
let conversation = try await ElevenLabsSDK.Conversation.startSession(
config: config,
callbacks: callbacks
)
```
becomes
```swift
let conversation = try await OutspeedSDK.Conversation.startSession(
config: config,
callbacks: callbacks
apiKey: ""
)
```
Refer to a working example on [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/swift-ios-basic).
# Swift SDK: Getting Started
Source: https://docs.outspeed.com/iOS/outspeed-swift-getting-started
Add real-time voice conversations to your iOS app with the Outspeed Swift SDK
## Features
* Real-time voice conversations with AI
* Support for both Outspeed and OpenAI providers
* WebRTC-based audio streaming
* Customizable voice and model selection
* ElevenLabs Conversational AI compatibility
The Outspeed Swift SDK is fully compatible with the ElevenLabs Swift SDK. See our{" "}
ElevenLabs Compatibility guide for migration details.
## Installation
1. Open Your Project in Xcode
2. Go to `File` > `Add Packages...`
3. Enter Repository URL: `https://github.com/outspeed-ai/OutspeedSwift`
4. Import the SDK
```swift
import OutspeedSDK
```
Ensure `NSMicrophoneUsageDescription` is added to your Info.plist to explain microphone access.
## Requirements
* iOS 15.2 or later
* Swift 6.1+
## Basic Usage
Here's the minimal code to get a voice conversation working:
```swift
import OutspeedSDK
// Create a session configuration
let config = OutspeedSDK.SessionConfig()
// Create callbacks to handle events
let callbacks = OutspeedSDK.Callbacks()
callbacks.onMessage = { message, role in
print("Received message from \(role.rawValue): \(message)")
}
callbacks.onError = { message, error in
print("Error: \(message)")
}
callbacks.onStatusChange = { status in
print("Status changed to: \(status.rawValue)")
}
// Start the conversation
Task {
do {
let conversation = try await OutspeedSDK.Conversation.startSession(
config: config,
callbacks: callbacks,
apiKey: ""
)
// When done with the conversation
conversation.endSession()
} catch {
print("Failed to start conversation: \(error)")
}
}
```
## Customization
### System Prompt and First Message
```swift
let agentConfig = OutspeedSDK.AgentConfig(
prompt: "You are a helpful assistant with a witty personality.",
firstMessage: "Hey there, how can I help you with Outspeed today?"
)
let config = OutspeedSDK.SessionConfig(
overrides: OutspeedSDK.ConversationConfigOverride(agent: agentConfig)
)
```
### Voice Selection
```swift
let ttsConfig = OutspeedSDK.TTSConfig(voiceId: OutspeedSDK.Voice.david.rawValue)
let config = OutspeedSDK.SessionConfig(
overrides: OutspeedSDK.ConversationConfigOverride(tts: ttsConfig)
)
```
## Complete Example
See a full iOS voice bot application example at our [Examples Repository](https://github.com/outspeed-ai/outspeed-examples/tree/main/swift-ios-basic).
# React SDK Example
Source: https://docs.outspeed.com/react/example
Build your first voice AI application with the Outspeed React SDK
## Basic Voice Chat Component
Here's a complete example of a React component that creates a voice conversation using the `useConversation` hook:
```tsx
import React, { useState } from "react";
import { type SessionConfig } from "@outspeed/client";
import { useConversation } from "@outspeed/react";
const getEphemeralKeyFromServer = async (config: SessionConfig) => {
const tokenResponse = await fetch("/token", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(config),
});
const data = await tokenResponse.json();
if (!tokenResponse.ok) {
throw new Error("Failed to get ephemeral key");
}
return data.client_secret.value;
};
const sessionConfig: SessionConfig = {
model: "outspeed-v1",
instructions: "You are a helpful but witty assistant named Alfred.",
voice: "david", // see the voices page for all available voices
turn_detection: {
type: "semantic_vad",
},
first_message: "Hello, how can I assist you with Outspeed today?",
};
export default function VoiceChat() {
const [sessionCreated, setSessionCreated] = useState(false);
const [isConnecting, setIsConnecting] = useState(false);
const conversation = useConversation({
sessionConfig: sessionConfig,
});
const startSession = async () => {
try {
setIsConnecting(true);
const ephemeralKey = await getEphemeralKeyFromServer(sessionConfig);
await conversation.startSession(ephemeralKey, {
source: "react-example" // Optional: custom string to identify session source
});
// Listen for session creation event
conversation.on("session.created", (event) => {
console.log("Session created:", event);
setSessionCreated(true);
setIsConnecting(false);
});
} catch (error) {
console.error("Error starting session:", error);
setIsConnecting(false);
}
};
const endSession = async () => {
try {
await conversation.endSession();
setSessionCreated(false);
} catch (error) {
console.error("Error ending session:", error);
}
};
if (isConnecting) {
return (
Connecting...
Please wait while we establish the connection.
);
}
if (sessionCreated) {
return (
🎙️ Voice Chat Active
You can now speak with the AI assistant!
);
}
return (
Voice AI Assistant
Click the button below to start a voice conversation.
);
}
```
Want to use a different voice? See all [available voices](/features/voices) you can choose from.
## Connection Management
### onDisconnect Callback
Use `onDisconnect` to handle cleanup when the conversation ends:
```tsx
const conversation = useConversation({
onDisconnect: () => {
console.log("Disconnected! cleaning up...");
setSessionCreated(false);
// Add any cleanup logic here
},
});
```
**Important**: Use `onDisconnect` for cleanup logic, not `onError`. The `onError` callback is called for runtime errors as well, so it's not ideal for cleanup tasks.
### onError Callback
Use `onError` specifically for handling errors:
```tsx
const conversation = useConversation({
onError: (err) => {
console.error("Conversation error:", err);
// Handle error display/logging only
},
onDisconnect: () => {
console.log("Session ended");
setSessionCreated(false);
// Handle cleanup here instead
},
});
```
## Advanced Features
### Event Handling
You can listen to various events during the conversation:
```tsx
// Listen for speech detection
conversation.on("input_audio_buffer.speech_started", () => {
console.log("User started speaking");
});
conversation.on("input_audio_buffer.speech_stopped", () => {
console.log("User stopped speaking");
});
// Listen for AI responses
conversation.on("response.text.delta", (event) => {
console.log("AI response text:", event.delta);
});
conversation.on("response.audio_transcript.delta", (event) => {
console.log("AI speech transcript:", event.delta);
});
```
### Text Input
You can also send text messages programmatically:
```tsx
const sendTextMessage = () => {
conversation.sendText("Tell me about the weather today");
};
return (
{/* ... other components */}
);
```
### Mute Control
Control the microphone state by passing `micMuted` prop to `useConversation`:
```tsx
const [isMicMuted, setIsMicMuted] = useState(false);
const conversation = useConversation({
micMuted: isMicMuted,
});
const toggleMute = () => {
setIsMicMuted(!isMicMuted);
};
return (
{/* ... other components */}
);
```
### Volume Control
Control the AI's voice volume by passing `volume` prop to `useConversation` (value between 0 and 1):
```tsx
const [volume, setVolume] = useState(0.8);
const conversation = useConversation({
volume: volume,
});
return (
{/* ... other components */}
);
```
## Complete Example with Error Handling
Here's a more complete example showing proper error and disconnect handling:
```tsx
const [error, setError] = useState(null);
const [sessionCreated, setSessionCreated] = useState(false);
const conversation = useConversation({
onError: (err) => {
console.error("Conversation error:", err);
setError(err.message);
// Only handle error display here
},
onDisconnect: () => {
console.log("Session ended");
setSessionCreated(false);
setError(null); // Clear any previous errors
// Handle all cleanup logic here
},
});
// Display error to user
if (error) {
return (
❌ Error
{error}
);
}
```
## Next Steps
* [API Reference](/api-spec/client) - Explore all available methods and events
* [Starter Templates](/get-started/templates) - See template implementations
# React SDK Setup
Source: https://docs.outspeed.com/react/setup
Get started with the Outspeed React SDK for realtime voice AI applications
## Installation
Install the React SDK using your preferred package manager:
```bash npm
npm install @outspeed/react
```
```bash pnpm
pnpm add @outspeed/react
```
```bash yarn
yarn add @outspeed/react
```
## Prerequisites
Before using the React SDK, you'll need:
1. **Outspeed API Key**: Get your API key from the [Outspeed Dashboard](https://dashboard.outspeed.com)
2. **Backend Token Endpoint**: A server endpoint to generate ephemeral keys for client authentication
## Backend Setup
```javascript Express.js
app.use(express.json());
app.post("/token", async (req, res) => {
try {
const response = await fetch("https://api.outspeed.com/v1/realtime/sessions", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(req.body),
});
if (!response.ok) {
const error = await response.text();
console.error("failed to generate ephemeral key:", error);
res.status(response.status).json({ error: "Failed to generate token" });
return;
}
const data = await response.json();
res.json(data);
} catch (error) {
console.error("failed to generate ephemeral key:", error);
res.status(500).json({ error: "Internal server error" });
}
});
```
```python FastAPI
import os
import httpx
from fastapi import FastAPI, HTTPException, Request
OUTSPEED_API_KEY = os.getenv("OUTSPEED_API_KEY")
app = FastAPI()
@app.post("/token")
async def create_token(request: Request):
try:
session_config = await request.json()
async with httpx.AsyncClient() as client:
response = await client.post(
"https://api.outspeed.com/v1/realtime/sessions",
headers={
"Authorization": f"Bearer {OUTSPEED_API_KEY}",
"Content-Type": "application/json",
},
json=session_config,
)
if not response.is_success:
print("error generating ephemeral key:", response.text)
raise HTTPException(status_code=response.status_code, detail=response.text)
return response.json()
except Exception as e:
print("error generating ephemeral key:", e)
raise HTTPException(status_code=500, detail="Internal server error")
```
## Environment Variables
Add your Outspeed API key to your server's environment variables:
```bash
OUTSPEED_API_KEY=your_outspeed_api_key_here
```
Never expose your Outspeed API key in client-side code. Always generate ephemeral tokens on your backend server.
## Session Configuration
The React SDK uses a `SessionConfig` object to configure voice sessions:
```typescript
import { type SessionConfig } from "@outspeed/client";
const sessionConfig: SessionConfig = {
model: "outspeed-v1",
instructions: "You are a helpful assistant.",
voice: "david", // see the voices page for all available voices
turn_detection: {
type: "semantic_vad",
},
first_message: "Hello! How can I help you today?", // Optional welcome message
};
```
Want to use a different voice? See all [available voices](/features/voices) you can choose from.
### Configuration Options
| Option | Type | Description |
| ---------------- | -------- | --------------------------------------- |
| `model` | `string` | Must be `"outspeed-v1"` |
| `instructions` | `string` | System prompt for the AI assistant |
| `voice` | `string` | Voice ID to use for speech synthesis |
| `turn_detection` | `object` | Voice activity detection settings |
| `first_message` | `string` | Optional initial message from assistant |
You can find available voices [here](/features/voices)
## Next Steps
Now that you have the SDK installed and configured, you can start building voice AI applications:
* [Basic Example](/react/example) - Learn how to create a simple voice conversation
* [API Reference](/api-spec/client) - Explore all available methods and events