# Client Events Source: https://docs.outspeed.com/api-spec/client These are the events that Realtime servers accept from the client ## Overview These events are accepted by the Realtime servers via WebRTC datachannel from the client. Each event has a specific structure and purpose for managing realtime interactions. ## `session.update` Send this event to update the session's default configuration. The client may send this event at any time to update any field, except for voice. Note that once a session has been initialized with a particular model, it can't be changed to another model using `session.update`. When the server receives a `session.update`, it will respond with a `session.updated` event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string. ### Properties Client-generated ID used to identify this event. The event type, must be `session.update`. Realtime session object configuration. ```json { "event_id": "event_123", "type": "session.update", "session": { "modalities": ["text", "audio"], "instructions": "You are a helpful assistant.", "voice": "sage", "input_audio_format": "pcm16", "output_audio_format": "pcm16", "input_audio_transcription": { "model": "whisper-1" }, "turn_detection": { "type": "server_vad", "threshold": 0.5, "prefix_padding_ms": 300, "silence_duration_ms": 500, "create_response": true }, "tools": [ { "type": "function", "name": "get_weather", "description": "Get the current weather...", "parameters": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"] } } ], "tool_choice": "auto", "temperature": 0.8, "max_response_output_tokens": "inf" } } ``` ## `conversation.item.create` Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages. ### Properties Client-generated ID used to identify this event. The event type, must be `conversation.item.create`. The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation. The item to add to the conversation. ```json { "event_id": "event_345", "type": "conversation.item.create", "previous_item_id": null, "item": { "id": "msg_001", "type": "message", "role": "user", "content": [ { "type": "input_text", "text": "Hello, how are you?" } ] } } ``` ## `response.create` This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically. ### Properties Client-generated ID used to identify this event. The event type, must be `response.create`. Create a new Realtime response with these parameters. ```json { "event_id": "event_234", "type": "response.create", "response": { "modalities": ["text", "audio"], "instructions": "Please assist the user.", "voice": "sage", "output_audio_format": "pcm16", "tools": [ { "type": "function", "name": "calculate_sum", "description": "Calculates the sum of two numbers.", "parameters": { "type": "object", "properties": { "a": { "type": "number" }, "b": { "type": "number" } }, "required": ["a", "b"] } } ], "tool_choice": "auto", "temperature": 0.8, "max_output_tokens": 1024 } } ``` # Outspeed API Source: https://docs.outspeed.com/api-spec/live-api Embed low-latency voice interaction in your apps with Outspeed API The Outspeed API enables you to build fast, natural voice experiences by connecting to our hosted speech-to-speech stack. With Outspeed, you can send and receive both text and audio in real time, leverage voice activity detection, call external functions, and more. Outspeed API is fully compatible with the OpenAI Realtime API. ## How It Works Outspeed API is event-driven. After connecting via WebRTC, your app sends [Client Events](/api-spec/client) and listens for [Server Events](/api-spec/server) to drive the conversation. Getting started is easy: * Install the [JS SDK](https://www.npmjs.com/package/@outspeed/client) * Install the [React SDK](https://www.npmjs.com/package/@outspeed/react) * Install the [Swift SDK](https://github.com/outspeed-ai/OutspeedSwift) Each SDK manages the WebRTC connection and event flow for you, so you can focus on building your app. ## Key Capabilities * **Text & Audio**: Send and receive both text and audio * **Realtime Responses**: Low-latency replies for natural conversations * **Function Calling**: Call external tools and services ## API Events The API is organized around events: Events your app can send to the Realtime server. Events your app receives from the Realtime server. # Server Events Source: https://docs.outspeed.com/api-spec/server Events that the Realtime WebRTC server accepts from the client # Overview These are events emitted from the Realtime servers and sent via WebRTC datachannel to the client. ## `error` Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be error. * **error**: `object` - Details of the error. ```json { "event_id": "event_890", "type": "error", "error": { "type": "invalid_request_error", "code": "invalid_event", "message": "The 'type' field is missing.", "param": null, "event_id": "event_567" } } ``` ## `session.created` Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be session.created. * **session**: `object` - Realtime session object configuration. ```json { "event_id": "event_1234", "type": "session.created", "session": { "id": "sess_001", "object": "realtime.session", "model": "gpt-4o-realtime-preview-2024-12-17", "modalities": ["text", "audio"], "instructions": "...model instructions here...", "voice": "sage", "input_audio_format": "pcm16", "output_audio_format": "pcm16", "input_audio_transcription": null, "turn_detection": { "type": "server_vad", "threshold": 0.5, "prefix_padding_ms": 300, "silence_duration_ms": 200 }, "tools": [], "tool_choice": "auto", "temperature": 0.8, "max_response_output_tokens": "inf" } } ``` ## `session.updated` Returned when a session is updated with a session.update event, unless there is an error. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be session.updated. * **session**: `object` - Realtime session object configuration. ```json { "event_id": "event_5678", "type": "session.updated", "session": { "id": "sess_001", "object": "realtime.session", "model": "gpt-4o-realtime-preview-2024-12-17", "modalities": ["text"], "instructions": "New instructions", "voice": "sage", "input_audio_format": "pcm16", "output_audio_format": "pcm16", "input_audio_transcription": { "model": "whisper-1" }, "turn_detection": null, "tools": [], "tool_choice": "none", "temperature": 0.7, "max_response_output_tokens": 200 } } ``` ## `conversation.item.created` Returned when a conversation item is created. There are several scenarios that produce this event: * The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function\_call. * The input audio buffer has been committed, either by the client or the server (in server\_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item. * The client has sent a conversation.item.create event to add a new Item to the Conversation. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be conversation.item.created. * **previous\_item\_id**: `string` - The ID of the preceding item in the Conversation context, allows the client to understand the order of the conversation. * **item**: `object` - The item to add to the conversation. ```json { "event_id": "event_1920", "type": "conversation.item.created", "previous_item_id": "msg_002", "item": { "id": "msg_003", "object": "realtime.item", "type": "message", "status": "completed", "role": "user", "content": [ { "type": "input_audio", "transcript": "hello how are you", "audio": "base64encodedaudio==" } ] } } ``` ## `input_audio_buffer.committed` Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item\_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be input\_audio\_buffer.committed. * **previous\_item\_id**: `string` - The ID of the preceding item after which the new item will be inserted. * **item\_id**: `string` - The ID of the user message item that will be created. ```json { "event_id": "event_1121", "type": "input_audio_buffer.committed", "previous_item_id": "msg_001", "item_id": "msg_002" } ``` ## `input_audio_buffer.speech_started` Sent by the server when in server\_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user. The client should expect to receive a input\_audio\_buffer.speech\_stopped event when speech stops. The item\_id property is the ID of the user message item that will be created when speech stops and will also be included in the input\_audio\_buffer.speech\_stopped event (unless the client manually commits the audio buffer during VAD activation). ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be input\_audio\_buffer.speech\_started. * **audio\_start\_ms**: `integer` - Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix\_padding\_ms configured in the Session. * **item\_id**: `string` - The ID of the user message item that will be created when speech stops. ```json { "event_id": "event_1516", "type": "input_audio_buffer.speech_started", "audio_start_ms": 1000, "item_id": "msg_003" } ``` ## `input_audio_buffer.speech_stopped` Returned in server\_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be input\_audio\_buffer.speech\_stopped. * **audio\_end\_ms**: `integer` - Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min\_silence\_duration\_ms configured in the Session. * **item\_id**: `string` - The ID of the user message item that will be created. ```json { "event_id": "event_1718", "type": "input_audio_buffer.speech_stopped", "audio_end_ms": 2000, "item_id": "msg_003" } ``` ## `output_audio_buffer.started` Returned when the server begins playing audio output to the client. This event is sent at the start of audio playback for a response, allowing clients to synchronize their UI with the audio stream. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be `output_audio_buffer.started`. * **response\_id**: `string` - The ID of the response that triggered this audio output to start ```json { "event_id": "event_1718", "type": "output_audio_buffer.started", "response_id": "resp_3876604f-7a35-4c95-ad80-e553ddb9a166" } ``` ## `output_audio_buffer.stopped` Returned when the output audio buffer stops playing audio. This event is sent by the server when the audio output for a response has been fully generated and transmitted. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be `output_audio_buffer.stopped`. * **response\_id**: `string` - The ID of the response that triggered this audio output. ```json { "event_id": "event_1920", "type": "output_audio_buffer.stopped", "response_id": "resp_3876604f-7a35-4c95-ad80-e553ddb9a166" } ``` ## `response.done` Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be response.done. * **response**: `object` - The response resource. ```json { "event_id": "event_3132", "type": "response.done", "response": { "id": "resp_001", "object": "realtime.response", "status": "completed", "status_details": null, "output": [ { "id": "msg_006", "object": "realtime.item", "type": "message", "status": "completed", "role": "assistant", "content": [ { "type": "text", "text": "Sure, how can I assist you today?" } ] } ], "usage": { "total_tokens":275, "input_tokens":127, "output_tokens":148, "input_token_details": { "cached_tokens":384, "text_tokens":119, "audio_tokens":8, "cached_tokens_details": { "text_tokens": 128, "audio_tokens": 256 } }, "output_token_details": { "text_tokens":36, "audio_tokens":112 } } } } ``` ## `response.audio_transcript.delta` Returned when the model-generated transcription of audio output is updated. ### Properties * **event\_id**: `string` - The unique ID of the server event. * **type**: `string` - The event type, must be response.audio\_transcript.delta. * **response\_id**: `string` - The ID of the response. * **item\_id**: `string` - The ID of the item. * **output\_index**: `integer` - The index of the output item in the response. * **content\_index**: `integer` - The index of the content part in the item's content array. * **delta**: `string` - The transcript delta. ```json { "event_id": "event_4546", "type": "response.audio_transcript.delta", "response_id": "resp_001", "item_id": "msg_008", "output_index": 0, "content_index": 0, "delta": "Hello, how can I a" } ``` # Text-to-Speech (HTTP) Source: https://docs.outspeed.com/api-spec/tts Generate WAV audio from text using the HTTP TTS endpoint. ## Overview Generate audio from text using our TTS endpoints. We support both single-voice and multi-voice dialogue generation. * **Single Voice TTS**: Convert text to speech with one voice * **Dialogue Generation**: Mix narrator and character voices in the same audio file ## Single Voice TTS POST `https://api.outspeed.com/v1/tts/` ### Request Body ```json { "model": "outspeed-tts-v2", "voice": "clark", "text": "Hello, world!", "stream": false } ``` * **model**: TTS model to use. Use `outspeed-tts-v2` (`outspeed-tts-v1` is deprecated) * **voice**: the voice identifier. Find all available voices and their models at [TTS Playground](https://dashboard.outspeed.com/tts) * **text**: the text to synthesize * **stream**: set to `true` to stream audio chunks; `false` returns the full WAV ### Response * Content-Type: `audio/pcm` * Headers: * `X-Sample-Rate`: Sample rate (default: 24000) * `X-Channels`: Number of audio channels (default: 1) * `X-Bit-Depth`: Bit depth (default: 16) * Body: Raw PCM audio bytes (little-endian int16), 24kHz, mono. No WAV header is included. Wrap with a WAV header or convert with a tool like ffmpeg. Authenticate with Authorization: Bearer \. ## Examples (non-streaming) ```bash curl curl \ -X POST \ 'https://api.outspeed.com/v1/tts/' \ -H 'Authorization: Bearer YOUR_OUTSPEED_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"model":"outspeed-tts-v2","voice":"clark","text":"Hello, world!","stream":false}' \ --output tts.pcm ``` ```bash javascript // Node.js 18+ //Use a WAV helper library to write a proper WAV file from PCM: //npm install wav import wav from "wav"; const res = await fetch("https://api.outspeed.com/v1/tts/", { method: "POST", headers: { Authorization: "Bearer YOUR_OUTSPEED_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ model: "outspeed-tts-v2", voice: "clark", text: "Hello, world!", stream: false }), }); if (!res.ok) throw new Error(`TTS failed: ${res.status}`); const pcm = Buffer.from(await res.arrayBuffer()); const writer = new wav.FileWriter("tts.wav", { channels: 1, sampleRate: 24000, bitDepth: 16 }); writer.write(pcm); writer.end(); ``` ## Examples (streaming) ```bash curl curl \ -X POST \ 'https://api.outspeed.com/v1/tts/' \ -H 'Authorization: Bearer YOUR_OUTSPEED_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"model":"outspeed-tts-v2","voice":"clark","text":"Hello, world!","stream":true}' \ --no-buffer \ --output tts.pcm ``` ```bash javascript // Node.js 18+ (streaming) //Use a WAV helper library to write a proper WAV file from PCM: //npm install wav import { Readable } from "node:stream"; import wav from "wav"; const res = await fetch("https://api.outspeed.com/v1/tts/", { method: "POST", headers: { Authorization: "Bearer YOUR_OUTSPEED_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ model: "outspeed-tts-v2", voice: "clark", text: "Hello, world!", stream: true }), }); if (!res.ok || !res.body) throw new Error(`TTS failed: ${res.status}`); const nodeReadable = Readable.fromWeb(res.body); const writer = new wav.FileWriter("tts.wav", { channels: 1, sampleRate: 24000, bitDepth: 16 }); await new Promise((resolve, reject) => { nodeReadable.pipe(writer); writer.on("finish", resolve); writer.on("error", reject); nodeReadable.on("error", reject); }); ``` ## Dialogue Generation Generate audio with multiple voices (narrator + character) using the `outspeed-tts-v2` model. Try it visually at the [Dialogue Playground](https://dashboard.outspeed.com/tts/dialogue) ### Endpoint POST `https://api.outspeed.com/v1/tts/dialogue` ### Request Body ```json { "model": "outspeed-tts-v2", "text": "*The old library stood silent.* I pushed open the heavy door. *Inside, dust particles danced in my flashlight beam.*", "speaker_voice": "9c5c73f4-1cb7-46cf-91d8-24c80b6288f0", "narrator_voice": "a42c84b2-0e6b-4c9f-a8e7-3f5d1c2e8a9b", "narrator_delimiter": "*" } ``` ### Parameters | Parameter | Type | Required | Description | | -------------------- | ------ | -------- | --------------------------------------------------- | | `model` | string | Yes | Must be `outspeed-tts-v2` | | `text` | string | Yes | Dialogue text with delimiters for narrator parts | | `speaker_voice` | string | Yes | Voice ID for character dialogue | | `narrator_voice` | string | No | Voice ID for narrator parts (omit to skip narrator) | | `narrator_delimiter` | string | No | Delimiter: `*`, `(`, `[`, or `{` (default: `*`) | ### Delimiter Usage * **Text in delimiters** = narrator voice * **Text outside delimiters** = character voice Supported delimiters: * `*` → `*narrator text*` * `(` → `(narrator text)` * `[` → `[narrator text]` * `{` → `{narrator text}` ### Examples ```bash Audiobook Style curl -X POST https://api.outspeed.com/v1/tts/dialogue \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "outspeed-tts-v2", "text": "*The old library stood silent.* I pushed open the heavy door. *Inside, dust particles danced in my flashlight beam.*", "speaker_voice": "", "narrator_voice": ", "narrator_delimiter": "*" }' \ --output dialogue.wav ``` ### Voice Selection Both voices require **voice IDs** (not names). Get voice IDs from: * [TTS Playground](https://dashboard.outspeed.com/tts) - Browse and copy existing voice IDs * [Voice Upload](/features/voice-cloning) - Upload custom voices or create [voice clones](/features/voice-cloning) ### Response Same as single voice TTS: * Content-Type: `audio/wav` * Headers: `X-Sample-Rate`, `X-Channels`, `X-Bit-Depth` * Body: Raw PCM audio bytes # Auto Reconnection Source: https://docs.outspeed.com/features/auto-reconnection Handle connection drops on mobile browsers when app goes to background ## Overview Mobile browsers have privacy protections that affect voice conversations when your app goes to the background. **Context is preserved**: When auto-reconnection happens, your conversation context and history are automatically restored. You don't need to worry about losing the conversation state. ## What Happens on Mobile ### 1. App Goes to Background When users switch apps or lock their phone: * **Microphone access is revoked** within seconds (privacy protection) * User's voice won't reach the AI, even though connection appears active * WebRTC connection may drop after extended background time ### 2. App Returns to Foreground When users return to your app: * SDK automatically attempts to reconnect * Microphone permissions are restored * Conversation resumes seamlessly ## Handling Reconnection ### Listen for Connection State Changes ```typescript const conversation = useConversation({ onStatusChange: (status) => { // Status values: "connected" | "connecting" | "disconnecting" | "disconnected" | "reconnecting" if (status === "reconnecting") { // Show "Reconnecting..." message to user showReconnectingIndicator(); } else if (status === "connected") { // Hide loading indicators hideReconnectingIndicator(); } }, }); ``` ### Detect New vs Reconnected Sessions Use the `session.created` event to know if this is a fresh start or reconnection: ```typescript conversation.on("session.created", (event) => { if (event.run_id === 0) { // This is a brand new session console.log("New conversation started"); } else { // This is a reconnection (run_id will be 1, 2, 3, etc.) console.log(`Reconnected - Run #${event.run_id}`); // showToast("Connection restored!"); } }); ``` ## Best Practices ### User Experience * **Show clear status**: Use visual indicators for connection states * **Inform about background**: Educate users that voice won't work in background due to privacy restrictions * **Seamless return**: Make reconnection feel automatic and smooth ## Summary 1. **Mobile browsers revoke mic access** when app goes to background 2. **SDK auto-reconnects** when app returns to foreground 3. **Use `onStatusChange`** to show connection status to users 4. **Use `session.created` with `run_id`** to detect new vs reconnected sessions 5. **Provide clear visual feedback** so users understand what's happening # Client Tools Source: https://docs.outspeed.com/features/client-tools Create custom tools that extend AI agents with your own functions and APIs ## Overview Client tools are custom functions that the voice agent can use during the conversation. They are defined and implemented on the client side. ## Simple Example Here's a complete example with one tool: ### Step 1: Define the Tool ```typescript const getTimeSchema = { name: "get_time", type: "function", description: "Get the current time", parameters: { type: "object", properties: {}, required: [], }, }; ``` Tool schemas are OpenAI compatible. See the [OpenAI Function Calling guide](https://platform.openai.com/docs/guides/function-calling) for more details on defining functions. ### Step 2: Implement the Function ```typescript import { type ClientTool } from "@outspeed/client"; // params & context can be skipped here since this tool doesn't use them const getTime: ClientTool<{}> = (params, context) => { return new Date().toLocaleTimeString(); }; ``` The return value from your function is sent directly to the AI model, which uses it to generate its response to the user. **Always return a value** - even for action-based tools: * **Data tools** (weather, calculations): Return the actual data * **Action tools** (generate image, open browser): Return acknowledgment like "Image generated successfully" or "Browser tab opened" * **On failure**: Return error description like "Failed to generate image: rate limit error" This tells the AI whether your tool succeeded or failed. See the [implementation section](#implementation) for more details. ### Step 3: Configure Session ```typescript const sessionConfig = { // rest of config... tools: [getTimeSchema], }; const conversation = useConversation({ clientTools: { get_time: getTime, }, }); ``` That's it! When the user asks "What time is it?", the agent will: 1. Call your `getTime()` function 2. Receive the return value (e.g., "2:30:45 PM") 3. Use that information to respond to the user ## Advanced Example with Context Here's a more complex tool with typed parameters and context usage: ```typescript import { type ClientTool } from "@outspeed/client"; const setTimer: ClientTool<{ time: number; prompt: string }> = ({ time, prompt }, context) => { setTimeout(() => { // show a toast if you want to // toast.info("Timer completed!"); // we let the model know that the timer is done so that it can respond to the user context.sendText( `🔔 TIMER ALERT: The timer you set ${time} seconds ago has finished. Timer prompt: "${prompt}" Completed at: ${new Date().toLocaleString()} This is an automated system notification. Please proceed with any actions related to this timer.`, ); }, time * 1000); // we set the timer and let the model know that the timer is set return "Timer set"; }; ``` The `context` parameter provides access to conversation methods like `sendText()` for sending messages back to the AI after your tool completes. ## Tool Schema Format Tool schemas are OpenAI compatible. See the [OpenAI Function Calling guide](https://platform.openai.com/docs/guides/function-calling) for more details on defining functions. ```typescript { name: string, // Unique tool identifier type: "function", // Always "function" for client tools description: string, // Clear description for the AI parameters: { type: "object", properties: { [paramName]: { type: string, // "string", "number", "boolean", "array", "object" description: string // Parameter description } }, required: string[] // Required parameter names } } ``` ## Best Practices ### Tool Design * **Clear descriptions**: Help the AI understand when and how to use each tool * **Specific parameters**: Define precise parameter types and descriptions * **Single purpose**: Each tool should do one thing well * **Predictable naming**: Use descriptive, consistent naming conventions ### Implementation **Always return meaningful values** - the AI uses your return value to respond to the user: ```typescript import { type ClientTool } from "@outspeed/client"; // ✅ Good: Return actual data const getWeather: ClientTool<{ city: string }> = ({ city }, context) => { return "72°F and sunny in San Francisco"; }; // ✅ Good: Return success confirmation const sendEmail: ClientTool<{ to: string; subject: string }> = ({ to, subject }, context) => { // ... send email logic return `Email sent to ${to}`; }; // ✅ Good: Return error details const uploadFile: ClientTool<{ filename: string }> = ({ filename }, context) => { try { // ... upload logic return "File uploaded successfully"; } catch (error) { return `Upload failed: ${error.message}`; } }; // ❌ Bad: Don't return undefined/null const badTool: ClientTool<{}> = (params, context) => { // The AI gets nothing to work with return null; }; ``` **Handle errors gracefully**: ```typescript const robustTool: ClientTool<{ param: string }> = ({ param }, context) => { try { if (!param?.trim()) { return "Parameter is required"; } const result = performOperation(param); // notice that we're returning something that model can use to respond to the user return result || "Operation completed but no data returned"; } catch (error) { return `Error: ${error.message}`; // for the model to understand the error } }; ``` **Use async/await for API calls**: ```typescript const fetchData: ClientTool<{ query: string }> = async ({ query }, context) => { try { const response = await fetch(`/api/search?q=${query}`); const data = await response.json(); return `Found ${data.results.length} results for "${query}"`; } catch (error) { return "Search service unavailable"; // again, for the model to understand what went wrong } }; ``` ### Performance * **Cache results**: Cache API responses when appropriate * **Timeout handling**: Set reasonable timeouts for external calls * **Rate limiting**: Respect API rate limits * **Graceful degradation**: Provide fallbacks when tools fail ## Error Handling ```typescript export async function robustToolFunction({ param }: { param: string }) { try { // Validate input if (!param || param.trim() === "") { return "Parameter is required"; } // Perform operation const result = await someApiCall(param); // Validate result if (!result) { return "No data available"; } return result; } catch (error) { console.error("Tool error:", error); // Return user-friendly error message if (error instanceof Error) { return `Error: ${error.message}`; } return "An unexpected error occurred"; } } ``` ## Combining with System Tools You can use client tools alongside system tools: ```typescript const sessionConfig = { // rest of config... tools: [getTimeSchema], system_tools: [ { name: "end_call", enabled: true }, { name: "skip_turn", enabled: true }, ], }; ``` # Context Management Source: https://docs.outspeed.com/features/context-management Manage conversation context during voice interactions ## Overview Context updates allow you to dynamically add information to the conversation's context during voice interactions. This helps provide the AI agent with additional context about the user, conversation state, or UI state. ## Automatic Context Updates During voice conversations, several types of items are automatically added to the context: * **User speech**: Everything the user says is automatically transcribed and added as message items * **Agent responses**: All agent responses are added to maintain conversation flow * **Function calls**: When agents call tools, the function call details are automatically added * **Function outputs**: Tool results are automatically added by the SDK when tools complete ## Manual Context Updates Adding context doesn't trigger an agent response. You can manually add items to the conversation context using the `conversation.item.create` event: ```typescript import { useConversation } from "@outspeed/react"; // Inside a React component (useConversation is a React hook) export default function MyComponent() { const conversation = useConversation({}); const addToContext = async (text: string) => { const event = { type: "conversation.item.create", item: { type: "message", role: "user", content: [ { type: "input_text", text, }, ], }, }; await conversation.send(JSON.stringify(event)); }; // rest of component... } ``` ## Event Notifications The server sends events for all context operations: ```typescript // Item added (automatic or manual) conversation.on("conversation.item.created", (event) => { console.log("Item added:", event.item); }); // Item retrieved conversation.on("conversation.item.retrieved", (event) => { console.log("Item retrieved:", event.item); }); // Item deleted conversation.on("conversation.item.deleted", (event) => { console.log("Item deleted:", event.item_id); }); ``` ## Retrieving Context Items You can retrieve a specific item from the conversation context by its ID: ```typescript const retrieveEvent = { type: "conversation.item.retrieve", item_id: "item_12345", }; await conversation.send(JSON.stringify(retrieveEvent)); // Listen for the response conversation.on("conversation.item.retrieved", (event) => { console.log("Retrieved item:", event.item); }); ``` ## Deleting Context Items You can remove items from the conversation context: ```typescript const deleteEvent = { type: "conversation.item.delete", item_id: "item_12345", }; await conversation.send(JSON.stringify(deleteEvent)); // Listen for confirmation conversation.on("conversation.item.deleted", (event) => { console.log("Deleted item ID:", event.item_id); }); ``` ## Item Structure Context items follow this structure: | Field | Type | Description | | --------- | ----------------- | ---------------------------------------------------------------- | | `type` | string | Item type: "message", "function\_call", "function\_call\_output" | | `role` | string | Message role: "user", "assistant", "system" | | `content` | array | Message content with type and text | | `status` | string (optional) | Item status: "in\_progress", "completed" | ## Context Item Types ### Message Items Standard conversation messages from users, assistants, or system ### Function Call Items Tool invocations with parameters (automatically managed) ### Function Output Items Results from tool executions (automatically managed) The context helps the AI agent maintain awareness of the conversation history and make more informed responses based on the full context of the interaction. # End Call Source: https://docs.outspeed.com/features/end-session End Call is a system tool that enables AI agents to gracefully terminate voice conversations when users explicitly indicate they want to end the call. ## Overview The End Call system tool helps agents handle situations where users are: * Saying goodbye or farewell * Clearly indicating they want to end the call * Using phrases like "bye", "goodbye", "talk to you later", "I'm hanging up" * Concluding the conversation with clear intent to leave The agent uses conversation context to distinguish between genuine farewell intentions and brief acknowledgments that don't require ending the session. ## Configuration Enable `end_call` system tool in your session configuration: ```javascript const sessionConfig = { // rest of config... system_tools: [{ name: "end_call", enabled: true }], }; ``` ## Tool parameters | Parameter | Type | Required | Description | | ------------------ | ------ | -------- | ---------------------------------------------------------------------------- | | `farewell_message` | string | Yes | Final message to say before ending the call | | `reason` | string | Yes | Why the call is ending (e.g., 'user said goodbye', 'conversation concluded') | ## Override tool descriptions You can override the default tool description and parameter descriptions to customize the agent's behavior: ```javascript const sessionConfig = { // rest of config... system_tools: [ { name: "end_call", description: "", parameter_descriptions: { farewell_message: "", reason: "", }, }, ], }; ``` ## When end call is used The agent automatically calls this tool when it detects: Clear goodbye scenarios like: * **Direct goodbye**: User says "goodbye", "bye", "farewell" * **Conversation conclusion**: User says "talk to you later", "see you soon" * **Explicit end request**: User says "I'm hanging up", "I need to go" * **Formal conclusion**: User indicates the conversation is finished ### Example situations ```json // User says goodbye Agent would call: end_call({ farewell_message: "Goodbye! Have a great day!", reason: "user said goodbye" }) // User says "I need to go" Agent would call: end_call({ farewell_message: "Of course! Take care!", reason: "user requested to end" }) // User says "thanks" (should NOT end call) Agent continues conversation instead of ending // User says "talk to you later" Agent would call: end_call({ farewell_message: "Talk to you later! Goodbye!", reason: "conversation concluded" }) ``` ## Default implementation Here's how the End Call tool is defined on our server: ```python tool_def = FunctionDefinition( name="end_call", description="End the conversation ONLY when the user explicitly says goodbye, farewell, or clearly indicates they want to end the call (e.g., 'bye', 'goodbye', 'talk to you later', 'I'm hanging up'). Do NOT end the call for brief responses like 'thanks', 'okay', 'got it', or simple acknowledgments - just return to listening mode instead.", parameters=FunctionParameters( type="object", properties={ "farewell_message": FunctionProperty( type="string", description="Final message to say before ending the call", ), "reason": FunctionProperty( type="string", description="Why the call is ending (e.g., 'user said goodbye', 'conversation concluded', 'user requested to end')", ), }, required=["farewell_message", "reason"], ), ) ``` ## Troubleshooting If you notice the agent ending sessions when it shouldn't: 1. **Review Context**: Check if the conversation context provides clear goodbye signals 2. **Adjust Instructions**: Refine your system prompt to distinguish between acknowledgments and farewells 3. **Override description**: You can override the tool description to be more or less restrictive # Input Language Source: https://docs.outspeed.com/features/input-language Configure the language users can speak in during voice conversations ## Overview The `input_language` field in your session configuration provides a **hint** about the primary language users will speak. However, the model can understand multiple languages simultaneously, regardless of the hint provided. The AI agent will understand their speech and respond in the language corresponding to the chosen voice. The response language depends on your chosen voice. See [available voices](/features/voices) for language support. The `input_language` setting is a hint to optimize recognition, but users can speak in any supported language. ## Supported Languages Currently supported input language hints: * **en** - English (default) * **zh** - Chinese (Mandarin) * **hi** - Hindi * **de** - German * **es** - Spanish * **fr** - French * **it** - Italian * **ja** - Japanese * **ko** - Korean * **nl** - Dutch * **pl** - Polish * **pt** - Portuguese * **ru** - Russian * **sv** - Swedish * **tr** - Turkish ## Configuration Add `input_language` to your session configuration as a hint for the primary expected language: ```typescript const sessionConfig = { model: "outspeed-v1", instructions: "You are a helpful assistant.", voice: "sophie", input_language: "zh", // Hint: Primary language is Chinese, but users can still switch to English mid-conversation turn_detection: { type: "semantic_vad", }, first_message: "Hello! How can I help you today?", }; ``` ## Language Examples ### English (Default) ```typescript const sessionConfig = { // rest of config... input_language: "en", // or omit this field entirely }; // User speaks: "What's the weather like?" // Agent responds: "The weather is sunny and 72°F." (English voice responds in English) ``` ### Chinese (Mandarin) with Multilingual Support ```typescript const sessionConfig = { // rest of config... input_language: "zh", // Hint: Primary language is Chinese, but users can still switch to English mid-conversation }; // User speaks: "今天天气怎么样?" (How's the weather today?) // Agent responds: "The weather is sunny and 72°F." (English voice responds in English) // User can also mix languages: // User speaks: "今天 how are you?" // Agent responds: "Hello! I'm doing well, thank you for asking." (English voice responds in English) ``` ### Hindi with Multilingual Support ```typescript const sessionConfig = { // rest of config... voice: "apoorva", // Hindi voice input_language: "hi", // Hint for Hindi, but users can mix languages }; // User speaks: "आज मौसम कैसा है?" (How's the weather today?) // Agent responds: "मौसम धूप है और 72°F है।" (Hindi voice responds in Hindi) // User can also mix languages: // User speaks: "हेलो, सब बढ़िया? Everything is fine?" // Agent responds: "हैलो! हाँ, सब कुछ ठीक है। मैं आपकी कैसे मदद कर सकती हूँ?" (Hindi voice responds in Hindi) ``` ## Language Detection The system automatically detects which languages are spoken in each user input and includes this information in the transcription event: ```json { "event_id": "bf03fb6b-1160-411d-b332-f2a97b743d2c", "type": "conversation.item.input_audio_transcription.completed", "server_sent": true, "item_id": "item_5c49066c55c549d29132a467d01e8e3b", "transcript": "हेलो, सब बढ़िया? Everything is fine?", "languages": ["hi", "en"], "content_index": 0 } ``` The `languages` array contains all detected languages in the user's input, allowing you to understand the linguistic composition of each utterance. You can listen to this event to understand the languages spoken in the user's input. ```typescript conversation.on("conversation.item.input_audio_transcription.completed", (event) => { console.log("Languages spoken:", event.languages); }); ``` ## Important Notes * **Hint, not restriction**: `input_language` is a hint to optimize recognition, not a limitation * **Multilingual support**: Users can speak multiple languages in a single session or even single utterance * **Voice-dependent output**: The AI agent responds in the language of the chosen voice (see [available voices](/features/voices)) * **Default behavior**: Default input language value is `en` for English * **Language detection**: Each transcription includes detected languages in the `languages` array # Pronunciation Rules Source: https://docs.outspeed.com/features/pronunciation Customize how the AI pronounces specific words and terms ## Overview Custom pronunciation rules allow you to control how the AI pronounces specific words, acronyms, and technical terms in its speech output. This is an optional feature. ## Configuration Add `custom_pronunciation_rules` to your session configuration: ```typescript const sessionConfig = { model: "outspeed-v1", instructions: "You are a helpful assistant.", voice: "sophie", custom_pronunciation_rules: [ { pattern: "Node.js", pronunciation: "Node JS", }, { pattern: "React.js", pronunciation: "React JS", }, { pattern: "API", pronunciation: "A P I", }, ], // rest of config... }; ``` ## Common Examples ```typescript custom_pronunciation_rules: [ { pattern: "AWS", pronunciation: "A W S" }, { pattern: "JSON", pronunciation: "Jason" }, { pattern: "API", pronunciation: "A P I" }, { pattern: "Node.js", pronunciation: "Node JS" }, { pattern: "React.js", pronunciation: "React JS" }, ]; ``` # Resume Session Source: https://docs.outspeed.com/features/resume-session Continue voice conversations from where they left off ## Overview Session resumption allows you to continue voice conversations from where they left off, even after page refreshes or application restarts. This provides a seamless user experience for ongoing conversations. ## How It Works To resume a session, pass the `resume_session` query parameter when requesting an ephemeral key from your backend. ### Backend Implementation Update your `/token` endpoint to handle session resumption: ```typescript app.post("/token", async (req, res) => { const { resume_session } = req.query; // resume_session is the ID of the session to resume let url = "https://api.outspeed.com/v1/realtime/sessions"; if (resume_session) { url += `?resume_session=${resume_session}`; } const response = await fetch(url, { method: "POST", headers: { Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify(req.body), }); const data = await response.json(); res.json(data); }); ``` ### Frontend Usage Request an ephemeral key with the session ID to resume: ```typescript const getEphemeralKeyFromServer = async (config: SessionConfig, sessionId?: string) => { let url = `${SERVER_URL}/token`; if (sessionId) { url += `?resume_session=${sessionId}`; } const tokenResponse = await fetch(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(config), }); const data = await tokenResponse.json(); return data.client_secret.value; }; // Resume existing session const ephemeralKey = await getEphemeralKeyFromServer(sessionConfig, "session_123"); await conversation.startSession(ephemeralKey); ``` ## Context Restoration When a session is resumed, the context is restored and you'll receive a `context.restored` event: ```typescript conversation.on("context.restored", (event) => { console.log("Context restored with items:", event.items); // Handle restored conversation items event.items.forEach(item => { console.log(`${item.role}: ${item.content}`); }); }); ``` ## Use Cases * **Page refreshes**: Continue conversations after accidental refreshes * **Application restarts**: Resume conversations across app sessions * **Multi-device**: Continue conversations on different devices # Send Text Messages Source: https://docs.outspeed.com/features/send-text Send text messages to the AI agent programmatically ## Overview You can send text messages to the AI agent programmatically, either by sending events directly or using the convenient `sendText` method. ## Using sendText Method The React SDK provides a convenient `sendText` method that handles both events for you: ```typescript import { useConversation } from "@outspeed/react"; // inside a React component const conversation = useConversation({}); // Send text and get AI response await conversation.sendText("What's the weather like?"); // Cancel ongoing response and send new text await conversation.sendText("What's the weather like?", { cancelOngoing: true }); ``` `sendText` will generate a response only if the model is currently **not** generating anything. ## Options The `sendText` method accepts an optional second parameter with these options: | Option | Type | Default | Description | | --------------- | ------- | ------- | ------------------------------------------------------------------ | | `cancelOngoing` | boolean | `false` | Cancel any ongoing response generation before sending the new text | ### Cancel Ongoing Responses When the AI is generating a response, you can cancel it and send a new message: ```typescript // This will cancel any ongoing response and send the new text await conversation.sendText("Actually, tell me about the weather in Tokyo instead", { cancelOngoing: true, }); ``` ## Using Events To send a text message, you need to send two events: ```typescript // 1. Add text message to conversation context const messageEvent = { type: "conversation.item.create", item: { type: "message", role: "user", content: [ { type: "input_text", text: "What's the weather like?", }, ], }, }; await conversation.send(JSON.stringify(messageEvent)); // 2. Request AI response const responseEvent = { type: "response.create", }; await conversation.send(JSON.stringify(responseEvent)); ``` ## Example Usage ```typescript function ChatInput() { const [message, setMessage] = useState(""); const conversation = useConversation({}); // add the logic to start & end a conversation session const handleSubmit = async (e: FormEvent) => { e.preventDefault(); if (!message.trim()) return; try { await conversation.sendText(message); setMessage(""); // Clear input after sending } catch (error) { console.error("Failed to send message:", error); } }; return (
setMessage(e.target.value)} placeholder="Type a message..." />
); } ``` ## Important Notes ### Concurrent sendText Calls By default, `sendText` will generate a response only if the model is currently not generating anything. If you call `sendText` while the AI is responding, you have two options: **Option 1: Handle the error (default behavior)** ```typescript const conversation = useConversation({}); conversation.on("error", (event) => { if (event.error.code === "conversation_already_has_active_response") { console.log("AI is currently responding, please wait"); // Show user feedback or queue the message } }); ``` **Option 2: Cancel ongoing response** ```typescript // This will cancel the current response and send the new message await conversation.sendText("New message", { cancelOngoing: true }); ``` ### Error Response If you don't use `cancelOngoing: true`, simultaneous calls will fail with this error: ```json { "event_id": "b5ea58f4-8dfe-4590-959c-0c85ef108b9b", "type": "error", "server_sent": true, "created_at_": 1756734784.1917815, "error": { "type": "invalid_request_error", "code": "conversation_already_has_active_response", "message": "Conversation already has an active response", "param": null, "event_id": null } } ``` # Skip Turn Source: https://docs.outspeed.com/features/skip-turn Skip Turn is a system tool that enables AI agents to intelligently decide when to skip their speaking turn based on conversation context. The agent uses this tool when users are not directly addressing them or when no response is needed. ## Overview The Skip Turn system tool helps agents handle situations where users are: * Singing songs * Talking to someone else * Having background conversations * Talking to themselves * Saying something unrelated to the conversation The agent uses conversation context to make intelligent decisions about when to skip turns, creating more natural interaction patterns. ## Configuration Enable Skip Turn in your session configuration: ```javascript const sessionConfig = { // rest of config... system_tools: [{ name: "skip_turn", enabled: true }], }; ``` ## Tool parameters | Parameter | Type | Required | Description | | --------- | ------ | -------- | --------------------------------------------------- | | `reason` | string | Yes | Brief explanation for why the turn is being skipped | ## Override tool descriptions You can override the default tool description and parameter descriptions to customize the agent's behavior: ```javascript const sessionConfig = { // rest of config... system_tools: [ { name: "skip_turn", description: "", parameter_descriptions: { reason: "", }, }, ], }; ``` ## When skip turn is used The agent automatically calls this tool when it detects: ### Direct scenarios * **Singing**: User is singing a song * **Third-party conversation**: User is talking to someone else in the room * **Background chatter**: Casual conversation not directed at the agent * **Self-talk**: User talking to themselves * **Unrelated content**: User saying something unrelated to the current conversation ### Example situations ```json // User singing Agent would call: skip_turn({ reason: "singing song" }) // User talking to someone else Agent would call: skip_turn({ reason: "talking to someone else" }) // Background conversation Agent would call: skip_turn({ reason: "background conversation" }) // User talking to themselves Agent would call: skip_turn({ reason: "not addressing bot" }) ``` ## Default implementation Here's how the Skip Turn tool is defined on our server: ```python tool_def = FunctionDefinition( name="skip_turn", description="Call this when the user is not directly addressing you or when no response is needed. Skip if the user is: singing, talking to someone else, having background conversation, talking to themselves, or saying something unrelated to our conversation. Use conversation context to decide. If genuinely unsure whether they're addressing you, ask for clarification instead of skipping.", parameters=FunctionParameters( type="object", properties={ "reason": FunctionProperty( type="string", description="Brief reason for skipping this turn (e.g., 'singing song', 'talking to someone else', 'background conversation', 'not addressing bot')", ), }, required=["reason"], ), ) ``` ## Troubleshooting If you notice the agent skipping turns when it shouldn't: 1. **Review Context**: Check if the conversation context is clear enough 2. **Adjust Instructions**: Refine your system prompt to provide better guidance 3. **Override description**: You can override the tool description as per your need # Speak Source: https://docs.outspeed.com/features/speak Make the AI agent speak specific text directly ## Overview The `speak` method allows you to make the AI agent speak specific text directly, without any processing. It's basically Text-to-Speech. ## Using speak Method The React SDK provides a `speak` method on the conversation object: ```typescript import { useConversation } from "@outspeed/react"; // Inside a React component const conversation = useConversation({}); // Make the agent speak specific text await conversation.speak("Hey there, I'm Perry from Outspeed! How can I help you today?"); ``` The text will be added to the context as a message item. ## Key Differences | Method | Purpose | Behavior | | ------------ | -------------------- | ---------------------------------------------------- | | `sendText()` | Send user message | AI processes the message and generates a response | | `speak()` | Direct speech output | AI speaks the exact text provided without processing | ## Example Usage ```typescript function VoiceAnnouncements() { const conversation = useConversation({}); const makeAnnouncement = async (text: string) => { try { await conversation.speak(text); } catch (error) { console.error("Failed to speak:", error); } }; return (
); } ``` ## Common Use Cases * **System announcements**: Status updates or notifications * **Predefined responses**: Quick replies without AI processing * **Error messages**: Speaking error states directly * **Welcome messages**: Greeting users with specific text * **Instructions**: Providing step-by-step guidance ## Complete Example ```typescript function CustomerService() { const [isOnHold, setIsOnHold] = useState(false); const conversation = useConversation({}); const putOnHold = async () => { setIsOnHold(true); await conversation.speak("Please hold while I transfer you to a specialist. This may take a few moments."); }; const removeFromHold = async () => { setIsOnHold(false); await conversation.speak("Thank you for holding. How can I assist you today?"); }; const endCall = async () => { await conversation.speak("Thank you for contacting us. Have a great day!"); await conversation.endSession(); }; return (
{isOnHold ? ( ) : ( )}
); } ``` # Speak First Source: https://docs.outspeed.com/features/speak-first Control whether the agent initiates conversation or waits for user input ## Overview The `first_message` field allows your AI agent to start the conversation automatically with a predefined greeting, instead of waiting for the user to speak first. ## Basic Usage Add `first_message` to your session configuration: ```javascript const sessionConfig = { // rest of config... first_message: "Hello! How can I help you today?", }; ``` When the session starts, the agent will immediately speak this message to greet the user. ## Examples ### Professional Assistant ```javascript first_message: "Good morning! I'm your AI assistant. What can I help you with today?" ``` ### Customer Support ```javascript first_message: "Hi there! Welcome to our support chat. How can I assist you?" ``` ### Interview Bot ```javascript first_message: "Hello! I'm excited to conduct your interview today. Shall we get started?" ``` ### Casual Companion ```javascript first_message: "Hey! Great to see you. What's on your mind?" ``` ## Best Practices * **Keep it brief**: Short, welcoming messages work best * **Match your brand**: Use language that fits your application's tone * **Stay consistent**: Use similar greetings across sessions ## Optional Field The `first_message` field is optional. If you don't include it, the session will start in listening mode, waiting for the user to speak first. # System Tools Source: https://docs.outspeed.com/features/system-tools Pre-built tools that run on Outspeed's servers for conversation management ## Overview System tools are pre-built tools that run on Outspeed's servers. They handle common conversation management tasks and require no implementation on your part - simply enable them in your session configuration. System tools help agents manage conversation flow intelligently: * **Server-side execution**: No client-side implementation needed * **Automatic decisions**: Agents call tools based on conversation context * **Override capabilities**: Customize tool behavior with custom descriptions * **Zero maintenance**: Tools are maintained and updated by Outspeed ## Available System Tools ### Skip turn Allows agents to intelligently skip their speaking turn when users aren't directly addressing them. **Use Cases:** * User is singing or humming * Background conversations * User talking to someone else * Self-talk or thinking aloud 👉 **[Learn about Skip Turn →](skip-turn)** ### End session Enables agents to gracefully terminate conversations when users explicitly say goodbye. **Use Cases:** * User says "goodbye", "bye", "farewell" * Clear conversation conclusion * Explicit requests to end the call 👉 **[Learn about End Session →](end-session)** ## Basic sonfiguration Enable system tools in your session configuration: ```javascript const sessionConfig = { // rest of config... system_tools: [ { name: "skip_turn", enabled: true }, { name: "end_call", enabled: true } ], }; ``` ## Override tool descriptions You can customize how system tools behave by overriding their descriptions: ```javascript const sessionConfig = { // rest of config... system_tools: [ { name: "skip_turn", enabled: true, description: "Skip turns more conservatively - only when user is clearly singing or talking to others", parameter_descriptions: { reason: "Specific reason for skipping (singing or talking to others only)", }, }, { name: "end_call", enabled: true, description: "End calls only on very explicit goodbye phrases like 'goodbye' or 'bye'", parameter_descriptions: { farewell_message: "Brief, professional farewell message", reason: "Specific goodbye phrase that triggered the end", }, } ], }; ``` ## Best practices ### When to enable * **Skip Turn**: Recommended for most applications to handle natural conversation flow * **End Session**: Essential for applications where users initiate call termination ### Override examples **More Restrictive Skip Turn:** ```javascript { name: "skip_turn", description: "Only skip when absolutely certain user is not addressing the agent", parameter_descriptions: { reason: "Very specific reason with high confidence", }, } ``` **More Permissive End Session:** ```javascript { name: "end_call", description: "End calls when users show any sign of wanting to conclude", parameter_descriptions: { farewell_message: "Warm, understanding farewell", reason: "Any indication user wants to end conversation", }, } ``` ## Troubleshooting ### Tools not working 1. **Check configuration**: Ensure tools are properly enabled 2. **Verify session**: Confirm tools are included in session config 3. **Review logs**: Check for error messages in session events ### Unexpected behavior 1. **Monitor context**: Review conversation context when tools trigger 2. **Adjust descriptions**: Override tool descriptions to fine-tune behavior 3. **System prompt**: Ensure your system prompt doesn't conflict with tool usage ### Performance impact * System tools have minimal performance impact * No client-side processing required * Server-side execution is optimized and fast # Tool Use Source: https://docs.outspeed.com/features/tool-use Enable AI agents to use external tools and APIs during conversations ## Overview Tools extend your AI agents with external capabilities, allowing them to interact with APIs, perform actions, and access real-time information during voice conversations. ## Tool Types Custom functions you define and implement in your application. Pre-built tools that like `skip-turn` and `end-call` that run on Outspeed's servers. ## Examples See real implementations of tools in action in our [Examples Repository](https://github.com/outspeed-ai/outspeed-examples). The repository includes working examples with: * **Weather tool integration** (client tool) * **Skip turn and end call** (system tools) * **Multiple frameworks**: React, Next.js, Swift iOS, and more ## Getting Started 1. **Choose your tools**: Decide which client tools you need and which system tools to enable 2. **Implement client tools**: Write functions for your custom tools 3. **Configure session**: Add tools to your session configuration # Voice Cloning Source: https://docs.outspeed.com/features/voice-cloning Create custom AI voices from audio samples ## Overview Voice cloning allows you to create highly realistic custom AI voices from your own audio samples. This enables a personalized and branded voice experience for your applications. 1. **Upload Audio**: Provide clear audio samples of the voice you want to clone 2. **Generate Voice**: Outspeed's AI processes the samples to create a new voice 3. **Use in Apps**: Integrate your custom voice into your agents and TTS requests Cloned voices are currently only available for use with our [Text-to-Speech (TTS) API](/api-spec/tts). ## Getting Started 1. **Visit Dashboard**: Go to the [Voice Upload page](https://dashboard.outspeed.com/tts/upload-voice) in your Outspeed Dashboard 2. **Upload Samples**: Follow the instructions to upload your audio files 3. **Create Voice**: Once processed, your new voice will be available to use 4. **Copy Voice ID**: Copy the generated Voice ID for use in your code ## Audio Requirements * **Length**: Minimum 10 seconds of clear speech * **Size**: Maximum 3 MB * **Quality**: High-quality audio (e.g., studio recording, no background noise) * **Format**: Only WAV files are supported * **Content**: Natural speech, varied tone and pitch **Do not upload copyrighted material or impersonate individuals without consent.** Ensure you have the necessary rights for all uploaded audio. ## Usage Once you have your custom Voice ID, use it with our [Text-to-Speech (HTTP)](/api-spec/tts): ### In TTS API Requests ```json { "model": "outspeed-tts-v2", "voice": "your-custom-voice-id", "text": "This is my custom voice." } ``` ### In Dialogue Generation ```json { "model": "outspeed-tts-v2", "text": "*Narrator:* This is my custom voice. Character: Hello!", "speaker_voice": "your-custom-voice-id", "narrator_voice": "another-custom-voice-id" } ``` ## Best Practices * **High-quality audio**: Crucial for best results * **Consent**: Always obtain consent if cloning a real person's voice * **Monitor usage**: Track how your custom voices perform in real applications # Voices Source: https://docs.outspeed.com/features/voices Available voices for Outspeed voice agents ## Overview ## Standard Voices * viktoria `de` * lena `de` * thomas `de` * lukas `de` * sophie `en` * savannah `en` * brooke `en` * zia `en` * corinne `en` * david `en` * griffin `en` * carson `en` * ethan `en` * daniela `es` * elena `es` * pedro `es` * luis `es` * camille `fr` * marie `fr` * antoine `fr` * philippe `fr` * apoorva `hi` * aarti `hi` * ishan `hi` * neeraj `hi` * liv `it` * francesca `it` * lucio `it` * luca `it` * sayuri `ja` * yumiko `ja` * daisuke `ja` * akira `ja` * jihyun `ko` * mimi `ko` * jaechul `ko` * byungtae `ko` * ana\_paula `pt` * luana `pt` * felipe `pt` * camilo `pt` * tatiana `ru` * irina `ru` * sergei `ru` * nikolai `ru` * chen `zh` * yue `zh` * liu `zh` * kai `zh` * sanne `nl` * daan `nl` * lucas\_dutch `nl` * bram `nl` * zofia `pl` * katarzyna `pl` * tomek `pl` * wojciech `pl` * ingrid `sv` * freja `sv` * anders `sv` * cees `sv` * leyla `tr` * aylin `tr` * emre `tr` * taylan `tr` ## Beta Voices Beta voices are still being improved. * clark `en` * linda `en` * isabella `en` * emily `en` * aria `en` * jasmine `en` * kate `en` * siren `en` * sienna `en` * claudia `en` * dominique `en` * hannah `en` * chloe `en` * chloe-whisper `en` ## Usage Use these voice names in your session configuration: #### JavaScript ```javascript const sessionConfig = { // rest of config... voice: "sophie", // Replace with any voice name from the list above }; ``` #### Swift ```swift let ttsConfig = OutspeedSDK.TTSConfig(voiceId: "sophie") ``` # Authentication Source: https://docs.outspeed.com/get-started/authentication Secure your voice agents with authentication and authorization ## Overview Outspeed provides flexible authentication options to balance ease of development with production security needs. ## Authentication Modes ### Development Mode (No Auth) Perfect for prototyping and development - no backend required. 1. **Create agent** in the [Outspeed Dashboard](https://dashboard.outspeed.com/agents) 2. **Disable authentication** in agent settings 3. **Use directly** from your client app ```typescript await conversation.startSession({ agentId: "your-agent-id", source: "my-app", }); ``` ### Production Mode (Auth Enabled) Recommended for production apps - requires backend for security. 1. **Enable authentication** in agent settings 2. **Set up backend** to [generate ephemeral keys](#backend-implementation) 3. **Pass ephemeral key** when starting sessions ```typescript const ephemeralKey = await getEphemeralKeyFromServer("your-agent-id"); await conversation.startSession({ agentId: "your-agent-id", source: "my-app", ephemeralKey: ephemeralKey, }); ``` ## Dashboard Configuration ### Creating an Agent 1. Go to [Outspeed Dashboard](https://dashboard.outspeed.com) 2. Click **Create Agent** 3. Configure your agent: * **Name**: Give your agent a name * **Voice**: Choose from available voices 4. Once created, you can configure other settings like system instructions, first message, etc. in the dashboard. ### Authentication Settings In your agent settings, you can: * **Toggle Authentication**: Enable/disable auth requirements * **Set Allowed Hostnames**: Restrict which domains can use your agent * **View Agent ID**: Copy the ID to use in your code ### Hostname Security Add allowed hostnames to restrict where your agent can be used: * `example.com` - Allow only this exact domain * `app.domain.com` - Allow specific subdomain We do not support wildcard domains yet. Sessions will only start if the request origin matches your allowlist. Leave empty to allow any origin. ## Backend Implementation ### Simple Agent-Based Auth For agents created in the dashboard, you only need the agent ID: ```javascript Express.js app.use(express.json()); app.post("/token", async (req, res) => { try { const { agentId } = req.body; const response = await fetch(`https://api.outspeed.com/v1/realtime/sessions?agent_id=${agentId}`, { method: "POST", headers: { Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`, "Content-Type": "application/json", }, }); if (!response.ok) { const error = await response.text(); console.error("Failed to generate ephemeral key:", error); res.status(response.status).json({ error: "Failed to generate token" }); return; } const data = await response.json(); res.json(data); } catch (error) { console.error("Failed to generate ephemeral key:", error); res.status(500).json({ error: "Internal server error" }); } }); ``` ```python FastAPI import os import httpx from fastapi import FastAPI, HTTPException OUTSPEED_API_KEY = os.getenv("OUTSPEED_API_KEY") app = FastAPI() @app.post("/token") async def create_token(request: dict): try: agent_id = request.get("agentId") async with httpx.AsyncClient() as client: response = await client.post( f"https://api.outspeed.com/v1/realtime/sessions?agent_id={agent_id}", headers={ "Authorization": f"Bearer {OUTSPEED_API_KEY}", "Content-Type": "application/json", }, ) if not response.is_success: print("Error generating ephemeral key:", response.text) raise HTTPException(status_code=response.status_code, detail=response.text) return response.json() except Exception as e: print("Error generating ephemeral key:", e) raise HTTPException(status_code=500, detail="Internal server error") ``` ### Frontend Integration ```typescript const getEphemeralKeyFromServer = async (agentId: string) => { const tokenResponse = await fetch("/token", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ agentId }), }); const data = await tokenResponse.json(); if (!tokenResponse.ok) { throw new Error("Failed to get ephemeral key"); } return data.client_secret.value; }; // Usage const ephemeralKey = await getEphemeralKeyFromServer("your-agent-id"); await conversation.startSession({ agentId: "your-agent-id", source: "my-app", ephemeralKey: ephemeralKey, }); ``` ## Legacy: Programmatic Configuration The new dashboard approach is recommended, but you can still configure agents programmatically for advanced use cases. ### Backend with Session Config ```javascript Express.js app.use(express.json()); app.post("/token", async (req, res) => { try { const response = await fetch("https://api.outspeed.com/v1/realtime/sessions", { method: "POST", headers: { Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify(req.body), }); if (!response.ok) { const error = await response.text(); console.error("failed to generate ephemeral key:", error); res.status(response.status).json({ error: "Failed to generate token" }); return; } const data = await response.json(); res.json(data); } catch (error) { console.error("failed to generate ephemeral key:", error); res.status(500).json({ error: "Internal server error" }); } }); ``` ```python FastAPI import os import httpx from fastapi import FastAPI, HTTPException, Request OUTSPEED_API_KEY = os.getenv("OUTSPEED_API_KEY") app = FastAPI() @app.post("/token") async def create_token(request: Request): try: session_config = await request.json() async with httpx.AsyncClient() as client: response = await client.post( "https://api.outspeed.com/v1/realtime/sessions", headers={ "Authorization": f"Bearer {OUTSPEED_API_KEY}", "Content-Type": "application/json", }, json=session_config, ) if not response.is_success: print("error generating ephemeral key:", response.text) raise HTTPException(status_code=response.status_code, detail=response.text) return response.json() except Exception as e: print("error generating ephemeral key:", e) raise HTTPException(status_code=500, detail="Internal server error") ``` ### Frontend with Session Config ```typescript import { type SessionConfig } from "@outspeed/client"; const getEphemeralKeyFromServer = async (config: SessionConfig) => { const tokenResponse = await fetch("/token", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(config), }); const data = await tokenResponse.json(); if (!tokenResponse.ok) { throw new Error("Failed to get ephemeral key"); } return data.client_secret.value; }; const sessionConfig: SessionConfig = { model: "outspeed-v1", instructions: "You are a helpful assistant.", voice: "sophie", temperature: 0.5, turn_detection: { type: "semantic_vad", }, first_message: "Hello! How can I help you today?", }; // Usage const ephemeralKey = await getEphemeralKeyFromServer(sessionConfig); await conversation.startSession(ephemeralKey, { source: "my-app", }); ``` ## Environment Setup ### Required Environment Variables ```bash # Your Outspeed API key from the dashboard OUTSPEED_API_KEY=your_outspeed_api_key_here ``` ### Security Best Practices **Never expose your API key in client-side code**. Always generate ephemeral tokens on your backend server. * **API Keys**: Keep server-side only, never in frontend code * **Ephemeral Keys**: Short-lived tokens for client authentication * **Hostname Allowlist**: Restrict origins that can use your agent * **HTTPS**: Always use HTTPS in production * **Rate Limiting**: Implement rate limiting on your token endpoint ## When to Use Each Mode ### Development Mode (No Auth) * ✅ **Prototyping** and development * ✅ **Internal tools** with trusted users * ✅ **Quick demos** and experiments * ❌ **Production apps** with external users ### Production Mode (Auth Enabled) * ✅ **Production applications** * ✅ **Public-facing** voice agents * ✅ **Hostname restrictions** for security ## Summary 1. **Start simple**: Use dashboard agents with auth disabled for development 2. **Add security**: Enable auth and set up backend for production 3. **Restrict access**: Use hostname allowlist for additional security Both approaches give you full control - choose based on your security and deployment needs. # OpenAI Compatibility Source: https://docs.outspeed.com/get-started/openai-compatibility Leverage OpenAI Compatibility with Outspeed Live API Outspeed Live API offers a seamless integration experience by being compatible with OpenAI's Realtime API. This means you can often use similar integration patterns and even reuse existing code. You can connect to Outspeed Live API in your applications using [WebRTC](https://webrtc.org/). The process mirrors the one described in [OpenAI's Realtime API WebRTC Guide](https://platform.openai.com/docs/guides/realtime#connect-with-webrtc). The key difference is the API endpoint: * **Outspeed Live API:** `api.outspeed.com` * **OpenAI API:** `api.openai.com` Here's a summary of the integration steps: ### 1. Obtain an Ephemeral Key First, your backend needs to obtain an ephemeral key from Outspeed. This process is similar to OpenAI's. You'll make a POST request to the Outspeed sessions endpoint. **Example: Creating a session and obtaining a key** ```javascript const outspeedApiHost = "api.outspeed.com"; // Or your model provider's URL const apiKey = "YOUR_OUTSPEED_API_KEY"; // Replace with your actual API key async function createRealtimeSession(sessionRequestBody) { const url = `https://${outspeedApiHost}/v1/realtime/sessions`; console.log(`👉 Creating session using: ${url}`); try { const response = await fetch(url, { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, "Content-Type": "application/json", }, body: JSON.stringify(sessionRequestBody), }); if (!response.ok) { const errorText = await response.text(); console.error("Session creation failed:", response.status, errorText); // Handle error appropriately in your application throw new Error(`Failed to create session: ${errorText}`); } const sessionData = await response.json(); console.log("🎉 Session created successfully!", sessionData); return sessionData; // This contains the ephemeral_key } catch (error) { console.error("Error during session creation:", error); throw error; } } ``` *Remember to replace `YOUR_OUTSPEED_API_KEY` with your actual Outspeed API key.* ### 2. Establish WebRTC Connection on Frontend Once you have the ephemeral key, use it on your application's frontend to establish a WebRTC connection with Outspeed Live API. You can find a client-side code snippet demonstrating this in our [Voice Devtools repository on GitHub](https://github.com/outspeed-ai/voice-devtools/blob/main/src/client/helpers/webrtc.ts#L30). ### 3. Reuse OpenAI Event Handlers A significant advantage of Outspeed's OpenAI compatibility is that the Live API emits the same events as OpenAI's Realtime API. This allows you to retain your existing event handling logic if you are migrating from or familiar with OpenAI's API. For more details on the specific events and API specifications, please refer to the [Outspeed Live API documentation](./api-spec/live-api). # Build with Outspeed Source: https://docs.outspeed.com/get-started/overview Add voice interactivity to your app
Outspeed Voice AI Platform

Embed voice in your mobile or web app

Outspeed helps you add a voice interface to your mobile and web apps. Build an empathetic therapist, a chatty companion or a professional AI interviewer: possibilities are endless.

## Overview Get started with Outspeed in minutes. Explore pre-built templates to jumpstart your voice AI projects. ## Examples Voice agent for HR interviews and onboarding. Empathetic voice agent for therapy and support. Friendly AI companion for conversation. Two AIs conversing with each other using voice. {/*

Templates

*/}

Capabilities

Remove unwanted background noise from audio. Identify and distinguish between speakers. Detect and express emotions in voice.
# Quickstart Source: https://docs.outspeed.com/get-started/quickstart Add a voice agent to your React app in minutes. ## What you'll learn In this quickstart you'll learn how to: 1. Create an agent in the Outspeed Dashboard 2. Install the Outspeed SDK 3. Build a voice agent that can talk to users ## Step 1: Create Agent in Dashboard 1. Go to the [Outspeed Dashboard](https://dashboard.outspeed.com) and sign in 2. Create a new agent with your desired configuration: * **Instructions**: "You are a helpful assistant" * **Voice**: Choose from available voices like `sophie`, `david`, etc. 3. **Important**: Make sure **Authentication** is **disabled** for this quickstart 4. Copy your **Agent ID** - you'll need it in the next step For production apps, you'll want to enable authentication. See our [Authentication Guide](/get-started/authentication) for details. ## Step 2: Install Outspeed Install the React SDK using your preferred package manager: ```bash npm npm install @outspeed/client @outspeed/react ``` ```bash pnpm pnpm add @outspeed/client @outspeed/react ``` ```bash yarn yarn add @outspeed/client @outspeed/react ``` ## Step 3: Create React component Here's a complete working example: ```tsx import { useConversation } from "@outspeed/react"; import { useState } from "react"; // Replace with your Agent ID from the dashboard const AGENT_ID = "your-agent-id-here"; export default function App() { const [sessionCreated, setSessionCreated] = useState(false); const conversation = useConversation({ onDisconnect: () => { console.log("Disconnected! cleaning up..."); setSessionCreated(false); }, }); const startSession = async () => { try { conversation.on("session.created", () => { setSessionCreated(true); }); await conversation.startSession({ agentId: AGENT_ID, source: "quickstart" }); } catch (error) { console.error("Error starting session", error); } }; const endSession = async () => { try { await conversation.endSession(); } catch (error) { console.error("Error ending session", error); } finally { setSessionCreated(false); } }; if (sessionCreated) { return (

Voice session active!

Start talking to your AI assistant

); } return (

Voice AI Assistant

Click the button to start talking

); } ``` Want to customize your agent? Edit the configuration in the [Outspeed Dashboard](https://dashboard.outspeed.com) - changes apply instantly without code updates! ## That's it! You now have a working voice agent that can: * Start and end voice sessions with zero backend setup * Talk to users with AI-powered responses * Handle voice activity detection automatically * Be customized entirely through the dashboard ## More Examples Want to see complete implementations with different frameworks? Check out our [Examples Repository](https://github.com/outspeed-ai/outspeed-examples) which includes: * **React + Vite + FastAPI** - Full-featured voice assistant with weather tools * **React + Vite + Cloudflare Functions** - Serverless voice assistant * **Create React App + FastAPI** - Simple React setup with Python backend * **Next.js** - complete Next.js voice assistant (also includes AI-to-AI chat) * **Swift iOS** - Native iOS voice bot with SwiftUI ## Next Steps * **Add Authentication**: For production apps, enable authentication in your agent settings and follow our [Authentication Guide](/get-started/authentication) * **Add Tools**: Give your agent superpowers with [client tools](/features/client-tools) and [system tools](/features/system-tools) * **Customize Behavior**: Explore [voices](/features/voices), [context management](/features/context-management), and more features # Templates Source: https://docs.outspeed.com/get-started/templates Ready-to-use templates for building voice applications with Outspeed *** Get started quickly with our pre-built templates for different frameworks and platforms. | Template | Framework | Platform | Link | | ---------------- | -------------------- | -------- | --------------------------------------------------------------------------------------------- | | Next.js | Next.js | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/next-js-basic) | | Vite + FastAPI | React, Vite, FastAPI | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/react-vite-fastapi) | | Cloudflare Pages | React, Vite | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/react-vite-cloudflare) | | Create React App | React, FastAPI | Web | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/create-react-app-fastapi) | | Mobile App (iOS) | Swift | Mobile | [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/swift-ios-basic) | ## Need Help? If you need assistance with any template or want to request a new one: * Visit our [Quickstart Guide](/get-started/quickstart) for step-by-step instructions * Join our community for support and discussions * Raise an issue on [GitHub](https://github.com/outspeed-ai/outspeed-examples/issues) # Migrate from ElevenLabs SwiftSDK Source: https://docs.outspeed.com/iOS/outspeed-swift-elevenlabs How to migrate from ElevenLabs Swift SDK to Outspeed Swift SDK. # ElevenLabs Swift Compatibility OutspeedSDK is fully compatible with Elevenlabs Swift SDK specifications (with exception of some features.) If you are migrating from ElevenLabsSDK, follow these steps: ## 1. Replace Imports and Configs Replace all occurrences of `ElevenLabsSDK` with `OutspeedSDK`. For example: ```swift import ElevenLabsSDK let config = ElevenLabsSDK.SessionConfig( agentId : "testagent") ``` becomes ```swift import OutspeedSDK let config = OutspeedSDK.SessionConfig( agentId : "testagent") ``` ## 2. Add Your Outspeed API Key Update your `startSession` call to include your Outspeed API key: ```swift let conversation = try await ElevenLabsSDK.Conversation.startSession( config: config, callbacks: callbacks ) ``` becomes ```swift let conversation = try await OutspeedSDK.Conversation.startSession( config: config, callbacks: callbacks apiKey: "" ) ``` Refer to a working example on [GitHub](https://github.com/outspeed-ai/outspeed-examples/tree/main/swift-ios-basic). # Swift SDK: Getting Started Source: https://docs.outspeed.com/iOS/outspeed-swift-getting-started Add real-time voice conversations to your iOS app with the Outspeed Swift SDK ## Features * Real-time voice conversations with AI * Support for both Outspeed and OpenAI providers * WebRTC-based audio streaming * Customizable voice and model selection * ElevenLabs Conversational AI compatibility The Outspeed Swift SDK is fully compatible with the ElevenLabs Swift SDK. See our{" "} ElevenLabs Compatibility guide for migration details. ## Installation 1. Open Your Project in Xcode 2. Go to `File` > `Add Packages...` 3. Enter Repository URL: `https://github.com/outspeed-ai/OutspeedSwift` 4. Import the SDK ```swift import OutspeedSDK ``` Ensure `NSMicrophoneUsageDescription` is added to your Info.plist to explain microphone access. ## Requirements * iOS 15.2 or later * Swift 6.1+ ## Basic Usage Here's the minimal code to get a voice conversation working: ```swift import OutspeedSDK // Create a session configuration let config = OutspeedSDK.SessionConfig() // Create callbacks to handle events let callbacks = OutspeedSDK.Callbacks() callbacks.onMessage = { message, role in print("Received message from \(role.rawValue): \(message)") } callbacks.onError = { message, error in print("Error: \(message)") } callbacks.onStatusChange = { status in print("Status changed to: \(status.rawValue)") } // Start the conversation Task { do { let conversation = try await OutspeedSDK.Conversation.startSession( config: config, callbacks: callbacks, apiKey: "" ) // When done with the conversation conversation.endSession() } catch { print("Failed to start conversation: \(error)") } } ``` ## Customization ### System Prompt and First Message ```swift let agentConfig = OutspeedSDK.AgentConfig( prompt: "You are a helpful assistant with a witty personality.", firstMessage: "Hey there, how can I help you with Outspeed today?" ) let config = OutspeedSDK.SessionConfig( overrides: OutspeedSDK.ConversationConfigOverride(agent: agentConfig) ) ``` ### Voice Selection ```swift let ttsConfig = OutspeedSDK.TTSConfig(voiceId: OutspeedSDK.Voice.david.rawValue) let config = OutspeedSDK.SessionConfig( overrides: OutspeedSDK.ConversationConfigOverride(tts: ttsConfig) ) ``` ## Complete Example See a full iOS voice bot application example at our [Examples Repository](https://github.com/outspeed-ai/outspeed-examples/tree/main/swift-ios-basic). # React SDK Example Source: https://docs.outspeed.com/react/example Build your first voice AI application with the Outspeed React SDK ## Basic Voice Chat Component Here's a complete example of a React component that creates a voice conversation using the `useConversation` hook: ```tsx import React, { useState } from "react"; import { type SessionConfig } from "@outspeed/client"; import { useConversation } from "@outspeed/react"; const getEphemeralKeyFromServer = async (config: SessionConfig) => { const tokenResponse = await fetch("/token", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(config), }); const data = await tokenResponse.json(); if (!tokenResponse.ok) { throw new Error("Failed to get ephemeral key"); } return data.client_secret.value; }; const sessionConfig: SessionConfig = { model: "outspeed-v1", instructions: "You are a helpful but witty assistant named Alfred.", voice: "david", // see the voices page for all available voices turn_detection: { type: "semantic_vad", }, first_message: "Hello, how can I assist you with Outspeed today?", }; export default function VoiceChat() { const [sessionCreated, setSessionCreated] = useState(false); const [isConnecting, setIsConnecting] = useState(false); const conversation = useConversation({ sessionConfig: sessionConfig, }); const startSession = async () => { try { setIsConnecting(true); const ephemeralKey = await getEphemeralKeyFromServer(sessionConfig); await conversation.startSession(ephemeralKey, { source: "react-example" // Optional: custom string to identify session source }); // Listen for session creation event conversation.on("session.created", (event) => { console.log("Session created:", event); setSessionCreated(true); setIsConnecting(false); }); } catch (error) { console.error("Error starting session:", error); setIsConnecting(false); } }; const endSession = async () => { try { await conversation.endSession(); setSessionCreated(false); } catch (error) { console.error("Error ending session:", error); } }; if (isConnecting) { return (

Connecting...

Please wait while we establish the connection.

); } if (sessionCreated) { return (

🎙️ Voice Chat Active

You can now speak with the AI assistant!

); } return (

Voice AI Assistant

Click the button below to start a voice conversation.

); } ``` Want to use a different voice? See all [available voices](/features/voices) you can choose from. ## Connection Management ### onDisconnect Callback Use `onDisconnect` to handle cleanup when the conversation ends: ```tsx const conversation = useConversation({ onDisconnect: () => { console.log("Disconnected! cleaning up..."); setSessionCreated(false); // Add any cleanup logic here }, }); ``` **Important**: Use `onDisconnect` for cleanup logic, not `onError`. The `onError` callback is called for runtime errors as well, so it's not ideal for cleanup tasks. ### onError Callback Use `onError` specifically for handling errors: ```tsx const conversation = useConversation({ onError: (err) => { console.error("Conversation error:", err); // Handle error display/logging only }, onDisconnect: () => { console.log("Session ended"); setSessionCreated(false); // Handle cleanup here instead }, }); ``` ## Advanced Features ### Event Handling You can listen to various events during the conversation: ```tsx // Listen for speech detection conversation.on("input_audio_buffer.speech_started", () => { console.log("User started speaking"); }); conversation.on("input_audio_buffer.speech_stopped", () => { console.log("User stopped speaking"); }); // Listen for AI responses conversation.on("response.text.delta", (event) => { console.log("AI response text:", event.delta); }); conversation.on("response.audio_transcript.delta", (event) => { console.log("AI speech transcript:", event.delta); }); ``` ### Text Input You can also send text messages programmatically: ```tsx const sendTextMessage = () => { conversation.sendText("Tell me about the weather today"); }; return (
{/* ... other components */}
); ``` ### Mute Control Control the microphone state by passing `micMuted` prop to `useConversation`: ```tsx const [isMicMuted, setIsMicMuted] = useState(false); const conversation = useConversation({ micMuted: isMicMuted, }); const toggleMute = () => { setIsMicMuted(!isMicMuted); }; return (
{/* ... other components */}
); ``` ### Volume Control Control the AI's voice volume by passing `volume` prop to `useConversation` (value between 0 and 1): ```tsx const [volume, setVolume] = useState(0.8); const conversation = useConversation({ volume: volume, }); return (
{/* ... other components */}
); ``` ## Complete Example with Error Handling Here's a more complete example showing proper error and disconnect handling: ```tsx const [error, setError] = useState(null); const [sessionCreated, setSessionCreated] = useState(false); const conversation = useConversation({ onError: (err) => { console.error("Conversation error:", err); setError(err.message); // Only handle error display here }, onDisconnect: () => { console.log("Session ended"); setSessionCreated(false); setError(null); // Clear any previous errors // Handle all cleanup logic here }, }); // Display error to user if (error) { return (

❌ Error

{error}

); } ``` ## Next Steps * [API Reference](/api-spec/client) - Explore all available methods and events * [Starter Templates](/get-started/templates) - See template implementations # React SDK Setup Source: https://docs.outspeed.com/react/setup Get started with the Outspeed React SDK for realtime voice AI applications ## Installation Install the React SDK using your preferred package manager: ```bash npm npm install @outspeed/react ``` ```bash pnpm pnpm add @outspeed/react ``` ```bash yarn yarn add @outspeed/react ``` ## Prerequisites Before using the React SDK, you'll need: 1. **Outspeed API Key**: Get your API key from the [Outspeed Dashboard](https://dashboard.outspeed.com) 2. **Backend Token Endpoint**: A server endpoint to generate ephemeral keys for client authentication ## Backend Setup ```javascript Express.js app.use(express.json()); app.post("/token", async (req, res) => { try { const response = await fetch("https://api.outspeed.com/v1/realtime/sessions", { method: "POST", headers: { Authorization: `Bearer ${process.env.OUTSPEED_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify(req.body), }); if (!response.ok) { const error = await response.text(); console.error("failed to generate ephemeral key:", error); res.status(response.status).json({ error: "Failed to generate token" }); return; } const data = await response.json(); res.json(data); } catch (error) { console.error("failed to generate ephemeral key:", error); res.status(500).json({ error: "Internal server error" }); } }); ``` ```python FastAPI import os import httpx from fastapi import FastAPI, HTTPException, Request OUTSPEED_API_KEY = os.getenv("OUTSPEED_API_KEY") app = FastAPI() @app.post("/token") async def create_token(request: Request): try: session_config = await request.json() async with httpx.AsyncClient() as client: response = await client.post( "https://api.outspeed.com/v1/realtime/sessions", headers={ "Authorization": f"Bearer {OUTSPEED_API_KEY}", "Content-Type": "application/json", }, json=session_config, ) if not response.is_success: print("error generating ephemeral key:", response.text) raise HTTPException(status_code=response.status_code, detail=response.text) return response.json() except Exception as e: print("error generating ephemeral key:", e) raise HTTPException(status_code=500, detail="Internal server error") ``` ## Environment Variables Add your Outspeed API key to your server's environment variables: ```bash OUTSPEED_API_KEY=your_outspeed_api_key_here ``` Never expose your Outspeed API key in client-side code. Always generate ephemeral tokens on your backend server. ## Session Configuration The React SDK uses a `SessionConfig` object to configure voice sessions: ```typescript import { type SessionConfig } from "@outspeed/client"; const sessionConfig: SessionConfig = { model: "outspeed-v1", instructions: "You are a helpful assistant.", voice: "david", // see the voices page for all available voices turn_detection: { type: "semantic_vad", }, first_message: "Hello! How can I help you today?", // Optional welcome message }; ``` Want to use a different voice? See all [available voices](/features/voices) you can choose from. ### Configuration Options | Option | Type | Description | | ---------------- | -------- | --------------------------------------- | | `model` | `string` | Must be `"outspeed-v1"` | | `instructions` | `string` | System prompt for the AI assistant | | `voice` | `string` | Voice ID to use for speech synthesis | | `turn_detection` | `object` | Voice activity detection settings | | `first_message` | `string` | Optional initial message from assistant | You can find available voices [here](/features/voices) ## Next Steps Now that you have the SDK installed and configured, you can start building voice AI applications: * [Basic Example](/react/example) - Learn how to create a simple voice conversation * [API Reference](/api-spec/client) - Explore all available methods and events