WebSocket API Guide

Overview

The VoxNexus WebSocket API provides real-time bidirectional communication for voice services. It’s ideal for applications requiring low-latency, interactive voice processing such as:

Real-time voice assistants
Live transcription services
Interactive voice response (IVR) systems
Voice-controlled applications
Real-time captioning

Connection

Endpoints

Text-to-Speech: wss://api.voxnexus.ai/v1/tts/realtime
Speech-to-Text: wss://api.voxnexus.ai/v1/stt/realtime

Authentication

VoxNexus WebSocket API supports two authentication methods: Option 1: Query Parameter (Recommended) Append your API key as a query parameter. This method is recommended as it works with all WebSocket clients:

const ws = new WebSocket('wss://api.voxnexus.ai/v1/tts/realtime?token=YOUR_API_KEY');

Option 2: Header Use X-Api-Key header (may not work with all clients):

const ws = new WebSocket('wss://api.voxnexus.ai/v1/tts/realtime', {
  headers: {
    'X-Api-Key': 'YOUR_API_KEY'
  }
});

Query parameter authentication is recommended for browser-based applications as the standard WebSocket API doesn’t support custom headers.

Connection Lifecycle

Connect: Establish WebSocket connection with authentication
Initialize: Send initialization message with configuration
Ready: Receive ready confirmation from server
Exchange: Send/receive data messages
Close: Gracefully close connection when done

Text-to-Speech WebSocket

For compatible voices, the optional intervention_group_id in init selects one pronunciation group owned by your account. Omission selects default; the session uses only that group and system rules. The selected group is fixed for the session: send a new init on a new connection to use another group, not a group field on text messages. Explicit selection on an unsupported provider, including default, is rejected with INTERVENTION_NOT_SUPPORTED. IDs outside your account cannot select another user’s rules. The initialization language follows the same BCP 47 locale rules as HTTP synthesis. Omit it to use the available primary locale of the selected model and voice, or its sole supported locale. A broad tag such as en uses a matching primary locale or sole match. If several matches remain without a default, select a locale explicitly. An explicit region such as en-GB must be supported and does not fall back to en-US. Use model-filtered voice capabilities to populate locale choices. For a compatible model and voice, initialize a session with a group returned by the group API:

{
  "type": "init",
  "model_id": "YOUR_COMPATIBLE_MODEL_ID",
  "voice_id": "YOUR_COMPATIBLE_VOICE_ID",
  "intervention_group_id": "YOUR_GROUP_ID",
  "language": "en",
  "format": "pcm",
  "sample_rate": 24000
}

Message Types

Client Messages

Initialization (init)

{
  "type": "init",
  "model_id": "vn-tts-basic",
  "voice_id": "vn-xiaoxiao",
  "language": "zh-CN",
  "format": "wav",
  "sample_rate": 16000,
  "speed": 1.0,
  "pitch": 0,
  "volume": 1.0,
  "voice_config": {
    "style": "cheerful",
    "role": "Girl",
    "degree": 0.5
  }
}

Text (text)

{
  "type": "text",
  "text": "Hello, this is a test message",
  "is_final": false
}

Server Messages

Ready (ready)

{
  "type": "ready",
  "request_id": "req_1234567890",
  "voice_id": "vn-xiaoxiao",
  "language": "zh-CN",
  "format": "wav",
  "sample_rate": 16000
}

Audio (audio)

{
  "type": "audio",
  "data": "base64-encoded-audio-data",
  "is_final": false
}

Error (error)

{
  "type": "error",
  "error": "Invalid voice_id",
  "code": "VOICE_NOT_FOUND",
  "request_id": "req_1234567890"
}

Complete Example

class TTSWebSocketClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.ws = null;
    this.ready = false;
  }
  
  connect() {
    return new Promise((resolve, reject) => {
      // Use query parameter for authentication (works with all clients)
      this.ws = new WebSocket(`wss://api.voxnexus.ai/v1/tts/realtime?token=${this.apiKey}`);
      
      this.ws.onopen = () => {
        console.log('Connected');
        resolve();
      };
      
      this.ws.onmessage = (event) => {
        this.handleMessage(JSON.parse(event.data));
      };
      
      this.ws.onerror = (error) => {
        console.error('WebSocket error:', error);
        reject(error);
      };
      
      this.ws.onclose = () => {
        console.log('Connection closed');
        this.ready = false;
      };
    });
  }
  
  initialize(config) {
    this.ws.send(JSON.stringify({
      type: 'init',
      ...config
    }));
  }
  
  synthesize(text, isFinal = false) {
    if (!this.ready) {
      throw new Error('Not ready. Wait for ready message.');
    }
    
    this.ws.send(JSON.stringify({
      type: 'text',
      text: text,
      is_final: isFinal
    }));
  }
  
  handleMessage(message) {
    switch (message.type) {
      case 'ready':
        this.ready = true;
        console.log('Ready:', message.request_id);
        break;
        
      case 'audio':
        this.onAudio(message.data, message.is_final);
        break;
        
      case 'error':
        console.error('Error:', message.error);
        this.onError(message);
        break;
    }
  }
  
  onAudio(base64Data, isFinal) {
    // Decode and handle audio
    const audioData = atob(base64Data);
    // Play audio or save to buffer
  }
  
  onError(error) {
    // Handle error
  }
  
  close() {
    if (this.ws) {
      this.ws.close();
    }
  }
}

// Usage
const client = new TTSWebSocketClient('YOUR_API_KEY');
await client.connect();
client.initialize({
  model_id: 'vn-tts-basic',
  voice_id: 'vn-xiaoxiao',
  format: 'wav',
  sample_rate: 16000
});

// Wait for ready, then synthesize
setTimeout(() => {
  client.synthesize('Hello, world!', true);
}, 1000);

Speech-to-Text WebSocket

Message Types

Client Messages

Initialization (init)

{
  "type": "init",
  "model_id": "vn-stt-basic",
  "language": "zh-CN",
  "format": "pcm",
  "sample_rate": 16000,
  "enable_timestamps": true,
  "enable_llm_transform": true,
  "llm_prompt": "Correct punctuation and remove filler words",
  "llm_mode": "per_segment"
}

Audio (audio)

{
  "type": "audio",
  "data": "base64-encoded-audio-chunk"
}

Command (command)

{
  "type": "command",
  "command": "flush"
}

The flush command tells the server that no more audio will be sent. The server will respond with a flush_done message after completing recognition. Subsequent audio will start a new recognition session.

Server Messages

Ready (ready)

{
  "type": "ready",
  "request_id": "req_1234567890",
  "language": "zh-CN",
  "format": "pcm",
  "sample_rate": 16000
}

Transcript (transcript)

{
  "type": "transcript",
  "request_id": "req_1234567890",
  "segment_id": "550e8400-e29b-41d4-a716-446655440000",
  "text": "Hello, this is a test message.",
  "is_final": true,
  "language": "en",
  "confidence": 0.95,
  "offset": 0,
  "duration": 2500,
  "words": [
    {
      "word": "hello",
      "offset": 0,
      "duration": 500,
      "confidence": 0.98
    }
  ]
}

The is_final field distinguishes between partial results (false) and complete sentences (true). Confidence scores and word-level information are only valid when is_final is true. segment_id is only present when is_final=true and LLM transform is enabled.

LLM Transform (llm)

{
  "type": "llm",
  "request_id": "req_1234567890",
  "segment_id": "550e8400-e29b-41d4-a716-446655440000",
  "delta": "Hello,",
  "is_final": false
}

Sent only when enable_llm_transform=true. Each final transcript segment triggers a sequence of llm messages streaming the LLM output incrementally. Concatenate all delta values until is_final=true. segment_id links the output to its corresponding transcript message (absent for a full-text post-flush pass).

Flush Done (flush_done)

{
  "type": "flush_done",
  "request_id": "req_1234567890"
}

Error (error)

{
  "type": "error",
  "error": "Invalid audio format",
  "code": "UNSUPPORTED_FORMAT",
  "request_id": "req_1234567890"
}

Complete Example

class STTWebSocketClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.ws = null;
    this.ready = false;
    this.audioContext = null;
    this.processor = null;
  }
  
  async connect() {
    return new Promise((resolve, reject) => {
      // Use query parameter for authentication (works with all clients)
      this.ws = new WebSocket(`wss://api.voxnexus.ai/v1/stt/realtime?token=${this.apiKey}`);
      
      this.ws.onopen = () => {
        console.log('Connected');
        resolve();
      };
      
      this.ws.onmessage = (event) => {
        this.handleMessage(JSON.parse(event.data));
      };
      
      this.ws.onerror = (error) => {
        console.error('WebSocket error:', error);
        reject(error);
      };
      
      this.ws.onclose = () => {
        console.log('Connection closed');
        this.ready = false;
      };
    });
  }
  
  initialize(config) {
    this.ws.send(JSON.stringify({
      type: 'init',
      ...config
    }));
  }
  
  async startRecording(sampleRate = 16000) {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    this.audioContext = new AudioContext({ sampleRate });
    const source = this.audioContext.createMediaStreamSource(stream);
    
    this.processor = this.audioContext.createScriptProcessor(4096, 1, 1);
    this.processor.onaudioprocess = (e) => {
      if (!this.ready) return;
      
      const audioData = e.inputBuffer.getChannelData(0);
      const pcm16 = new Int16Array(audioData.length);
      
      for (let i = 0; i < audioData.length; i++) {
        pcm16[i] = Math.max(-32768, Math.min(32767, audioData[i] * 32768));
      }
      
      const base64 = btoa(String.fromCharCode(...new Uint8Array(pcm16.buffer)));
      this.ws.send(JSON.stringify({
        type: 'audio',
        data: base64
      }));
    };
    
    source.connect(this.processor);
    this.processor.connect(this.audioContext.destination);
  }
  
  stopRecording() {
    if (this.processor) {
      this.processor.disconnect();
      this.processor = null;
    }
    if (this.audioContext) {
      this.audioContext.close();
      this.audioContext = null;
    }
  }
  
  flush() {
    if (!this.ready) {
      throw new Error('Not ready. Wait for ready message.');
    }
    
    this.ws.send(JSON.stringify({
      type: 'command',
      command: 'flush'
    }));
  }
  
  handleMessage(message) {
    switch (message.type) {
      case 'ready':
        this.ready = true;
        console.log('Ready:', message.request_id);
        break;
        
      case 'transcript':
        if (message.is_final) {
          this.onFinal(message);
        } else {
          this.onPartial(message);
        }
        break;
        
      case 'llm':
        this.onLLMDelta(message);
        break;
        
      case 'flush_done':
        console.log('Flush done:', message.request_id);
        break;
        
      case 'error':
        console.error('Error:', message.error);
        this.onError(message);
        break;
    }
  }
  
  onPartial(transcript) {
    console.log('Partial:', transcript.text);
    // Update UI with interim results
  }
  
  onFinal(transcript) {
    console.log('Final:', transcript.text);
    if (transcript.language) {
      console.log('Detected language:', transcript.language);
    }
    if (transcript.confidence) {
      console.log('Confidence:', transcript.confidence);
    }
    // Handle final transcription
  }
  
  onLLMDelta(message) {
    // Accumulate streaming LLM output
    const key = message.segment_id ?? '__post_flush__';
    if (!this._llmBuffers) this._llmBuffers = {};
    this._llmBuffers[key] = (this._llmBuffers[key] ?? '') + message.delta;
    if (message.is_final) {
      console.log('LLM output:', this._llmBuffers[key]);
      delete this._llmBuffers[key];
    }
  }
  
  onError(error) {
    // Handle error
  }
  
  close() {
    this.stopRecording();
    if (this.ws) {
      this.ws.close();
    }
  }
}

// Usage
const client = new STTWebSocketClient('YOUR_API_KEY');
await client.connect();
client.initialize({
  model_id: 'vn-stt-basic',
  format: 'pcm',
  sample_rate: 16000,
  enable_timestamps: true
});

// Wait for ready, then start recording
setTimeout(async () => {
  await client.startRecording(16000);
}, 1000);

Best Practices

Connection Management

Reconnection Logic

class ReconnectingWebSocket {
  constructor(url, options) {
    this.url = url;
    this.options = options;
    this.ws = null;
    this.reconnectAttempts = 0;
    this.maxReconnectAttempts = 5;
    this.reconnectDelay = 1000;
  }
  
  connect() {
    this.ws = new WebSocket(this.url, this.options);
    
    this.ws.onclose = () => {
      if (this.reconnectAttempts < this.maxReconnectAttempts) {
        setTimeout(() => {
          this.reconnectAttempts++;
          this.connect();
        }, this.reconnectDelay * this.reconnectAttempts);
      }
    };
    
    this.ws.onopen = () => {
      this.reconnectAttempts = 0;
    };
  }
}

Heartbeat/Ping

// Send periodic ping to keep connection alive
setInterval(() => {
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: 'ping' }));
  }
}, 30000); // Every 30 seconds

Error Handling

ws.onerror = (error) => {
  console.error('WebSocket error:', error);
  // Implement retry logic or notify user
};

ws.onclose = (event) => {
  if (event.code !== 1000) { // Not a normal closure
    console.error('Unexpected closure:', event.code, event.reason);
    // Attempt reconnection
  }
};

Audio Processing

Chunk Size Optimization

Send audio chunks of 100-200ms for optimal latency
Too small: Increased overhead
Too large: Increased latency

Buffer Management

class AudioBuffer {
  constructor() {
    this.buffer = [];
    this.chunkSize = 1600; // 100ms at 16kHz
  }
  
  addAudio(audioData) {
    this.buffer.push(...audioData);
    
    while (this.buffer.length >= this.chunkSize) {
      const chunk = this.buffer.splice(0, this.chunkSize);
      this.sendChunk(chunk);
    }
  }
  
  sendChunk(chunk) {
    const base64 = btoa(String.fromCharCode(...chunk));
    ws.send(JSON.stringify({
      type: 'audio',
      data: base64
    }));
  }
  
  flush() {
    if (this.buffer.length > 0) {
      this.sendChunk(this.buffer);
      this.buffer = [];
    }
  }
}

Performance Optimization

Batch Text Messages

// For TTS, batch multiple text segments
const texts = ['Hello', 'world', 'this', 'is', 'a', 'test'];
texts.forEach((text, index) => {
  setTimeout(() => {
    client.synthesize(text, index === texts.length - 1);
  }, index * 100);
});

Throttle Audio Sending

class ThrottledAudioSender {
  constructor(ws, interval = 100) {
    this.ws = ws;
    this.interval = interval;
    this.queue = [];
    this.timer = null;
  }
  
  send(audioData) {
    this.queue.push(audioData);
    
    if (!this.timer) {
      this.timer = setInterval(() => {
        if (this.queue.length > 0) {
          const data = this.queue.shift();
          this.ws.send(JSON.stringify({
            type: 'audio',
            data: data
          }));
        } else {
          clearInterval(this.timer);
          this.timer = null;
        }
      }, this.interval);
    }
  }
}

Common Patterns

Bidirectional Voice Conversation

// Combine TTS and STT for voice conversation
class VoiceConversation {
  constructor(apiKey) {
    this.ttsClient = new TTSWebSocketClient(apiKey);
    this.sttClient = new STTWebSocketClient(apiKey);
  }
  
  async start() {
    await Promise.all([
      this.ttsClient.connect(),
      this.sttClient.connect()
    ]);
    
    this.ttsClient.initialize({ model_id: 'vn-tts-basic', voice_id: 'vn-xiaoxiao' });
    this.sttClient.initialize({ model_id: 'vn-stt-basic', format: 'pcm', sample_rate: 16000, enable_timestamps: false});
    
    // Handle STT results and respond with TTS
    this.sttClient.onFinal = (result) => {
      const response = this.processUserInput(result.text);
      this.ttsClient.synthesize(response, true);
    };
    
    await this.sttClient.startRecording();
  }
  
  processUserInput(text) {
    // Process user input and generate response
    return `You said: ${text}`;
  }
}

Real-time Captioning

class LiveCaptioning {
  constructor(apiKey, captionElement) {
    this.sttClient = new STTWebSocketClient(apiKey);
    this.captionElement = captionElement;
  }
  
  async start() {
    await this.sttClient.connect();
    this.sttClient.initialize({
      model_id: 'vn-stt-basic',
      format: 'pcm',
      sample_rate: 16000,
      enable_timestamps: true
    });
    
    this.sttClient.onPartial = (transcript) => {
      this.captionElement.textContent = transcript.text;
      this.captionElement.classList.add('interim');
    };
    
    this.sttClient.onFinal = (transcript) => {
      this.captionElement.textContent = transcript.text;
      this.captionElement.classList.remove('interim');
      // Save final caption with timestamp
      this.saveCaption(transcript);
    };
    
    await this.sttClient.startRecording();
  }
  
  saveCaption(transcript) {
    // Save caption with timing information
    const endTime = transcript.offset + transcript.duration;
    console.log(`[${transcript.offset}-${endTime}ms] ${transcript.text}`);
  }
}

Troubleshooting

Connection Issues

Problem: Connection fails immediately

Solution: Verify API key is correct and has proper permissions
Solution: Check network connectivity and firewall settings

Problem: Connection drops frequently

Solution: Implement reconnection logic with exponential backoff
Solution: Check for network instability or proxy issues

Audio Issues

Problem: No audio received (TTS)

Solution: Verify initialization message was sent and ready message received
Solution: Check that text messages are being sent correctly

Problem: Recognition not working (STT)

Solution: Verify audio format and sample rate match initialization
Solution: Check that audio chunks are being sent continuously
Solution: Ensure audio quality is sufficient (no excessive noise)

Performance Issues

Problem: High latency

Solution: Reduce audio chunk size for faster processing
Solution: Use appropriate sample rates (16kHz is usually sufficient)
Solution: Optimize network connection (use closer server if available)

Problem: High memory usage

Solution: Process and discard audio chunks after sending
Solution: Limit buffer sizes for audio data
Solution: Close unused connections promptly

Getting Started

REST API

WebSocket API

WebSocket API Guide

Overview

Connection

Endpoints

Authentication

Connection Lifecycle

Text-to-Speech WebSocket

Message Types

Client Messages

Server Messages

Complete Example

Speech-to-Text WebSocket

Message Types

Client Messages

Server Messages

Complete Example

Best Practices

Connection Management

Error Handling

Audio Processing

Performance Optimization

Common Patterns

Bidirectional Voice Conversation

Real-time Captioning

Troubleshooting

Connection Issues

Audio Issues

Performance Issues

​Overview

​Connection

​Endpoints

​Authentication

​Connection Lifecycle

​Text-to-Speech WebSocket

​Message Types

​Client Messages

​Server Messages

​Complete Example

​Speech-to-Text WebSocket

​Message Types

​Client Messages

​Server Messages

​Complete Example

​Best Practices

​Connection Management

​Error Handling

​Audio Processing

​Performance Optimization

​Common Patterns

​Bidirectional Voice Conversation

​Real-time Captioning

​Troubleshooting

​Connection Issues

​Audio Issues

​Performance Issues

Overview

Connection

Endpoints

Authentication

Connection Lifecycle

Text-to-Speech WebSocket

Message Types

Client Messages

Server Messages

Complete Example

Speech-to-Text WebSocket

Message Types

Client Messages

Server Messages

Complete Example

Best Practices

Connection Management

Error Handling

Audio Processing

Performance Optimization

Common Patterns

Bidirectional Voice Conversation

Real-time Captioning

Troubleshooting

Connection Issues

Audio Issues

Performance Issues