> ## Documentation Index
> Fetch the complete documentation index at: https://docs.voxnexus.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocket API Guide

> Complete guide to using VoxNexus WebSocket API for real-time voice services

## Overview

The VoxNexus WebSocket API provides real-time bidirectional communication for voice services. It's ideal for applications requiring low-latency, interactive voice processing such as:

* Real-time voice assistants
* Live transcription services
* Interactive voice response (IVR) systems
* Voice-controlled applications
* Real-time captioning

## Connection

### Endpoints

* **Text-to-Speech**: `wss://api.voxnexus.ai/v1/tts/realtime`
* **Speech-to-Text**: `wss://api.voxnexus.ai/v1/stt/realtime`

### Authentication

VoxNexus WebSocket API supports two authentication methods:

**Option 1: Query Parameter (Recommended)**

Append your API key as a query parameter. This method is recommended as it works with all WebSocket clients:

```javascript theme={null}
const ws = new WebSocket('wss://api.voxnexus.ai/v1/tts/realtime?token=YOUR_API_KEY');
```

**Option 2: Header**

Use `X-Api-Key` header (may not work with all clients):

```javascript theme={null}
const ws = new WebSocket('wss://api.voxnexus.ai/v1/tts/realtime', {
  headers: {
    'X-Api-Key': 'YOUR_API_KEY'
  }
});
```

<Note>
  Query parameter authentication is recommended for browser-based applications as the standard WebSocket API doesn't support custom headers.
</Note>

### Connection Lifecycle

1. **Connect**: Establish WebSocket connection with authentication
2. **Initialize**: Send initialization message with configuration
3. **Ready**: Receive ready confirmation from server
4. **Exchange**: Send/receive data messages
5. **Close**: Gracefully close connection when done

## Text-to-Speech WebSocket

### Message Types

#### Client Messages

**Initialization (`init`)**

```json theme={null}
{
  "type": "init",
  "model_id": "vn-tts-basic",
  "voice_id": "vn-xiaoxiao",
  "language": "zh-CN",
  "format": "wav",
  "sample_rate": 16000,
  "speed": 1.0,
  "pitch": 0,
  "volume": 1.0,
  "voice_config": {
    "style": "cheerful",
    "role": "Girl",
    "degree": 0.5
  }
}
```

**Text (`text`)**

```json theme={null}
{
  "type": "text",
  "text": "Hello, this is a test message",
  "is_final": false
}
```

#### Server Messages

**Ready (`ready`)**

```json theme={null}
{
  "type": "ready",
  "request_id": "req_1234567890",
  "voice_id": "vn-xiaoxiao",
  "language": "zh-CN",
  "format": "wav",
  "sample_rate": 16000
}
```

**Audio (`audio`)**

```json theme={null}
{
  "type": "audio",
  "data": "base64-encoded-audio-data",
  "is_final": false
}
```

**Error (`error`)**

```json theme={null}
{
  "type": "error",
  "error": "Invalid voice_id",
  "code": "VOICE_NOT_FOUND",
  "request_id": "req_1234567890"
}
```

### Complete Example

```javascript theme={null}
class TTSWebSocketClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.ws = null;
    this.ready = false;
  }
  
  connect() {
    return new Promise((resolve, reject) => {
      // Use query parameter for authentication (works with all clients)
      this.ws = new WebSocket(`wss://api.voxnexus.ai/v1/tts/realtime?token=${this.apiKey}`);
      
      this.ws.onopen = () => {
        console.log('Connected');
        resolve();
      };
      
      this.ws.onmessage = (event) => {
        this.handleMessage(JSON.parse(event.data));
      };
      
      this.ws.onerror = (error) => {
        console.error('WebSocket error:', error);
        reject(error);
      };
      
      this.ws.onclose = () => {
        console.log('Connection closed');
        this.ready = false;
      };
    });
  }
  
  initialize(config) {
    this.ws.send(JSON.stringify({
      type: 'init',
      ...config
    }));
  }
  
  synthesize(text, isFinal = false) {
    if (!this.ready) {
      throw new Error('Not ready. Wait for ready message.');
    }
    
    this.ws.send(JSON.stringify({
      type: 'text',
      text: text,
      is_final: isFinal
    }));
  }
  
  handleMessage(message) {
    switch (message.type) {
      case 'ready':
        this.ready = true;
        console.log('Ready:', message.request_id);
        break;
        
      case 'audio':
        this.onAudio(message.data, message.is_final);
        break;
        
      case 'error':
        console.error('Error:', message.error);
        this.onError(message);
        break;
    }
  }
  
  onAudio(base64Data, isFinal) {
    // Decode and handle audio
    const audioData = atob(base64Data);
    // Play audio or save to buffer
  }
  
  onError(error) {
    // Handle error
  }
  
  close() {
    if (this.ws) {
      this.ws.close();
    }
  }
}

// Usage
const client = new TTSWebSocketClient('YOUR_API_KEY');
await client.connect();
client.initialize({
  model_id: 'vn-tts-basic',
  voice_id: 'vn-xiaoxiao',
  format: 'wav',
  sample_rate: 16000
});

// Wait for ready, then synthesize
setTimeout(() => {
  client.synthesize('Hello, world!', true);
}, 1000);
```

## Speech-to-Text WebSocket

### Message Types

#### Client Messages

**Initialization (`init`)**

```json theme={null}
{
  "type": "init",
  "model_id": "vn-stt-basic",
  "language": "zh-CN",
  "format": "pcm",
  "sample_rate": 16000,
  "enable_timestamps": true,
  "enable_llm_transform": true,
  "llm_prompt": "Correct punctuation and remove filler words",
  "llm_mode": "per_segment"
}
```

**Audio (`audio`)**

```json theme={null}
{
  "type": "audio",
  "data": "base64-encoded-audio-chunk"
}
```

**Command (`command`)**

```json theme={null}
{
  "type": "command",
  "command": "flush"
}
```

<Note>
  The `flush` command tells the server that no more audio will be sent. The server will respond with a `flush_done` message after completing recognition. Subsequent audio will start a new recognition session.
</Note>

#### Server Messages

**Ready (`ready`)**

```json theme={null}
{
  "type": "ready",
  "request_id": "req_1234567890",
  "language": "zh-CN",
  "format": "pcm",
  "sample_rate": 16000
}
```

**Transcript (`transcript`)**

```json theme={null}
{
  "type": "transcript",
  "request_id": "req_1234567890",
  "segment_id": "550e8400-e29b-41d4-a716-446655440000",
  "text": "Hello, this is a test message.",
  "is_final": true,
  "language": "en",
  "confidence": 0.95,
  "offset": 0,
  "duration": 2500,
  "words": [
    {
      "word": "hello",
      "offset": 0,
      "duration": 500,
      "confidence": 0.98
    }
  ]
}
```

<Note>
  The `is_final` field distinguishes between partial results (`false`) and complete sentences (`true`). Confidence scores and word-level information are only valid when `is_final` is `true`. `segment_id` is only present when `is_final=true` and LLM transform is enabled.
</Note>

**LLM Transform (`llm`)**

```json theme={null}
{
  "type": "llm",
  "request_id": "req_1234567890",
  "segment_id": "550e8400-e29b-41d4-a716-446655440000",
  "delta": "Hello,",
  "is_final": false
}
```

<Note>
  Sent only when `enable_llm_transform=true`. Each final transcript segment triggers a sequence of `llm` messages streaming the LLM output incrementally. Concatenate all `delta` values until `is_final=true`. `segment_id` links the output to its corresponding `transcript` message (absent for a full-text post-flush pass).
</Note>

**Flush Done (`flush_done`)**

```json theme={null}
{
  "type": "flush_done",
  "request_id": "req_1234567890"
}
```

**Error (`error`)**

```json theme={null}
{
  "type": "error",
  "error": "Invalid audio format",
  "code": "UNSUPPORTED_FORMAT",
  "request_id": "req_1234567890"
}
```

### Complete Example

```javascript theme={null}
class STTWebSocketClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.ws = null;
    this.ready = false;
    this.audioContext = null;
    this.processor = null;
  }
  
  async connect() {
    return new Promise((resolve, reject) => {
      // Use query parameter for authentication (works with all clients)
      this.ws = new WebSocket(`wss://api.voxnexus.ai/v1/stt/realtime?token=${this.apiKey}`);
      
      this.ws.onopen = () => {
        console.log('Connected');
        resolve();
      };
      
      this.ws.onmessage = (event) => {
        this.handleMessage(JSON.parse(event.data));
      };
      
      this.ws.onerror = (error) => {
        console.error('WebSocket error:', error);
        reject(error);
      };
      
      this.ws.onclose = () => {
        console.log('Connection closed');
        this.ready = false;
      };
    });
  }
  
  initialize(config) {
    this.ws.send(JSON.stringify({
      type: 'init',
      ...config
    }));
  }
  
  async startRecording(sampleRate = 16000) {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    this.audioContext = new AudioContext({ sampleRate });
    const source = this.audioContext.createMediaStreamSource(stream);
    
    this.processor = this.audioContext.createScriptProcessor(4096, 1, 1);
    this.processor.onaudioprocess = (e) => {
      if (!this.ready) return;
      
      const audioData = e.inputBuffer.getChannelData(0);
      const pcm16 = new Int16Array(audioData.length);
      
      for (let i = 0; i < audioData.length; i++) {
        pcm16[i] = Math.max(-32768, Math.min(32767, audioData[i] * 32768));
      }
      
      const base64 = btoa(String.fromCharCode(...new Uint8Array(pcm16.buffer)));
      this.ws.send(JSON.stringify({
        type: 'audio',
        data: base64
      }));
    };
    
    source.connect(this.processor);
    this.processor.connect(this.audioContext.destination);
  }
  
  stopRecording() {
    if (this.processor) {
      this.processor.disconnect();
      this.processor = null;
    }
    if (this.audioContext) {
      this.audioContext.close();
      this.audioContext = null;
    }
  }
  
  flush() {
    if (!this.ready) {
      throw new Error('Not ready. Wait for ready message.');
    }
    
    this.ws.send(JSON.stringify({
      type: 'command',
      command: 'flush'
    }));
  }
  
  handleMessage(message) {
    switch (message.type) {
      case 'ready':
        this.ready = true;
        console.log('Ready:', message.request_id);
        break;
        
      case 'transcript':
        if (message.is_final) {
          this.onFinal(message);
        } else {
          this.onPartial(message);
        }
        break;
        
      case 'llm':
        this.onLLMDelta(message);
        break;
        
      case 'flush_done':
        console.log('Flush done:', message.request_id);
        break;
        
      case 'error':
        console.error('Error:', message.error);
        this.onError(message);
        break;
    }
  }
  
  onPartial(transcript) {
    console.log('Partial:', transcript.text);
    // Update UI with interim results
  }
  
  onFinal(transcript) {
    console.log('Final:', transcript.text);
    if (transcript.language) {
      console.log('Detected language:', transcript.language);
    }
    if (transcript.confidence) {
      console.log('Confidence:', transcript.confidence);
    }
    // Handle final transcription
  }
  
  onLLMDelta(message) {
    // Accumulate streaming LLM output
    const key = message.segment_id ?? '__post_flush__';
    if (!this._llmBuffers) this._llmBuffers = {};
    this._llmBuffers[key] = (this._llmBuffers[key] ?? '') + message.delta;
    if (message.is_final) {
      console.log('LLM output:', this._llmBuffers[key]);
      delete this._llmBuffers[key];
    }
  }
  
  onError(error) {
    // Handle error
  }
  
  close() {
    this.stopRecording();
    if (this.ws) {
      this.ws.close();
    }
  }
}

// Usage
const client = new STTWebSocketClient('YOUR_API_KEY');
await client.connect();
client.initialize({
  model_id: 'vn-stt-basic',
  format: 'pcm',
  sample_rate: 16000,
  enable_timestamps: true
});

// Wait for ready, then start recording
setTimeout(async () => {
  await client.startRecording(16000);
}, 1000);
```

## Best Practices

### Connection Management

**Reconnection Logic**

```javascript theme={null}
class ReconnectingWebSocket {
  constructor(url, options) {
    this.url = url;
    this.options = options;
    this.ws = null;
    this.reconnectAttempts = 0;
    this.maxReconnectAttempts = 5;
    this.reconnectDelay = 1000;
  }
  
  connect() {
    this.ws = new WebSocket(this.url, this.options);
    
    this.ws.onclose = () => {
      if (this.reconnectAttempts < this.maxReconnectAttempts) {
        setTimeout(() => {
          this.reconnectAttempts++;
          this.connect();
        }, this.reconnectDelay * this.reconnectAttempts);
      }
    };
    
    this.ws.onopen = () => {
      this.reconnectAttempts = 0;
    };
  }
}
```

**Heartbeat/Ping**

```javascript theme={null}
// Send periodic ping to keep connection alive
setInterval(() => {
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: 'ping' }));
  }
}, 30000); // Every 30 seconds
```

### Error Handling

```javascript theme={null}
ws.onerror = (error) => {
  console.error('WebSocket error:', error);
  // Implement retry logic or notify user
};

ws.onclose = (event) => {
  if (event.code !== 1000) { // Not a normal closure
    console.error('Unexpected closure:', event.code, event.reason);
    // Attempt reconnection
  }
};
```

### Audio Processing

**Chunk Size Optimization**

* Send audio chunks of 100-200ms for optimal latency
* Too small: Increased overhead
* Too large: Increased latency

**Buffer Management**

```javascript theme={null}
class AudioBuffer {
  constructor() {
    this.buffer = [];
    this.chunkSize = 1600; // 100ms at 16kHz
  }
  
  addAudio(audioData) {
    this.buffer.push(...audioData);
    
    while (this.buffer.length >= this.chunkSize) {
      const chunk = this.buffer.splice(0, this.chunkSize);
      this.sendChunk(chunk);
    }
  }
  
  sendChunk(chunk) {
    const base64 = btoa(String.fromCharCode(...chunk));
    ws.send(JSON.stringify({
      type: 'audio',
      data: base64
    }));
  }
  
  flush() {
    if (this.buffer.length > 0) {
      this.sendChunk(this.buffer);
      this.buffer = [];
    }
  }
}
```

### Performance Optimization

**Batch Text Messages**

```javascript theme={null}
// For TTS, batch multiple text segments
const texts = ['Hello', 'world', 'this', 'is', 'a', 'test'];
texts.forEach((text, index) => {
  setTimeout(() => {
    client.synthesize(text, index === texts.length - 1);
  }, index * 100);
});
```

**Throttle Audio Sending**

```javascript theme={null}
class ThrottledAudioSender {
  constructor(ws, interval = 100) {
    this.ws = ws;
    this.interval = interval;
    this.queue = [];
    this.timer = null;
  }
  
  send(audioData) {
    this.queue.push(audioData);
    
    if (!this.timer) {
      this.timer = setInterval(() => {
        if (this.queue.length > 0) {
          const data = this.queue.shift();
          this.ws.send(JSON.stringify({
            type: 'audio',
            data: data
          }));
        } else {
          clearInterval(this.timer);
          this.timer = null;
        }
      }, this.interval);
    }
  }
}
```

## Common Patterns

### Bidirectional Voice Conversation

```javascript theme={null}
// Combine TTS and STT for voice conversation
class VoiceConversation {
  constructor(apiKey) {
    this.ttsClient = new TTSWebSocketClient(apiKey);
    this.sttClient = new STTWebSocketClient(apiKey);
  }
  
  async start() {
    await Promise.all([
      this.ttsClient.connect(),
      this.sttClient.connect()
    ]);
    
    this.ttsClient.initialize({ model_id: 'vn-tts-basic', voice_id: 'vn-xiaoxiao' });
    this.sttClient.initialize({ model_id: 'vn-stt-basic', format: 'pcm', sample_rate: 16000, enable_timestamps: false});
    
    // Handle STT results and respond with TTS
    this.sttClient.onFinal = (result) => {
      const response = this.processUserInput(result.text);
      this.ttsClient.synthesize(response, true);
    };
    
    await this.sttClient.startRecording();
  }
  
  processUserInput(text) {
    // Process user input and generate response
    return `You said: ${text}`;
  }
}
```

### Real-time Captioning

```javascript theme={null}
class LiveCaptioning {
  constructor(apiKey, captionElement) {
    this.sttClient = new STTWebSocketClient(apiKey);
    this.captionElement = captionElement;
  }
  
  async start() {
    await this.sttClient.connect();
    this.sttClient.initialize({
      model_id: 'vn-stt-basic',
      format: 'pcm',
      sample_rate: 16000,
      enable_timestamps: true
    });
    
    this.sttClient.onPartial = (transcript) => {
      this.captionElement.textContent = transcript.text;
      this.captionElement.classList.add('interim');
    };
    
    this.sttClient.onFinal = (transcript) => {
      this.captionElement.textContent = transcript.text;
      this.captionElement.classList.remove('interim');
      // Save final caption with timestamp
      this.saveCaption(transcript);
    };
    
    await this.sttClient.startRecording();
  }
  
  saveCaption(transcript) {
    // Save caption with timing information
    const endTime = transcript.offset + transcript.duration;
    console.log(`[${transcript.offset}-${endTime}ms] ${transcript.text}`);
  }
}
```

## Troubleshooting

### Connection Issues

**Problem**: Connection fails immediately

* **Solution**: Verify API key is correct and has proper permissions
* **Solution**: Check network connectivity and firewall settings

**Problem**: Connection drops frequently

* **Solution**: Implement reconnection logic with exponential backoff
* **Solution**: Check for network instability or proxy issues

### Audio Issues

**Problem**: No audio received (TTS)

* **Solution**: Verify initialization message was sent and ready message received
* **Solution**: Check that text messages are being sent correctly

**Problem**: Recognition not working (STT)

* **Solution**: Verify audio format and sample rate match initialization
* **Solution**: Check that audio chunks are being sent continuously
* **Solution**: Ensure audio quality is sufficient (no excessive noise)

### Performance Issues

**Problem**: High latency

* **Solution**: Reduce audio chunk size for faster processing
* **Solution**: Use appropriate sample rates (16kHz is usually sufficient)
* **Solution**: Optimize network connection (use closer server if available)

**Problem**: High memory usage

* **Solution**: Process and discard audio chunks after sending
* **Solution**: Limit buffer sizes for audio data
* **Solution**: Close unused connections promptly