Audio Streaming

WebSocket endpoints enable real-time audio streaming — send audio and receive transcripts as they're spoken, or send text and receive audio as it's generated.

For a full-duplex speech-to-speech session — the model listens and replies on one socket, with turn taking and interruption handled for you — see Live voice sessions.

Streaming Speech-to-Text

`GET /v1/audio/transcriptions/ws`

Connect via WebSocket for real-time transcription. Ideal for live microphone input, voice assistants, and call centers.

Protocol

1. Client opens WebSocket with Authorization header
2. Client sends: {"type":"configure","model":"u3-rt-pro","encoding":"pcm_s16le","sample_rate":16000}
3. Server sends: {"type":"session_started","session_id":"..."}
4. Client sends: [binary PCM audio frames]
5. Server sends: {"type":"transcript","text":"hello","is_final":false}     ← interim
6. Server sends: {"type":"transcript","text":"Hello world.","is_final":true}  ← final
7. Client sends: {"type":"stop"}
8. Server sends: {"type":"done","audio_duration_seconds":30,"session_duration_seconds":35}

Configure Message

Field	Type	Required	Default	Description
`type`	string	Yes		Must be `"configure"`
`model`	string	Yes		Streaming STT model ID
`encoding`	string	No	`pcm_s16le`	Audio encoding
`sample_rate`	int	No	`16000`	Sample rate in Hz
`language`	string	No	auto-detect	ISO-639-1 code

Audio Requirements

Format: Raw PCM only (no MP3, WAV, or other containers)
Channels: Mono (single channel)
Chunk size: 20-100ms per frame (e.g., 640 bytes at 16kHz s16le for 20ms)

Transcript Messages

{
  "type": "transcript",
  "text": "Hello, how are you?",
  "is_final": true,
  "words": [
    { "word": "Hello,", "start": 0.0, "end": 0.4 },
    { "word": "how", "start": 0.5, "end": 0.7 },
    { "word": "are", "start": 0.8, "end": 0.9 },
    { "word": "you?", "start": 1.0, "end": 1.3 }
  ]
}

is_final: false — interim result, may change as more audio arrives
is_final: true — final result for this utterance, will not change

JavaScript Example (Browser Microphone)

The browser WebSocket API does not support custom headers. Pass the API key as a query parameter instead. The server accepts token as an alternative to the Authorization header.

// Browser WebSocket — auth via query parameter (headers not supported in browser WS API)
const ws = new WebSocket(
  `wss://api.llmapi.ai/v1/audio/transcriptions/ws?token=${encodeURIComponent(API_KEY)}`
);

ws.onopen = () => {
  ws.send(
    JSON.stringify({
      type: "configure",
      model: "u3-rt-pro",
      encoding: "pcm_s16le",
      sample_rate: 16000,
    })
  );
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === "session_started") {
    startMicrophone(ws);
  } else if (msg.type === "transcript") {
    console.log(msg.is_final ? "[FINAL]" : "[interim]", msg.text);
  } else if (msg.type === "done") {
    console.log("Session complete:", msg.audio_duration_seconds, "seconds");
  }
};

// Uses AudioWorklet for modern, non-blocking audio processing.
// Falls back to ScriptProcessor for older browsers.
async function startMicrophone(ws) {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: { sampleRate: 16000, channelCount: 1 },
  });
  const ctx = new AudioContext({ sampleRate: 16000 });
  const source = ctx.createMediaStreamSource(stream);

  // Register the AudioWorklet processor
  await ctx.audioWorklet.addModule(
    URL.createObjectURL(
      new Blob(
        [
          `
    class PCMProcessor extends AudioWorkletProcessor {
      process(inputs) {
        const input = inputs[0][0]; // mono channel
        if (input) {
          const int16 = new Int16Array(input.length);
          for (let i = 0; i < input.length; i++) {
            int16[i] = Math.max(-32768, Math.min(32767, input[i] * 32768));
          }
          this.port.postMessage(int16.buffer, [int16.buffer]);
        }
        return true;
      }
    }
    registerProcessor('pcm-processor', PCMProcessor);
  `,
        ],
        { type: "application/javascript" }
      )
    )
  );

  const workletNode = new AudioWorkletNode(ctx, "pcm-processor");
  workletNode.port.onmessage = (e) => {
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(e.data);
    }
  };
  source.connect(workletNode);
  workletNode.connect(ctx.destination);
}

Python Example

import asyncio, json, websockets

async def transcribe(audio_path):
    uri = "wss://api.llmapi.ai/v1/audio/transcriptions/ws"
    headers = {"Authorization": f"Bearer {API_KEY}"}

    async with websockets.connect(uri, extra_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "configure",
            "model": "u3-rt-pro",
            "encoding": "pcm_s16le",
            "sample_rate": 16000,
        }))

        msg = json.loads(await ws.recv())
        assert msg["type"] == "session_started"

        # Send audio in 100ms chunks
        with open(audio_path, "rb") as f:
            while chunk := f.read(3200):
                await ws.send(chunk)

        await ws.send(json.dumps({"type": "stop"}))

        while True:
            msg = json.loads(await ws.recv())
            if msg["type"] == "transcript":
                prefix = "[FINAL]" if msg["is_final"] else "[interim]"
                print(f"{prefix} {msg['text']}")
            elif msg["type"] == "done":
                break

asyncio.run(transcribe("recording.pcm"))

Budget Protection

The server monitors your credit balance during streaming sessions. If estimated cost reaches 90% of your remaining credits, the session terminates with:

{ "type": "error", "message": "session terminated: credit budget exhausted", "code": 402 }

This prevents a long-running session from overdrawing your balance.

Streaming Text-to-Speech

`GET /v1/audio/speech/ws`

WebSocket endpoint for real-time TTS. Currently supports Cartesia models only.

Protocol

1. Client opens WebSocket with Authorization header
2. Client sends: {"type":"tts","model":"sonic-3","input":"Hello","voice":"Sarah"}
3. Server sends: {"type":"audio","data":"<base64>","context_id":"ctx_..."}  ← chunks
4. Server sends: {"type":"done","context_id":"ctx_..."}

Request Message

Field	Type	Required	Default	Description
`type`	string	Yes		Must be `"tts"`
`model`	string	Yes		Cartesia model ID (e.g., `sonic-3`)
`input`	string	Yes		Text to synthesize
`voice`	string	Yes		Voice name or ID
`output_format`	string	No	raw PCM 24kHz	Audio format
`speed`	number	No	`1.0`	Speed multiplier
`language`	string	No		ISO-639-1 code

Response Messages

Audio chunk:

{ "type": "audio", "data": "<base64-encoded audio>", "context_id": "ctx_1234" }

Done:

{ "type": "done", "context_id": "ctx_1234" }

Audio data is base64-encoded raw PCM. Decode and play directly with your audio output device.

Features

Cancel: Send {"type":"cancel"} to stop an in-progress generation
Multiple requests: You can send a new TTS request while one is in progress — the previous generation is automatically cancelled
Connection keepalive: The server sends ping frames every 30 seconds to prevent proxy timeouts

Error Handling

WebSocket errors are sent as JSON text messages:

{ "type": "error", "message": "unsupported encoding \"opus\" for model u3-rt-pro", "code": 400 }

Code	Meaning
`400`	Invalid configure message, unsupported model/encoding/sample_rate/language
`401`	Authentication failed
`402`	Credits exhausted (budget protection)
`403`	Access denied (suspended org, IAM restriction)
`502`	Provider connection error

Connection Limits

Idle timeout: Connections close after 60 seconds of no messages
Max session: Depends on model (1 hour for Cartesia, 3 hours for AssemblyAI)
Keepalive: Server sends pings every 30 seconds — ensure your client handles pong responses