Batch Processing

The Batch API allows you to submit large numbers of requests asynchronously and receive results when they are complete. Batches are processed with up to a 24-hour turnaround and come with significant cost savings compared to synchronous requests.

INFO

The Batch API is fully compatible with the OpenAI Batch API. Any existing code that uses the OpenAI Batch API will work with LLM API without modification.

Supported Providers

The Batch API supports OpenAI and Google Vertex AI models. For the full list of models that support batch, see the Models page in your dashboard — look for the Batch capability indicator.

INFO

ElevenLabs prerecorded speech-to-text uses the dedicated /v1/audio/transcriptions workflow rather than the JSONL Batch API.

Supported Endpoints

Each batch targets one of the following endpoints:

Endpoint	Description
`/v1/chat/completions`	Standard chat completions — supported by all batch-capable models
`/v1/responses`	OpenAI Responses API — supported by OpenAI models with the Responses API capability

All lines in a batch file must use the same endpoint. The url field in each JSONL line must match the endpoint you specify when creating the batch.

Routing to a Specific Provider

By default, the gateway automatically routes to the best available provider for the model. You can explicitly target a provider using the provider/model format in the model field:

{
  "custom_id": "req-1",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "google-vertex/gemini-2.5-pro",
    "messages": [{ "role": "user", "content": "Hello" }],
    "max_tokens": 100
  }
}

Format	Example	Behaviour
Bare model	`gemini-2.5-pro`	Auto-routed to the first configured provider that supports batch
Explicit provider	`google-vertex/gemini-2.5-pro`	Forced to that provider; returns an error if unavailable
Explicit provider	`openai/gpt-4o`	Forced to OpenAI

INFO

All lines in a single batch file must use the same model. Mixing models across lines is not supported.

Minimum Balance Requirement

A minimum credit balance of $50 is required to submit a batch. Batch jobs can generate large volumes of tokens before the final cost is known, so this threshold ensures the job can complete without interruption.

WARNING

If your balance is below $50 at the time of submission, batch creation will be rejected with an insufficient_balance error. Top up your credits before submitting.

How It Works

Upload a JSONL file

Prepare a .jsonl file where each line is a JSON object containing a custom_id, method, url, and body.

Create a batch

Submit the uploaded file ID and target endpoint to create a batch job.

Poll for status

Check the batch status periodically until it reaches completed or failed.

Download results

Fetch the output file and error file to retrieve per-request results.

Step 1 — Prepare Your Input File

Using `/v1/chat/completions`

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "What is 2+2?"}], "max_tokens": 100}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 50}}

Using `/v1/responses`

For OpenAI models that support the Responses API, use input instead of messages and max_output_tokens instead of max_tokens:

{"custom_id": "req-1", "method": "POST", "url": "/v1/responses", "body": {"model": "gpt-4o-mini", "input": "What is 2+2?", "max_output_tokens": 100}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/responses", "body": {"model": "gpt-4o-mini", "input": "Say hello", "max_output_tokens": 50}}

Field	Description
`custom_id`	Your identifier for this request — echoed back in the output
`method`	Always `"POST"`
`url`	`/v1/chat/completions` or `/v1/responses`
`body`	A standard request body matching the target endpoint

Step 2 — Upload the File

curl -X POST "https://api.llmapi.ai/v1/files" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F "purpose=batch" \
  -F "file=@requests.jsonl"

{
  "id": "file-abc123",
  "object": "file",
  "purpose": "batch",
  "filename": "requests.jsonl",
  "bytes": 512,
  "created_at": 1715000000
}

Step 3 — Create the Batch

curl -X POST "https://api.llmapi.ai/v1/batches" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

For a /v1/responses batch, set "endpoint": "/v1/responses".

{
  "id": "batch_xyz789",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-abc123",
  "status": "validating",
  "created_at": 1715000000,
  "expires_at": 1715086400
}

Batch Statuses

Status	Description
`validating`	Input file is being validated
`in_progress`	Batch is being processed by the provider
`finalizing`	Results are being written to output files
`completed`	All requests are done — output file is ready
`failed`	Batch failed before any requests were processed
`expired`	Batch was not completed within 24 hours
`cancelling`	Cancellation in progress
`cancelled`	Batch was cancelled

Step 4 — Poll for Status

curl "https://api.llmapi.ai/v1/batches/batch_xyz789" \
  -H "Authorization: Bearer $LLM_API_KEY"

{
  "id": "batch_xyz789",
  "status": "completed",
  "output_file_id": "file-output123",
  "error_file_id": "file-errors456",
  "request_counts": {
    "total": 2,
    "completed": 2,
    "failed": 0
  }
}

INFO

We recommend polling every 30–60 seconds. Most batches complete well within the 24-hour window.

Step 5 — Download Results

Once status is completed, download the output file:

curl "https://api.llmapi.ai/v1/files/file-output123/content" \
  -H "Authorization: Bearer $LLM_API_KEY"

Each line in the output is a JSON object:

{"id": "batch_req_1", "custom_id": "req-1", "response": {"status_code": 200, "body": {"id": "chatcmpl-abc", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "2 + 2 equals 4."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 14, "completion_tokens": 8, "total_tokens": 22}}}, "error": null}
{"id": "batch_req_2", "custom_id": "req-2", "response": {"status_code": 200, "body": {"id": "chatcmpl-def", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello!"}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 8, "completion_tokens": 2, "total_tokens": 10}}}, "error": null}

Error File

If some requests failed, they appear in the error file (error_file_id). The format is identical to the output file — each line has the original custom_id and an error field with details:

{
  "id": "batch_req_3",
  "custom_id": "req-3",
  "response": {
    "status_code": 400,
    "body": {
      "error": {
        "message": "max_tokens is too large",
        "type": "invalid_request_error",
        "code": "invalid_value"
      }
    }
  },
  "error": null
}

Gemini Audio Input Batches

Gemini models on Google Vertex AI support audio input inside batch requests. This lets you transcribe or analyse large volumes of audio files asynchronously at the batch discount rate.

Step 1.1 — Upload your audio files

Upload each audio file with purpose=audio_batch. The file is stored and you receive a file_id to reference inside the JSONL.

curl -X POST "https://api.llmapi.ai/v1/files" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F "purpose=audio_batch" \
  -F "file=@recording.ogg"

{
  "id": "file-abc123",
  "object": "file",
  "purpose": "audio_batch",
  "filename": "recording.ogg",
  "bytes": 92415,
  "created_at": 1715000000
}

Supported audio formats: mp3, wav, ogg, flac, m4a, webm, mp4.

Step 1.2 — Prepare your JSONL file

Use the input_audio content block inside a messages array, referencing a previously uploaded file by file_id:

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-2.5-flash", "messages": [{"role": "user", "content": [{"type": "input_audio", "input_audio": {"file_id": "file-abc123"}}]}], "max_tokens": 200}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-2.5-flash", "messages": [{"role": "user", "content": [{"type": "input_audio", "input_audio": {"file_id": "file-def456"}}]}], "max_tokens": 200}}

You can also combine input_audio with a text prompt in the same message:

{
  "custom_id": "req-1",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "gemini-2.5-flash",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "input_audio", "input_audio": { "file_id": "file-abc123" } },
          { "type": "text", "text": "Transcribe this recording and summarise the key points." }
        ]
      }
    ],
    "max_tokens": 500
  }
}

Then continue with Steps 2–5 of the standard batch flow above.

ElevenLabs Batch Speech-to-Text

ElevenLabs Scribe v2 can transcribe prerecorded audio through POST /v1/audio/transcriptions. It supports multichannel recordings where each channel contains a distinct speaker, which is useful for interviews, podcasts, and call-center recordings with isolated audio tracks.

INFO

This is a synchronous file-upload workflow, not a /v1/batches JSONL job. The request stays open until the transcription is ready, so there is no batch ID to poll or output file to download.

Transcribe each channel separately

Set use_multi_channel=true to process every channel independently. The default multichannel_output_style is separate:

curl -X POST "https://api.llmapi.ai/v1/audio/transcriptions" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F "model=scribe_v2" \
  -F "file=@stereo-interview.wav" \
  -F "use_multi_channel=true" \
  -F "multichannel_output_style=separate"

The response contains one transcript for each channel:

{
  "transcripts": [
    {
      "text": "Welcome to the interview.",
      "channel_index": 0
    },
    {
      "text": "Thanks for having me.",
      "channel_index": 1
    }
  ]
}

Use response_format=verbose_json to include the detected language, duration, and word-level timestamps for every channel:

curl -X POST "https://api.llmapi.ai/v1/audio/transcriptions" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F "model=scribe_v2" \
  -F "file=@stereo-interview.wav" \
  -F "use_multi_channel=true" \
  -F "response_format=verbose_json"

Combine channels by timestamp

Set multichannel_output_style=combined to receive one conversation ordered by word start time. Combined output requires response_format=verbose_json:

curl -X POST "https://api.llmapi.ai/v1/audio/transcriptions" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F "model=scribe_v2" \
  -F "file=@stereo-interview.wav" \
  -F "use_multi_channel=true" \
  -F "multichannel_output_style=combined" \
  -F "response_format=verbose_json"

{
  "text": "Welcome. Thanks.",
  "language": "en",
  "words": [
    {
      "channel_index": 0,
      "word": "Welcome.",
      "speaker_id": "speaker_0",
      "start": 0.1,
      "end": 0.7
    },
    {
      "channel_index": 1,
      "word": "Thanks.",
      "speaker_id": "speaker_1",
      "start": 0.8,
      "end": 1.3
    }
  ],
  "duration": 1.3
}

Requirements and billing

A recording can contain up to 5 channels.
Keep one speaker isolated on each channel. Speaker IDs are assigned from the channel number: channel 0 is speaker_0, channel 1 is speaker_1, and so on.
Multichannel transcription handles speaker separation, so you do not need diarization or a speaker count.
Usage is billed by audio duration multiplied by the number of channels. For example, a 10-minute stereo recording is billed as 20 transcription minutes.

See the ElevenLabs multichannel transcription guide for advice on preparing multichannel audio.

Function Calling (Tools)

Both OpenAI and Vertex AI batch support function calling. Include tools and tool_choice in the request body:

{
  "custom_id": "req-tools-1",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": { "city": { "type": "string" } },
            "required": ["city"]
          }
        }
      }
    ],
    "tool_choice": "auto",
    "max_tokens": 200
  }
}

When the model invokes a tool, the output will contain tool_calls instead of text content:

{
  "custom_id": "req-tools-1",
  "response": {
    "status_code": 200,
    "body": {
      "choices": [
        {
          "message": {
            "role": "assistant",
            "tool_calls": [
              {
                "id": "call_abc123",
                "type": "function",
                "function": {
                  "name": "get_weather",
                  "arguments": "{\"city\": \"Paris\"}"
                }
              }
            ]
          },
          "finish_reason": "tool_calls"
        }
      ]
    }
  }
}

List and Manage Batches

List all batches

curl "https://api.llmapi.ai/v1/batches" \
  -H "Authorization: Bearer $LLM_API_KEY"

Supports pagination via after (cursor) and limit query parameters.

Cancel a batch

curl -X POST "https://api.llmapi.ai/v1/batches/batch_xyz789/cancel" \
  -H "Authorization: Bearer $LLM_API_KEY"

WARNING

Cancellation is best-effort. If the batch has already been submitted to the provider, it may continue processing until the provider-side job is stopped.

List uploaded files

curl "https://api.llmapi.ai/v1/files?purpose=batch" \
  -H "Authorization: Bearer $LLM_API_KEY"

Supports purpose, after, and limit query parameters.

Webhooks

You can receive a webhook notification when a batch reaches a terminal state (completed, failed, expired, cancelled). Pass webhook_url and optionally webhook_secret when creating the batch:

curl -X POST "https://api.llmapi.ai/v1/batches" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "webhook_url": "https://your-server.example.com/webhook",
    "webhook_secret": "your-secret"
  }'

The webhook payload is a JSON object:

{
  "id": "evt_abc123",
  "type": "batch.completed",
  "created_at": 1715000000,
  "data": {
    "id": "batch_xyz789",
    "object": "batch",
    "status": "completed",
    "request_counts": { "total": 2, "completed": 2, "failed": 0 },
    "output_file_id": "file-output123",
    "completed_at": 1715003600,
    "failed_at": null
  }
}

Signature verification

We follow the Standard Webhooks specification.

Using the Standard Webhooks library (recommended):

npm install standardwebhooks

import { Webhook } from "standardwebhooks";

const wh = new Webhook(webhookSecret);
wh.verify(rawBody, headers); // throws on invalid signature

from standardwebhooks import Webhook

wh = Webhook(webhook_secret)
wh.verify(raw_body, headers)  # raises on invalid signature

Without a library:

When webhook_secret is set, each request includes three headers:

Header	Description
`webhook-id`	Unique event ID
`webhook-timestamp`	Unix timestamp of the delivery attempt
`webhook-signature`	HMAC-SHA256 signature prefixed with `v1,`

The signed string is {webhook-id}.{webhook-timestamp}.{body}:

import hmac, hashlib, base64

def verify(secret, headers, body):
    msg = f"{headers['webhook-id']}.{headers['webhook-timestamp']}.".encode() + body
    expected = "v1," + base64.b64encode(
        hmac.new(secret.encode(), msg, hashlib.sha256).digest()
    ).decode()
    return headers["webhook-signature"] == expected

import crypto from "crypto";

function verify(secret: string, headers: Record<string, string>, body: string): boolean {
  const msg = `${headers["webhook-id"]}.${headers["webhook-timestamp"]}.${body}`;
  const expected = "v1," + crypto.createHmac("sha256", secret).update(msg).digest("base64");
  return headers["webhook-signature"] === expected;
}

INFO

Webhooks are retried with exponential backoff (up to 5 attempts over ~2 minutes) if your endpoint returns a non-2xx status code.

Python Example (OpenAI SDK)

Since the Batch API is OpenAI-compatible, you can use the official openai Python SDK:

import time
from openai import OpenAI

client = OpenAI(
    api_key="your-llmapi-key",
    base_url="https://api.llmapi.ai/v1",
)

# 1. Upload input file
with open("requests.jsonl", "rb") as f:
    file = client.files.create(file=f, purpose="batch")

# 2. Create batch
batch = client.batches.create(
    input_file_id=file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
print(f"Batch created: {batch.id}")

# 3. Poll until complete
while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)
    print(f"Status: {batch.status}")

# 4. Download results
if batch.status == "completed" and batch.output_file_id:
    content = client.files.content(batch.output_file_id)
    for line in content.text.splitlines():
        import json
        result = json.loads(line)
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])

TypeScript Example (OpenAI SDK)

import OpenAI from "openai";
import fs from "fs";

const client = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: "https://api.llmapi.ai/v1",
});

async function runBatch() {
  // 1. Upload input file
  const file = await client.files.create({
    file: fs.createReadStream("requests.jsonl"),
    purpose: "batch",
  });

  // 2. Create batch
  let batch = await client.batches.create({
    input_file_id: file.id,
    endpoint: "/v1/chat/completions",
    completion_window: "24h",
  });
  console.log("Batch created:", batch.id);

  // 3. Poll until complete
  while (!["completed", "failed", "expired", "cancelled"].includes(batch.status)) {
    await new Promise((r) => setTimeout(r, 30_000));
    batch = await client.batches.retrieve(batch.id);
    console.log("Status:", batch.status);
  }

  // 4. Download results
  if (batch.status === "completed" && batch.output_file_id) {
    const content = await client.files.content(batch.output_file_id);
    const text = await content.text();
    for (const line of text.split("\n").filter(Boolean)) {
      const result = JSON.parse(line);
      console.log(result.custom_id, result.response.body.choices[0].message.content);
    }
  }
}

runBatch();

Quota Limits

Limit	Default
Max active batches per organization	1
Max lines per batch file	1,000
Max in-flight lines across all active batches	1,000

INFO

Need higher limits? Contact support to increase quotas for your organization.