Observability

LLM API can send a trace of every request your organization makes to the observability platform you already use. Connect Langfuse, LangSmith, any OpenTelemetry (OTLP/HTTP) endpoint — Datadog, Grafana, Arize Phoenix, Braintrust and others — or a signed webhook once in the dashboard, and every request made with your API keys shows up there. You don't need an SDK, a wrapper or code changes.

Broadcast is free. Traces are exported in the background after the response is sent, so they add no latency to your requests.

Setting it up

Owners and admins manage observability under Settings → Observability in the dashboard.

Click Add destination and pick the platform.
Enter its connection details (see below) and click Test connection. We send a test trace; nothing is saved unless your platform accepts it.
Click Connect, then turn on Broadcast requests.

An organization can connect up to 5 destinations. Each one can be turned off on its own, and the Broadcast requests switch turns all of them off at once.

Credentials are write-only. When you edit a destination, its key fields are empty: leave them blank to keep the stored credentials, or enter new ones to replace them. Changing the URL, region or credentials runs the connection test again before anything is saved; renaming a destination or changing its filters does not.

Platform	You provide
Langfuse	Region (EU, US, Japan, HIPAA) or your self-hosted URL; the project's public key and secret key
LangSmith	Region (US, EU, APAC) or your self-hosted URL; an API key; optionally the project name and workspace ID
OpenTelemetry	The full OTLP/HTTP traces URL (for example `https://otel.example.com/v1/traces`); up to 10 headers, such as an API key
Webhook	Any HTTPS URL and a signing secret (generate one in the dialog); optionally up to 10 headers

Datadog, Grafana Cloud, Honeycomb, Arize Phoenix, Braintrust, PostHog, Comet Opik, W&B Weave, New Relic and Sentry are listed by name. Pick one and enter that vendor's own credentials, such as the Datadog API key or the Honeycomb ingest key; we fill in the OpenTelemetry endpoint and headers for you.

For a self-hosted server, enter its base URL and we add the traces path. A URL that already ends in /otel/v1/traces is used as given, which also works for a vendor region missing from the list.

INFO

URLs must use HTTPS and resolve to a public address. Credentials are stored encrypted and are never shown again — only their last characters.

Per-destination options

Privacy mode — send usage, cost and timing only, without prompts or completions. Off by default.
Sampling — send a percentage of requests. Requests that share a session or trace, including a W3C traceparent, are kept or dropped together, so a sampled conversation stays complete.
API keys — send requests from all keys, only from the keys you pick, or from every key except the ones you pick.
Include tool definitions — also send the tools each request declares. Off by default: agent requests can carry tens of thousands of tokens of tool schemas. Never sent in Privacy mode.

Under Advanced:

Always send errors — failed requests are sent even when sampling would skip them.
Detailed timeline — send provider selection, fallback planning and response streaming as separate spans (see below).
Projects and Environments — send only requests from the projects you pick, or tagged with the environments you list (the X-LLMAPI-Environment header or trace.environment).
Resource attributes — for OpenTelemetry and webhook destinations, labels added to every trace, such as service.name.

The destinations table shows when each destination last received a trace and how many requests were delivered or dropped in the last 24 hours.

What a trace contains

Each request becomes a generation span named after the operation and model (for example chat gpt-4o), using the OpenTelemetry GenAI conventions (gen_ai.*). Langfuse and LangSmith also get their own attributes, so model, usage and cost land in their native fields.

When a request needed more than one provider attempt — a retry or a fallback — each attempt is a child span (Provider attempt 1, 2, …), failed ones marked as errors. Otherwise the generation is the only span: most platforms, Langfuse included, bill per span, so routing time and the time until the provider call started are attributes on the generation instead. Turn on Detailed timeline to get provider selection, fallback planning and response streaming as their own spans too.

Always sent:

request id, and the ids and names of the project and API key
endpoint, the model you requested, provider, and request parameters such as temperature and max tokens
start, end, duration and time to first token; for streamed responses, also seconds per output token and output tokens per second after the first token
input, output, cached and reasoning tokens
finish reason; whether the request was streamed, cached, canceled or used your own provider key (BYOK); whether a provider fallback happened and how many attempts it took
the cost we billed (not sent for BYOK requests)
the error code and message your client received
your tags (see Tagging traces)

Token counts match the usage in our API responses: output tokens count the visible output, and reasoning tokens are reported separately. In OpenTelemetry attributes that is gen_ai.usage.output_tokens plus gen_ai.usage.reasoning.output_tokens; Langfuse and LangSmith receive them in their own usage fields, so their totals match the tokens you were billed for.

Sent unless Privacy mode is on — for chat completions, /v1/messages and /v1/responses:

the input messages, including the system prompt or instructions
the output text, tool calls and readable reasoning
the request's tool definitions, if Include tool definitions is on for the destination

Images, audio, files and documents are replaced by a {"type": …, "stripped": true} placeholder, and text over 1 MiB is truncated.

WARNING

Prompts and outputs are sent whatever your data retention setting is: they go to your own platform and are not stored by LLM API. Turn on Privacy mode for a destination that must not receive them.

Tagging traces

Group requests into sessions and traces, and label them, with headers or body fields. Headers take precedence over body fields.

Purpose	Header	Body field
End user	`X-LLMAPI-User-Id`	`user` (OpenAI), `metadata.user_id` (Anthropic)
Session / conversation	`X-LLMAPI-Session-Id`, `x-session-id`	`session_id`
Group calls into one trace	`X-LLMAPI-Trace-Id`, W3C `traceparent`	`trace.trace_id`
Trace, span and generation names	—	`trace.trace_name`, `trace.span_name`, `trace.generation_name`
Environment and release	`X-LLMAPI-Environment`	`trace.environment`, `trace.release`
Tags and metadata	`X-LLMAPI-Team`, `X-LLMAPI-Feature`, `X-LLMAPI-Label`, `X-LLMAPI-Type`, `X-LLMAPI-App-Id`	any other key inside `trace`

curl https://api.llmapi.ai/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{ "role": "user", "content": "Summarize this ticket" }],
    "user": "user-42",
    "session_id": "conv-8d1f",
    "trace": {
      "trace_name": "support-summary",
      "environment": "production",
      "ticket_tier": "enterprise"
    }
  }'

session_id and trace are used only for observability: they are never forwarded to the model provider and don't affect routing.
A trace object larger than 16 KiB is ignored. At most 32 extra keys are kept, and values are cut to 256 bytes.
With a valid traceparent header, the LLM API span nests under your application's span in Langfuse and OpenTelemetry backends. LangSmith drops spans whose parent it never received, so there the span stays a root.

Where tags appear

Tag	Langfuse	LangSmith	OpenTelemetry and webhook
End user	trace user	`user_id` metadata	`user.id`
Session	trace session	`thread_id` metadata (thread)	`session.id`
Trace name	trace name	trace name	`llmapi.trace.name`
Environment	environment	`environment` metadata	`deployment.environment.name`
Release	release	—	`llmapi.release`
Team, feature, `trace.*`…	trace tags (`key:value`) and metadata	tags (`key:value`) and metadata	`llmapi.metadata.<key>`
Generation name	observation name	run name	span name

Every destination also gets the request id, project and API key as llmapi.request_id, llmapi.project.name and llmapi.api_key.name (and their ids), so you can filter by key or project in any platform.

Keeping one request private

Send X-LLMAPI-Broadcast-Privacy: true to keep a single request's prompts, outputs and tool definitions out of every destination, as if Privacy mode were on for that request. Usage, cost and timing are still sent. Only a false boolean (false or 0) leaves content on, so a typo such as yes or on keeps it private; an empty header is ignored.

Delivery and health

Delivery is best-effort. When your platform is slow or unavailable, we retry with backoff for up to 15 minutes before dropping that batch.

Payload limits: when a platform answers HTTP 413, we split the batch and resend the parts, keeping each request's spans together. Only a single request that is still too large is dropped.
Duplicates: span ids are derived from the request id, so a resent trace updates the existing one instead of duplicating it.
Rejected credentials (HTTP 401 or 403) pause the destination right away.
Failing for an hour: owners and admins get one in-app notification and one email.
Failing for 24 hours: the destination is paused.

A paused destination shows its reason in the dashboard. Fix the credentials or the endpoint, then click Resume — resuming runs the connection test again.

Verifying webhook deliveries

A webhook receives the same traces as an OpenTelemetry destination, as OTLP/JSON in a POST with Content-Type: application/json. Each delivery is signed with Standard Webhooks: verify the webhook-signature header with your signing secret using any Standard Webhooks library, and reject old webhook-timestamp values. A retried delivery keeps its webhook-id, so you can drop duplicates. Answer with any 2xx status.

from standardwebhooks import Webhook

payload = Webhook(signing_secret).verify(request_body, request_headers)

Any custom headers you add to the webhook, such as a receiver token, are sent with every delivery.

Troubleshooting

Connection test failed

The dashboard says	What happened	What to do
…rejected these credentials	The platform answered HTTP 401 or 403.	Check the keys. For Langfuse, the public and secret key must come from the same project and region.
…the endpoint was not found	The platform answered HTTP 404.	Pick the region your account lives in, or for OpenTelemetry enter the full traces URL, usually ending in `/v1/traces`.
…rejected the test trace	Another 4xx answer, a redirect, or the platform refused the span.	Use the exact traces URL — redirects are not followed — and check that the endpoint accepts OTLP/HTTP.
…did not answer in time	No answer within 10 seconds, or HTTP 408 or 504.	Check that the endpoint is up, then test again.
…answered with HTTP 429 (or 5xx)	The platform is rate-limiting or failing.	Try again later.
…could not be reached	DNS, TLS or the connection failed.	The URL must use HTTPS with a publicly trusted certificate and resolve to a public address; private and internal addresses are blocked.

Traces don't appear

Check that Broadcast requests is on and the destination is enabled and not paused.
Check the destination's filters: sampling, API keys, projects and environments all have to match the request.
Settings changes take effect within about 30 seconds; traces usually arrive a few seconds after the response.
Requests rejected before they reach a model, such as those with an invalid API key or an unknown model name, don't produce a trace.
Privacy mode never hides a trace — only its prompts and outputs.

INFO

Organizations managed by a partner can't configure observability; their partner owns it.