Promptfoo Security Integration

Promptfoo is an open-source tool for testing and evaluating LLM outputs. In addition to benchmarking, it provides red-teaming and security testing capabilities for identifying vulnerabilities in LLM applications.

Promptfoo's security testing features work with any OpenAI-compatible provider, including LLM API.

Prerequisites

An LLM API account with an API key
Promptfoo Security installed or accessible

Setup

Get Your LLM API Key

Log in to your LLM API dashboard
Click Create Key to Start
Copy your new API key immediately — it will only be shown once
Store the key securely (e.g., in a password manager or .env file)

LLM API is an OpenAI-compatible gateway that gives you access to dozens of AI models through a single API key and endpoint.

Use LLM API with Promptfoo Red-Teaming

Set environment variables:

export OPENAI_API_KEY="your-llm-api-key-here"
export OPENAI_BASE_URL="https://api.llmapi.ai/v1"

Run red-team evaluations:

promptfoo redteam run

Promptfoo will test your LLM API application for vulnerabilities.

Test the Integration

Verify that Promptfoo Security can successfully communicate with LLM API by sending a test request. All requests will now be routed through LLM API.

Promptfoo's red-teaming helps ensure your LLM API application is secure before deployment.

Benefits of Using LLM API with Promptfoo Security

Multi-Provider Access: Use models from OpenAI, Anthropic, Google, and more through a single API
Cost Control: Track and limit your AI spending with detailed usage analytics
Unified Billing: One account for all providers instead of managing multiple API keys
Caching: Reduce costs with response caching for repeated requests

View all available models on the models page.