promptfoo/promptfoo

Overview

promptfoo is a command-line tool and Node library for testing LLM prompts, providers and agents. You write a promptfooconfig.yaml with prompts, providers and tests. Each test has variables and a list of assertions. promptfoo eval expands this into a matrix (every prompt × every provider × every test, optionally repeated), calls each provider, grades every output, and stores the results in a local SQLite database. promptfoo view serves a React UI over that database.

The same engine runs red-team scans. promptfoo redteam run first generates adversarial test cases with plugins (what to attack) and strategies (how to wrap the attack), writes them to redteam.yaml, and then runs an ordinary eval over that file. Red-teaming is a test generator in front of the eval engine, not a separate system.

The whole repository is MIT-licensed. There is no ee/ folder or enterprise-licensed code. The commercial boundary is a hosted service instead. Many red-team plugins, some grading and part of attack generation call promptfoo’s API, and a free monthly probe limit applies when you are not logged in to promptfoo Cloud (details below).

Architecture

flowchart LR
  CFG["promptfooconfig.yaml"] --> CLI["CLI: doEval"]
  RT["redteam generate"] --> YAML["redteam.yaml"]
  YAML --> CLI
  CLI --> EV["Evaluator (evaluator.ts)"]
  EV --> RUN["runEval: render + call"]
  RUN --> PROV["Providers (src/providers)"]
  PROV --> CACHE["Cache (memory + disk)"]
  RUN --> ASSERT["runAssertions"]
  ASSERT --> GRADE["llmGrading matchers"]
  ASSERT --> TRACE["Trace store"]
  OTLP["OTLP receiver /v1/traces"] --> TRACE
  EV --> DB["SQLite via Drizzle"]
  DB --> UI["Express server + React UI"]
  RT --> REMOTE["Remote generation API"]

Component	Path	Role
CLI and commands	src/main.ts, src/commands/, src/node/doEval.ts	Parse flags, resolve configs, call evaluate
Evaluator	src/evaluator.ts	Builds the test matrix, schedules steps, runs comparisons, persists rows
Single step	runEval in src/evaluator.ts	Renders the prompt with Nunjucks, calls the provider, transforms, grades
Assertions	src/assertions/	ASSERTION_HANDLERS registry and runAssertions aggregation
Model graders	src/matchers/llmGrading.ts	llm-rubric, factuality, closed-QA, G-Eval and others
Providers	src/providers/	Over 100 provider modules, including generic HTTP, WebSocket, script and MCP targets
Cache	src/cache.ts	cache-manager over Keyv: memory plus a KeyvFile disk store
Storage	src/database/tables.ts	Drizzle schema for evals, results, prompts, datasets, traces, spans
Tracing	src/tracing/	OTel SDK setup, local OTLP receiver, trace store, external trace providers
Red team	src/redteam/	Plugins, strategies, graders, risk scoring, remote generation
Web UI	src/server/, src/app/	Express API and React app for results and red-team setup

How a request flows

For promptfoo eval:

1. Resolve. doEval loads and merges configs and builds a TestSuite. The library evaluate sets base path, env and config context, picks the Node runtime and constructs an Evaluator (evaluator.ts).
2. Expand. The evaluator merges defaultTest and scenarios into each test, expands array-valued vars into combinations (generateVarCombinations), and creates one RunEvalOptions per prompt × provider × test × repeat.
3. Schedule. Concurrency defaults to 4 (-j overrides it). It drops to 1 when a prompt uses the _conversation variable, a test uses storeOutputAs, or a browser provider keeps a persistent session (evaluator.ts). Concurrent steps run through async.forEachOfLimit and periodically flush prompt metrics to the store (L4394-L4425).
4. Run a step. runEval wraps the step in a cache namespace keyed by repeat index. runEvalInternal then fills runtime vars, renders the prompt, optionally creates an agent workspace, and generates a W3C trace context when tracing is on (L1618-L1700). It then calls the provider, applies transforms and grades the response.
5. Grade. runAssertions expands assert-set groups and runs assertions with bounded concurrency (default 3). Each result is added to an AssertionsResult with its weight and metric name, and the final pass/score applies the test threshold or a custom scoring function (assertions/index.ts, L860-L911).
6. Compare. After all rows finish, select-best and max-score assertions grade outputs for the same test across providers. Rows with model-graded assertions can be deferred and graded in groups per grading provider.
7. Persist. Rows go to eval_results and the run summary to evals in SQLite. evals.config keeps a snapshot of the resolved config (tables.ts). Optional JSON, CSV or HTML output files are written too.

Key components

Assertions

ASSERTION_HANDLERS maps about 70 base types to handlers (assertions/index.ts). They fall into four groups:

• deterministic checks: equals, contains*, regex, is-json, is-sql, cost, latency, word-count;
• reference metrics: bleu, rouge-n, levenshtein, similar (embeddings), tool-call-f1;
• model-graded checks: llm-rubric, g-eval, factuality, answer-relevance, context-*, moderation;
• trace and trajectory checks over recorded spans: trace-span-count, trace-error-spans, trajectory:tool-sequence, trajectory:goal-success.

Any type can be negated with a not- prefix. javascript, python, ruby and webhook cover custom logic.

LLM grading

matchesLlmRubric renders a grading prompt, calls the grading provider and parses a JSON {pass, score, reason} (llmGrading.ts). The grader comes from options.provider (or --grader), otherwise a default grading provider. Inside a red-team config with no explicit grader, it prefers promptfoo’s remote grading endpoint. Grading calls are not deduplicated. ProviderGroupedCallQueue only groups queued calls by provider id so one grader’s calls run together (providerCallQueue.ts).

Cache

Provider responses go to a cache-manager cache with a memory store and a KeyvFile disk store at ~/.promptfoo/cache/cache.json, with a 14-day default TTL that PROMPTFOO_CACHE_TTL can change (cache.ts, L80-L105). Repeats get separate namespaces, so --repeat 3 makes three real calls.

Tracing

With tracing enabled, promptfoo starts a local Express OTLP receiver. It accepts /v1/traces (JSON or protobuf) and /v1/logs and writes spans into SQLite traces/spans tables (otlpReceiver.ts). The target app receives a traceparent and exports its spans there. Trace assertions then poll until the span count settles. Langfuse, Braintrust and Tempo can also serve as trace sources. Tracing exists to grade agent runs, not to monitor production.

Red team

synthesize in src/redteam/index.ts drives plugins (dozens, under plugins/) and strategies (about 30, under strategies/) to produce test cases (index.ts). doRedteamRun calls doGenerateRedteam, then doEval on the generated file (shared.ts). Results are scored by per-plugin graders and summarised with severity-weighted risk scores.

How much runs locally is the main thing to know. REMOTE_ONLY_PLUGIN_IDS lists plugins that have no local implementation and always call the remote API: BOLA/BFLA, SSRF, indirect prompt injection, RAG poisoning, the MCP plugin, the coding-agent sets and all the medical, financial, pharmacy, insurance, e-commerce, telecom and real-estate packs (plugins.ts). Other generation goes remote when you are logged in to Cloud or have no OPENAI_API_KEY. The default endpoint is api.promptfoo.app (remoteGeneration.ts, L164-L189). PROMPTFOO_DISABLE_REMOTE_GENERATION turns all of this off (L73-L91), which also disables the remote-only plugins. Generation also checks a 100,000-probe monthly limit, counted from the local database, unless you are logged in to Cloud (redteamProbeLimit.ts, generate.ts).

Extending it

• Providers. Use file://provider.js or .py custom providers, the generic http and websocket providers, or exec: scripts. Any id: string is resolved through providerRegistry.ts.
• Assertions. Use javascript, python and ruby assertions, either inline or from a file. They return a boolean, a score or a full GradingResult. assertScoringFunction replaces the default weighted aggregation.
• Tests. Tests can come from CSV, JSON, YAML, XLSX, Google Sheets, HuggingFace datasets or a JS/Python generator.
• Hooks. extensions run code before and after all tests or each test.
• Red team. Custom plugins load from a file (plugins/custom.ts). Custom strategies are file:// JS modules.

Running it

Run npx promptfoo@latest init, then promptfoo eval and promptfoo view (the UI defaults to port 15500). Node.js 22.22 or newer is required. All state lives under ~/.promptfoo (SQLite database and cache). Results can be shared to promptfoo’s hosted service, but this is optional. A Dockerfile and Helm chart serve the web UI for teams. For CI, run promptfoo eval with a non-zero exit on failures. Telemetry can be turned off with PROMPTFOO_DISABLE_TELEMETRY.

Strengths and caveats

• Strength: breadth of checks. About 70 assertion types cover deterministic checks, reference metrics, LLM judges and trace/trajectory checks. Weights, thresholds and named metrics make mixed scorecards easy.
• Strength: zero infrastructure. A single Node process plus SQLite, with response caching. It is easy to put in CI and cheap to re-run.
• Strength: the most complete red-team generator in this group. Plugins, wrapping strategies, multi-turn attackers and per-plugin graders feed the same eval and report pipeline.
• Caveat: red team is partly a client of a hosted API. Many plugins only exist server-side, generation goes remote by default without an OpenAI key, and there is a free-tier probe cap. Air-gapped use means disabling remote generation and losing those plugins.
• Caveat: offline by design. It has no production trace ingestion, sampling or online scoring. Traces exist only for runs it starts itself.
• Caveat: one very large file. src/evaluator.ts is over 5,600 lines and holds scheduling, comparison, persistence and timeouts, which makes changes to the core risky.
• Caveat: single-machine storage. SQLite works for one user. Team history needs the hosted service or the self-hosted server image.

Sources: code at 1df3eb8, deepwiki-open wiki (12 pages), verified Q&A.