langfuse/langfuse

Langfuse provides three evaluator types that produce scores, plus deterministic sampling and blocking. LLM-as-a-judge (evalService.ts lines 941–1233) — the primary mechanism — constructs a prompt from a template, calls a configurable LLM (OpenAI, Anthropic, etc.) with Zod-validated structured output, and normalizes the response into NUMERIC, BOOLEAN, or CATEGORICAL scores (outputDefinition.ts lines 1–428). The output definition is a persisted schema (PersistedEvalOutputDefinitionSchema) supporting numeric ranges (minValue/maxValue), boolean verdicts, and categorical choices (single or multi-match). The toNormalizedScores helper (evalService.ts lines 1235–1270) converts the LLM response — e.g. a numeric 0–1, a boolean true/false, or a string array — into CodeEvalScoreWithName[] records with a shared comment (the model's reasoning). Each score carries a dataType, value, and optionally configId and metadata. Code-based evaluators (codeEvalExecution.ts) dispatch user-supplied code (JS/Python) to either an AWS Lambda dispatcher or a local process, returning scores in a {scores: [...]} JSON contract — NO built-in heuristic or statistical metrics exist outside these three types. Decision-model evaluators (decisionModelEvaluatorExecution.ts) call a separate "TypeSafe" API for choice/score/boolean questions and map responses into scores. Deterministic sampling (deterministicSampling.ts lines 1–31) controls what fraction of traces are scored, using a SHA‑256 over target ID and sampling rate. Per-trace vs per-observation scoring is handled via targetObject: TRACE scores the whole trace at the trace level, DATASET ties scores to dataset items, and EVENT (observation-level) schedules per-observation evaluations (observationEval/). All scores are persisted with source: "EVAL" and share an EvalScoreWritePayload structure (evalScoreEvent.ts).

LLM-as-a-judge is implemented through a chain of modules converging on runLLMAsJudgeEvaluation in evalService.ts (lines 941–1233). Judge prompts and rubrics are stored as EvalTemplate records with a prompt (template string using {{variable}} syntax) and promptMessages (an array of system/user/assistant messages). The prompt is compiled by substituting extracted variables — trace fields like input, output, or dataset-item columns — via compileEvalPrompt in llmEvaluatorExecution.ts (lines 40–57). Structured output is enforced through Zod schemas: the outputDefinition stored on the template (legacy {score, reasoning} strings or v2 {dataType, score, reasoning} objects) is compiled into a Zod schema via compilePersistedEvalOutputDefinition (outputDefinition.ts lines 362–375). The schema is serialized to JSON Schema and passed to the AI SDK, which constraints the LLM's response shape. Results are validated with validateEvalOutputResult — numeric ranges, allowed categorical values, duplicates detection, and boolean type strictness are all checked (test file lines 541–679). Judge model choice is per-evaluator: each template stores provider, model, and modelParams which are resolved through fetchModelConfig (evalExecutionDeps.ts lines 327–337). The system supports OpenAI, Anthropic, Google, AWS Bedrock, and any OpenAI-compatible endpoint. No built-in multi-sample/consensus — each eval job produces a single LLM call. No explicit calibration exists, but bias controls are available through the prompt template itself (the user authors the rubric). Auto-blocking (classifyEvaluatorLlmError.ts) is key: when the judge LLM fails (auth, billing, endpoint unreachable), the evaluator is automatically paused at the database level so all rules using it stop until the user intervenes (lines 60–84). Internal tracing records each judge execution as an LLMJudge-environment trace in the user's project for debugging.

Datasets are stored as Dataset and DatasetItem records in PostgreSQL, with ClickHouse for analytics queries. A dataset has an inputSchema and expectedOutputSchema (optional JSON Schema) for validation (dataset-items.ts lines 62–72). Dataset items carry input, expectedOutput, metadata, plus a sourceTraceId/sourceObservationId when created from traced data. Items are versioned via a temporal model: validFrom/validTo columns enable point-in-time snapshots — fetching an item at a specific date retrieves the version valid then (code in evalService.ts extractVariablesFromTracingData at line 1635: ...(datasetItemValidFrom ? { validFrom: datasetItemValidFrom } : { validTo: null })). Dataset runs (dataset-runs.ts) execute a prompt experiment against dataset items: the experimentServiceClickhouse.ts takes a prompt template, replaces variables with each dataset item's input, calls the LLM, and creates DatasetRunItem records linking the experiment's output traces back to the original dataset items (lines 74–78). No synthetic data generation DSL exists — items are created via the UI, the public API (POST /api/public/dataset-items), or batch CSV/JSON upload. Golden sets are just datasets used as eval benchmarks: you create a dataset, map its columns to evaluator variables in an evaluation rule, and the rule scores every new trace that matches (or you run a batch evaluation). Eval jobs can target TRACE (live scoring) or DATASET (scoring dataset-run items). Benchmark task registration is handled through the evaluation rules system (evaluationRule Prisma model) — rules connect evaluators (judge templates) to dataset items or traces via filters, sampling rates, and variable mappings.

Eval execution is a job-queue pipeline built on BullMQ (Redis). Scheduling (evalService.ts createEvalJobs lines 302–913): when a trace is upserted or a dataset run item created, the system fetches active evaluation rules for the project, filters them against the trace (filter state and sampling), deduplicates via deterministic job IDs, and enqueues EvalExecutionQueue jobs. Three parallel queues exist for trace-level and observation-level evals, routed by EvalExecutionQueue.getInstance({shardingKey}) (line 841). Parallelism is handled by BullMQ worker concurrency; no built-in batching per model provider. Caching (deterministicSampling.ts) uses SHA‑256 hashing of the target ID to produce a deterministic [0,1) value for sampling decisions. A trace-cache optimization (evalService.ts lines 389–442) fetches trace data once for multiple configs. Execution calls into runLLMAsJudgeEvaluation (or code/decision-model executors), which calls the LLM, validates the structured output, and persists scores. Results flow through completeEvalExecution (evalCompletion.ts lines 23–98): scores are uploaded to S3 as JSON event files, queued for ClickHouse ingestion via IngestionQueue, and the JobExecution row is updated to COMPLETED with the primary score ID and execution trace ID. Error handling (evalExecutionMetrics.ts) classifies outcomes into success, platform_error, upstream_error, customer_error, or cancelled. LLM errors are categorized and can auto-block the evaluator. CI integration is user-defined (no built-in harness); users call the public API or run evaluations ad-hoc via the UI's "Run evaluation" batch dialog. Results appear as scores on traces/observations in the Langfuse UI, filterable and searchable via the Eval Logs table (eval-log.tsx). Dashboards and regression are user-built on top of the score data — the platform stores all scores with timestamps, trace IDs, and metadata, enabling external charting or the built-in analytics views.

Langfuse is an observability platform, so traces are captured via a first-party Python/JS/TS SDK (langfuse pip package, JS SDK in packages/shared/src/server/llm/), and it also ingests OpenTelemetry data through the OTel ingestion pipeline (OtelIngestionProcessor.ts). Traces consist of a root trace with nested observations (spans, generations, events), all stored in ClickHouse. Linking evals to traces works through the executionTraceId mechanism: when an LLM-as-a-judge evaluator runs, it creates a new internal trace (environment langfuse-llm-as-a-judge or langfuse-code-eval) in the same project, passing traceSinkParams with targetProjectId, traceId, and metadata (evalExecutionDeps.ts lines 302–312). This internal trace records the judge's input prompt, output score, model, latency, and reasoning. The score event's executionTraceId field links the score back to this internal trace. Online vs offline evals: "online" (trace-upsert-triggered) runs eval rules immediately when traces arrive; "offline" evaluations are triggered via the UI batch-action dialog or the public API with a timestamp range for historical data. Feedback and annotation is a separate feature: Langfuse has scores with source: "ANNOTATION" as distinct from source: "EVAL", both stored in the same scores table but tagged differently. The scores repository (scores.ts) handles both types. The eval log (eval-log.tsx) provides a UI table of all eval job executions with status, score, comment, and links to the target trace. Each evaluation also writes its own internal trace with environment labels that are excluded from triggering further evaluations (the isInternalEvalEnvironment guard in isEvalTargetEnvironmentAllowed.ts prevents infinite eval loops).

Langfuse does NOT have a dedicated red-teaming framework (no adversarial-probe runners, attack plugins, or automated jailbreak testing pipelines). Instead, it provides evaluator templates in the safety category of its managed templates catalog (managedTemplatesCatalog.ts lines 53–58, 1193–1297) that users can use for manual or automated safety testing. These include: Detect Prompt Injection — an LLM-as-a-judge template that classifies whether an input contains a credible manipulation attempt (override instructions, reveal system prompt, bypass safeguards), returning a boolean verdict (lines 1242–1297). Check Rule Adherence — a boolean evaluator that checks if an output follows a defined policy or instruction set (lines 1193–1241). The safety category description (lines 54–57) states it "Monitors policy adherence, privacy leakage, and adversarial prompts." These are template starters — users must adapt the prompt and deploy an evaluation rule that runs on their traces. There are no built-in safety-specific tools, no adversarial dataset generators, no automated red-team loops, and no vulnerability-reporting workflow (the repo's SECURITY.md at root only links to a generic security policy). Safety testing must be authored by the user: create a dataset of adversarial inputs, write an evaluator template (or use the managed one), and run batch evaluations against it. The platform stores all results and can track regressions over time via the scores API.