# LLM as judge

The use of a language model to score, compare, classify, or critique another system’s output or trajectory.

technical-term · Evaluation & verification · emerging · Reviewed 2026-08-09

## Definition

An LLM judge applies a rubric to candidate outputs or agent traces and returns a score, preference, explanation, or failure label. Model graders can scale subjective evaluation, but they require calibration against experts and are not automatically independent, unbiased, or authorized to accept risk.

Use is widespread and evaluation methods are developing; bias, stability, contamination, and calibration remain material limitations.

## Why it matters

A judge can measure qualities that deterministic tests cannot easily express.

The verification design must disclose judge model lineage, rubric, sampling, disagreement handling, and human calibration.

## System anatomy

- **Rubric:** Observable criteria and decision labels.
- **Judge:** The model, version, context, and configuration performing evaluation.
- **Calibration set:** Expert-scored examples used to measure agreement and bias.
- **Disposition:** What the score is allowed to block, route, or inform.



## Important distinctions

- **Independent verifier:** A judge becomes one verifier method only when ownership, lineage, scope, trigger, block authority, and recorded disposition satisfy independence requirements.
- **Deterministic test:** A model grader is probabilistic and rubric-dependent; a deterministic test has reproducible program logic.



## Implementation signals

- Calibrate against domain experts
- Randomize or counterbalance comparison order
- Fuse model judgment with deterministic checks and human review

## Failure modes

- Position or verbosity bias determines the winner
- Builder and judge share the same model lineage and blind spots
- A scalar score hides disqualifying failures



## Related knowledge

- [Eval-driven development](https://aisdlc.ai/agentic-engineering/eval-driven-development) — An engineering loop that converts expected behavior and observed failures into repeatable evaluations combining deterministic checks, environment inspection, security testing, model graders, repeated trials, and human judgment.
- [Independent verifier systems](https://aisdlc.ai/agentic-engineering/multi-agent-verification) — A separation-of-judgment architecture in which builder agents, evaluator agents, deterministic checks, domain experts, and authorization authorities challenge different failure surfaces.
- [Reflection](https://aisdlc.ai/agentic-engineering/reflection) — A feedback pattern in which a model critiques an attempt and records lessons or a revised approach for another attempt.

## Sources and further study

- [Anthropic — Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) — A practical treatment of evaluating trajectories, outcomes, graders, tasks, and agent-environment interaction. **Use:** Official guidance. **Limitation:** This is first-party engineering guidance drawn from Anthropic deployments. Evaluation designs remain task- and environment-specific, and the article does not establish independent verifier ownership by itself.
- [arXiv — Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://arxiv.org/abs/2406.07791) — Documents position-related bias in model-based comparative judgment and motivates calibration and bias controls. **Use:** Empirical evidence. **Limitation:** The study addresses selected judge models, prompts, and tasks; other grader failure modes and later systems require separate evaluation.
- [NIST — Towards best practices for automated benchmark evaluations](https://www.nist.gov/news-events/news/2026/01/towards-best-practices-automated-benchmark-evaluations) — Guidance on the integrity and limitations of automated benchmark evaluation practices. **Use:** Official guidance. **Limitation:** Guidance describes recommended practice; citation does not prove that a control is implemented or effective in a particular environment.

---

This library synthesizes cited research, standards, official documentation, and clearly attributed practitioner perspectives. Maturity describes the state of a concept—not vendor endorsement, production readiness, or permission to deploy. Benchmarks and demonstrations do not replace use-case evaluation, governed controls, independent verification, or named human release authority.