LLM Judges

LLM judges are AI systems that use large language models to make or assist in making legal and administrative decisions, such as evaluating evidence, predicting case outcomes, or recommending penalties. Organizations deploying LLM judges face significant governance challenges around bias, explainability, and accountability, since these systems can affect people's rights and livelihoods. This category matters for AI governance because it raises critical questions about algorithmic fairness, human oversight requirements, and whether automated judicial decision-making should be regulated differently than other AI applications.

1 item

ResearchGlobal2026-07-22

LLM Judges Can Silently Shift Benchmark Rankings by Six Positions While Aggregate Metrics Show Nothing Wrong, CrucibleBench Research Finds

CrucibleBench, an independent research project, published findings showing that a single LLM-judge component inside a benchmark scoring stack reordered model rankings by up to six positions without triggering any warning from standard aggregate reliability statistics. Per-model judge agreement varied from 21.7% to 84.8% across 13 models and 650 runs. The researchers recommend that any benchmark using LLM judges report per-subject agreement rates and test ranking stability under judge substitution.

AI evaluation benchmark integrity LLM judges model procurement measurement risk AI vendor due diligence model risk management