AI Vendor Due Diligence

AI vendor due diligence is the process of evaluating third-party AI providers and their systems before integration into enterprise operations, examining factors like data handling practices, model transparency, security measures, and compliance certifications. This practice is critical for enterprise AI governance because vendors often introduce significant risks around data privacy, algorithmic bias, regulatory compliance, and operational continuity that directly impact the organization's legal and reputational standing. Effective vendor due diligence helps organizations assess whether AI solutions meet internal governance standards and external regulatory requirements before deployment.

1 item

ResearchGlobal2026-07-22

LLM Judges Can Silently Shift Benchmark Rankings by Six Positions While Aggregate Metrics Show Nothing Wrong, CrucibleBench Research Finds

CrucibleBench, an independent research project, published findings showing that a single LLM-judge component inside a benchmark scoring stack reordered model rankings by up to six positions without triggering any warning from standard aggregate reliability statistics. Per-model judge agreement varied from 21.7% to 84.8% across 13 models and 650 runs. The researchers recommend that any benchmark using LLM judges report per-subject agreement rates and test ranking stability under judge substitution.

AI evaluation benchmark integrity LLM judges model procurement measurement risk AI vendor due diligence model risk management