AI Benchmarking

AI benchmarking refers to the standardized tests used to measure and compare how well AI models perform on tasks such as reasoning, coding, or answering questions. For governance teams, benchmark scores often appear in vendor claims, procurement evaluations, and regulatory documentation, so understanding what a benchmark does and does not measure is essential. Scores can be inflated by training on test data or by narrow test conditions, which means a high ranking may not reflect how a model will behave in a company's real-world use cases.

1 item

Research2026-10-28

DualChem Benchmark Exposes a Gap Between AI Risk Recognition and Model Behavior

Scale AI, working with NIST's Center for Advancing Innovation and Standards for Super Intelligence (CAISSI), has published DualChem. It is a 600-question benchmark testing how frontier AI models handle ambiguous chemistry queries with dual-use potential. Testing across roughly 30 frontier models found that models frequently answer queries they simultaneously flag as potentially dangerous. The benchmark gives government agencies and enterprises a new tool for assessing AI risk in chemistry and related sensitive domains.

Scale AI NIST CAISSI DualChem dual-use risk AI benchmarking AI safety evaluation export controls