Scale AI

Scale AI is a San Francisco company that supplies labeled training data and evaluation services to AI developers, including frontier model labs and government agencies. Its business rests on human annotators who label, rank, and test model outputs, so its data-handling and worker-classification practices carry governance weight for any organization that buys or relies on its work. Following a 2025 investment by Meta, several AI labs reduced or ended their use of Scale, which has made its client concentration and data provenance a recurring topic for vendor-risk reviews.

1 item

Research2026-10-28

DualChem Benchmark Exposes a Gap Between AI Risk Recognition and Model Behavior

Scale AI, working with NIST's Center for Advancing Innovation and Standards for Super Intelligence (CAISSI), has published DualChem. It is a 600-question benchmark testing how frontier AI models handle ambiguous chemistry queries with dual-use potential. Testing across roughly 30 frontier models found that models frequently answer queries they simultaneously flag as potentially dangerous. The benchmark gives government agencies and enterprises a new tool for assessing AI risk in chemistry and related sensitive domains.

Scale AI NIST CAISSI DualChem dual-use risk AI benchmarking AI safety evaluation export controls