AI Governance Institute
← News
Research2026-08-04

LLMs Fail on High-Dimensional Tabular Data, Exposing Fitness-for-Purpose Gaps

Source

Why Large Language Models Fail at Tabular Prediction

arXiv / Independent Researchers

What happened

Researchers Marta Garnelo and Wojciech Czarnecki published Why Large Language Models Fail at Tabular Prediction on arXiv, reporting results from controlled experiments across 31 benchmark datasets. The study tested a frontier LLM in a pure inference regime without fine-tuning or external tools, systematically evaluating five candidate explanations for why LLMs underperform classical models on tabular prediction. Of the hypotheses tested, input dimensionality emerged as the decisive factor: as the number of input features grows, LLM accuracy declines, while classical methods hold flat or improve. The finding is directly relevant to enterprise use cases that rely on high-dimensional structured data, including fraud detection, credit risk scoring, anti-money laundering screening, and automated compliance monitoring. Organizations that selected LLMs for these tasks based on general capability benchmarks rather than domain-specific tabular evaluations may be operating with models that are materially less accurate than available alternatives.

Why it matters

  • ·Enterprises that deployed LLMs in fraud detection, AML screening, or risk scoring workflows without high-dimensional tabular benchmarking may be accepting degraded accuracy without knowing it, creating direct regulatory exposure in sectors where model performance standards are enforced.
  • ·Procurement and intake controls built around general capability benchmarks rather than task-specific evaluation are structurally insufficient to catch this failure mode, meaning the gap is as much a process problem as a model problem.
  • ·If classical baselines demonstrably outperform LLMs on the actual data structures used in high-stakes decisions, continued use of LLMs in those roles may be difficult to justify under explainability and fitness-for-purpose obligations increasingly embedded in frameworks like the EU AI Act Implementation Timeline Update and sector-specific guidance from financial regulators.

Governance controls affected

PRC-003Third-Party AI Model EvaluationHOC-001AI System Risk ClassificationMON-001AI Performance BaselineMON-005Continuous Model EvaluationMGV-006RAI Benchmark-Aligned Evaluation Framework

What to do now

  • ☐Audit every active LLM deployment used for tabular prediction tasks (fraud scoring, risk modeling, AML, compliance monitoring) and document the number of input features used in each workflow.
  • ☐Re-run fitness-for-purpose evaluations comparing LLM performance against classical baselines specifically on high-dimensional subsets of your production data, not just general benchmarks used at procurement.
  • ☐Update your model intake and approval workflow (MGV-002) to require tabular-specific benchmark validation for any LLM being considered for structured data prediction tasks.
  • ☐Flag existing deployments where dimensionality was not tested as a risk variable and escalate to model owners for remediation prioritization.
  • ☐Review vendor model cards and documentation for any claims about tabular or structured data performance to determine whether those claims were validated under high-dimensional conditions.

What to watch next

Regulatory guidance on model fitness-for-purpose in financial services is tightening, with the Treasury Department AI Risk Management Framework for Financial Services already establishing expectations around performance validation in risk-sensitive workflows. Academic findings like this one are increasingly cited in enforcement proceedings and supervisory reviews as evidence of what a reasonable organization should have known at deployment time. Compliance teams should also monitor whether sector regulators in insurance and healthcare issue updated model validation guidance that incorporates structured data performance requirements for LLMs specifically.

Stay ahead of stories like this

Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.

Powered by Buttondown.

Related Coverage

Research2026-09-01

SR 26-2 Forces Banks to Rethink Model Governance From Inventory to Board Oversight

The OCC and Federal Reserve's revised model risk management guidance, SR 26-2, resets supervisory expectations for U. .S. banks by shifting to a materiality-based approach that covers both traditional statistical models and AI systems. Replacing the SR 11-7 framework that had governed bank model governance since 2011. Practitioner analysis from CRA identifies four areas banks must redesign: inventory scope, model tiering, validation independence. Governance alignment up to the board. A companion implementation guide from Lumenova AI adds concrete steps. Inventory rationalization and a distinct governance lane for agentic and generative AI. A proposed academic framework maps a six-layer control architecture for bringing GenAI systems into SR 26-2 scope. Banks that still run AI governance and model risk management. Separate programs face the most immediate pressure to harmonize them.

Research2026-09-12

IEEE Survey Links Explainability and Fairness as a Single Audit Obligation

A peer-reviewed survey published by the IEEE Computer Society examines the relationship between explainability and fairness in machine learning. Finding that the two properties are increasingly inseparable in practice. The survey, titled 'On the Interplay of Explainability. Fairness in AI,' documents how each capability reinforces the other when diagnosing and mitigating bias. The work has direct implications for model validation, audit readiness, and bias documentation programs.

Research2026-09-01

PwC Banking AI Framework Maps Five Gaps SR 26-2 Left Unresolved

PwC Germany published a whitepaper structuring AI governance for banks around five core challenges: scope definition, three-lines-of-defense adaptation, proportionality. Third-party risk, and AI-specific model validation. The paper offers a practical implementation scaffold for financial institutions working through model risk management reform. It does not introduce regulatory obligations, but provides detailed control-ownership guidance banks can use to close gaps left by existing. Supervisory requirements.