AI Governance Institute
← News
Research2026-10-26

Old CNN Beats GPT-6 Astra and Claude Fable 5.1 on Skin Cancer Diagnosis

What happened

Dr. Magnus Lynch of Lynch Lab published A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis, comparing GPT-6 Astra and Claude Fable 5.1 against a fine-tuned Inception v3 convolutional neural network. Inception v3 is a specialized image-recognition model trained on skin lesions. The comparison covered 200 biopsy-confirmed cases. The specialized model achieved 62% overall diagnostic accuracy versus 42% for each frontier LLM. It also correctly identified melanomas at a higher rate. It was also dramatically better at recognizing benign lesions. The frontier models were more likely to raise false alarms or miss cancers outright. Both LLMs showed systematic over-prediction biases, consistently leaning toward specific diagnoses regardless of the actual image. The study adds clinical evidence to a growing pattern. AI models that perform impressively on general tasks can underperform purpose-built tools in high-stakes specialist contexts. Procurement and clinical governance programs must explicitly test for this gap rather than assume it away.

Why it matters

  • ·Fitness-for-purpose validation is now a concrete procurement requirement, not a checklist item. The EU AI Act (Regulation (EU) 2024/1689) classifies AI used in medical diagnosis as high-risk, requiring documented conformity assessment before deployment. This study shows that frontier-model benchmark scores do not substitute for domain-specific clinical testing.
  • ·Organizations deploying general-purpose AI in clinical or other high-stakes specialist settings carry liability exposure if they cannot demonstrate that the model was evaluated against task-specific performance standards. Regulators and plaintiffs will ask whether the procurement team compared the tool to the specialized alternative.
  • ·Both LLMs systematically favored certain diagnoses regardless of the actual evidence. This is exactly the kind of systematic error that bias and fairness monitoring programs are designed to catch. Yet such programs rarely test for it in specialist domains. Compliance teams that approved frontier AI for clinical use without domain-specific bias testing may need to revisit those approvals.

Stay ahead of stories like this

Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.

Free every Thursday. Unsubscribe anytime.

Governance controls affected

What to do now

  • ☐Identify every deployment where a general-purpose AI model is being used for a specialist task, such as clinical diagnosis, legal analysis, or financial underwriting, and confirm whether it was benchmarked against the best available purpose-built alternative before approval.
  • ☐Ask your clinical or domain leads to review whether the AI tools currently in use have been tested for systematic over-prediction or under-prediction biases specific to the task, not just general accuracy figures.
  • ☐Review procurement records for any AI tool deployed in a high-risk context to confirm that vendor-supplied benchmark results were supplemented by independent, task-specific evaluation on representative data from your own environment.
  • ☐Escalate to your AI governance committee any clinical AI deployment that was approved based solely on general-purpose frontier-model reputation or published leaderboard scores rather than domain-specific clinical validation evidence.
  • ☐For any AI tool used in medical, legal, or other regulated specialist decisions, add a standing review trigger for when the underlying model is updated, since a model change can alter task-specific performance without changing headline benchmark scores.

What to watch next

Regulators overseeing clinical AI, including the U.S. Food and Drug Administration and EU AI Act enforcement bodies, are increasing scrutiny of AI validation before deployment in high-risk settings. As the EU AI Act (Regulation (EU) 2024/1689) conformity assessment obligations mature, expect inspectors to request evidence of domain-specific testing rather than accepting general capability claims. The EU AI Office Inspections targeting hiring, credit, and healthcare AI signal that regulators are already moving into these sectors. Healthcare and medtech compliance teams should also monitor FDA's FDA AI/ML Software as Medical Device Guidance. Updates may address the growing use of frontier general-purpose models in diagnostic workflows.

Related Coverage

Standards2026-11-04

First AI Company Earns FedRAMP 20x Class C, Resetting Federal Vendor Eligibility

Perplexity has become the first AI company to receive a full FedRAMP 20x Class C authorization under the 2026 Consolidated Rules. The certification covers Perplexity Enterprise and its application programming interface (API) Platform for use by the General Services Administration (GSA) and the Department of Agriculture (USDA). Security is now measured through outcome-based Key Security Indicators rather than narrative documentation, changing what vendors must demonstrate to qualify.

Corporate Policy2026-11-02

OpenAI's Biology Model Is Classified High-Risk. Applicants Must Prove Their Own Controls.

OpenAI published the [GPT-Rosalind-6 System Card Addendum](https://deploymentsafety.openai.com/gpt-rosalind-6) on October 30, 2026, for a frontier reasoning model specialized in biology, drug discovery, and translational medicine. The model carries a High capability designation under OpenAI's internal Preparedness Framework for biological and chemical risk. Access is limited to qualified researchers whose own governance practices and security controls pass a trust-based review conducted by OpenAI.

Corporate Policy2026-11-09

OpenAI's Transparency Blueprint Puts AI Infrastructure Risk on the Compliance Agenda

After the 2026 midterm elections, ballot measures restricted data center construction across the US. OpenAI responded by publishing a compute campus policy blueprint. It commits to ending non-disclosure agreements with local officials and disclosing environmental impacts. Compliance teams need to add vendor infrastructure exposure to procurement and vendor risk programs.