Old CNN Beats GPT-6 Astra and Claude Fable 5.1 on Skin Cancer Diagnosis
What happened
Dr. Magnus Lynch of Lynch Lab published A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis, comparing GPT-6 Astra and Claude Fable 5.1 against a fine-tuned Inception v3 convolutional neural network. Inception v3 is a specialized image-recognition model trained on skin lesions. The comparison covered 200 biopsy-confirmed cases. The specialized model achieved 62% overall diagnostic accuracy versus 42% for each frontier LLM. It also correctly identified melanomas at a higher rate. It was also dramatically better at recognizing benign lesions. The frontier models were more likely to raise false alarms or miss cancers outright. Both LLMs showed systematic over-prediction biases, consistently leaning toward specific diagnoses regardless of the actual image. The study adds clinical evidence to a growing pattern. AI models that perform impressively on general tasks can underperform purpose-built tools in high-stakes specialist contexts. Procurement and clinical governance programs must explicitly test for this gap rather than assume it away.
Why it matters
- ·Fitness-for-purpose validation is now a concrete procurement requirement, not a checklist item. The EU AI Act (Regulation (EU) 2024/1689) classifies AI used in medical diagnosis as high-risk, requiring documented conformity assessment before deployment. This study shows that frontier-model benchmark scores do not substitute for domain-specific clinical testing.
- ·Organizations deploying general-purpose AI in clinical or other high-stakes specialist settings carry liability exposure if they cannot demonstrate that the model was evaluated against task-specific performance standards. Regulators and plaintiffs will ask whether the procurement team compared the tool to the specialized alternative.
- ·Both LLMs systematically favored certain diagnoses regardless of the actual evidence. This is exactly the kind of systematic error that bias and fairness monitoring programs are designed to catch. Yet such programs rarely test for it in specialist domains. Compliance teams that approved frontier AI for clinical use without domain-specific bias testing may need to revisit those approvals.
Governance controls affected
What to do now
- ☐Identify every deployment where a general-purpose AI model is being used for a specialist task, such as clinical diagnosis, legal analysis, or financial underwriting, and confirm whether it was benchmarked against the best available purpose-built alternative before approval.
- ☐Ask your clinical or domain leads to review whether the AI tools currently in use have been tested for systematic over-prediction or under-prediction biases specific to the task, not just general accuracy figures.
- ☐Review procurement records for any AI tool deployed in a high-risk context to confirm that vendor-supplied benchmark results were supplemented by independent, task-specific evaluation on representative data from your own environment.
- ☐Escalate to your AI governance committee any clinical AI deployment that was approved based solely on general-purpose frontier-model reputation or published leaderboard scores rather than domain-specific clinical validation evidence.
- ☐For any AI tool used in medical, legal, or other regulated specialist decisions, add a standing review trigger for when the underlying model is updated, since a model change can alter task-specific performance without changing headline benchmark scores.
What to watch next
Regulators overseeing clinical AI, including the U.S. Food and Drug Administration and EU AI Act enforcement bodies, are increasing scrutiny of AI validation before deployment in high-risk settings. As the EU AI Act (Regulation (EU) 2024/1689) conformity assessment obligations mature, expect inspectors to request evidence of domain-specific testing rather than accepting general capability claims. The EU AI Office Inspections targeting hiring, credit, and healthcare AI signal that regulators are already moving into these sectors. Healthcare and medtech compliance teams should also monitor FDA's FDA AI/ML Software as Medical Device Guidance. Updates may address the growing use of frontier general-purpose models in diagnostic workflows.
Stay ahead of stories like this
Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.
Recent issues
- Regulators, litigants, and attackers are all moving faster than most organizations' AI governance frameworks, shifting leverage away from deployers and toward those who can prove control, accountability, and honest disclosure.21 Oct
- Three separate incidents this week reveal that AI vendors cannot be trusted to self-police, and that harm from deployed agents can run unchecked for months before anyone inside the organization notices.15 Oct
Free every Thursday. Unsubscribe anytime.