A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
- Dr. Magnus Lynch evaluated frontier LLMs (GPT-6 Astra and Claude Fable 5.1) against a fine-tuned, 10-year-old Inception v3 CNN on diagnosing biopsy-confirmed skin lesions from the MILK10k dataset.
- In a test of 200 biopsy-proven lesions (100 unsafe to discharge, 100 safe to discharge), the CNN significantly outperformed both frontier models:
- Overall accuracy: The CNN achieved 62% (125/200), whereas GPT-6 Astra reached 42% (84/200) and Claude Fable 5.1 reached 42% (85/200).
- Top-two accuracy: The CNN reached 173/200, compared to 134/200 for GPT-6 Astra and 138/200 for Claude Fable 5.1.
- Melanoma detection: Out of 30 melanomas, the CNN correctly diagnosed 13 as its first choice, beating Claude Fable 5.1 (11/30) and GPT-6 Astra (8/30).
- The models exhibited distinct failure modes and class biases:
- GPT-6 Astra over-diagnosed basal cell carcinoma (75 times, including 14 of the 30 melanomas, when only 25 were BCC).
- Claude Fable 5.1 over-diagnosed actinic keratosis (53 times, when only 20 were present).
- The CNN primarily failed on harmless lesions by leaning toward benign misclassifications (misclassifying 14 of 17 missed melanomas as naevi or seborrhoeic keratoses).
- Specialized lightweight models remain drastically more efficient: the Inception v3 model was fine-tuned in ~20 minutes on a single H100 and runs locally inside a phone browser via WebGPU.
- When tested from a general patient perspective asking "do I need to see a doctor?" using clinical photos alone, both frontier LLMs showed extreme risk aversion, recommending medical consultation in almost all cases (98–99%).
Hacker News Discussion
- Commenters emphasized the architectural mismatch of using massive general-purpose LLMs for tasks best served by dedicated, task-specific classifiers (CNNs, Bayesian models, or traditional ML).
- The discussion highlighted "Jev-style" decision models, debating whether their popularity stems from brilliant marketing to developers who treat LLMs as a universal hammer, or from genuine convenience for zero-shot tasks where collecting labeled training data is uneconomical.
- Participants observed a pervasive lack of first-principles ML knowledge in the software industry, where newer developers overlook traditional ML despite its superior efficiency, lower latency, and lower resource requirements.
- Multiple commenters argued that the most effective workflow is using LLMs to build, label, or scaffold traditional models and deterministic tools, rather than using raw LLM inference directly for classification or numerical computation.