IPHS 300: AI for Humanity
Document Type
Poster
Publication Date
Spring 2026
Abstract
Psychology gave us the Dunning-Kruger effect, the observed pattern that a person is often overconfident when they fall in the lowest quartile of that skill’s proficiency. This research is interested in seeing if large language models (LLMs) suffer the same miscalibration of metacognitive skills. Using thirty deliberately ambiguous college-level math question prompts, this research assesses Claude Sonnet 4.6’s ability to interpret each question, provide an answer, and rate its own confidence without asking for clarification. Results show that while confidence appropriately drops at the moderate tier, it rebounds on the hardest questions (0.840) even as accuracy falls to 0.450. This pattern suggests that the Dunning-Kruger effect may be observed in LLMs. An Expected Calibration Error of 0.1033 confirms notable miscalibration, suggesting users should not treat expressed LLM confidence as a reliable signal on complex or ambiguous inputs.
Recommended Citation
Grespin, Maggie, "Confident and Wrong : How LLM Miscalibration Mirrors the Dunning-Kruger Effect Across Ambiguity Levels" (2026). IPHS 300: AI for Humanity. Paper 65.
https://digital.kenyon.edu/dh_iphs_ai/65
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.