OpenAI - Unpacking LLM Hallucinations - Statistical Roots, Evaluation Incentives, and Solutions

why-language-models-hallucinate.pdf

Excerpt: Why do LLMs confidently generate false information, and is it our current evaluation systems that are inadvertently encouraging this "hallucination" behavior?

Unpacking LLM Hallucinations: Statistical Underpinnings and the Deep Logic of Evaluation Incentives

In recent years, LLM technology has achieved remarkable advancements, demonstrating unprecedented capabilities in understanding and generating human language. However, a persistent specter—"Hallucination"—continues to plague LLMs, undermining user trust in their outputs. Hallucination, in this context, does not refer to sensory illusions but rather to the LLM's generation of plausible but factually incorrect or fabricated information, often with excessive confidence. A recent study by OpenAI and its collaborators delves deep into the statistical roots of LLM hallucinations and reveals the reinforcing role played by current evaluation mechanisms.

To intuitively grasp the prevalence of LLM hallucinations, we can review several typical cases:

• The Adam Tauman Kalai Birthday Problem: When an advanced open-source LLM was asked for Adam Tauman Kalai's birthday, even when explicitly instructed to "answer only if you know," it confidently provided 3 different incorrect dates (03-07, 15-06, 01-01), while the correct date is actually in the autumn. This exposed the model's propensity to "guess" under uncertainty.
• The DEEPSEEK Letter Counting Problem: When asked to count the number of "D"s in the word "DEEPSEEK", DeepSeek-V3 yielded answers of "2" or "3" (the correct answer is 2) in 10 independent tests, while other models even produced egregious errors like "6" and "7". This indicates that hallucinations exist not only in factual information but can also occur in seemingly simple logical reasoning tasks.

Adam Kalai's Doctoral Dissertation Title Problem: Adam Kalai's Doctoral Dissertation Title Hallucination Case

Source: OpenAI - Why Language Models HallucinateThe table above shows the responses of ChatGPT, DeepSeek, and Llama, 3 mainstream LLMs, to the question "Adam Kalai's doctoral dissertation title." Although they all provided a seemingly professional title and year, none were correct (the correct year is 2001). These cases collectively point to a core question: Why do LLMs confidently generate false information?

Statistical Roots of Hallucination: Inevitability in the Pre-training Phase

One of the core insights from the OpenAI study is that the roots of LLM hallucination are deeply embedded in the statistical nature of their pre-training phase. During this stage, foundation models learn the language distribution from vast text corpora, essentially solving an "Is-It-Valid (IIV) classification problem"—that is, determining whether a given LLM output is valid.

1. The "Is-It-Valid (IIV) Classification Problem" and Generation Errors

The authors connect the LLM's ability to generate valid outputs with a binary classification problem: Is-It-Valid (IIV). This means that during the learning process, the model needs to determine whether each segment it generates is "valid" (labeled as +) or "invalid" (labeled as -).

IIV Classification Problem Diagram

Source: OpenAI - Why Language Models Hallucinate

The diagram above visually illustrates the working principle of the IIV classifier. The left panel shows how IIV learns to identify valid generations through labeled +/- examples. The right panel reveals the classifier's performance differences across various concepts: on tasks with clear patterns like "Spelling," the model can perform well; however, on tasks requiring precise reasoning like "Counting," if the model itself has flaws, it may perform poorly; and for "arbitrary facts" with no discernible patterns in the training data, such as "Birthdays," even a "good" model can hardly avoid errors.

The study indicates that any LLM can be viewed as an IIV classifier, and its generation error rate (i.e., the probability of the model generating incorrect information) is approximately 2x its IIV misclassification rate (i.e., the probability of the model making errors when judging information validity). This implies that as long as LLMs misjudge the validity of information, they will inevitably produce hallucinations when generating content. Even if the training data itself is 100% correct, the model may still err on facts that are inherently "unlearnable," such as a specific birthday missing from the training data.

2. Calibration and the Inevitability of Hallucination

"Calibration" refers to the consistency between a model's predicted probabilities and its actual correctness rate. For instance, if a model claims 90% confidence in an answer, then the answer should be correct approximately 90% of the time. Pre-trained LLMs, by minimizing the cross-entropy loss function, naturally tend towards calibration.

However, a subtle link exists between calibration and error: if a model is calibrated, it will inevitably produce errors when facing uncertain or unlearnable facts. An "error-free" model (e.g., one that always answers "I don't know" or only memorizes training data) would necessarily be uncalibrated. The significance of calibration lies in its ability to match the actual correctness rate with the model's confidence level.

GPT-4 Calibration Histogram

Source: OpenAI - Why Language Models Hallucinate

The chart above displays the calibration histograms of GPT-4 at different stages. The left panel shows that in the pre-training phase, before RL, GPT-4's calibration was excellent (ECE: 0.007), indicating a high consistency between its confidence level and actual correctness rate. However, the right panel shows that in the post-training phase, after RL, the model's calibration decreased (ECE: 0.074). This suggests that the post-training phase might have led the model away from "honestly" expressing uncertainty.

3. Multiple Inducements for Foundation Model Errors

In addition to the statistical inevitability mentioned above, LLM hallucinations can also be caused by various other factors:

• Arbitrary-fact hallucinations: When certain facts appear very rarely in the training data, or have no simple patterns to explain them, models encounter "cognitive uncertainty." For example, LLMs rarely err on frequently appearing facts like "Einstein's birthday," but are more prone to hallucinate about the birthday of an ordinary person mentioned only once in an obituary. The study introduced the "Singleton rate" concept, which measures the frequency with which a prompt appears only once in the training data, and points out that the lower bound of hallucination is positively correlated with this frequency.
• Errors due to poor models: Sometimes, hallucinations do not stem from data issues but rather from the inherent limitations of the LLM itself. For instance, a trigram model based on 2 words of context might have a generation error rate as high as 1/2 when distinguishing semantically similar but gender-different phrases. This indicates that if the model itself is not "good" enough, even if it can capture local dependencies, it might still produce a high error rate. DeepSeek-V3's errors in letter counting were attributed by the authors to its inherent model limitations.
• Other significant factors: These also include "computational complexity limitations" where LLMs err on computationally difficult problems (e.g., decryption); "distribution shift issues" caused by discrepancies between training and testing data distributions; and the "GIGO effect" where factual errors present in the training corpus are replicated by the model.

Evaluation Reinforcement of Hallucination: Challenges in the Post-training Phase

Although hallucinations in the pre-training phase have their statistical roots, the study points out that their persistence and even exacerbation in the model's subsequent optimization (post-training) phase are primarily attributable to deficiencies in current mainstream evaluation mechanisms.

1. The "All-or-Nothing" Dilemma of Binary Scoring

Current mainstream LLM evaluation benchmarks, such as GPQA, MMLU-Pro, IFEval, Omni-MATH, etc., generally employ binary scoring (0-1 point system): a correct answer receives 1 point, while incorrect, blank, or "I don't know" (IDK) answers all receive 0 points.

The fundamental flaw of this mechanism is that it penalizes the expression of uncertainty. Under such a scoring system, a model choosing "I don't know" (IDK) is always suboptimal. This is because saying "I don't know" yields 0 points, whereas if it "guesses" an answer, it might get 1 point if correct, and still 0 points if incorrect. This incentivizes LLMs to "guess" when uncertain, rather than honestly admitting uncertainty, in order to maximize their performance on leaderboards. The study explicitly states that for any binary scoring distribution, abstention is not the optimal response.

For example, most popular evaluation benchmarks award no points for "I don't know" (IDK) or abstention. Only WildBench provides partial scores, but its scoring rules (1-10 point system) may still result in IDK scores being lower than "acceptable" responses containing hallucinations, thereby implicitly encouraging guessing. Furthermore, many evaluation benchmarks use LLMs for scoring, which can lead to incorrect guesses being mistakenly judged as correct, further fueling hallucinatory behavior.

Solutions and Future Outlook: Recalibrating Incentive Mechanisms

Given the statistical roots of hallucination and the reinforcing effect of evaluation mechanisms, the OpenAI research team proposes a key solution: modifying the scoring methods of existing evaluation benchmarks to introduce "explicit confidence targets."

1. Introducing Explicit Confidence Targets

The study suggests explicitly defining confidence targets in evaluation instructions, for example: "Only answer if you have >t confidence, as incorrect answers will incur a penalty of t/(1-t) points, correct answers receive 1 point, and 'I don't know' receives 0 points."

• Incentivizing calibrated behavior: Under this mechanism, providing an answer only yields an expected score higher than "I don't know" (0 points) when the probability of the answer being correct is >t. This means the model will be incentivized to "behaviorally calibrate," deciding whether to answer or abstain based on its true confidence level.
• Distinction from existing work: This paper emphasizes that such confidence targets should be explicitly communicated to the LLM and integrated into existing mainstream evaluations (rather than merely introducing new hallucination evaluations) to more effectively reduce hallucination.

2. "Demystifying" Hallucination and Future Outlook

This research successfully "demystified" LLM hallucination. It is no longer an elusive black-box problem but rather a product of generation errors resulting from cross-entropy loss minimization during the pre-training phase, and its persistence in the post-training phase due to evaluation mechanisms rewarding guessing.

The key to solving hallucination lies in recalibrating incentive mechanisms. By simply modifying mainstream evaluations to reward appropriate expressions of uncertainty rather than penalizing them, the barriers inhibiting hallucination can be removed. This will pave the way for developing LLMs that are more nuanced, "honest," and pragmatically capable (i.e., knowing what to say in which context), enabling them to provide information not only accurately but also with the appropriate wisdom of "knowing what you know, and knowing what you don't know."