When a large language model (LLM) hallucinates, developers often assume the model lacks the necessary information. The usual response is to increase model size, expand training data, deploy retrieval-augmented generation (RAG), or build a more complex search architecture.
However, new research from Google Research and Technion suggests that missing knowledge is not always the problem. In many cases, an LLM has already encoded the relevant fact in its parameters but cannot reliably retrieve or reproduce it during generation.
The researchers found that frontier models, including GPT-5 and Gemini-3, encoded approximately 95% to 98% of the facts tested. This suggests that factual accuracy is often limited by knowledge recall—not knowledge storage.
Understanding how to unlock an LLM’s existing parametric knowledge through inference-time computation could help engineering teams build more reliable AI applications without always relying on larger models or external databases.
Knowledge profiling reveals what an LLM really knows
To measure the gap between knowledge storage and knowledge retrieval, the researchers recommend shifting LLM evaluation from simple question-level accuracy to fact-level knowledge profiling.
Instead of measuring whether a model answers one prompt correctly, fact-level profiling tests the same underlying fact under multiple conditions. This approach examines whether the fact is stored in the model’s parameters, whether it can be retrieved using different wording and representations, and how much inference-time computation is needed to produce the correct answer.
The framework distinguishes between a fact being encoded and a fact being reliably known. A model has encoded a fact when it can reproduce it after being prompted with the original training context. A model reliably knows the fact when it can answer questions about it across different phrasings, formats, contexts, and instructions.
“Encoding and recall failures are indistinguishable under accuracy metrics but imply different limitations and solutions,” the researchers wrote. “Encoding failures require pre-training interventions, such as scaling model size or data coverage. Recall failures often suggest post-training interventions that improve how the model makes use of what it has already encoded.”
The paper illustrates this distinction with the fact that Oasis played its first gig at the Boardwalk Club. The study categorizes model behavior into five knowledge profiles.
-
Direct recall: The model has encoded the fact and can retrieve it immediately when asked a direct question. No additional reasoning or inference-time computation is required.
-
Encoding failure, or “empty shelf”: The model has neither stored nor recognized the fact. It cannot complete a relevant Wikipedia-style passage about Oasis or answer questions about the band’s early performances. This type of failure may require more pre-training data, broader data coverage, or greater model capacity.
-
Recall failure, or “lost key”: The fact is encoded in the model’s parameters but is not accessible through a particular prompt. For example, the model may complete the original training passage about Oasis but fail to answer, “Where did Oasis play its first show?” Even additional thinking may not always recover the information.
-
Recall through reasoning: The fact is encoded but cannot be produced directly. The model successfully retrieves it only after using inference-time computation, such as step-by-step reasoning. Generating intermediate thoughts about Oasis, Manchester, and the band’s early history can help activate the relevant information.
-
Inference without encoding: The model has not explicitly stored the fact but arrives at a plausible answer by combining other information. For example, it may know that Oasis formed in Manchester, that the Boardwalk was a prominent music venue, and that emerging bands performed there. These related facts can support an educated—but potentially unreliable—inference.
LLM scaling does not automatically solve factual recall
The researchers evaluated 13 language models across more than 4 million responses. Their experiments used WikiProfile, a benchmark containing 2,150 facts extracted from Wikipedia. Each fact was tested in multiple formats, including context completion, direct questions, reverse questions, and multiple-choice validation.
For frontier models such as GPT-5 and Gemini-3, factual encoding was close to saturation. These systems encoded between 95% and 98% of the tested facts. Nevertheless, they could not directly recall approximately 26% to 34% of the facts they had already encoded.
Additional reasoning time acted as an important recovery mechanism. By allowing the model to perform more computation, researchers recovered approximately 40% to 65% of the encoded facts that the model initially failed to reproduce directly.
The researchers compared this behavior to the human “tip-of-the-tongue” experience. Deliberately reconstructing surrounding context can eventually make an otherwise inaccessible memory available.
Increasing model size alone did not eliminate the recall gap. In one experiment, expanding Gemma 3 from 1 billion to 27 billion parameters reduced encoding failures from 85% to 23%. However, recall failures increased and reached approximately 40%.
This finding suggests that scaling primarily improves knowledge storage, not knowledge access. As models memorize more facts, a larger portion of their information can become encoded but difficult to retrieve. Consequently, many LLM errors may shift from missing data to failed recall.
“Our findings suggest that memory is closely tied to the conditions under which facts were learned, and that memory declines when queries deviate from the training pattern,” the researchers wrote. In practice, the way a user phrases a question can determine whether an LLM can access an answer stored in its parameters.
The study also found that rare facts were encoded at rates similar to common facts. However, rare or long-tail facts showed a recall gap of more than 25% compared with highly popular facts in frontier models.
Language models also struggled with reverse questions—questions that ask for the subject rather than the object. For example, a model might correctly answer that Oasis played its first show at the Boardwalk Club but fail to answer which band played its first show there.
In some cases, the same model could identify the correct answer when presented with multiple-choice options. This indicates that the information was available for recognition even when the model could not generate it independently.
“While these failures are often interpreted as limitations of memory or bidirectional encoding, our results suggest a different picture: rare facts are often encoded but inaccessible, and opposite facts can be recognized even when they cannot be produced,” the researchers wrote. “This reframes both phenomena as failures of recall rather than ‘knowledge gaps.’”
What LLM knowledge profiling means for developers
The high encoding rates observed in frontier models suggest that developers should rethink how they respond to hallucinations and factual errors.
Do not treat every factual error as a search problem. When an LLM produces an incorrect answer, an enterprise may immediately deploy RAG, expand its vector database, or ingest additional domain documents. Those approaches remain essential for real-time or proprietary information, but the study suggests that many common facts are already encoded in the model. Some failures may be resolved through better prompting or inference rather than external retrieval.
Use inference-time reasoning selectively. Reasoning recovered 40% to 65% of encoded facts that models could not recall directly. However, extended reasoning increases latency and computational costs. A practical architecture could use a fast first-pass response, followed by a higher-compute retry when confidence is low. Teams can also route difficult queries—such as rare entities, unusual wording, and reverse questions—to reasoning-focused models.
Evaluate semantic access, not just answer accuracy. Conventional accuracy benchmarks can hide whether a model lacks a fact or simply cannot retrieve it. Evaluation suites should test the same facts across different contexts, phrasings, orientations, and response formats. This provides a clearer picture of what an LLM knows and how reliably it can access that knowledge.
Restructure queries and support retries. Factual recall is highly dependent on context. Rephrasing a question, providing relevant intermediate context, or asking the model to reason through the problem before answering can improve access to stored information. These techniques should be treated as reliability mechanisms—not merely prompt-engineering tricks.
Limitations of the LLM recall research
The WikiProfile benchmark focuses on encyclopedic facts from Wikipedia. The findings may not apply directly to highly specialized corporate information, private datasets, rapidly changing facts, or unique internal metrics. An LLM may store and recall proprietary enterprise knowledge differently from publicly available encyclopedic information.
Comprehensively profiling a frontier model with the WikiProfile benchmark costs approximately $500. Developers can reduce evaluation costs by removing multiple-choice tests or decreasing the number of answer samples collected for each question.
Teams can access the WikiProfile benchmark on Hugging Face to evaluate their own systems and create custom knowledge profiles. Identifying whether an AI application has an “empty shelf” or a “missing key” is an important first step toward reducing LLM hallucinations and improving factual reliability.
Source: venturebeat.com