Search Models Cite Pages They Never Opened
Search-augmented models often answer from snippets without opening a single page. A new Imperial College study finds that is where most of their stale answers and contradicted citations come from.
In a prompt dated 8 August 2026, researchers asked OpenAI's GPT-5.6 Luna, running on its own web search, whether "Alex Chriss is the CEO of PayPal" was accurate. The logged trace is short: "One search (site:paypal.com Alex Chriss CEO PayPal leadership); 18 results returned; 0 pages opened." The model said yes and attributed the claim to paypal.com. Enrique Lores had been PayPal's CEO since 1 March 2026.
That example opens Retrieved but Not Read, a paper by Owen Walters, Lihu Chen and Francesca Toni of Imperial College London, posted to arXiv on October 15. Its claim is narrow. When search-augmented models give a stale answer even though the right evidence sat one click away, the error usually comes from an episode where the model never clicked.
The setup
The authors built 225 facts whose values changed after 16 February 2026, the latest training cutoff among the models tested. They cover six domains: FDA drug labels, finance, sports transfers, deaths, SEC officer changes and chemical reclassifications. Each fact has an old value and a current one, so a stale answer can be graded separately from a plain wrong one, and a citation can be checked against what the model actually asserted.
There are two settings. In the deployed run, Luna uses OpenAI's native web search through the Responses API. In the frozen run, Luna and six open-weight configurations (GLM-5.3-Flash, gpt-oss-120b, gpt-oss-20b, gemma-4-31b with and without reasoning, and qwen3-8b) all get the same ten-result menu per fact, built from Brave Search, with the ground-truth page always reachable. Each model gets the same three tools: search, open page, find in page.
How often nobody opens the page
Search helps every model. Claim-check accuracy rises by 27 to 42 points once it is switched on, and accuracy on open questions lands between 85.8% and 95.4%. Most answers are right.
Reading depth is where the models split. Given identical menus, the share of episodes answered without opening a single page runs from 2.6% for gpt-oss-120b to 76.0% for qwen3-8b. Luna on its own live search did it 53.6% of the time. Across 1,747 deployed episodes it searched every time, averaging 1.58 searches and 23 results per search, and opened at least one page in only 46.4% of them.
The same Luna model opened no page in just 9.9% of frozen-menu episodes. The paper doesn't isolate why, and the two setups differ in result counts and tooling, so I wouldn't treat the gap as a grade for OpenAI's search product. It does suggest that the loop wrapped around a model changes how much it reads.
Where the errors live
Pooled across all seven frozen configurations, snippet-only answers were stale 6.4% of the time, against 1.3% when the model cited a page it had opened. Unfaithful citations, where the cited page contradicts the answer, ran 6.4% against 2.3%. Among the open models on sports facts, snippet-only answers were stale 20.2% of the time, compared with 2.1% when an opened page was cited.
The authors also checked what the snippets contained. For sports facts, 72% of menus had a snippet carrying the current club and 70% had one carrying the old club, so a model answering from snippets often sees both. FDA facts sit at the other end: only 8% of menus had the current value in any snippet and none had the old one, because the answer lives in long label sections.
On sourced open questions, 27 of 33 citations attached to a stale answer were unfaithful (81.8%). For current answers the figure was 6 of 694 (0.9%).
Handing over the page
When the authors pasted the player's current Wikipedia page straight into the prompt, gemma-4-31b went from endorsing the stale claim 14 times out of 50 to zero, and gpt-oss-20b went from 11 of 47 to 1 of 50. In all 14 of gemma's search-menu failures, it had opened no page.
They also tested a cheap change to the agent loop. After every opened page, the full original prompt is repeated so the question is the last thing the model reads. That recovered 41 of 157 failed fact-sheet checks (26.1%), ranging from 4.5% to 57.1% across the six configurations re-run.
Some models still fail with the page in the prompt. Using the logit lens and a Jacobian lens on transfer facts, the authors found that in those cases "the retrieved value is promoted, but the stale value is not suppressed enough to be displaced." In qwen3-8b and gpt-oss-20b the new club leads in the upper layers and the old club overtakes it near the output. In gemma-4-31b the old club rises first and stays ahead. The paper says this describes the failure without proving a mechanism, since there is no activation patching.
Caveats
Luna is "the lightest of the three GPT-5.6 tiers, chosen for cost," and it is the only proprietary model in the study. Most prompts required a fixed answer line, which in a pre-registered closed-book check lowered Luna's false endorsements by 17.5 points, so the authors treat the deployed failure rates as likely lower bounds. Sports drives the worst numbers, and the SEC and chemical domains are small (13 and 8 facts).
The ethics section flags a risk I hadn't considered. If a model repeats whatever a snippet says, then misleading snippet text can end up in an answer without anyone loading the page it came from. If I were running a search agent, I would start logging whether each cited page was actually opened and try the re-ask trick after every fetch. The code and data are public.
