Asking a Model to Think Longer Mostly Didn't Make It Think Longer

A paper Obada Kraishan of Texas Tech University posted to arXiv on October 7 asks a fair question about reasoning models. If the thinking phase works like slow, deliberate human thought, then giving a model more of it should weaken the decision biases that fast judgment produces. Across seven models, it didn't. The part I keep thinking about is Table II, which shows that the thinking budget the author requested and the thinking the models actually did were often unrelated.

That matters for anyone who sets these budgets. It is easy to treat "reasoning budget" as a dial, in a paper or in a production config. In this experiment, turning it often changed little or nothing about how long the model thought.

The setup

The battery is 30 decision vignettes, five each for six biases: anchoring, framing, loss aversion, escalation of commitment, availability, and confirmation. They come from an existing cognitive bias benchmark by Malberg and colleagues. Each vignette has a control version and a version with the bias manipulation, both answered on a fixed numbered scale, so a bias shows up as a shift in the chosen option between the two.

The panel is seven models from four families, each reasoning model paired with a non-reasoning sibling: Claude Haiku 4.5 with and without thinking, GPT-5.1 with reasoning on and off, DeepSeek-R1 against DeepSeek-V3, and Qwen3 thinking against Qwen3 Instruct. The reasoning models got requested thinking ceilings of 0, 1,024, 4,096, and 8,192 tokens. Every cell was sampled ten times at temperature 1.0, for 12,350 calls, all routed through OpenRouter so one parameter could set the budget for every provider.

The author also logged the reasoning tokens each response reported spending and used that count as the dose, instead of the number he asked for.

What the dial did

Table II has one row per reasoning model. Claude Haiku 4.5 honored the ceiling. Its mean realized thinking went from 0 at budget 0 to 437, 575, and 620 tokens at the three positive settings. No call went over its ceiling, and the model never came close to the larger ones. Asking for 8,192 tokens bought about 620.

GPT-5.1 used 116 tokens at the 1,024 setting and 118 at both 4,096 and 8,192. Through the gateway, OpenAI reasoning models take an effort level, and OpenRouter's docs say a token budget sent to those models is used to pick the effort. Three requested budgets produced one realized level of thinking.

For DeepSeek-R1 and Qwen3 thinking, the ceiling was not enforced. Requests to turn thinking off were rejected or ignored, and at budget 0 the models thought anyway: 1,188 tokens on average for R1 and 1,074 for Qwen3. At the 1,024 ceiling, 79% of R1 calls and 52% of Qwen3 calls went over it. Consumption rose a little from 1,024 to 4,096 and was flat after that.

So of four reasoning models, one had a budget that worked as a control, and even that one stopped well short of what was offered. A plot of bias against the requested ceilings would have shown three models whose x-axis barely tracked any change in thinking.

Why it changes how to read the headline

Because the paper measures the realized dose, the headline result holds up, within limits the author states plainly. Paired by item, reasoning models were not less biased than their siblings in any family. The pooled difference leaned slightly toward more bias for the reasoning models (+0.031 on a bias magnitude scale that tops out at 1, p = .157), which the paper declines to call a real difference. Regressing bias on realized tokens, no model's bias magnitude fell significantly as it thought more.

The limits come from Tables II and IV. Claude is the one model whose thinking the ceiling controlled, and its realized range ran roughly 0 to 800 tokens per cell. GPT-5.1 spanned 0 to about 225. R1 and Qwen3 had wider ranges, up to about 3,400 tokens, but the models set those ranges themselves, and their zero point is a different set of weights. The paper says it cannot speak to deliberation beyond roughly 3,400 tokens per cell. Going by the requested numbers alone, the study would have appeared to test up to 8,192.

What the thinking did instead

The bias results are odd in their own right. Anchoring was the one bias that ran in the human direction, and it was strong: a mean of +0.163 with an effect size of d = 1.89, present in every model. Four of the other five leaned the opposite way from people. Framing reversed reliably (mean -0.336). Escalation of commitment, confirmation, and loss aversion leaned backwards in all seven models, though with five items each the paper calls those directional. Loss aversion had the largest mean reversal at -0.586. Availability was small and inconsistent, so the paper classes it as absent.

The author offers one reading, flagged as a hypothesis: on these choice pairs, the loss aversion reversal is what a risk-neutral expected value maximizer would produce, so models trained toward "rational" answers may have overshot. The paper's practical point holds either way. A review built from the human bias catalog would ask whether a model is framing-sensitive like a person, get "no," and miss that it moves just as far the other way.

Where more realized thinking did move anything, it pushed the signed score further from the human direction, for Claude and GPT-5.1. Anchoring kept its pull at the highest deliberation the paper observed.

The one intervention in the study that reduced anchoring cost a single prompt line. Adding one instruction to restate every number in the prompt before answering lowered anchoring on all five anchoring items and in five of the seven models. The effect is small (a pooled change of -0.015) and with five items it misses significance at p = .057, so the paper reports it as a cheap mitigation worth testing.

What I'd take from it

The practical lesson I'd carry into my own projects is about logging. If you call reasoning models through a gateway or an SDK and set a budget, read the reasoning token count in the response usage field and check it against what you asked for. The paper recommends reporting realized reasoning tokens per condition as standard practice, and Table II is a good argument for doing that in evaluation code too. A comparison of "low" and "high" budgets means little if both settings produce about 117 tokens.

Some caveats. All calls went through one gateway in one collection window, so Table II describes these models as served there at that time, and going straight to each provider's own API could behave differently. Five items per bias is thin, which is why most of the reversals are labeled directional. The panel is Haiku 4.5, GPT-5.1, DeepSeek-R1, Qwen3, and their siblings, so current flagship models were not tested. And the study measures behavior on decision vignettes, so it does not test math or code, where the paper notes longer thinking demonstrably helps.

The frozen battery, collection code, scored data, and per-call token logs are on GitHub, so Table II can be rerun on other models and other gateways.