Think Longer or Check Again: How Language Models Spend Extra Time

Picture a student in a math exam. They finish a hard problem with ten minutes to spare. They could spend that time thinking harder about the next problem. They could go back and check the answer they just wrote. Or they could solve the same problem again in a new way and see if the two answers match. Each choice costs the same ten minutes. So which one earns the most marks?

Language models face the very same choice. Training is over but we can still give a model more compute when it answers a question. People call this test-time compute. In this post we will walk through the main ways to spend it. At the end we will share a surprise from our own work about how these tricks get scored.

Figure 1. Four common ways to spend extra compute on one question. The model can think longer or answer many times and vote. It can ask a trained grader to pick the best answer. Or it can check its own first answer and decide whether to keep it.

Trick 1: Show your work

The simplest trick is to let the model write out its steps. Wei and colleagues (2022) found that asking a model to think step by step makes it much better at math and logic. They called this chain-of-thought. Each extra step costs a little more compute. In return the model gets more room to work the problem out.

Newer reasoning models take this idea much further. DeepSeek-R1 (DeepSeek-AI 2025) was trained with reinforcement learning to think for thousands of tokens before it answers. You can often watch it try an idea and then catch its own slip halfway through the thought.

Trick 2: Ask many times and vote

Now try a small question. Three pens cost 2 dollars. How much do 12 pens cost? Ask a model five times and you might get 8 and then 8 and then 6 and then 8 and then 9. Three of the five answers say 8. So we go with 8.

This trick is called self-consistency and it comes from Wang and colleagues (2022). It works because wrong answers tend to scatter in many directions while right answers tend to land in the same spot. The catch is the bill. Five answers cost about five times as much as one.

Trick 3: Hire a grader

Instead of a vote we can train a second model to grade each answer and keep the best one. Cobbe and colleagues (2021) did this for school math problems in their GSM8K dataset. Later Lightman and colleagues (2023) showed that it works even better to grade every step and not just the final answer. A grader like this is called a process reward model.

A good grader is a strong tool. But it needs a lot of labeled examples to train and it means running a whole second model.

Trick 4: Let the model check itself

The cheapest idea of all is to ask the model to read its own answer and fix any mistakes. Early methods like Self-Refine (Madaan and colleagues 2023) and Reflexion (Shinn and colleagues 2023) reported nice gains this way.

Then people took a closer look. Huang and colleagues (2023) found that a model with no outside help often fails to fix its own reasoning. Sometimes it even turns a right answer into a wrong one. Studies on planning tasks told a similar story (Valmeekam and colleagues 2023 and Stechly and colleagues 2024). Models will even drop a correct answer when a user simply asks “Are you sure?” (Laban and colleagues 2023).

Self-checking works much better when real feedback comes from outside the model. CRITIC (Gou and colleagues 2023) lets the model use tools such as a search engine or a code runner to test its answer. SCoRe (Kumar and colleagues 2024) trains the model to correct itself with reinforcement learning. If you want the full map of when self-correction works and when it fails then the survey by Kamoi and colleagues (2024) is a great place to start.

So which trick wins?

Snell and colleagues (2024) asked this question head on. Their answer was that it depends on how hard the question is. On easier questions it pays to let the model revise its answer. On harder questions it pays to try many answers and let a grader search among them. When the budget is split in this smart way a small model can even beat a much larger one on questions it has a fair chance to solve.

How long the model thinks matters too. The s1 paper (Muennighoff and colleagues 2025) showed a very simple knob. You can force a model to keep thinking or force it to stop early. Accuracy rises and falls in a clear way as you turn that knob. But more thinking is not always better. Chen and colleagues (2024) found that reasoning models often overthink easy questions. They can write pages of thought for something as small as 2 plus 3.

A surprise from our own work

In our ongoing work we asked a narrow question. Does a model get better when it checks an answer it has already finished? We expected a clear yes or a clear no. Instead we found that the answer depended on a step that almost nobody talks about.

That step is the reader. A model writes free text and we need a rule that pulls the final answer out of that text before we can score it. Most studies use a simple pattern rule for this job. The trouble is that the two sides of the test look very different. A first answer is often short and clean. A check is often long and chatty.

Here is a small example. The question asks where you put a dirty plate and the choices are A for table and B for sink and C for shelf. The first answer just says “B”. The check says “Let me look again. A table is where we eat. The sink is where we wash. Final answer: B.”

Now picture a reader that grabs the first choice letter it sees. On the first answer it finds B and marks it right. On the check it finds the A at the start of “A table” and marks it wrong. The model never changed its mind. Yet the score now says that checking broke a right answer.

Figure 2. Both texts give the same final answer B. A reader that takes the first choice letter it sees scores the short first answer as right. It scores the chatty check as wrong because it stops at the word “A” in “A table”.

This is not just a made up worry. In our runs different common readers pushed the measured effect in different directions. In some settings they showed a gain or a loss of about 25 points where a careful answer key found almost no change at all.

So we built a fairer key. Outside judge models read each response and copy out only the final answer it settles on. They never see the correct answer. They also do not know which side of the test a text came from. With this key the story became much calmer.

On finished answers a self-check changed very little on average. It still hurt in a few settings so “harmless” is not quite the right word either. The real action was on answers that had been cut off by the token limit. There a check helped a lot and accuracy rose from 44% to 62%. So much of what looks like self-correction is really the model getting a second chance to finish its thought. We also saw that the checker matters. A stronger model as the checker helped while a weaker checker hurt.

We are not alone in this worry. Biderman and colleagues (2024) showed how small details in an evaluation setup can change the results. The Calibration Floor (Chen and colleagues 2026) found that simply fixing the format of an answer can look like self-correction.

What to take away

Let us go back to our student in the exam. The best use of ten spare minutes depends on the problem in front of them. A problem they never finished clearly needs more time. A hard problem may need a fresh try. A finished easy problem may need nothing at all. Language models work the same way and there is no single best trick for every question.

Help from outside tends to beat looking inward. A tool or a test or a trained grader or a stronger checker will usually do more than a model rereading its own work.

Keep an eye on the token limit. If the first try was cut off then a second pass can help a lot. If the first try finished then the gain is often small.

Finally check your reader before you trust a result. Read some outputs by hand from both sides of the comparison. If the reader misses answers more often on one side then the effect you measured may not be real.

If you work on test-time compute or self-correction or LLM evaluation then I would love to hear from you. You can find me through the links on the left.

References

1. Biderman S. and others (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv 2405.14782.
2. Chen M. and others (2026). The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale. arXiv 2608.04355.
3. Chen X. and others (2024). Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv 2412.21187.
4. Cobbe K. and others (2021). Training Verifiers to Solve Math Word Problems. arXiv 2110.14168.
5. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948.
6. Gou Z. and others (2023). CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. arXiv 2305.11738.
7. Huang J. and others (2023). Large Language Models Cannot Self-Correct Reasoning Yet. arXiv 2310.01798.
8. Kamoi R. and others (2024). When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. arXiv 2406.01297.
9. Kumar A. and others (2024). Training Language Models to Self-Correct via Reinforcement Learning. arXiv 2409.12917.
10. Laban P. and others (2023). Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment. arXiv 2311.08596.
11. Lightman H. and others (2023). Let’s Verify Step by Step. arXiv 2305.20050.
12. Madaan A. and others (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv 2303.17651.
13. Muennighoff N. and others (2025). s1: Simple Test-Time Scaling. arXiv 2501.19393.
14. Shinn N. and others (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv 2303.11366.
15. Snell C. and others (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arXiv 2408.03314.
16. Stechly K. and others (2024). On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks. arXiv 2402.08115.
17. Valmeekam K. and others (2023). Can Large Language Models Really Improve by Self-Critiquing Their Own Plans? arXiv 2310.08118.
18. Wang X. and others (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv 2203.11171.
19. Wei J. and others (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv 2201.11903.