Chain-of-Thought Prompting: Reasoning Without Fine-Tuning

Nothing about the model changes

The technique is to replace the usual in-context examples, which pair a question with its answer, by triples that pair a question with a series of intermediate reasoning steps and then the answer. Eight hand-written exemplars, the authors note without prompt engineering, are enough. The model then produces its own intermediate steps before answering. [1]

Weights, checkpoint and decoding are untouched. The paper states it in a sentence that is unusual to find in a machine learning paper: no language models were finetuned in the process of writing this paper. One checkpoint performs many reasoning tasks with no gradient updates and no labeled training set, which is the property the authors argue for rather than the benchmark deltas. [1]

The paper is careful about what the visible steps are. It describes them as an interpretable window into the model's behavior while noting that fully characterizing the computation supporting an answer remains open, says the work does not establish whether the network is reasoning, and states there is no guarantee of correct reasoning paths. [1]

The numbers, and which version they came from

On the GSM8K grade-school maths benchmark with eight exemplars and greedy decoding, PaLM at 540 billion parameters goes from 17.9 percent with standard prompting to 56.9 with intermediate steps, against a previous best of 55 from a fine-tuned system. GPT-3 at 175B goes from 15.6 to 46.9, and Codex from 19.7 to 63.1. Nine further benchmarks across arithmetic and commonsense, plus two symbolic tasks the paper calls toy, mostly move the same way. [1]

Atlas interpretation: Those figures are not what was published on this date. The January 2022 version predates PaLM and makes its case on a 137 billion parameter LaMDA model instead, where the same benchmark moves from 6.3 to 14.8 percent. The results everyone quotes arrived across six revisions through January 2023, which also added two authors and the hyphens to the title. The event is the arrival of the technique; the paper cited today is a materially different document from the one posted on this date. [1][5]

The claim about scale

The paper's own framing is that this is an emergent ability of model scale: it does not positively affect performance for small models and only produces gains at around 100 billion parameters, where smaller models were found to produce fluent but illogical chains. The claim is measured across the GPT, LaMDA and PaLM size ladders. It is not merely that the technique fails below the threshold. It reverses: on GSM8K, LaMDA at 420M drops from 2.6 to 0.4 and GPT at 1.3B from 2.4 to 0.5 when asked to show its work. [1]

The reversal is not confined to small models either. LaMDA at 137B falls from 25.5 to 20.6 on AQuA, and GPT-3 at 175B from 79.5 to 73.5 on CommonsenseQA. The paper also treats the scale requirement as a practical problem rather than a triumph, noting that emergence only at large scale makes the technique costly to serve. [1]

What has been contested since

Three separate lines of later work push back, on different points. On emergence itself, a 2023 paper argues that apparent emergent abilities are produced by discontinuous metrics, and that linear or continuous metrics show smooth and predictable improvement instead, concluding that emergence may reflect the researcher's choice of measure rather than a change in model behavior. Its target is emergent abilities generally rather than a re-measurement of these specific curves. [2]

On whether the stated steps describe the actual computation, a 2023 paper added biasing features to model inputs and found the models produced explanations that rationalized the biased answer without ever mentioning the bias, with accuracy dropping by up to 36 percent across 13 tasks. A related study found that faithfulness varies by task and that larger models produced less faithful reasoning on most tasks tested. Anthropic extended the question to trained reasoning models in 2025 and reported hint-reveal rates often below 20 percent. [3][6]

On where the technique helps at all, a 2024 meta-analysis of over 100 papers plus evaluation across 20 datasets and 14 models reports that the gains are concentrated in mathematics and symbolic reasoning, with much smaller gains elsewhere, and that on MMLU direct answering matches the technique unless the question or response contains an equals sign. [4]

Atlas interpretation: None of these retract the original result, which is a measurement that held: the outputs get better on these benchmarks at these scales. What they question is the story attached to it, that a model given room to think is showing you its thinking. That distinction became load-bearing once the visible chain moved from the prompt into training, because a reasoning trace that can be read is only a safety property if it reflects the computation. [3][6][4]

From the prompt into the training run

The artifact this paper elicited with eight hand-written examples is the one that later models are trained to produce. OpenAI describes the o1 series as trained with large-scale reinforcement learning to reason using chain of thought, which is this paper's vocabulary applied to a training objective rather than a prompt. [7]