Neural Scaling Laws: How Model Size and Data Shape Loss

Three quantities, one curve each

Ten authors at OpenAI, the first of them also at Johns Hopkins, trained decoder-only Transformers from 768 to 1.5 billion non-embedding parameters on 22 million to 23 billion tokens of WebText2, and fitted test loss against three quantities separately: parameter count, dataset size, and compute. Each fit is a power law, and each holds only when the other two are not the binding constraint. The reported exponents are small: roughly 0.076 for parameters and 0.095 for data, meaning a doubling of model size buys about a five percent reduction in loss. [1]

The paper's more striking claim is what does not matter. Depth against width, and most other architectural choices, move the loss very little inside a reasonable range. Scale does the work. The fits are claimed across eight orders of magnitude in compute and six in parameters, though only about two in dataset size. [1]

Atlas interpretation: The dataset range is the weak leg. Two orders of magnitude is thin next to eight in compute, and it is the leg the compute-allocation conclusion leans on hardest, since that conclusion is a claim about how much data a budget should buy. [1]

The advice: buy parameters, not epochs

Minimizing the fitted loss at a fixed compute budget gives an allocation rule. Optimal model size grows as compute to the power 0.73, batch size as compute to the 0.24, and serial training steps as compute to the 0.03, an exponent the authors note is small enough to be consistent with zero. Data requirements follow at roughly compute to the 0.27. The summary sentence is the paper's own: as the budget increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size. [1]

This is paired with a second conclusion that survived everything that followed: convergence is inefficient. The compute-optimal move is to train a very large model and stop well short of the lowest loss it could reach. The paper also marks its own expiry, observing in a section titled "Contradictions and a Conjecture" that two of its curves intersect several orders of magnitude above anything tested, so the laws must break down before that point. [1]

A paper that spent other people's money

Four months later the GPT-3 paper cited it as the reason for its own shape, in a figure caption that reads: based on the analysis in Scaling Laws For Neural Language Models we train much larger models on many fewer tokens than is typical. The same paper credits Kaplan and McCandlish with applying scaling laws to guide its model and data scaling decisions, and follows the paper's guidance on batch size and learning rate too. GPT-3 trained on roughly 300 billion tokens, and the models that followed it copied that number. [5][1][2]

Atlas interpretation: It is rare to be able to trace a training run's budget to a specific citation, and rarer still for a rival lab to name the same paper as the cause of a field-wide error. That is what makes this preprint an event rather than a reference. Its influence is documented in the two directions that matter: a lab saying it sized a model this way because of this paper, and a different lab two years later diagnosing that reasoning as the reason everyone's models were undertrained. [5][2]

The exponent that turned out to be wrong

In March 2022 DeepMind's Chinchilla paper restated this work's rule as: a tenfold increase in budget should grow the model 5.5 times and the training tokens only 1.8 times. Its own three estimation approaches put the model-size exponent at 0.46 to 0.50 rather than 0.73, meaning parameters and tokens should scale together. Chinchilla kept the finding that compute-optimal training stops short of convergence, and disagreed only about the split. [2][1]

Chinchilla offered an explanation: the earlier work used one fixed learning-rate schedule for every run, so intermediate losses along a schedule tuned for 130 billion tokens overstate the loss of a model actually trained on fewer, which biases the fit toward larger models. It gave a second reason in the same passage, that its own analysis mostly used models above 500 million parameters while many of the earlier runs were under 100 million. [2]

Atlas interpretation: That explanation is widely repeated as settled and is not. Two 2024 papers re-ran the comparison and landed elsewhere. One attributes much of the gap to counting non-embedding rather than total parameters, combined with the small scale of the original runs. The other, a NeurIPS spotlight, isolates last-layer computational cost, warmup duration and scale-dependent optimizer tuning, and states directly that careful learning rate decay is not essential for the Chinchilla law to hold, contradicting the premise of Chinchilla's own account. A page that says the learning-rate schedule explains the error is repeating a 2022 hypothesis that later work did not confirm. [4][3][2]