Chinchilla Scaling Laws: Tokens and Model Size Together

Undertrained, not undersized

Twenty-two authors at DeepMind trained more than 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and concluded that the large models of the day were significantly undertrained. The rule they derived is that for compute-optimal training, model size and training tokens should scale equally: double the parameters and you should double the tokens. [1]

Three independent approaches back it. The first takes minima over training curves at fixed model sizes, the second builds isoFLOP profiles by varying model size at nine fixed compute budgets, and the third fits a parametric loss function in parameters and tokens. They give exponents on compute for optimal model size of 0.50, 0.49 and 0.46, against 0.73 in the earlier work, with the token exponent taking up the remainder. [1]

Atlas interpretation: The ratio everyone quotes, roughly 20 tokens per parameter, never appears in the paper as a phrase. It is read off how Chinchilla itself was trained and off the paper's projection table, whose rows sit close to it: 400 million parameters to 8.0 billion tokens, 10 billion to 205 billion, 175 billion to 3.7 trillion. By the time it was re-examined in 2024 it had long since become, in the replication authors' own words, a rule of thumb. [1][3]

The demonstration: 70B against 280B

Rather than leave it as a curve fit, DeepMind spent the compute. Chinchilla is 70 billion parameters trained on 1.4 trillion tokens, against Gopher's 280 billion parameters on 300 billion tokens, at the same training budget: four times smaller, four times the data. It reports 67.5 percent average accuracy on MMLU against Gopher's 60.0, and a 10.7 point gain on BIG-bench, while also outperforming GPT-3 and Jurassic-1. [1]

There is also a direct test against the older rule. At a budget of 10 to the 21 FLOPs the two laws predict different optimal sizes, 4.68 billion parameters against 2.86 billion. DeepMind trained a model near each figure, 4.74 billion and 2.80 billion, and reports that its own prediction won. Its blog put the conclusion about its own previous flagship plainly: for the compute used to train Gopher, a four times smaller model trained on four times more data would have been preferable. [1][2]

Why the earlier law was off, and how settled that account is

The paper characterises the 2020 scaling laws as recommending that a tenfold budget increase grow the model 5.5 times and the tokens only 1.8 times, and gives two reasons for the divergence. The first is that the earlier work used one fixed learning-rate schedule for every run, so losses read off partway through a schedule tuned for 130 billion tokens overstate what a shorter run would achieve, biasing the result toward larger models. The second is scale: its own analysis mostly used models above 500 million parameters, where many of the earlier runs were under 100 million. [1]

Atlas interpretation: The learning-rate account has hardened into folklore and deserves less confidence than it gets. Two 2024 papers re-examined the discrepancy and reached different conclusions, one attributing much of it to non-embedding parameter counting plus small scale, the other isolating last-layer cost, warmup duration and optimizer tuning, and stating directly that careful learning rate decay is not essential for this paper's own law to hold. What is not in dispute is the conclusion itself, which both later papers reproduce. [5][6]

It is also worth being clear about what was not overturned. The paper explicitly agrees with the earlier finding that large models should not be trained to their lowest achievable loss, and it borrows the earlier work's compute approximation wholesale. This is a correction of one exponent inside a shared framework, not a rejection of it. [1]

What it changed downstream

Meta's LLaMA paper cites this work as showing that the best performance at a given budget comes from smaller models trained on more data, and then departs from it deliberately: this paper optimizes training compute and disregards the inference budget, which is what dominates when a model is actually served. LLaMA reports that a 7B model kept improving past a trillion tokens, well beyond the compute-optimal point, and that its 13B model outperformed the 175B GPT-3 on most benchmarks. [4][1]

Atlas interpretation: So the lineage runs in two steps and is usually collapsed into one. This paper made small-and-long-trained the compute-optimal choice; LLaMA then trained past even that point for a reason this paper does not address, because a model you serve to millions of people is paid for twice and only one of those payments is in the training run. [4]

The replication, scoped narrowly

In April 2024, researchers at Epoch AI attempted to replicate the third of the three estimation approaches by extracting data from the paper's own plots. They report that the published estimates are inconsistent with the other two approaches, fit the extracted data poorly, and carry confidence intervals so narrow they would require over 600,000 experiments where fewer than 500 were likely run. The replication paper states that the original authors confirmed the cause as averaging rather than summing Huber loss values during minimization. [3]

Atlas interpretation: This is a correction to one of three procedures, and the corrected version agrees with the paper's headline better than the original did: the refit moves the third approach toward equal scaling and implies roughly 20 tokens per parameter, consistent with how Chinchilla was actually trained. Anyone citing it as a refutation of Chinchilla scaling has the direction backwards. [3]