Scaling laws and Chinchilla: why small models keep getting stronger

Why do AI companies pour billions into GPUs and data? Because there's a pattern stable enough to plan around: scaling laws. Increase model size, data and compute, and the model's loss falls along a predictable curve. Two studies shaped how the whole industry trains LLMs: OpenAI's (Kaplan et al., 2020) and DeepMind's Chinchilla (Hoffmann et al., 2022). They reached different conclusions, and the difference is valuable.

Kaplan 2020: loss falls as a power law

• Language model loss falls as a power law in parameters, data and compute, across many orders of magnitude.
• Architectural details such as depth or width matter much less than overall scale.
• Larger models are more sample-efficient.
• The practical takeaway at the time: with a fixed compute budget, train very large models and stop well before convergence.

Chinchilla 2022: balance parameters and data

DeepMind trained over 400 models of various sizes and found that the large models of the day were under-trained on data. To be compute-optimal, parameters and training tokens should scale in equal proportion.

With the same compute budget as Gopher, Chinchilla was 4 times smaller but trained on more data, and it beat Gopher, GPT-3 (175B) and Megatron-Turing NLG (530B) on many benchmarks. The rule of thumb commonly drawn from it is about 20 tokens per parameter.

After Chinchilla: optimising for serving

"Compute-optimal" only optimises training cost. But a model is called billions of times, so inference cost dominates. That's why many later open models deliberately train small models on far more tokens than the Chinchilla ratio: spend extra compute in training to get a small model that's cheap and fast to serve.

What it means for users and product builders

• Modern small models aren't as "weak" as they used to be: trained on more data, they're good enough for classification, extraction and summarisation.
• High-quality data is a scarce resource: hence the investment in synthetic data and data filtering.
• Progress comes on several axes: beyond parameters and data, inference-time compute (thinking, reasoning) is a new scaling axis.

Risks of misreading it

• Lower loss ≠ better at everything: scaling laws predict average loss, not exactly when a specific skill will appear.
• Extrapolating too far: the coefficients were measured within a range; extending the line many orders of magnitude further is a bet.
• "Just pick the biggest model": for many tasks, a well-trained small model gives similar quality at much lower cost.

How to apply it

• Dev: Try the smallest model first against your eval set; upgrade only when you can measure a difference.
• Tech Lead: Compute total cost at real call volumes; inference cost usually dwarfs everything else.
• PM: Don't judge models by parameter count; ask about quality on your task and price per million tokens.