Neural Scaling Laws: Kaplan, Chinchilla, and Compute-Optimal Training

The papers collected in this part span two otherwise disjoint literatures: empirical scaling (Kaplan, Chinchilla, Hestness, Bahri) and information-theoretic dynamics (Tishby, Shwartz-Ziv, Saxe), connected by the question of whether deep networks' generalization curves admit a compact functional description. A third, more recent strand (Wei, Schaeffer, Power) asks whether the smooth power laws tell the whole story or whether discontinuous behaviours (emergence, grokking) live on top of them. A fourth strand reframes the bottleneck from quantity to quality. (Sorscher et al. 2022) (NeurIPS 2022 Outstanding) argue that the Hestness-Kaplan power law $L \propto D^{-\alpha_D}$ is not a property of the learning problem but an artefact of i.i.d. random subsetting. When training examples are ranked by difficulty and the easiest fraction is discarded, test error decays exponentially in the retained fraction $f$ in the over-parameterised regime rather than following the $\propto 1/\sqrt{D}$ rate that classical uniform-convergence theory predicts. The intuition: random subsets always retain both easy and redundant samples; a pruned subset concentrated on hard, near-decision-boundary examples provides more gradient signal per sample, so the model learns the same boundary geometry from a smaller dataset.

Ten difficulty metrics are benchmarked, including EL2N (early-training $\ell_2$ error norm), forgetting scores, memorization, ensemble uncertainty, and margin. Among supervised metrics, EL2N performs best on CIFAR-10: a ResNet-18 trained on the top-50% hardest examples (by EL2N) matches full-dataset accuracy. On ImageNet, training on 80% of examples selected by memorization score matches the 100% baseline; most supervised metrics degrade sharply below 60% retention. A new self-supervised metric is proposed: cosine distance to the nearest k-means cluster centroid in SWaV embeddings, which matches supervised baselines without requiring labels and enables pruning before annotation.

The exponential advantage is conditional. It holds only in the over-parameterised regime (model large enough to fit the retained set) and only when the pruning metric aligns well with the true difficulty ordering (measured by angular alignment $\theta$ between the probe gradient and the teacher gradient). When the probe misaligns by more than roughly 10 degrees, a minimum retained fraction $f_{\min}(\theta)$ exists below which further pruning does not help; the exponential phase ends and the standard power law resumes. The paper therefore does not supersede scaling laws for the frontier regime (where models are maximally over-parameterised and the full pre-training corpus is already curated); it applies most directly to fine-tuning and mid-scale training where aggressive dataset curation is practical.

The shared methodology is to fit a parametric family to train-and-test curves and to ask what the fit exponents mean. In Kaplan's setup, the test cross-entropy loss is modelled as three single-variable power laws plus an irreducible offset; in Chinchilla it is a single joint $N$-$D$ surface; in Hestness it is a dataset-size-only power law; in Bahri the exponent is derived from kernel spectra and manifold dimension. The SoTA landscape across papers is summarised below by the exponents appearing in the three canonical laws

$$\begin{equation} L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}. \end{equation}$$

The IB and emergence rows do not fit the $\alpha_N$/$\alpha_D$/$\alpha_C$ schema because those papers report different observables (mutual information trajectories, task-by-task accuracy jumps). They appear later in this chapter under their own headings.

References