The Neural Tangent Kernel and Infinite-Width Networks

Part III's papers sit at the intersection of three older traditions. The kernel-methods lineage (Schölkopf-Smola, Rasmussen-Williams) supplies the mathematical apparatus (RKHS, positive-definite kernels, kernel ridge regression) that the NTK slots into. The Bayesian-neural-network lineage (MacKay 1992, Neal 1996, Graves 2011, Blundell et al. 2015) supplies the NNGP prior and the variational-inference toolkit that extends to finite-width approximations. The statistical-physics-of-learning lineage (Sompolinsky-Crisanti-Sommers 1988, Seung-Sompolinsky-Tishby 1992, Saad-Solla 1995) supplies the mean-field/replica toolkit that underpins (Mei et al. 2018) and the effective theory of (Roberts et al. 2022).

Within Part III's own papers, the intellectual dependency graph runs as follows. (Neal 1996) sits at the root; it establishes the infinite-width-as-GP principle for single-layer networks. (Lee et al. 2018) and (Matthews et al. 2018) extend this to deep networks, giving the NNGP recursion (eq:nngp-recursion). (Jacot et al. 2018) then introduces the tangent-kernel counterpart that governs dynamics rather than priors, and (Lee et al. 2019) tightens the convergence to a uniform-linearization theorem. (Arora et al. 2019) operationalizes all of this by computing the NTK exactly at CIFAR-10 scale. The critique then emerges in three papers: (Chizat et al. 2019) diagnoses the NTK regime as lazy via the $\alpha$-scaling and shows that feature-learning (rich) regimes exist; (Mei et al. 2018) gives the mean-field PDE as an explicit feature-learning limit; (Yang and Hu 2021) unifies these under the abc-parameterization and identifies $\mu$P as the canonical rich-regime limit. Finally, (Hanin and Nica 2019) and (Roberts et al. 2022) quantify finite-width corrections as a $1/n$-expansion, and (Geiger et al. 2020) and (Woodworth et al. 2020) provide empirical and mathematical evidence that real networks are not purely in the NTK regime.

Part III's place in the statistical-DL story

Part III sits in a specific position relative to the other parts of the book. Part I established classical capacity-based generalization theory and its failure on over-parameterized networks. Part II introduced PAC-Bayes as a more refined framework that handles over-parameterization via posterior-averaging. Part III introduces the NTK as a third distinct path: not a generalization bound but a generalization model in which training dynamics are explicit and closed-form.

The three approaches are complementary. Classical bounds (Part I) are architecture-agnostic but loose; PAC-Bayes bounds (Part II) are architecture-aware and tight with good priors; NTK analysis (Part III) is architecture-specific and gives exact predictions for wide-network training. A complete theory of deep-network generalization would combine these: PAC-Bayes analysis with NTK-GP priors is a natural synthesis that has been pursued (Neyshabur et al. 2018) with some success.

Parts IV and V extend the narrative. Part IV examines benign overfitting and double descent, phenomena that the NTK can describe in its linearized form but that require finite-width corrections to fully explain. Part V examines scaling laws, for which the NTK provides a mathematical template (eq:kernel-scaling-law) but not a complete mechanism.

Collectively the five parts frame a research program: statistical-learning-theoretic explanation of deep learning proceeds by (i) ruling out classical bounds as sufficient (Parts I, II), (ii) establishing the kernel-regression baseline (Part III), (iii) characterizing the gap between this baseline and practice (Parts III, IV, V). No single part delivers a complete theory; together they outline the shape one would take.

Intellectual lineages beyond Part III

A useful orienting frame is to place Part III's papers in the broader intellectual genealogy of the theory of neural networks.

Kernel methods (Aronszajn 1950; (Schölkopf and Smola 2002; Rasmussen and Williams 2006)). The RKHS framework that the NTK slots into was developed in mid-twentieth-century analysis for reasons having nothing to do with neural networks. The NTK's contribution is to identify a specific family of kernels (those arising from neural-architecture parameter-gradients) that inherit the architectural inductive bias and are computationally tractable.

Statistical mechanics of learning (Sompolinsky-Crisanti-Sommers 1988; Seung-Sompolinsky-Tishby 1992; Saad-Solla 1995). The physics tradition of analyzing learning as a statistical-mechanical system, with temperature corresponding to training noise and magnetization corresponding to generalization, underlies the mean-field analysis of (Mei et al. 2018) and the effective-theory program of (Roberts et al. 2022). The Wasserstein-gradient-flow reformulation of (eq:mean-field-pde) is more recent and owes to optimal-transport theory (Ambrosio et al. 2005).

Bayesian neural networks (MacKay 1992; Neal 1996; Graves 2011). The NNGP sits at the infinite-width end of this lineage. Finite-width Bayesian neural networks remain an active research area with variational and Monte Carlo methods; the NNGP provides an analytic benchmark against which finite-width approximations can be measured.

High-dimensional probability and random-matrix theory (Wigner 1955; Marchenko-Pastur 1967; recent kernel-random-matrix literature). The analysis of kernel Gram matrices, their spectra, and their double-descent behavior leans on random-matrix theory. The NTK Gram matrix at a random training set has spectral properties determined by random-matrix ensembles, which connects to the benign-overfitting / double-descent analyses of Part IV.

Tangent methods in optimization (Amari 1998 natural gradient; Martens-Grosse 2015 K-FAC). The NTK is mathematically a specific empirical-Fisher-like object; natural-gradient methods that precondition by the Fisher information are close relatives. The theoretical insight of the NTK is that the linear-ODE structure emerges from the infinite-width limit, not the preconditioning itself.

Gradient-flow analysis of neural networks (Du-Lee-Li-Wang-Zhai 2019; Chizat-Bach 2019). A parallel line of work analyzes gradient-flow convergence on specific neural architectures (two-layer ReLU, deep linear networks) without necessarily taking width to infinity. The NTK provides the infinite-width limit point of this line; the finite-width analyses give convergence rates for specific architectures that the NTK subsumes asymptotically.

Cross-references into other parts

• Part I (classical foundations): VC/Rademacher bounds are replaced by RKHS generalization bounds in the NTK regime; the spectral decay of $\Theta_\infty(X,X)$ plays the role of capacity.
• Part II (PAC-Bayes): the NTK GP posterior is a natural PAC-Bayes prior, and (Lee et al. 2019)'s ensemble interpretation enables direct PAC-Bayes bounds on NTK-regime networks.
• Part IV (benign overfitting and double descent): the NTK gives a linearized test-bed in which benign overfitting and double descent can be computed exactly; the condition number of $\Theta_\infty(X,X)$ at the interpolation threshold is the NTK's double-descent divergence.
• Part V (scaling laws): the solvable neural-scaling-law model of (Maloney et al. 2022) is an NTK model; its power-law exponents come from the spectral decay of the NNGP/NTK eigenspectrum.
• Companion book (Deep Gaussian Processes in Prob DL): NNGP is the infinite-width endpoint of the DGP hierarchy; variational DGP approximations recover finite-width corrections in a different expansion.

Cross-paper SoTA positioning

How do the papers of Part III position against each other on published benchmarks? A few observations.

The two leaderboard tables below (tbl:ntk-sota-comparison and tbl:ntk-sgd-gap) compare accuracies, which only the papers that report accuracies can enter. Table tbl:ntk-part3-claims covers the rest of the part on its own terms, one row per paper, so that the theory papers are held to a stated claim and a stated number rather than surveyed in prose.

(Jacot et al. 2018) established the theoretical framework but did not push benchmark numbers. Its empirical section was limited to small synthetic experiments confirming kernel concentration; it did not claim that NTK regression was a competitive predictor.

(Lee et al. 2018) pushed the NNGP benchmark numbers first. It reported NNGP on MNIST matching or exceeding finite networks, which was the first public evidence that an infinite-width closed-form kernel could be competitive on standard benchmarks.

(Arora et al. 2019) pushed the NTK benchmark numbers to CIFAR-10. It delivered the CNTK vs CNN table we have referenced, and established the 5-7% gap as the empirical anchor for subsequent discussions.

(Lee et al. 2019) closed the theoretical loop by proving uniform linearization, which justifies treating infinite-width SGD-trained networks as exactly equivalent to kernel regression.

(Chizat et al. 2019) introduced the scaling-based diagnosis of the NTK regime, revealing that lazy training is a universal phenomenon rather than a neural-network-specific one.

(Mei et al. 2018) and (Yang and Hu 2021) offered feature-learning alternatives that explain phenomena the NTK cannot. (Yang and Hu 2021) is particularly influential in practice because $\mu$P-transfer is directly usable.

(Hanin and Nica 2019) and (Roberts et al. 2022) quantified finite-width corrections, providing a bridge between the idealized infinite-width theory and finite-network reality.

(Geiger et al. 2020) and (Woodworth et al. 2020) provided empirical and mathematical evidence that real-network regimes lie strictly between the lazy and rich extremes.

The collective arc is from idealization (NTK as exact theory of wide networks) through critique (empirical gap is real) to synthesis (finite-width corrections and feature-learning parameterizations together give a richer picture). Each paper in Part III plays a specific role in this arc, and none is self-sufficient.

Connecting to scaling laws (Part V preview)

Part V treats neural scaling laws as a distinct topic. A brief preview of the NTK connection is useful here. The observation that test error scales as a power law in compute, data, and parameters (Kaplan et al. 2020; Hoffmann et al. 2022) begs the theoretical question of why power laws. The NTK provides a partial answer (Maloney et al. 2022): for kernel regression with a kernel whose eigenvalues decay as $\lambda_k \sim k^{-\alpha}$ (power law with decay rate $\alpha$), the test error on a target with coefficients $\beta_k \sim k^{-\beta}$ scales as

$$\begin{equation} \mathbb{E}[\mathrm{test error}] \;\sim\; n_{\mathrm{train}}^{-\min(\alpha, 2\beta - 1)/\alpha} + \mathrm{regularization terms}, \end{equation}$$

which is a power law in training-set size. The exponent depends on the kernel's spectral decay (architecture-dependent) and the target's smoothness (task-dependent). For natural datasets whose eigenvalue spectra can be estimated, this predicts scaling exponents that match observed ones reasonably well.

The limitations of this NTK-based scaling story are: it assumes the network is in the NTK regime (not feature-learning); it predicts a single power-law exponent per (architecture, task) pair, whereas observed scaling laws are often better fit by broken power laws with regime transitions; it has no mechanism for the emergent abilities reported in large language models. So the NTK gives a mathematical template for scaling-law derivation but does not account for the full empirical picture.

SoTA table across NTK vs modern architectures

The generalization of this table is: infinite-width kernel methods are competitive with non-pretrained, non-augmented, unregularized finite networks of matching architecture, but fall far short of modern training recipes that exploit feature learning, data augmentation, and transfer learning. The NTK theory explains one regime (wide, lazily trained, from scratch) and does not explain the regime where modern accuracy lives.

Cross-architecture comparison of the gap

The pattern is clear once depth is held in view: the gap grows both with architectural sophistication and, within a fixed architecture, with depth. Fully-connected networks have essentially no gap at any depth. Vanilla CNNs go from 3.6 points at depth 11 to 11.5 at depth 21, because the kernel is falling while the network is still rising. Pooled CNNs sit at 6 to 7 points across the whole depth range, and modern architectures are further out still. Reporting "the CNTK gap" as a single number is therefore a category error; it is a function of depth, and its depth-dependence is the informative part.

The persistence of the gap across training recipes

A natural question is whether the NTK-to-SGD gap can be closed by adjusting training recipe. Several attempts have been made, with mixed results:

Training for longer: extending SGD training does not close the gap; SGD converges to a feature-learning predictor, and continued training does not revert to kernel regression.

Using larger widths: increasing $n$ reduces the kernel fluctuation gap but does not reduce the feature-learning gap; infinite-width $\mu$P networks still have feature-learning benefits that NTK lacks.

Using regularization: explicit regularization (weight decay, dropout) can improve generalization but does not close the gap; both NTK and SGD-trained networks benefit, and the SGD-trained network still wins.

Using better initialization: initialization schemes that keep networks closer to the NTK regime (smaller variance, or effective $\alpha$) move the SGD-trained network toward the NTK predictor but at the cost of worse final performance.

The persistence of the gap suggests it is structural: SGD-trained finite networks operate in a feature-learning regime that is inherently better than kernel regression for sufficiently complex tasks, and no adjustment of training recipe can move them into the pure kernel regime without sacrificing the performance benefit. The gap is a feature of practical training, not a bug in the NTK approximation.

Gap decomposition by architecture family

Different architecture families exhibit different NTK-to-SGD gaps. A rough taxonomy:

Fully-connected networks. The NTK gap is small for FC networks at moderate width. On MNIST with a 3-layer FC network, the infinite-width NNGP reaches 98.7%, the NTK-regression predictor reaches 99.0%, and an SGD-trained finite network reaches 99.1%. The gap is within sampling noise. The NTK theory is a quantitative theory of wide fully-connected networks.

Vanilla CNNs (no skip connections, no batch norm). The NTK gap is intermediate. On CIFAR-10, the gap between CNTK-GAP and the matching CNN-GAP is 5-7% in the depth range 11-21 (Table above). This is where the feature-learning story becomes essential.

Modern CNNs (ResNets, WideResNets). The NTK gap is large. Analogs of CNTK for ResNets have been proposed (Huang and Yau 2020) but do not capture the training-recipe benefits (augmentation, label smoothing, mixup, schedules) that contribute substantially to modern accuracy. A precise NTK-vs-SGD comparison on ResNets is not easy to set up fairly, but order-of-magnitude estimates place the gap at 10-15% on CIFAR-10.

Transformers. The NTK gap is unclear. Attention-NTKs (Hron et al. 2020) are computable but have not been benchmarked extensively against SGD-trained Transformers; the order of limits matters: $d_k$ must go to infinity before $H$, or the kernel degenerates. The resulting NTK has no closed form, because the softmax requires a double Gaussian integral evaluated by Hermite quadrature. On a single-layer infinite-attention experiment with IMDB sentiment, NNGP achieves $90.1\%$, NTK $90.4\%$, and finite-width attention $89.8\%$ (Hron et al. 2020), so for that controlled setting infinite-width predictions match finite networks within 1%. Skip connections are essential: without residuals the attention NNGP collapses to a constant as $L \to \infty$. Layer-norm reduces in the infinite-width limit to a covariance normalization $\Sigma \mapsto \Sigma/\mathrm{tr}(\Sigma)$. At larger scale, preliminary experiments suggest gaps comparable to modern CNNs (10-15% on language-modeling metrics), though the comparison is complicated by the pretraining-and-fine-tuning structure of Transformer workflows.

Graph networks. Graph NTKs (Du et al. 2019) are computable for message-passing architectures and on graph-classification benchmarks the kernel often matches or exceeds SGD-trained GNNs: on MUTAG, GNTK reaches $90.0\%$ vs $89.4\%$ for the best GIN; on COLLAB, $83.6\%$ vs $80.2\%$, and similar inversions hold for 7 of 9 standard benchmarks (PROTEINS, NCI1, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, REDDIT-5K). The kernel decomposes into node-level kernels $\Theta_{u,u'}(G, G')$ plus sum-pooling, with the number of message-passing rounds $k$ entering as a hyperparameter (sweet spot $k = 2$-$3$). The kernel-favourable picture for graphs may reflect the shallow (2-4 layer) depth of typical GNN architectures, which keeps them in a more NTK-favorable regime. It also suggests that feature learning matters less for graph classification than for vision, an inversion of the standard NTK-versus-feature-learning narrative.

The overall pattern is that simpler, shallower, more homogeneous architectures are better-described by the NTK, and more sophisticated architectures exhibit larger gaps. This is consistent with the diagnosis that the NTK-vs-SGD gap tracks the degree to which training benefits from feature learning, which in turn tracks architectural sophistication.

NTK's contribution to the theory of generalization

The NTK's principal contribution to generalization theory is to provide a computable baseline against which to measure. Before the NTK, generalization analyses of deep networks were either very loose (VC dimension, Rademacher complexity) or required strong unverified assumptions (PAC-Bayes with specific posterior priors). The NTK changed this by delivering:

1. An explicit upper bound on capacity: the NTK's RKHS norm ball of radius $R$ has Rademacher complexity $\mathcal{O}(R \cdot \sqrt{\mathrm{tr}(\Theta_\infty(X,X))} / n_{\mathrm{train}})$, which is computable and non-vacuous for wide networks.
2. A direct generalization bound: for functions in the NTK RKHS with bounded norm, test error is bounded by training error plus the Rademacher term, giving non-vacuous bounds of the form "test error $\le$ training error + $\mathcal{O}(R / \sqrt{n_{\mathrm{train}}})$".
3. A spectral explanation: components of $y^\star$ aligned with large eigenvalues of $\Theta_\infty(X,X)$ are fit easily and generalize well; components in small eigenvalues generalize poorly. This makes spectral decay a quantitative predictor of generalization.

For networks demonstrably in the NTK regime (very wide, NTK-parameterized, trained without aggressive recipes), these bounds are tight: predicted generalization matches observed generalization within small factors. For networks outside the NTK regime, the bounds are conservative: they upper-bound the kernel-regression generalization, which is worse than the feature-learning generalization of the actually-trained network.

This is the pattern that justifies the framing "NTK as baseline": wherever the NTK bound is tight, deep learning is kernel learning; wherever the NTK bound is loose, the slack is attributable to feature learning. The NTK is therefore a tool for diagnosing when feature learning matters, by comparison of NTK-predicted-test-error to actual test error.

Critiques and counter-evidence

Several authors have pushed back on the NTK-as-theory-of-deep-learning framing with evidence that finite networks behave in ways the NTK cannot predict.

Shift-invariance and data augmentation. (Li et al. 2019) show that flip augmentation is exactly a kernel symmetrization, while crop augmentation is not representable as a single Gram matrix, so the benefit of the latter is invisible to NTK regression. Yet augmentation adds 2-5% test accuracy on CIFAR-10 in practical training. The kernel framework therefore absorbs the group-averaged part of a training recipe and not the distribution-dependent part; some benefit of training comes from outside kernel regression.

Catastrophic forgetting and transfer learning. A trained network can be fine-tuned on a new task with transfer of learned features. The NTK has no mechanism for this: the kernel is fixed at initialization, so fine-tuning a kernel-regression predictor on a new task is just re-running kernel regression with the same kernel, which cannot use anything from the prior training run. The empirical fact of successful transfer learning in BERT, GPT and CLIP is therefore evidence against the NTK as a complete theory of deep-network training.

Adversarial robustness. (Tsipras et al. 2019) and subsequent work show that adversarial-robustly trained networks have different feature representations from naturally-trained ones, with robust features often being human-interpretable. This feature-selection process is a rich-regime phenomenon; adversarial robustness cannot emerge from kernel regression on a fixed kernel.

Emergent abilities. The observation that large language models exhibit qualitatively new capabilities (e.g., in-context learning, chain-of-thought reasoning) at threshold scales has no NTK explanation. The NTK predicts smooth, monotone, power-law-governed performance scaling, not phase-transition-like emergence. Even granting that emergence may be partly a measurement artifact (Schaeffer et al. 2023), the NTK-predicted behavior is insufficient to rationalize large-model practice.

Grokking. (Power et al. 2022) report that networks trained long past fitting the training data eventually suddenly generalize, with test accuracy rising from chance to 100% over a narrow training window. This is incompatible with the NTK's exponential-decay training dynamics (eq:ntk-residual-flow), which predicts that test accuracy should track training loss monotonically.

Taken together, these phenomena establish that something beyond kernel regression is happening in practical deep-learning training. The NTK's residual value is as a null model: behaviors the NTK predicts are explicable without appeal to feature learning; behaviors it does not are evidence for feature-learning mechanisms.

The role of scale in deep-learning theory

A recurring theme in Part III is that scale is not merely a practical scaling axis but a conceptual one: different scales correspond to different theoretical regimes. The NTK emerges as width goes to infinity; the mean-field regime as width goes to infinity with a different parameterization; double descent as training-set size approaches the critical threshold; emergent abilities as compute or parameters cross task-specific thresholds.

The theoretical challenge is that each regime is hard on its own terms, and transitions between regimes are hard to characterize. The NTK regime is tractable but narrow; the mean-field regime is tractable for simple architectures; the $\mu$P regime is tractable via Tensor Programs but computationally heavy. None of them overlap with the practical regime of deep-learning training, which involves finite but large networks with complex architectures and training recipes.

A concrete manifestation: the regime analysis of the lazy-versus-rich chapter says that practical networks with $\alpha$ in an intermediate range are in an interpolating regime where neither the lazy nor the rich theory directly applies. The description of this interpolating regime is an active research direction. Candidate approaches include:

• Perturbative expansion around $\mu$P: treat $\mu$P as the base point and expand in the departure from it. This requires understanding the structural stability of $\mu$P under perturbation, which is not yet understood.
• Interpolation via coupling constants: introduce one or more dimensionless coupling constants that smoothly interpolate between lazy and rich, and study how observables depend on them. The Chizat-Oyallon-Bach $\alpha$ is one such coupling; others (for specifically architectural perturbations) are under development.
• Empirical characterization with theoretical fits: train a large suite of networks at varying scales and parameterizations; extract the relevant regime-indicating observables; fit phenomenological models that predict the regime transitions. This is the empirical-first approach and has the virtue of directly addressing practical networks.

The conceptual shift Part III embodies is from "deep learning is a tricky optimization problem" to "deep learning is a multi-regime phenomenon whose regimes have different governing equations". This shift changes what a "theory of deep learning" should look like: not a single theorem, but a family of theorems for different regimes plus a characterization of when each applies.

References