Abstract
How can we make sense of large-scale recordings of neural activity across learning? Theories of neural network learning with their origins in statistical physics offer a potential answer: for a given task, there are often a small set of summary statistics that are sufficient to predict performance as the network learns. Here, we review recent advances in how summary statistics can be used to build theoretical understanding of neural network learning. We then argue for how this perspective can inform the analysis of neural data, enabling better understanding of learning in biological and artificial neural networks.
1 Introduction
Experience reshapes neural population activity, molding an animal's representations of the world as it learns to perform new tasks. Thanks to advances in experimental technologies, it is just now becoming possible to measure changes in the activity of large neural populations across the course of learning (; ; ; ; ; ; ). However, with this new capability comes the challenge of identifying which features of high-dimensional activity patterns are meaningful for understanding learning. While analyses of representations have begun how to elucidate how learning reshapes the structure of activity, it is not in general clear whether these measurements are sufficient to understand how representational changes relate to behavior (; ; ; ).
In this Perspective, we propose that the principled identification of summary statistics of learning offers a possible path forward. This framework is grounded in theories of the statistical physics of learning in neural networks, which show that low-dimensional summary statistics are often sufficient to predict task performance over the course of learning (; ; ). We argue that thinking systematically about summary statistics gives new insight into what existing approaches of quantifying neural representations reveal about learning, and allows identification of what additional measurements would be required to constrain models of plasticity. We emphasize that the goal of this Perspective is not to advocate for the use of a particular set of summary statistics, but rather to explain the general philosophy of this approach to understanding learning in high dimensions.
2 What is a summary statistic?
We posit that summary statistics of learning must satisfy two minimal desiderata:
They must be low-dimensional. That is, their dimension is low relative to the number of neurons in the network of interest. Indeed, most summary statistics we will encounter are determined by averages over the population of neurons.
They must be sufficient to predict behavior across learning. From a theoretical standpoint, there should exist a closed set of equations describing the evolution of the summary statistics that predict the network's performance.
As we will illustrate with concrete examples in Section 3, summary statistics satisfying these two desiderata are often highly interpretable thanks to their clear relationship to the network architecture and learning task. However, the summary statistics relevant for predicting performance may not be sufficient to predict all statistical properties of population activity. We will elaborate on this issue, and the resulting limitations of descriptions based on summary statistics alone, in Section 4.
Our use of the term “summary statistics” follows work by , . In the literature on the statistical physics of learning, the quantities that we refer to as summary statistics are often termed “order parameters” (; ; ; ). We prefer to use the former, more general term as it better captures the goal of these reduced descriptions in a neuroscientific context: we aim to summarize the features of neural activity relevant for learning.
3 Summary statistics in theories of neural network learning
We now review how summary statistics emerge naturally in theoretical analyses of neural network learning. Out of many theoretical results, we focus on two example settings: online learning from high-dimensional data in shallow networks, and batch learning in wide and deep networks (; ; ; ; ; ; ; ; ; ; ; ; ; ). These model problems illustrate how relevant summary statistics may be identified given a task, network architecture, and learning rule.
3.1 Online learning in shallow neural networks with high dimensional data
Classical models of online gradient descent learning in high dimensions can be often be summarized with simple summary statistics (; ; ; ; , ; ; ). In this section, we discuss how the generalization performance of perceptrons and shallow (two-layer) neural networks trained on large quantities of high dimensional data can be summarized by simple weight alignment measures. Most simply, the perceptron model seeks to learn a weight vector w∈ℝD which correctly classifies a finite set of randomly sampled training input-output pairs (xμ, yμ). If the inputs are random, , and the targets yμ = y(xμ) are generated by a teacher network, then the generalization performance (performance of the model on new unseen data, ) is completely determined by the overlap of w with itself and with the target direction w⋆
If the learning rate is scaled appropriately with the dimension D, the high-dimensional (large-D) limit of online stochastic gradient descent is given by a deterministic set of equations for Q and R:
where the continuous training “time” τ is the ratio of the number of samples seen to the dimension and F:ℝ2 → ℝ2 is a nonlinear function that depends on the learning rate, the loss function, and the link function σ(·) (; ; ; ; ). Integrating this update equation allows one to predict the evolution of the generalization error as more training data are provided to the algorithm. Despite the infinite dimensionality of the original optimization problem, only two dimensions are necessary to capture the dynamics of generalization error.
The analysis of online perceptron learning can be extended to two layer neural networks with a small number of hidden neurons N,
In this setting with isotropic random data, the relevant summary statistics are the readout weights a∈ℝN, along with overlap matricesQ∈ℝN×N and R∈ℝN×K with entries
For this system, we can track the gradient descent dynamics for a, Q, and R through a generalization of Equation 2 (; ; ; ). This reduces the dimensionality of the dynamics from the N+DN trainable parameters {ai}, {wj} to N+N2+NK summary statistics, which is significant when D≫N+K. This reduction enables the application of analyses that cannot scale to high dimensions, for instance control-theoretic methods to study optimal learning hyperparameters and curricula (; ). Recent works have also begun to study approximations to these summary statistics when the network width N is also large, as further dimensionality reduction if possible when Q and R have stereotyped structures (; ).
Under what conditions is this reduction possible? Fundamentally, the summary statistics a, Q, and R are sufficient to determine the network's performance so long as the preactivations hi and are approximately Gaussian. Thus, one can relax the assumption that the inputs x are exactly Gaussian so long as a central limit theorem applies to hi and (, ). Moreover, one can allow for correlations between the different input dimensions so long as hi and remain Gaussian. If E[xx⊤] = Σ, with a modification of the definition of the overlaps to and a similar reduction applies (). One can even consider extensions to plasticity rules other than stochastic gradient descent. For example, online node perturbation leads to a different effective dynamics for the same set of summary statistics (, ).
How could the overlaps Q and R be accessed from measurements of neural activity? And, in the absence of detailed knowledge of a teacher network, how could one identify the relevant overlaps? Under the simple structural assumptions of these models, one could estimate the overlaps from covariances of network activity across stimuli, i.e., with isotropic inputs one has and Ex[hihj] = Qij. Moreover, one can in some cases detect this underlying low-dimensional structure by examining the principal components of the learning trajectory (). However, more theoretical work is required in this vein.
3.2 Learning in wide and deep neural networks
Another strategy to reduce the complexity of multilayer deep neural networks is to analyze the dynamics of learning in terms of representational similarity matrices (kernels) for each hidden layer of the network. Consider, for example, a deep fully-connected network with input x∈ℝD,
where t denotes training time. Instead of using online stochastic gradient descent to train the weights as we did in the preceding section, suppose we use gradient flow to minimize the average error on a fixed set of training examples. Moreover, instead of considering a regime where the hidden layer width N is small relative to the input dimension D, let us now consider very wide networks with N≫D (Figure 1a).
Figure 1
What are the relevant summary statistics in this case? Applying the chain rule to the dynamics of the network outputs, one finds the differential equation
where is the loss function and denotes expectation over the training dataset (; ; ). Here,
are representational similarity matrices, and
are gradient similarity matrices, which respectively compare the hidden states and the gradient signals at each hidden layer ℓ for each pair of data points (x, x′) and each pair of training times (t, t′). Thus, as Φ(ℓ) and G(ℓ) determine the dynamics of f, these matrices are suitable summary statistics of learning if they are low-dimensional relative to the set of synaptic weights, and if we can write down a closed set of equations for their dynamics.
First, it is easy to see that the criterion of dimensionality reduction requires that the number of training examples P is much less than the network width N, as the number of similarity matrix elements and the number of synaptic weights are of order P2 and N2, respectively. Second, it turns out that one can close the equations for Φ(ℓ) and G(ℓ) provided that the width is large and that the synaptic weights start from an uninformed initial condition (i.e., Gaussian random matrices) (; ; ; ). Depending on how weights and learning rates are scaled, one can obtain different types of large-width (N → ∞) limits (Figure 1b). In the lazy/kernel limit where γ is constant, these representational similarity matrices are static over the course of learning (; ). However, an alternative scaling () can be adopted where these objects evolve in a task-dependent manner even as N → ∞ (Figure 1c) (; ).
While this provides a description of the training dynamics of a model under gradient flow, one can extend this description in terms of similarity matrices to other learning rules which use approximations of the backward pass variables , which we called pseudo-gradients in . Such rules include Hebbian learning, feedback alignment, and direct feedback alignment (; ; ). In this case, the relevant summary statistics to characterize the prediction dynamics of the network include the gradient-pseudogradient correlation, which measures the alignment between the gradients used by the learning rule and the gradients that one would have used with gradient flow,
as governs the evolution of the function output:
4 Implications for neural measurements
The two example settings detailed in Section 3 show how the relevant summary statistics of learning depend on network architecture and learning rule. Theoretical studies are just beginning to map out the full space of possible summary statistics for different network architectures (; ; ; ; ; ; ; ; ; ; ; ; ). Though details of the relevant summary statistics vary depending on the scaling regime and task—as illustrated by the examples above, where network width, training dataset size, and learning rule change the relevant statistics and their effective dynamics—they share broad structural principles. In all cases, summary statistics are defined by (weighted) averages over sub-populations of neurons within the network of interest, e.g., correlations of activity with task-relevant variables, or autocorrelations of activity within a particular layer in a deep network. Thanks to these common structural features, these varied theories of summary statistics have common implications for the analysis and interpretation of neuroscience experiments.
4.1 Benign sub-sampling
The summary statistics encountered in Section 3 are robust to sub-sampling thanks to their basic nature as averages over the population of neurons. These statistical theories in fact post a far stronger notion of benign sub-sampling: they result in neurons that are statistically exchangeable. This is highly advantageous from the perspective of long-term recordings of neural activity, as reliable measurement of summary statistics does not require one to track the exact same neurons over time. Instead, it suffices to measure a sufficiently large subpopulation on any given day. This obviates many of the challenges presented by tracking neurons over multiple recording sessions (). Moreover, the variability and bias introduced by estimating summary statistics from a limited subset of relevant neurons can be characterized systematically (; ; ). Taken together, these properties mean that summary statistics are relatively easy to estimate given limited neural measurements, provided that exchangability is not too strongly violated (). We will return to this question in the Discussion, as a detailed analysis of the effects of non-identical neurons will be an important topic for future theoretical work. There are limits, however, to how far one can sub-sample. For instance, representational similarity kernels are more affected by small, coordinated changes in the tuning of many neurons than large changes in single-neuron tuning (Figure 2) (). Determining the minimum number of neurons one must record in order to predict generalization dynamics across learning will be an important subject for future theoretical work (; ).
Figure 2
4.2 Invariances and representational drift
Though by our definition the summary statistics mentioned in Section 3 are sufficient to predict the network's performance, they are not sufficient statistics for all properties of the neural code. In particular, in part because they arise from theories in which neurons become exchangable, they have many invariances. These invariances mean that individual tuning curves can change substantially without altering the population-level computation (). For instance, the representational similarity kernels are invariant under rotation of the neural code at each layer, enabling complete reorganization of the single-neuron code without any effect on behavior. Similarly, overlaps with task-relevant directions are invariant to changes in the null space of those low-dimensional projections. These invariances mean that focusing on summary statistics of learning sets a particular aperture on what aspects of representations one can assay.
At the same time, the invariances of summary statistics have important consequences for functional robustness. In particular, they are closely related to theories of representational drift, the seemingly puzzling phenomenon of continuing changes in neural representations of task-relevant variables despite stable behavioral performance (; ). Many models of drift explicitly propose that representational changes are structured in such a way that certain summary statistics are preserved (Figure 2a) (; ; ). Identifying the invariances of the summary statistics sufficient to determine task performance can allow for a more systematic characterization of what forms of drift can be accommodated by a given network. Conversely, identifying the invariances of a representation once task performance stabilizes might suggest which summary statistics are relevant for the learning problem at hand.
4.3 Universality
An important lesson from the theory of high-dimensional statistics is that of universality: certain coarse-grained statistics are asymptotically insensitive to the details of the distribution. The most prominent example of statistical universality is the familiar central limit theorem: the distribution of the sample mean of independent random variables tends to a Gaussian as the number of samples becomes large. A broader class of universality principles arise in random matrix theory: the distribution of eigenvalues and eigenvectors of a random matrix often become insensitive to details of the distribution of the elements as the matrix becomes large. Most famously, the Marčenko-Pastur theorem specifies that the singular values of a matrix with independent elements have a distribution that depends only on the mean and variance of the elements (). In the context of learning problems, universality manifests through insensitivity of the model performance to details of the distributions of parameters or of features (; ).
From the perspective of summary statistics, statistical universality can allow simple theories to make informative macroscopic predictions even if they do not capture detailed properties of single neurons. For instance, the mean-field description of the learning dynamics of wide neural networks introduced in Section 3 are universal in that they depend on the initial distribution of hidden layer weights only through its mean and variance, even though the details of that distribution will affect the distribution of weights throughout training (Figures 2b–d) (; ). Like the invariances to transformations of the neural population code mentioned before, this is nonetheless a double-edged sword: these universality properties mean that focusing on predicting performance commits one to coarse-graining away certain microscopic aspects of neural activity. Though these features are not required to predict macroscopic behavior, they may be important for understanding biological mechanisms.
5 Discussion
The core insight of the statistical mechanics of learning in neural networks is the existence of low-dimensional summary statistics sufficient to predict behavioral performance. We have reviewed how different summary statistics emerge depending on network architecture and task, how summary statistics might be estimated from experimental recordings, and what this perspective reveals about existing approaches to quantifying representational changes over learning. We now conclude by discussing complementary summary statistics of neural representations that arise from alternative desiderata, and future directions for theoretical inquiry.
A significant line of recent work in neuroscience aims to quantify neural representations and compare them across networks through analysis of representational similarity matrices Φ(ℓ)(x, x′) (; ; ; ). Here, we see that these kernel matrices arise naturally as summary statistics of forward signal propagation in wide and deep neural networks (Figures 1c, d). At the same time, those results show that tracking only feature kernels is not in general sufficient to predict performance over the course of learning. One needs access also to coarse-grained information about the plasticity rule in the form of gradient kernels [either G(ℓ) or ], and to information about the network outputs (for instance ). More theoretical work is required to determine how to reliably estimate these gradient kernels from data, thereby providing a means to gain coarse-grained information about the underlying plasticity rule.
The summary statistics discussed here explicitly depend on the architecture and nature of plasticity in the neural network of interest, as they seek to predict its performance over learning. A distinct set of summary statistics arises if one aims to study what features of a representation are relevant for an independently-trained decoder. In this line of work, one regards the representation as fixed, rather than considering end-to-end training of the full network as we considered here. If the decoder is a simple linear regressor that predicts a continuous variable, the relevant summary statistics of the representation are just its mean and covariance across stimuli (; ). Given a particular task, the covariance can be further distilled into the rate of decay of its eigenvalues and of the projections of the task direction into its eigenvectors (; ; , ; ; ; ; ). For categorically-structured stimuli, a substantial body of work has elucidated the summary statistics that emerge from assuming that one wants to divide the data according to a random dichotomy (; ; ; ; ; ; ; ).
The models reviewed here are composed of exchangeable neurons, which simplifies the relevant summary statistics and renders them particularly robust to sub-sampling. However, the brain has rich structure that can affect which summary statistics are sufficient to track learning and how those summary statistics may be measured. Biological neural networks are embedded in space, and their connectivity and selectivity is shaped by spatial structure (; ; ). Notably, many sensory areas are topographically organized: neurons with similar response properties are spatially proximal (; ). Moreover, neurons can be classified into genetically-identifiable cell types (), which may play distinct functional roles during learning (; ). Future theoretical work must contend with these biological complexities in order to determine the relevant summary statistics of learning subject to these constraints.
Statements
Data availability statement
No experimental data were analyzed or generated in the preparation of this Perspective. Simulations of wide neural networks in Figures 1b, c, 2b–d following () are based on code available under an MIT License at https://github.com/Pehlevan-Group/dmft_wide_networks.
Author contributions
JZ-V: Conceptualization, Funding acquisition, Visualization, Writing – original draft, Writing – review & editing. BB: Conceptualization, Visualization, Writing – original draft, Writing – review & editing. CP: Conceptualization, Funding acquisition, Writing – review & editing.
Funding
The author(s) declare that financial support was received for the research and/or publication of this article. JZ-V is supported by the Office of the Director of the National Institutes of Health under Award Number DP5OD037354. JZ-V is further supported by a Junior Fellowship from the Harvard Society of Fellows. BB is supported by a Google PhD Fellowship. CP is supported by NSF grant DMS-2134157, NSF CAREER Award IIS-2239780, DARPA grant DIAL-FP-038, a Sloan Research Fellowship, and The William F. Milton Fund from Harvard University. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
Acknowledgments
We are indebted to Nikolaus Kriegeskorte for sharing Figure 10 of (), from which our Figure 1d is derived. We thank Paul Masset, Venkatesh Murthy, Farhad Pashakhanloo, and Ningjing Xia for helpful discussions and comments on previous versions of this manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that no Gen AI was used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Author disclaimer
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
References
1
ArnaboldiL.StephanL.KrzakalaF.LoureiroB. (2023). “From high-dimensional &mean-field dynamics to dimensionless ODEs: a unifying approach to SGD in two-layers networks,” in Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, eds. G. Neu, and L. Rosasco (PMLR), 1199–1227.
2
AtanasovA.Zavatone-VethJ. A.PehlevanC. (2024). Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592.
3
Ben ArousG.GheissariR.HuangJ.JagannathA. (2023). High-dimensional SGD aligns with emerging outlier eigenspaces. arXiv preprint arXiv:2310.03010.
4
Ben ArousG.GheissariR.JagannathA. (2022). “High-dimensional limit theorems for SGD: effective dynamics and critical scaling,” in Advances in Neural Information Processing Systems, eds. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc.), 25349–25362.
5
BernardiS.BennaM. K.RigottiM.MunueraJ.FusiS.SalzmanC. D. (2020). The geometry of abstraction in the hippocampus and prefrontal cortex. Cell183, 954–967.e21. 10.1016/j.cell.2020.09.031
6
BiehlM.SchwarzeH. (1995). Learning by on-line gradient descent. J. Phys. A Math. Gen. 28:643. 10.1088/0305-4470/28/3/018
7
BordelonB.CotlerJ.PehlevanC.Zavatone-VethJ. A. (2025). Dynamically learning to integrate in recurrent neural networks. arXiv preprint arXiv:2503.18754.
8
BordelonB.MassetP.KuoH.PehlevanC. (2023). “Loss dynamics of temporal difference reinforcement learning,” in Advances in Neural Information Processing Systems, eds. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Curran Associates, Inc.), 14469–14496.
9
BordelonB.PehlevanC. (2022). Population codes enable learning from few examples by shaping inductive bias. Elife11:e78606. 10.7554/eLife.78606
10
BordelonB.PehlevanC. (2023a). “The influence of learning rule on representation dynamics in wide neural networks,” in The Eleventh International Conference on Learning Representations.
11
BordelonB.PehlevanC. (2023b). Self-consistent dynamical field theory of kernel evolution in wide neural networks. J. Stat. Mech. 2023:114009. 10.1088/1742-5468/ad01b0
12
BordelonB.PehlevanC. (2024). Dynamics of finite width kernel and prediction fluctuations in mean field neural networks. J. Statist. Mech. 2024:104021. 10.1088/1742-5468/ad642b
13
CanatarA.BordelonB.PehlevanC. (2021). Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nat. Commun. 12:2914. 10.1038/s41467-021-23103-1
14
CanatarA.FeatherJ.WakhlooA.ChungS. (2024). “A spectral theory of neural prediction and alignment,” in Advances in Neural Information Processing Systems, 36.
15
ChklovskiiD. B.SchikorskiT.StevensC. F. (2002). Wiring optimization in cortical circuits. Neuron34, 341–347. 10.1016/S0896-6273(02)00679-7
16
ChungS.LeeD. D.SompolinskyH. (2018). Classification and geometry of general perceptual manifolds. Phys. Rev. X8:031003. 10.1103/PhysRevX.8.031003
17
CohenU.ChungS.LeeD. D.SompolinskyH. (2020). Separability and geometry of object manifolds in deep neural networks. Nat. Commun. 11:746. 10.1038/s41467-020-14578-5
18
CuiH.KrzakalaF.ZdeborovaL. (2023). “Bayes-optimal learning of deep random networks of extensive-width,” in Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, eds. A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (PMLR), 6468–6521.
19
EngelA.van den BroeckC. (2001). Statistical Mechanics of Learning. Cambridge: Cambridge University Press. 10.1017/CBO9781139164542
20
FarrellM.BordelonB.TrivediS.PehlevanC. (2022). “Capacity of group-invariant linear readouts from equivariant representations: How many objects can be linearly classified under all possible views?,” in International Conference on Learning Representations.
21
FinkA. J.MuscinelliS. P.WangS.HoganM. I.EnglishD. F.AxelR.et al. (2025). Experience-dependent reorganization of inhibitory neuron synaptic connectivity. bioRxiv.16.633450. 10.1101/2025.01.16.633450
22
GaoP.TrautmannE.YuB.SanthanamG.RyuS.ShenoyK.et al. (2017). A theory of multineuronal dimensionality, dynamics and measurement. bioRxiv, 214262. 10.1101/214262
23
GoldtS.AdvaniM.SaxeA. M.KrzakalaF.ZdeborováL. (2019). “Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup,” in Advances in Neural Information Processing Systems, 32. 10.1088/1742-5468/abc61e
24
GoldtS.MézardM.KrzakalaF.ZdeborováL. (2020). Modeling the influence of data structure on learning in neural networks: the hidden manifold model. Phys. Rev. X10:041044. 10.1103/PhysRevX.10.041044
25
GolikovE.YangG. (2022). “Non-gaussian tensor programs,” in Advances in Neural Information Processing Systems, eds. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc.), 21521–21533.
26
HaraK.KatahiraK.OkanoyaK.OkadaM. (2011). Statistical mechanics of on-line node-perturbation learning. IPSJ Online Trans. 4, 23–32. 10.2197/ipsjtrans.4.23
27
HaraK.KatahiraK.OkanoyaK.OkadaM. (2013). Statistical mechanics of node-perturbation learning for nonlinear perceptron. J. Phys. Soc. Japan82:054001. 10.7566/JPSJ.82.054001
28
HarveyS. E.LipshutzD.WilliamsA. H. (2024). “What representational similarity measures imply about decodable information,” in UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models.
29
HastieT.MontanariA.RossetS.TibshiraniR. J. (2022). Surprises in high-dimensional ridgeless least squares interpolation. Ann. Stat. 50, 949–986. 10.1214/21-AOS2133
30
HebbD. O. (2005). The Organization of Behavior: A Neuropsychological Theory. London: Psychology press.
31
HirokawaJ.VaughanA.MassetP.OttT.KepecsA. (2019). Frontal cortex neuron types categorically encode single decision variables. Nature576, 446–451. 10.1038/s41586-019-1816-9
32
HuH.LuY. M. (2022). Universality laws for high-dimensional learning with random features. IEEE Trans. Inf. Theory69, 1932–1964. 10.1109/TIT.2022.3217698
33
JacotA.GabrielF.HonglerC. (2018). “Neural tangent kernel: convergence and generalization in neural networks,” in Advances in Neural Information Processing Systems, 31.
34
KandlerK.ClauseA.NohJ. (2009). Tonotopic reorganization of developing auditory brainstem circuits. Nat. Neurosci. 12, 711–717. 10.1038/nn.2332
35
KangH.CanatarA.ChungS. (2025). Spectral analysis of representational similarity with limited neurons. arXiv preprint arXiv:2502.19648.
36
Khaligh-RazaviS.-M.KriegeskorteN. (2014). Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS Comput. Biol. 10, 1–29. 10.1371/journal.pcbi.1003915
37
KhonaM.ChandraS.FieteI. (2025). Global modules robustly emerge from local interactions and smooth gradients. Nature640, 155–164. 10.1038/s41586-024-08541-3
38
KrakauerJ. W.GhazanfarA. A.Gomez-MarinA.MacIverM. A.PoeppelD. (2017). Neuroscience needs behavior: correcting a reductionist bias. Neuron93, 480–490. 10.1016/j.neuron.2016.12.041
39
KriegeskorteN.MurM.BandettiniP. A. (2008). Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2:249. 10.3389/neuro.06.004.2008
40
KriegeskorteN.WeiX.-X. (2021). Neural tuning and representational geometry. Nat. Rev. Neurosci. 22, 703–718. 10.1038/s41583-021-00502-3
41
LeeJ.XiaoL.SchoenholzS.BahriY.NovakR.Sohl-DicksteinJ.et al. (2019). “Wide neural networks of any depth evolve as linear models under gradient descent,” in Advances in Neural Information Processing Systems, eds. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc.). 10.1088/1742-5468/abc62b
42
LillicrapT. P.CowndenD.TweedD. B.AkermanC. J. (2016). Random synaptic feedback weights support error backpropagation for deep learning. Nat. Commun. 7:13276. 10.1038/ncomms13276
43
MarchenkoV. A.PasturL. A. (1967). Distribution of eigenvalues for some sets of random matrices. Matematicheskii Sbornik114, 507–536.
44
MassetP.QinS.Zavatone-VethJ. A. (2022). Drifting neuronal representations: bug or feature?Biol. Cyber. 116, 253–266. 10.1007/s00422-021-00916-3
45
MézardM.ParisiG.VirasoroM. A. (1987). Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications. World Scientific Publishing Company. 10.1142/0271
46
MignaccoF.MoriF. (2025). A statistical physics framework for optimal learning. arXiv preprint arXiv:2507.07907.
47
MisiakiewiczT.SaeedB. (2024). A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and GCV estimator. arXiv preprint arXiv:2403.08938.
48
MontanariA.UrbaniP. (2025). Dynamical decoupling of generalization and overfitting in large two-layer networks. arXiv preprint arXiv:2502.21269.
49
MoriF.MannelliS. S.MignaccoF. (2025). “Optimal protocols for continual learning via statistical physics and control theory,” in The Thirteenth International Conference on Learning Representations.
50
MurthyV. N. (2011). Olfactory maps in the brain. Annu. Rev. Neurosci. 34, 233–258. 10.1146/annurev-neuro-061010-113738
51
NøklandA. (2016). “Direct feedback alignment provides learning in deep neural networks,” in Advances in Neural Information Processing Systems, 29.
52
PashakhanlooF.KoulakovA. (2023). “Stochastic gradient descent-induced drift of representation in a two-layer neural network,” in Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, eds. A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (PMLR), 27401–27419.
53
QinS.FarashahiS.LipshutzD.SenguptaA. M.ChklovskiiD. B.PehlevanC. (2023). Coordinated drift of receptive fields in Hebbian/anti-Hebbian network models during noisy representation learning. Nat. Neurosci. 26, 339–349. 10.1038/s41593-022-01225-z
54
RuleM. E.O'LearyT.HarveyC. D. (2019). Causes and consequences of representational drift. Curr. Opin. Neurobiol. 58, 141–147. 10.1016/j.conb.2019.08.005
55
SaadD.SollaS. A. (1995). On-line learning in soft committee machines. Phys. Rev. E52:4225. 10.1103/PhysRevE.52.4225
56
SaxeA. M.McClellandJ. L.GanguliS. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120.
57
SorscherB.GanguliS.SompolinskyH. (2022). Neural representational geometry underlies few-shot concept learning. Proc. Nat. Acad. Sci. 119:e2200800119. 10.1073/pnas.2200800119
58
SteinmetzN. A.AydinC.LebedevaA.OkunM.PachitariuM.BauzaM.et al. (2021). Neuropixels 2.0: a miniaturized high-density probe for stable, long-term brain recordings. Science372:eabf4588. 10.1126/science.abf4588
59
StisoJ.BassettD. S. (2018). Spatial embedding imposes constraints on neuronal network architectures. Trends Cogn. Sci. 22, 1127–1142. 10.1016/j.tics.2018.09.007
60
SucholutskyI.MuttenthalerL.WellerA.PengA.BobuA.KimB.et al. (2024). Getting aligned on representational alignment. arXiv preprint arXiv:2310.13018.
61
SunW.WinnubstJ.NatrajanM.LaiC.KajikawaK.BastA.et al. (2025). Learning produces an orthogonalized state machine in the hippocampus. Nature640, 165–175. 10.1038/s41586-024-08548-w
62
VaidyaS. P.LiG.ChitwoodR. A.LiY.MageeJ. C. (2025). Formation of an expanding memory representation in the hippocampus. Nat. Neurosci. 28, 1510–1518. 10.1038/s41593-025-01986-3
63
van MeegenA.SompolinskyH. (2025). Coding schemes in neural networks learning classification tasks. Nat. Commun. 16:3354. 10.1038/s41467-025-58276-6
64
WatkinT. L. H.RauA.BiehlM. (1993). The statistical mechanics of learning a rule. Rev. Mod. Phys. 65, 499–556. 10.1103/RevModPhys.65.499
65
WilliamsA. H. (2024). Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis. bioRxiv. 2024-10. 10.1101/2024.10.23.619871
66
WilliamsA. H.KunzE.KornblithS.LindermanS. (2021). “Generalized shape metrics on neural representations,” in Advances in Neural Information Processing Systems, eds. M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc.), 4738–4750.
67
WilliamsC. (1996). “Computing with infinite networks,” in Advances in Neural Information Processing Systems, eds. M. Mozer, M. Jordan, and T. Petsche (MIT Press).
68
YangG.HuE. J. (2021). “Tensor programs IV: feature learning in infinite-width neural networks,” in International Conference on Machine Learning (PMLR), 11727–11737.
69
Zavatone-VethJ. A.CanatarA.RubenB. S.PehlevanC. (2022a). Asymptotics of representation learning in finite Bayesian neural networks. J. Statistical Mech. 2022:114008. 10.1088/1742-5468/ac98a6
70
Zavatone-VethJ. A.PehlevanC. (2021). “Depth induces scale-averaging in overparameterized linear Bayesian neural networks,” in Asilomar Conference on Signals, Systems, and Computers, 55. 10.1109/IEEECONF53345.2021.9723137
71
Zavatone-VethJ. A.PehlevanC. (2022). On neural network kernels and the storage capacity problem. Neural Comput. 34, 1136–1142. 10.1162/neco_a_01494
72
Zavatone-VethJ. A.TongW. L.PehlevanC. (2022b). Contrasting random and learned features in deep Bayesian linear regression. Phys. Rev. E105:064118. 10.1103/PhysRevE.105.064118
73
ZdeborováL.KrzakalaF. (2016). Statistical physics of inference: thresholds and algorithms. Adv. Phys. 65, 453–552. 10.1080/00018732.2016.1211393
74
ZhangM.PanX.JungW.HalpernA. R.EichhornS. W.LeiZ.et al. (2023). Molecularly defined and spatially resolved cell atlas of the whole mouse brain. Nature624, 343–354. 10.1038/s41586-023-06808-9
75
ZhongL.BaptistaS.GattoniR.ArnoldJ.FlickingerD.StringerC.et al. (2025). Unsupervised pretraining in biological neural networks. Nature2025, 1–10. 10.1038/s41586-025-09180-y
Summary
Keywords
neural networks, learning, statistical physics, representation learning, summary statistics, representational similarity analysis
Citation
Zavatone-Veth JA, Bordelon B and Pehlevan C (2025) Summary statistics of learning link changing neural representations to behavior. Front. Neural Circuits 19:1618351. doi: 10.3389/fncir.2025.1618351
Received
25 April 2025
Accepted
11 August 2025
Published
29 August 2025
Volume
19 - 2025
Edited by
Nicoletta Berardi, University of Florence, Italy
Reviewed by
Alexander van Meegen, Swiss Federal Institute of Technology Lausanne, Switzerland
Updates
Copyright
© 2025 Zavatone-Veth, Bordelon and Pehlevan.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jacob A. Zavatone-Veth jzavatoneveth@fas.harvard.eduBlake Bordelon blake_bordelon@g.harvard.eduCengiz Pehlevan cpehlevan@seas.harvard.edu
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.