Neural Network Scaling Limits: From Theory to Practice

Harvard John A. Paulson School of Engineering and Applied Sciences WIDELY APPLIED MATHEMATICS SEMINAR Thursday, October 22 3 - 4pm Maxwell Dworkin 119 Neural Network Scaling Limits: From Theory to Practice Boris Hanin Associate Professor, Operations Research & Financial Engineering (ORFE) Princeton University Abstract: A fundamental question in deep learning is to describe the scaling limits of neural networks by answering how and what neural networks learn in regimes where both model and dataset size diverge. From the perspective of theory the goal is to identify what infinite size limits of neural network training dynamics exist and to analyze the mechanisms by which architecture and optimizer combine to learn data-dependent features. The aspiration is to develop an analog of the ideal gas law, where out of the collective motion of many weakly interacting molecules (in this analogy network parameters) emerge a small number of thermodynamic variables like pressure, temperature, and volume that give simple and predictive insights into the properties of realistic gasses. From the practical perspective the objective is to design a wind tunnel for neural networks. The goal is to predict reliably from experiments on small models and datasets the most performant ways to align architecture, data, and optimizer at large scale. In this talk I will give a survey of some of my work on neural network scaling limits emphasizing the interplay between theory and practice. Bio: Boris Hanin is an Associate Professor at Princeton ORFE. His research uses tools from probability and mathematical physics to study deep learning theory. He is currently supported by a 2024 Sloan Fellowship in Mathematics.