Making Data Count: Measuring, Stretching, and Amortizing Training Data

Title: Making Data Count: Measuring, Stretching, and Amortizing Training Data Speaker: David Alvarez-Melis, Assistant Professor of Computer Science at Harvard SEAS Abstract: Compute for training AI models is outpacing the supply of high-quality data. Data, not architectures or compute, is increasingly the binding constraint on what models can learn. Yet the tools we have to measure, reuse, and allocate data remain far less developed than those for models. In this talk, I will argue that data should be treated as the primary object to be optimized in AI, and present three lines of work in this direction. First, on measuring data, I will show how optimal transport yields a principled geometry on the space of datasets, which we use to predict transfer and synthesize new training data. Second, on stretching data, we derive scaling laws that quantify what a token is worth when fresh data runs out, whether it is repeated, paraphrased, or drawn from a domain that synergizes with the target. Third, on amortizing data, I will discuss our work on boomerang distillation, which turns a single training run into a continuous family of models of any intermediate size. Together, these results point toward a data-centric foundation for machine learning, and I will close with the open questions it raises. The Computer Science Colloquium Series will be held weekly on Thursdays at 2:30pm in SEC LL2.224. There will be refreshments before the talk at 2:15pm.