Better Data Led AI Training Efficiency Gains in Small-Scale Study
The six-year experiment separates changes in training data from changes in model design, but it cannot establish whether the same pattern holds at frontier scale.
Loading page…
The six-year experiment separates changes in training data from changes in model design, but it cannot establish whether the same pattern holds at frontier scale.
Listen to this story
Patel and Han’s 2019–2025 comparison suggests curated training data delivered far more measured pretraining efficiency than evolving model recipes in small-scale experiments: at 1e19 FLOPs, data changes yielded a 12× gain versus 3.7× from model changes. The findings support investing in corpus quality and filtering, but do not establish that data dominates at frontier scale.
At the largest budget, authors estimate data improvements were 3.24× larger than model improvements.
Data and model gains were largely independent; additive effects explained 88% of OLMES-score variation.
Runs paired GPT-2–to–OLMo-2-era recipes with public corpora from OpenWebText to UltraFineWeb.
A six-year comparison of open AI training methods found that newer data reduced the compute needed for tested capabilities more than newer model recipes did. The result sharpens the case for data curation as a source of pretraining progress, but it comes from small runs that may not represent frontier training.
Dwarkesh Patel and Jerry Han paired year-representative model recipes with public data corpora from 2019 through 2025. They trained those combinations at budgets up to 1e19 FLOPs, a measure of training computation, then assessed downstream capability with OLMES, an aggregate of 10 relatively easy benchmarks that are mostly multiple-choice question answering.
The data side moves from OpenWebText, a corpus of roughly 9 billion tokens, to later collections including UltraFineWeb, which use broader web scrapes and more sophisticated filtering. The model side spans GPT-2 to OLMo-2 and incorporates changes to optimization, positional encoding, normalization, activation functions and initialization.
At 1e19 FLOPs, the authors estimated a 12.0× compute-efficiency gain from data improvements relative to the 2019 baseline.
At that budget, model improvements produced an estimated 3.7× gain; the authors calculate the data gain as 3.24 times larger in this experiment.
The authors found the two sources of improvement were mostly independent: additive effects explained 88% of variation in OLMES scores. But they caution against treating the result as a broad ranking of data work over model work. Model research may instead enable larger, longer and more stable training runs by easing constraints such as instability, memory and bandwidth limits.
Patel and Han describe their experiments as extremely small scale and call for larger tests. They hypothesize that careful filtering may benefit smaller models more, while larger models may gain more from a greater volume of material even when its average quality is lower. That is an explanation to test, not a result established by these runs.
Synthetic data is especially consequential to the interpretation because the study did not test it. If data improvements are a key contributor to pretraining efficiency, the unanswered question is whether useful training material can be expanded beyond existing sources without weakening results.
Loading discussion...
Join the conversation
Explain which investment you think matters more and why.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.