Better Data Led AI Training Efficiency Gains in Small-Scale Study

A six-year comparison of open AI training methods found that newer data reduced the compute needed for tested capabilities more than newer model recipes did. The result sharpens the case for data curation as a source of pretraining progress, but it comes from small runs that may not represent frontier training.

Separate ingredients in a training run

Dwarkesh Patel and Jerry Han paired year-representative model recipes with public data corpora from 2019 through 2025. They trained those combinations at budgets up to 1e19 FLOPs, a measure of training computation, then assessed downstream capability with OLMES, an aggregate of 10 relatively easy benchmarks that are mostly multiple-choice question answering.

The data side moves from OpenWebText, a corpus of roughly 9 billion tokens, to later collections including UltraFineWeb, which use broader web scrapes and more sophisticated filtering. The model side spans GPT-2 to OLMo-2 and incorporates changes to optimization, positional encoding, normalization, activation functions and initialization.

Measured efficiency at the largest tested budget

0112.0×

Data improvements

At 1e19 FLOPs, the authors estimated a 12.0× compute-efficiency gain from data improvements relative to the 2019 baseline.

023.7×

Model improvements

At that budget, model improvements produced an estimated 3.7× gain; the authors calculate the data gain as 3.24 times larger in this experiment.

Efficiency is not the whole job of model research

The authors found the two sources of improvement were mostly independent: additive effects explained 88% of variation in OLMES scores. But they caution against treating the result as a broad ranking of data work over model work. Model research may instead enable larger, longer and more stable training runs by easing constraints such as instability, memory and bandwidth limits.

The central uncertainty is scale

Patel and Han describe their experiments as extremely small scale and call for larger tests. They hypothesize that careful filtering may benefit smaller models more, while larger models may gain more from a greater volume of material even when its average quality is lower. That is an explanation to test, not a result established by these runs.

Questions left open

• Whether the measured gap persists at frontier training scales.
• Whether synthetic data can expand a corpus without reducing model performance.
• Whether results on OLMES capture broader model capabilities.

Synthetic data is especially consequential to the interpretation because the study did not test it. If data improvements are a key contributor to pretraining efficiency, the unanswered question is whether useful training material can be expanded beyond existing sources without weakening results.