GPT-BERT Beats Llama 2 70B on One Grammar Test With 100 Million Words

Language models were built on the premise that enormous text corpora can produce fluent systems. BabyLM’s 2024 result complicates that story: GPT-BERT, pretrained on about 100 million words, beat Meta’s Llama 2 70B on one grammar benchmark despite the larger model having roughly 15,000 times more pretraining data. The result is a data-efficiency signal, not evidence that small models match commercial systems or children.

The contest targets a striking mismatch in language learning. Toddlers typically begin producing grammatically correct sentences after hearing about 10 million words, with estimates as high as 30 million. A linguistically rich preteen may have heard about 100 million words; with literacy, cumulative exposure could reach roughly 300 million by age 20.

BabyLM asks teams to train models on developmentally plausible corpora of 100 million words, or 10 million in its toddler-scale track. The data draws from storybooks, dialogue, subtitles, two forms of Wikipedia, and transcripts of speech directed at children. Its tests adapt human psycholinguistics: models receive grammatical and ungrammatical sequences, and surprisal measures how unexpected each sequence is to the model.

GPT-BERT is a transformer trained partly to predict the next token and partly as a masked language model that fills gaps in token sequences. Its setup differs sharply from a child’s: BabyLM systems learn from written text, while children take in the world through their senses, especially vision and hearing.

What the score leaves open

• It does not show broad capability parity. Many BabyLM models cannot generate text, and the source describes GPT-BERT as clunky beside modern commercial models.
• It does not confirm that a baby-like training sequence is best. Curriculum learning, which moves from simple inputs to more complex ones, was the most popular first-round approach but performed less well than organizers expected.

For the past decade, language models have largely improved by scaling up. Meta’s Llama 3.1 was pretrained on 15 trillion tokens, and Georgetown cognitive scientist Ethan Gotlieb Wilcox estimates that frontier systems could be using 10 times more data. Researchers hope more efficient learning could help train AI on video and support chatbots for minority language communities.

BabyLM is also a way to test theories of how people acquire language. Researchers still disagree over the roles of innate grammatical knowledge, experience, and universal constraints on what languages can be learned. Testing such hypotheses in machine models could help address those questions, but the competition’s current result is one benchmark outcome rather than a resolution of that debate.