Audit finds serious flaws in 14 research reviews claiming AI helps students learn

An audit of 14 research reviews claiming benefits from AI in education found recurring conceptual and statistical flaws, undermining broad claims that the technology improves learning. Patrick O’Neill’s findings, described by The Hechinger Report in an October 26 story published by KQED, challenge evidence meant to help educators make sense of classroom AI.

The papers are meta-analyses: studies that combine results from earlier research to draw a larger conclusion. O’Neill, a professor at Ivy Tech’s Bloomington, Indiana, campus, examined 14 out of 39 such reviews reporting substantial benefits from educational AI. The audited papers had attracted more than 3,000 citations and roughly 200 additional citations a month, according to Google Scholar figures cited in the story.

O’Neill checked a sample of the underlying studies and the statistical methods used to combine them. At an October 20 webinar, he described impossible statistics, references that led to unrelated papers, and conclusions that went beyond the evidence. Some problems, he said, were visible from the study abstracts alone. Others concerned what an experiment actually measured:

• Nursing students received three hours of weekly study with an instructor using AI. Their comparison group received no study sessions.
• Students played video games and answered AI-generated questions. The comparison group did neither, leaving more than AI use different between the groups.
• Another experiment measured whether chatting with an AI bot reduced students’ reported anxiety—not how much they learned.

Beth Tipton, a Northwestern University professor and meta-analysis expert, challenged the premise of combining every educational use of AI into one verdict. She argued that researchers need to specify the intervention first. An educational program containing AI and unrestricted access to a chatbot are different approaches; a useful synthesis needs a narrower question about how the technology is used.

O’Neill also raised publication bias: the risk that the published record favors positive findings over disappointing or inconclusive ones. He said two-thirds of the calls for papers he examined specifically invited research on AI’s benefits, potentially discouraging studies finding no benefit or harm. Combining a selectively positive research base can then give its conclusions an undeserved appearance of certainty.

There is already a related correction in the record. A meta-analysis of ChatGPT’s effects on learning performance, perceptions and higher-order thinking was retracted. Its retraction notice was published April 22, 2026; a July 2 update credited Magnus Ingebrigtsen and Marko Lukic with originally raising the concerns.

Anybody can sort of say that they have evidence for anything, and share it with anybody, and consumers, schools, teachers, parents are left to try to wade through that and make sense of it.

Beth Tipton, Northwestern University professor

O’Neill is conducting his own meta-analysis. His preliminary calculations from 44 well-designed experiments found effects ranging from very helpful to very harmful, averaging roughly zero. That does not establish that AI is harmless or useless: the average combines sharply different outcomes, and the researchers stressed that the way AI is used matters.

Stanford education professor Susanna Loeb described a separate quality screen: her team collected almost 900 studies but included only 20 in a recent evidence summary. It has since identified 32 more studies. Loeb nevertheless said it remains too early for a meta-analysis useful to school leaders or teachers; very few high-quality studies take place in U.S. K–12 schools.

Repairing the review process also takes resources. Tipton said checking underlying studies can take days, while peer review is unpaid volunteer work. The researchers appealed to philanthropic funders and the National Science Foundation to support rigorous research syntheses. For now, they advised educators to start with small student groups and monitor results rather than assume broad effectiveness.