AI help improved answers but weakened unaided performance in randomized trials

AI assistance helped participants answer questions, but the advantage did not survive losing access to the tool. An updated, peer-reviewed study presented this week at the Conference on Language Modeling found worse independent performance and less persistence after roughly 10 minutes of AI use—a distinction between completing a task and strengthening the person doing it.

The paper, “AI Assistance Reduces Persistence and Hurts Independent Performance,” first appeared as a preliminary draft in April. Its latest version was posted October 3, before the conference presentation. Berkeley News described the updated findings on October 9.

First the help, then the withdrawal

The researchers recruited 1,222 people online for three randomized experiments. The tasks covered mathematical reasoning and reading comprehension, allowing the team to examine whether the pattern extended beyond one kind of question.

In the first experiment, 354 participants were assigned 15 basic fraction problems. One group worked without help; another had ChatGPT available alongside the questions and could ask it for assistance or the answer itself. The AI-assisted group initially answered more accurately. After the twelfth problem, however, the researchers removed access, and that group’s accuracy dropped sharply.

A second, larger experiment involved 667 participants. After assistance was withdrawn, people who had used AI answered incorrectly or gave up, while those who had worked without it persisted and finished more successfully. A third experiment used an SAT reading-comprehension prompt with 201 participants. Persistence and accuracy again fell when the AI tool was removed.

The researchers’ explanation: expecting instant answers

The authors propose that AI conditions people to expect immediate answers, leaving them less willing to work through challenges alone. They describe the lost experience as “productive struggle”: the effort involved in reaching an answer rather than receiving one. That explanation is their proposed mechanism, distinct from the measured changes in accuracy and willingness to continue.

In the paper’s abstract, the authors describe persistence as foundational to acquiring skills. They call for model development that supports long-term competence alongside immediate task completion.

From answer machine to tutor

Christian suggested making responses more instructive, like a tutor, instead of defaulting to quick answers. He also raised concerns about research and academia, where staying close to data, working deeply with material and developing ideas with colleagues are part of becoming a capable scientist.

The short-task results should not be read as a measurement of permanent skill loss. The proposed tutor-style response is a design recommendation, not a demonstrated remedy in these three experiments.