Loading page…
The pilot gives outside labs a larger window into real AI use than public chat datasets offer, but Anthropic still runs the analysis, screens the outputs and limits what researchers can inspect.
Listen to this story
Anthropic is turning production Claude usage into a research dataset without releasing chat transcripts: Stanford, Oxford and METR analyzed roughly 250,000 conversations each from April–May through company-run classification. The pilot found substantial consequential work and frequent user correction, but its value is constrained by opaque categorization and Anthropic’s review of every cluster, including removals from 1.8%–3.33% of outputs.
The samples totaled about 750,000 conversations; Claude.ai data excluded Team, Enterprise and API traffic.
Stanford found 56% of actionable-task conversations involved consequential-or-higher-impact work; 12% were high-stakes.
Researchers received counts and cluster descriptions, not source chats, limiting their ability to audit classification errors.
Anthropic has created a controlled route for outside researchers to study how people use Claude at production scale. Three groups examined separate samples totaling about 750,000 conversations, but received aggregated results rather than the chats themselves. The arrangement opens a new evidence source for AI-use research while leaving Anthropic at the center of the data pipeline.
Stanford’s Social and Language Technologies Lab, Oxford’s Human Information Processing Lab and the evaluation group METR each worked from a separate sample of roughly 250,000 conversations collected in April and May. Stanford and Oxford studied Claude.ai use; METR examined Claude Code sessions. The Claude.ai material covered Free, Pro and Max users, not Team, Enterprise or API traffic.
The distinction is central. Researchers chose analytical questions, known as facets, but Anthropic ran them on its systems. Claude classified conversations, grouped similar responses and returned category counts, percentages and cluster descriptions. The researchers did not view the underlying conversations.
The groups had publication independence, and Anthropic’s contractual review rights were limited to privacy, policy violations, confidential information and research accuracy. Yet the company manually reviewed every cluster before release. It removed or altered 1.9% of Stanford’s clusters, 3.33% of Oxford’s and 1.8% of METR’s.
Anthropic has published the resulting aggregate outputs under a CC BY 4.0 license on Hugging Face. That makes the released categories available for inspection, but not the source material that produced them.
Stanford’s completed study analyzed 249,834 Claude.ai conversations. Among conversations involving an actionable task, 56% involved work classified as consequential or higher-impact: work affecting other people or difficult to reverse. Twelve percent fell into the high-stakes category, including legal and financial guidance.
The privacy design creates an analytical trade-off. Because researchers cannot inspect source chats, poorly phrased facets can generate misleading categories that are hard to detect. Tests on the public WildChat dataset did not always transfer to Claude traffic, and Anthropic removed one Oxford facet after determining its descriptions were unreliable.
Oxford’s study remains underway. Its early patterns link warmer Claude interactions with more positive user behavior, while refusals and disagreement coincide with user pushback; those are associations, not evidence that one response style caused the other. METR’s unfinished coding study has preliminary indications that newer Claude models deliver significant speedups over older versions, using model estimates of no-AI task time as part of its method.
A privacy red team at Imperial College London reported that it could not reidentify users from released material. It did, however, connect one cluster to a widely used open-source project through distinctive language. Anthropic says future studies will use higher minimum-user thresholds and less distinctive cluster descriptions.
Anthropic is soliciting proposals for future access, while warning that privacy and review requirements will make expansion slow. The pilot’s value will rest on whether that controlled path can support more varied questions without weakening the protections that keep raw conversations private.
Story updates
People are bringing consequential work to Claude—and often asking it to explain rather than simply deliver. A Stanford-led study described on October 12 classified 56% of roughly 250,000 anonymized conversations as consequential or higher. Its findings challenge the picture of passive chatbot use, while putting greater weight on how people question and steer the system.
The team used Anthropic’s Insights analytical tool to examine conversations between people and the company’s chatbot. Led by Stanford computer science assistant professor Diyi Yang, the research was supported by Stanford’s Institute for Human-Centered AI. Stanford says the paper has been accepted by Nature Human Behavior.
Rather than asking only what tasks people assigned to AI, the researchers examined three aspects of collaboration: how meaningful the work was, how much control users exercised, and where the exchange encountered friction. That last term describes tension that can either help people learn or cause a conversation to break down.
Users brought legal, medical, professional and financial questions to Claude, alongside mental health and relationship advice. Co-first author Yijia Shao described much of the work as high-stakes and hard to reverse. His warning was straightforward: people should not hand over these decisions and accept the chatbot’s answers without scrutiny.
The researchers found greater engagement as the stakes increased. Co-first author Dora Zhao said conversation length nearly doubled compared with exchanges about more mundane tasks. The result runs against a simple account of AI as a shortcut: in these conversations, consequential work was associated with more interaction, not less.
In two-thirds of interactions, users asked the AI to explain or teach rather than merely generate content. The authors also identified a user type that pressed for deeper explanations. They reported that AI teaching was most beneficial in high-consequence areas including law, software engineering, home improvement and health.
About half of the conversations involved friction. The researchers distinguish productive tension—which leads to clarification, refinement and learning—from negative friction that disrupts collaboration. Their account treats questioning and pushback as part of working together, rather than assuming that the smoothest exchange necessarily produces the most useful result.
Zhao compared a user’s willingness to speak up to a healthy workplace team, where people can advocate for themselves. The authors’ finding is not that every disagreement helps. When friction turns negative, the exchange can break down, and users face the separate task of getting the collaboration back on track.
In nearly 80% of collaborations involving negative friction, users attempted to recover the interaction. That figure measures attempts, not successful repairs. The researchers say whether a difficult exchange ultimately becomes productive depends on how the person uses the AI; persistence alone is not their explanation for a beneficial outcome.
We could see a widening gap between those who recognize AI as a tool for skill development and career advancement, while others accept it as a substitute for their own good judgment and reasoning
The researchers recommend improving AI literacy: helping people understand the tool’s strengths and weaknesses while retaining or developing their own skills. Shao frames that as a question of access to benefits, not simply getting more people to use chatbots. Responsible use and continued human learning are central to that recommendation.
Their concern is that useful effects could concentrate in higher-stakes knowledge sectors, leaving others behind. Yang’s warning about a widening gap is a possible consequence, not a measured outcome of this study. The team’s next work addresses both the user’s choices during an exchange and the system’s longer-term design:
Shao is developing tools to help users recognize when AI output needs closer scrutiny, when friction is worth engaging with, and how to recover when an exchange goes wrong.
Zhao is investigating personalization and designs that support longer-term benefits, including greater cognitive engagement. That work looks beyond finishing the immediate task to how people continue thinking while using AI.
stanford.eduEditorial analysis
this is a more meaningful opening than publishing a lab-authored usage report, because Stanford, Oxford and METR set their own questions and retain control of their conclusions. But it is not independent access to production data in the conventional sense: Anthropic selects samples, operates the classification system and reviews released clusters. The next test is whether future projects can produce findings that meaningfully challenge Anthropic’s own priorities or framing while still surviving those privacy controls. The program’s slow scaling warning makes that question operational, not merely philosophical.
Loading discussion...
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.