Stanford researchers estimate AI use in 38% of Santa Clara non-debt complaints
A new benchmark estimates AI-assisted drafting across four county courts, but the researchers warn against using detectors to identify individual filings.
Loading page…
A new benchmark estimates AI-assisted drafting across four county courts, but the researchers warn against using detectors to identify individual filings.
Listen to this story
A Stanford-affiliated study estimates that signs of AI-assisted writing appeared in 36%–38% of non-debt complaints in two California counties in 2026, compared with 16%–17% in two Oklahoma counties. Its calibrated method is intended to estimate prevalence across filings, not determine whether a particular person used AI: detector settings alone made one Santa Clara estimate range from 10% to 98%. The findings highlight both rising use among self-represented filers and the limits of treating AI detectors as evidence in individual cases.
In Santa Clara, estimated AI use among self-represented filers climbed from 19% in 2024 to 72% in 2026; the estimate for attorney-filed complaints rose from 5% to 34%.
Employment and business complaints had the highest estimated use in California, while debt-collection complaints showed almost none and were excluded from the headline estimates.
PhantomDocket uses legal cases involving hallucinated citations as a reference set, while ScoreMix calibrates detector outputs against that benchmark.
AI assistance is showing up in a substantial share of court complaints, Stanford researchers estimate—but proving who used it is a different problem. In a study presented November 19, they estimated AI use in 38% of Santa Clara County’s non-debt-collection complaints in 2026, while warning that detectors remain unreliable tools for identifying individual AI-assisted filings.
The work, from researchers affiliated with Stanford’s Institute for Human-Centered AI and Regulation, Evaluation, and Governance Lab, examines complaints in four county courts across California and Oklahoma. It introduces PhantomDocket, a benchmark built around documented legal hallucinations, and ScoreMix, a method for turning detector outputs into estimates of how common AI-assisted writing is.
Legal language presents a particular detection problem. It uses a narrower vocabulary than everyday speech and repeats standard wording, leaving fewer distinctive signals for a detector to find. Commercial detectors also may not have been trained on legal writing. The researchers found that existing tools, including Pangram and EditLens, either missed AI text or needed extensive tuning.
To build a legal-specific reference set, the team turned to the AI Hallucinations Cases Database maintained by legal researcher Damien Charlotin. It tracks cases in which judges accuse or admonish parties for fictitious citations. The researchers combined documented hallucination cases with a carefully constructed set of controls to create PhantomDocket.
ScoreMix then calibrates detector scores to estimate the share of filings that look more like the hallucination examples than documents written before ChatGPT. That comparison underpins the study’s prevalence estimates. Without calibration, settings alone produced an enormous swing: one detector flagged anywhere from 10% to 98% of the same 2026 Santa Clara complaints.
The calibrated results show substantial differences between courts. Alongside Santa Clara’s 38% estimate, the researchers put AI use at 36% in San Francisco County’s non-debt-collection complaints in 2026. For comparable cases in Oklahoma, the estimates were lower: 17% in Tulsa County and 16% in Oklahoma County. These are estimates for the studied complaints, not a nationwide measure.
Who filed also made a difference. In Santa Clara, the estimated share of complaints showing signs of AI use among people representing themselves rose from 19% in 2024 to 72% in 2026. For attorney-filed complaints, it rose from 5% to 34%. Both groups showed growth, but the self-represented group had much higher estimated adoption.
Case type matters, too. Employment and business complaints had the highest estimated use in California. Debt-collection complaints showed almost none. Those documents rely heavily on templates and account for nearly half of filings in some courts, making their exclusion important when reading the headline percentages. Within documents, AI signals appeared more often in statements of facts and causes of action.
The distinction between population estimates and individual judgments is central to the researchers’ warning. Even the best detectors in their benchmark produced meaningful numbers of false positives and false negatives—human writing labeled as AI, and AI writing missed. They caution that off-the-shelf detection is risky in professional fields with distinctive language.
The study leaves a separate question unresolved: whether AI helps people who cannot afford lawyers or contributes to inadequate legal work. The authors raise both possibilities. They call for better disclosure policies across courts so that accountability and measurement depend less on guessing how a document was written.
Loading discussion...
Join the conversation
Explain whether disclosure adds useful accountability or unnecessary scrutiny.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.